mirror of
https://github.com/paperless-ngx/paperless-ngx.git
synced 2026-07-26 13:54:56 +00:00
Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 stumpylog
|
5d8bbe1592 |
@@ -272,8 +272,6 @@ optional arguments:
|
||||
-sm, --split-manifest
|
||||
-z, --zip
|
||||
-zn, --zip-name
|
||||
--zip-compression
|
||||
--zip-compression-level
|
||||
--data-only
|
||||
--no-progress-bar
|
||||
--passphrase
|
||||
@@ -336,19 +334,6 @@ If `-z` or `--zip` is provided, the export will be a zip file
|
||||
in the target directory, named according to the current local date or the
|
||||
value set in `-zn` or `--zip-name`.
|
||||
|
||||
The compression method for the zip can be set with `--zip-compression`
|
||||
(`stored`, `deflated` (default), `bzip2`, `lzma`, or `zstd`) and tuned with
|
||||
`--zip-compression-level` (deflated: 0–9, bzip2: 1–9, zstd: -22–22; ignored
|
||||
for `stored` and `lzma`). Both options require `--zip`.
|
||||
|
||||
!!! warning
|
||||
|
||||
`zstd` compression requires Python 3.14 or newer on **both** the machine
|
||||
creating the export and any machine importing it. An archive compressed with
|
||||
`zstd` (or `lzma`/`bzip2` where those modules are unavailable) cannot be
|
||||
imported on a runtime that lacks the codec; the importer will refuse it with
|
||||
a clear error. The default `deflated` is universally readable.
|
||||
|
||||
If `--data-only` is provided, only the database will be exported. This option is intended
|
||||
to facilitate database upgrades without needing to clean documents and thumbnails from the media directory.
|
||||
|
||||
|
||||
@@ -1,839 +0,0 @@
|
||||
# Export Zip Compression Control Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add `--zip-compression {stored,deflated,bzip2,lzma,zstd}` and `--zip-compression-level N` flags to `document_exporter`, threaded into `ZipExportSink`, with import-side safety for codecs the running Python can't read.
|
||||
|
||||
**Architecture:** A new pure-data module `documents/export/compression.py` owns the method↔constant map, per-method level bounds, the runtime availability probe, and a compress-type readability check. `ZipExportSink` gains `compression`/`compresslevel` constructor params. The command validates flags up front (fail-fast `CommandError`) and constructs the sink; the importer pre-checks entry compress types before extracting.
|
||||
|
||||
**Tech Stack:** Python ≥3.11 (zstd only on 3.14+), `zipfile`, `compression.zstd` (PEP 784), pytest + pytest-mock + factory-boy. Backend tests run on the Linux VM (Python 3.11 — zstd positive tests are `skipif`-guarded); `ruff` runs locally.
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-06-16-export-zip-compression-design.md`
|
||||
|
||||
**PREREQUISITE:** The base refactor `docs/superpowers/plans/2026-06-16-export-sink-architecture.md` MUST be merged first. This plan assumes `src/documents/export/sinks.py` exists with `ZipExportSink(target, zip_name, *, delete=False)` opening its `ZipFile` in `_open()`.
|
||||
|
||||
---
|
||||
|
||||
## Verified facts (CPython 3.14.3, via `uv run --python 3.14 --no-project`)
|
||||
|
||||
- Constants: `ZIP_STORED=0`, `ZIP_DEFLATED=8`, `ZIP_BZIP2=12`, `ZIP_LZMA=14`, `ZIP_ZSTANDARD=93` (zstd added 3.14; absent on < 3.14).
|
||||
- `ZipFile(file, "w", compression=…, compresslevel=…)` applies both as the default for every `write`/`writestr` — no per-entry args needed (verified).
|
||||
- Level bounds: `deflated` 0–9, `bzip2` 1–9, `lzma`/`stored` ignore level, `zstd` -131072…22 (`compression.zstd.CompressionParameter.compression_level.bounds() == (-131072, 22)`).
|
||||
- An invalid level fails at the **first write** (`ValueError: Invalid initialization option` / `compresslevel must be between 1 and 9`), plus GC-time `AttributeError` noise on close — hence up-front validation.
|
||||
- zstd is backed by `compression.zstd`; `zipfile` raises `RuntimeError` if it's unavailable.
|
||||
|
||||
## Conventions for every task
|
||||
|
||||
- **Run backend tests on the VM:** `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "<targets>"` (never locally).
|
||||
- **Lint locally:** `ruff check <paths> && ruff format <paths>` (global ruff, not `uv run`).
|
||||
- **Tests are pytest-style:** classes, `@pytest.mark.django_db` on the class only where DB is needed (the `compression.py` and sink tests need no DB), factory-boy, `mocker`, `parametrize`, full type annotations.
|
||||
- The VM runs Python 3.11, so **zstd positive tests must be `@pytest.mark.skipif(...)`-guarded**; they will simply not run there. zstd _rejection_ tests (the < 3.14 path) DO run on the VM.
|
||||
|
||||
## File structure
|
||||
|
||||
- **Create** `src/documents/export/compression.py` — method map, CLI choices, level bounds, `compression_available()`, `level_error()`, `compress_type_readable()`, `unreadable_method_names()`. Pure, no Django.
|
||||
- **Create** `src/documents/tests/export/test_compression.py` — unit tests for the above.
|
||||
- **Modify** `src/documents/export/sinks.py` — `ZipExportSink.__init__` gains `compression`/`compresslevel`; `_open()` passes them to `ZipFile`.
|
||||
- **Modify** `src/documents/tests/export/test_sinks.py` — assert the chosen `compress_type` is applied.
|
||||
- **Modify** `src/documents/management/commands/document_exporter.py` — add the two CLI flags, up-front validation, and pass resolved values to `ZipExportSink`.
|
||||
- **Modify** `src/documents/tests/test_management_exporter.py` — flag validation + default-unchanged tests.
|
||||
- **Modify** `src/documents/management/commands/document_importer.py` — pre-extract compress-type check.
|
||||
- **Modify** `src/documents/tests/test_management_importer.py` — unsupported-codec → `CommandError`.
|
||||
- **Modify** `docs/administration.md` — document both flags + zstd portability caveat.
|
||||
|
||||
---
|
||||
|
||||
## Task 1: `documents/export/compression.py` (pure compression policy)
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `src/documents/export/compression.py`
|
||||
- Test: `src/documents/tests/export/test_compression.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing tests**
|
||||
|
||||
Create `src/documents/tests/export/test_compression.py`:
|
||||
|
||||
```python
|
||||
import sys
|
||||
import zipfile
|
||||
|
||||
import pytest
|
||||
|
||||
from documents.export import compression
|
||||
|
||||
|
||||
class TestCompressionMethods:
|
||||
def test_choices_always_include_zstd(self) -> None:
|
||||
# zstd is offered regardless of runtime; availability is checked separately
|
||||
assert compression.COMPRESSION_CHOICES == (
|
||||
"stored",
|
||||
"deflated",
|
||||
"bzip2",
|
||||
"lzma",
|
||||
"zstd",
|
||||
)
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("name", "constant"),
|
||||
[
|
||||
("stored", zipfile.ZIP_STORED),
|
||||
("deflated", zipfile.ZIP_DEFLATED),
|
||||
("bzip2", zipfile.ZIP_BZIP2),
|
||||
("lzma", zipfile.ZIP_LZMA),
|
||||
],
|
||||
)
|
||||
def test_method_maps_to_zipfile_constant(self, name: str, constant: int) -> None:
|
||||
assert compression.COMPRESSION_METHODS[name] == constant
|
||||
|
||||
def test_stored_and_deflated_always_available(self) -> None:
|
||||
assert compression.compression_available("stored")
|
||||
assert compression.compression_available("deflated")
|
||||
|
||||
def test_zstd_availability_tracks_runtime(self) -> None:
|
||||
expected: bool = sys.version_info >= (3, 14)
|
||||
assert compression.compression_available("zstd") == expected
|
||||
|
||||
|
||||
class TestLevelError:
|
||||
@pytest.mark.parametrize(
|
||||
("method", "level"),
|
||||
[
|
||||
("deflated", 0),
|
||||
("deflated", 9),
|
||||
("bzip2", 1),
|
||||
("bzip2", 9),
|
||||
("deflated", None),
|
||||
("stored", None),
|
||||
],
|
||||
)
|
||||
def test_valid_levels_return_none(self, method: str, level: int | None) -> None:
|
||||
assert compression.level_error(method, level) is None
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("method", "level"),
|
||||
[
|
||||
("deflated", 10),
|
||||
("deflated", -1),
|
||||
("bzip2", 0),
|
||||
("bzip2", 10),
|
||||
],
|
||||
)
|
||||
def test_out_of_range_levels_return_message(
|
||||
self,
|
||||
method: str,
|
||||
level: int,
|
||||
) -> None:
|
||||
msg: str | None = compression.level_error(method, level)
|
||||
assert msg is not None
|
||||
assert "between" in msg
|
||||
|
||||
@pytest.mark.parametrize("method", ["stored", "lzma"])
|
||||
def test_level_on_levelless_method_is_rejected(self, method: str) -> None:
|
||||
msg: str | None = compression.level_error(method, 5)
|
||||
assert msg is not None
|
||||
assert "no effect" in msg
|
||||
|
||||
|
||||
class TestCompressTypeReadable:
|
||||
@pytest.mark.parametrize("ct", [zipfile.ZIP_STORED, zipfile.ZIP_DEFLATED])
|
||||
def test_stored_and_deflated_always_readable(self, ct: int) -> None:
|
||||
assert compression.compress_type_readable(ct)
|
||||
|
||||
def test_zstd_compress_type_readability_tracks_runtime(self) -> None:
|
||||
# 93 = ZIP_ZSTANDARD; 20 = legacy zstd method id (read-only)
|
||||
expected: bool = sys.version_info >= (3, 14)
|
||||
assert compression.compress_type_readable(93) == expected
|
||||
assert compression.compress_type_readable(20) == expected
|
||||
|
||||
def test_unknown_compress_type_is_unreadable(self) -> None:
|
||||
assert not compression.compress_type_readable(9999)
|
||||
|
||||
def test_unreadable_method_names_lists_methods(self) -> None:
|
||||
# An unknown method id maps to no name and is reported generically.
|
||||
names: set[str] = compression.unreadable_method_names({9999})
|
||||
assert names == {"method 9999"}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run to verify it fails**
|
||||
|
||||
Run: `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests/export/test_compression.py -v"`
|
||||
Expected: FAIL with `ModuleNotFoundError: No module named 'documents.export.compression'`.
|
||||
|
||||
- [ ] **Step 3: Implement `compression.py`**
|
||||
|
||||
Create `src/documents/export/compression.py`:
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import zipfile
|
||||
|
||||
# ZIP_ZSTANDARD exists only on Python 3.14+ (PEP 784). None elsewhere.
|
||||
ZSTD: int | None = getattr(zipfile, "ZIP_ZSTANDARD", None)
|
||||
|
||||
# CLI choices are fixed across runtimes so argparse never hides zstd; runtime
|
||||
# availability is enforced separately in compression_available().
|
||||
COMPRESSION_CHOICES: tuple[str, ...] = (
|
||||
"stored",
|
||||
"deflated",
|
||||
"bzip2",
|
||||
"lzma",
|
||||
"zstd",
|
||||
)
|
||||
|
||||
# Method name -> zipfile compression constant (zstd only when supported).
|
||||
COMPRESSION_METHODS: dict[str, int] = {
|
||||
"stored": zipfile.ZIP_STORED,
|
||||
"deflated": zipfile.ZIP_DEFLATED,
|
||||
"bzip2": zipfile.ZIP_BZIP2,
|
||||
"lzma": zipfile.ZIP_LZMA,
|
||||
}
|
||||
if ZSTD is not None:
|
||||
COMPRESSION_METHODS["zstd"] = ZSTD
|
||||
|
||||
# Inclusive (min, max) level bounds per method; None => level not applicable.
|
||||
# Verified on CPython 3.14.3.
|
||||
LEVEL_BOUNDS: dict[str, tuple[int, int] | None] = {
|
||||
"stored": None,
|
||||
"deflated": (0, 9),
|
||||
"bzip2": (1, 9),
|
||||
"lzma": None,
|
||||
"zstd": (-131072, 22),
|
||||
}
|
||||
|
||||
# zipfile compress_type id -> method name. 93 = current zstd id, 20 = legacy
|
||||
# zstd id that zipfile can still read.
|
||||
_COMPRESS_TYPE_TO_METHOD: dict[int, str] = {
|
||||
zipfile.ZIP_STORED: "stored",
|
||||
zipfile.ZIP_DEFLATED: "deflated",
|
||||
zipfile.ZIP_BZIP2: "bzip2",
|
||||
zipfile.ZIP_LZMA: "lzma",
|
||||
93: "zstd",
|
||||
20: "zstd",
|
||||
}
|
||||
|
||||
|
||||
def compression_available(method: str) -> bool:
|
||||
"""Whether the running interpreter can actually use the given method."""
|
||||
if method in ("stored", "deflated"):
|
||||
# zlib is a hard CPython dependency; stored needs nothing.
|
||||
return True
|
||||
if method == "bzip2":
|
||||
return _module_importable("bz2")
|
||||
if method == "lzma":
|
||||
return _module_importable("lzma")
|
||||
if method == "zstd":
|
||||
return ZSTD is not None and _module_importable("compression.zstd")
|
||||
return False
|
||||
|
||||
|
||||
def _module_importable(name: str) -> bool:
|
||||
try:
|
||||
importlib.import_module(name)
|
||||
except ImportError:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def level_error(method: str, level: int | None) -> str | None:
|
||||
"""Return a human message if (method, level) is invalid, else None."""
|
||||
if level is None:
|
||||
return None
|
||||
bounds = LEVEL_BOUNDS[method]
|
||||
if bounds is None:
|
||||
return f"--zip-compression-level has no effect for '{method}'"
|
||||
low, high = bounds
|
||||
if not (low <= level <= high):
|
||||
return (
|
||||
f"--zip-compression-level for '{method}' must be between "
|
||||
f"{low} and {high}"
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
def compress_type_readable(compress_type: int) -> bool:
|
||||
"""Whether this interpreter can decompress an entry of the given type."""
|
||||
method = _COMPRESS_TYPE_TO_METHOD.get(compress_type)
|
||||
if method is None:
|
||||
return False
|
||||
return compression_available(method)
|
||||
|
||||
|
||||
def unreadable_method_names(compress_types: set[int]) -> set[str]:
|
||||
"""Map a set of compress_type ids to human method names for error messages."""
|
||||
names: set[str] = set()
|
||||
for ct in compress_types:
|
||||
names.add(_COMPRESS_TYPE_TO_METHOD.get(ct, f"method {ct}"))
|
||||
return names
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run to verify it passes**
|
||||
|
||||
Run: `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests/export/test_compression.py -v"`
|
||||
Expected: PASS (on the 3.11 VM, `test_zstd_availability_tracks_runtime` and `test_zstd_compress_type_readability_tracks_runtime` assert `False`).
|
||||
|
||||
- [ ] **Step 5: Lint**
|
||||
|
||||
Run: `ruff check src/documents/export/compression.py src/documents/tests/export/test_compression.py && ruff format src/documents/export/compression.py src/documents/tests/export/test_compression.py`
|
||||
Expected: no errors.
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add src/documents/export/compression.py src/documents/tests/export/test_compression.py
|
||||
git commit -m "Feature: add export compression policy module"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: `ZipExportSink` accepts compression method + level
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `src/documents/export/sinks.py`
|
||||
- Test: `src/documents/tests/export/test_sinks.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
Append to `src/documents/tests/export/test_sinks.py` (the top-of-file block already imports `zipfile`, `Path`, `pytest`, `ZipExportSink`, `StreamingManifestWriter` from the base-refactor plan):
|
||||
|
||||
```python
|
||||
class TestZipExportSinkCompression:
|
||||
@pytest.fixture()
|
||||
def source_file(self, tmp_path: Path) -> Path:
|
||||
src: Path = tmp_path / "src" / "doc.pdf"
|
||||
src.parent.mkdir(parents=True)
|
||||
src.write_bytes(b"PDF-CONTENT" * 100)
|
||||
return src
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("method", "constant"),
|
||||
[
|
||||
("stored", zipfile.ZIP_STORED),
|
||||
("deflated", zipfile.ZIP_DEFLATED),
|
||||
("bzip2", zipfile.ZIP_BZIP2),
|
||||
("lzma", zipfile.ZIP_LZMA),
|
||||
],
|
||||
)
|
||||
def test_compression_method_is_applied_to_file_entries(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
source_file: Path,
|
||||
method: str,
|
||||
constant: int,
|
||||
) -> None:
|
||||
target: Path = tmp_path / "out"
|
||||
target.mkdir()
|
||||
with ZipExportSink(
|
||||
target,
|
||||
"export",
|
||||
delete=False,
|
||||
compression=constant,
|
||||
) as sink:
|
||||
sink.add_file(source_file, "doc.pdf")
|
||||
with zipfile.ZipFile(target / "export.zip") as zf:
|
||||
info = zf.getinfo("doc.pdf")
|
||||
assert info.compress_type == constant
|
||||
|
||||
def test_compressing_method_beats_stored(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
source_file: Path,
|
||||
) -> None:
|
||||
# Robust size invariant: a compressing method must be <= stored on
|
||||
# compressible content (avoids flaky level-9-vs-level-1 comparisons).
|
||||
sizes: dict[str, int] = {}
|
||||
for name, constant in (("stored", zipfile.ZIP_STORED), ("deflated", zipfile.ZIP_DEFLATED)):
|
||||
target: Path = tmp_path / name
|
||||
target.mkdir()
|
||||
with ZipExportSink(target, "export", delete=False, compression=constant) as sink:
|
||||
sink.add_file(source_file, "doc.pdf")
|
||||
sizes[name] = (target / "export.zip").stat().st_size
|
||||
assert sizes["deflated"] <= sizes["stored"]
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run to verify it fails**
|
||||
|
||||
Run: `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests/export/test_sinks.py::TestZipExportSinkCompression -v"`
|
||||
Expected: FAIL with `TypeError: __init__() got an unexpected keyword argument 'compression'`.
|
||||
|
||||
- [ ] **Step 3: Add the params to `ZipExportSink`**
|
||||
|
||||
In `src/documents/export/sinks.py`, change `ZipExportSink.__init__` to accept the new keyword-only params and store them, and pass them in `_open()`:
|
||||
|
||||
```python
|
||||
def __init__(
|
||||
self,
|
||||
target: Path,
|
||||
zip_name: str,
|
||||
*,
|
||||
delete: bool = False,
|
||||
compression: int = zipfile.ZIP_DEFLATED,
|
||||
compresslevel: int | None = None,
|
||||
) -> None:
|
||||
self._target = target.resolve()
|
||||
self._zip_path = (self._target / zip_name).with_suffix(".zip")
|
||||
self._tmp_path = self._zip_path.with_name(self._zip_path.name + ".tmp")
|
||||
self._delete = delete
|
||||
self._compression = compression
|
||||
self._compresslevel = compresslevel
|
||||
self._zip: zipfile.ZipFile | None = None
|
||||
self._dirs: set[str] = set()
|
||||
self._pending_manifest: tuple[Path, str] | None = None
|
||||
self._stream_open = False
|
||||
```
|
||||
|
||||
And in `_open()`:
|
||||
|
||||
```python
|
||||
def _open(self) -> None:
|
||||
settings.SCRATCH_DIR.mkdir(parents=True, exist_ok=True)
|
||||
self._zip = zipfile.ZipFile(
|
||||
self._tmp_path,
|
||||
"w",
|
||||
compression=self._compression,
|
||||
compresslevel=self._compresslevel,
|
||||
allowZip64=True,
|
||||
)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run to verify it passes**
|
||||
|
||||
Run: `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests/export/test_sinks.py -v"`
|
||||
Expected: PASS (all sink tests, including the four method params and the size invariant). `bzip2`/`lzma` are present on the VM's CPython, so those params pass.
|
||||
|
||||
- [ ] **Step 5: Lint**
|
||||
|
||||
Run: `ruff check src/documents/export/sinks.py && ruff format src/documents/export/sinks.py`
|
||||
Expected: no errors.
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add src/documents/export/sinks.py src/documents/tests/export/test_sinks.py
|
||||
git commit -m "Feature: ZipExportSink accepts compression method and level"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Wire CLI flags + validation into `document_exporter`
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `src/documents/management/commands/document_exporter.py`
|
||||
- Test: `src/documents/tests/test_management_exporter.py`
|
||||
|
||||
- [ ] **Step 1: Add the argparse flags**
|
||||
|
||||
In `document_exporter.py`, add the import near the other `documents.export` import:
|
||||
|
||||
```python
|
||||
from documents.export.compression import COMPRESSION_CHOICES
|
||||
from documents.export.compression import COMPRESSION_METHODS
|
||||
from documents.export.compression import compression_available
|
||||
from documents.export.compression import level_error
|
||||
from documents.export.compression import ZSTD
|
||||
```
|
||||
|
||||
In `add_arguments`, after the `--zip-name` argument, add:
|
||||
|
||||
```python
|
||||
parser.add_argument(
|
||||
"--zip-compression",
|
||||
choices=COMPRESSION_CHOICES,
|
||||
default=None,
|
||||
help=(
|
||||
"Compression method for the export zip (requires --zip). "
|
||||
"Default: deflated. 'zstd' requires Python 3.14+ on both the "
|
||||
"exporting and importing machine."
|
||||
),
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--zip-compression-level",
|
||||
type=int,
|
||||
default=None,
|
||||
help=(
|
||||
"Compression level for the export zip (requires --zip). "
|
||||
"deflated: 0-9, bzip2: 1-9, zstd: -131072..22; ignored for "
|
||||
"stored/lzma."
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Read + validate the flags in `handle()`**
|
||||
|
||||
In `handle()`, after the existing `--compare-*` + `--zip` guard, add the compression flag handling. Insert before the sink construction:
|
||||
|
||||
```python
|
||||
zip_compression: str | None = options["zip_compression"]
|
||||
zip_compression_level: int | None = options["zip_compression_level"]
|
||||
|
||||
if not self.zip_export and (
|
||||
zip_compression is not None or zip_compression_level is not None
|
||||
):
|
||||
raise CommandError(
|
||||
"--zip-compression and --zip-compression-level require --zip",
|
||||
)
|
||||
|
||||
compression_method = zip_compression or "deflated"
|
||||
if self.zip_export:
|
||||
if not compression_available(compression_method):
|
||||
if compression_method == "zstd" and ZSTD is None:
|
||||
raise CommandError(
|
||||
"zstd compression requires Python 3.14 or newer",
|
||||
)
|
||||
raise CommandError(
|
||||
f"Compression method '{compression_method}' is not "
|
||||
f"available on this Python runtime",
|
||||
)
|
||||
level_msg = level_error(compression_method, zip_compression_level)
|
||||
if level_msg is not None:
|
||||
raise CommandError(level_msg)
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Pass the resolved values into `ZipExportSink`**
|
||||
|
||||
Change the `ZipExportSink(...)` construction in `handle()` to:
|
||||
|
||||
```python
|
||||
if self.zip_export:
|
||||
sink = ZipExportSink(
|
||||
self.target,
|
||||
options["zip_name"],
|
||||
delete=self.delete,
|
||||
compression=COMPRESSION_METHODS[compression_method],
|
||||
compresslevel=zip_compression_level,
|
||||
)
|
||||
else:
|
||||
sink = DirectoryExportSink(
|
||||
self.target,
|
||||
compare_checksums=self.compare_checksums,
|
||||
compare_json=self.compare_json,
|
||||
delete=self.delete,
|
||||
)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Write the command-level tests**
|
||||
|
||||
Add to the `TestExportImport` class in `src/documents/tests/test_management_exporter.py` (imports `call_command`, `CommandError`, `ZipFile`, `timezone` already present):
|
||||
|
||||
```python
|
||||
def test_compression_flags_require_zip(self) -> None:
|
||||
for args in (
|
||||
["--zip-compression", "lzma"],
|
||||
["--zip-compression-level", "5"],
|
||||
):
|
||||
with self.assertRaises(CommandError):
|
||||
call_command(
|
||||
"document_exporter",
|
||||
self.target,
|
||||
*args,
|
||||
skip_checks=True,
|
||||
)
|
||||
|
||||
def test_zip_compression_level_out_of_range_raises(self) -> None:
|
||||
with self.assertRaises(CommandError):
|
||||
call_command(
|
||||
"document_exporter",
|
||||
self.target,
|
||||
"--zip",
|
||||
"--zip-compression",
|
||||
"deflated",
|
||||
"--zip-compression-level",
|
||||
"99",
|
||||
skip_checks=True,
|
||||
)
|
||||
|
||||
def test_zip_compression_level_rejected_for_stored(self) -> None:
|
||||
with self.assertRaises(CommandError):
|
||||
call_command(
|
||||
"document_exporter",
|
||||
self.target,
|
||||
"--zip",
|
||||
"--zip-compression",
|
||||
"stored",
|
||||
"--zip-compression-level",
|
||||
"5",
|
||||
skip_checks=True,
|
||||
)
|
||||
|
||||
def test_zip_lzma_compression_round_trips(self) -> None:
|
||||
call_command(
|
||||
"document_exporter",
|
||||
self.target,
|
||||
"--zip",
|
||||
"--zip-compression",
|
||||
"lzma",
|
||||
skip_checks=True,
|
||||
)
|
||||
expected = str(
|
||||
self.target / f"export-{timezone.localdate().isoformat()}.zip",
|
||||
)
|
||||
self.assertIsFile(expected)
|
||||
with ZipFile(expected) as zip_file:
|
||||
info = zip_file.getinfo("manifest.json")
|
||||
# manifest.json carries the chosen method; deflated is the default
|
||||
self.assertEqual(info.compress_type, 14) # ZIP_LZMA
|
||||
|
||||
def test_default_zip_uses_deflate(self) -> None:
|
||||
call_command(
|
||||
"document_exporter",
|
||||
self.target,
|
||||
"--zip",
|
||||
skip_checks=True,
|
||||
)
|
||||
expected = str(
|
||||
self.target / f"export-{timezone.localdate().isoformat()}.zip",
|
||||
)
|
||||
with ZipFile(expected) as zip_file:
|
||||
info = zip_file.getinfo("manifest.json")
|
||||
self.assertEqual(info.compress_type, 8) # ZIP_DEFLATED
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run the tests**
|
||||
|
||||
Run: `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests/test_management_exporter.py -v"`
|
||||
Expected: PASS — the new tests plus all existing exporter tests stay green.
|
||||
|
||||
- [ ] **Step 6: Lint**
|
||||
|
||||
Run: `ruff check src/documents/management/commands/document_exporter.py src/documents/tests/test_management_exporter.py && ruff format src/documents/management/commands/document_exporter.py src/documents/tests/test_management_exporter.py`
|
||||
Expected: no errors.
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add src/documents/management/commands/document_exporter.py src/documents/tests/test_management_exporter.py
|
||||
git commit -m "Feature: add --zip-compression and --zip-compression-level flags"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Importer pre-check for unreadable codecs
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `src/documents/management/commands/document_importer.py`
|
||||
- Test: `src/documents/tests/test_management_importer.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
The importer test file `src/documents/tests/test_management_importer.py` is
|
||||
`TestCase`-style (`class TestCommandImport(... TestCase)`, `self.assertRaises`,
|
||||
`DirectoriesMixin` gives `self.dirs.scratch_dir`). Match that style. Add this
|
||||
method to `TestCommandImport`. It builds a valid zip and patches the readability
|
||||
probe so the check fires deterministically on any runtime:
|
||||
|
||||
```python
|
||||
def test_import_rejects_unreadable_compression(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- A zip archive with an entry whose compression this Python can't read
|
||||
WHEN:
|
||||
- Import is attempted
|
||||
THEN:
|
||||
- A CommandError naming the issue is raised, before extraction
|
||||
"""
|
||||
import zipfile
|
||||
from unittest import mock
|
||||
|
||||
archive = Path(self.dirs.scratch_dir) / "export.zip"
|
||||
with zipfile.ZipFile(archive, "w") as zf:
|
||||
zf.writestr("manifest.json", "[]")
|
||||
|
||||
with mock.patch(
|
||||
"documents.management.commands.document_importer.compress_type_readable",
|
||||
return_value=False,
|
||||
):
|
||||
with self.assertRaises(CommandError) as e:
|
||||
call_command(
|
||||
"document_importer",
|
||||
str(archive),
|
||||
"--no-progress-bar",
|
||||
skip_checks=True,
|
||||
)
|
||||
self.assertIn("compression", str(e.exception))
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run to verify it fails**
|
||||
|
||||
Run: `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests/test_management_importer.py -k unreadable_compression -v"`
|
||||
Expected: FAIL — no pre-check exists yet, so the import proceeds (or fails with a different error).
|
||||
|
||||
- [ ] **Step 3: Implement the pre-check**
|
||||
|
||||
In `document_importer.py`, add the import:
|
||||
|
||||
```python
|
||||
from documents.export.compression import compress_type_readable
|
||||
from documents.export.compression import unreadable_method_names
|
||||
```
|
||||
|
||||
Find the zip-handling block (around `document_importer.py:453`):
|
||||
|
||||
```python
|
||||
with ZipFile(self.source) as zf:
|
||||
zf.extractall(tmp_dir)
|
||||
```
|
||||
|
||||
Replace it with a pre-check before extraction:
|
||||
|
||||
```python
|
||||
with ZipFile(self.source) as zf:
|
||||
unsupported = {
|
||||
info.compress_type
|
||||
for info in zf.infolist()
|
||||
if not compress_type_readable(info.compress_type)
|
||||
}
|
||||
if unsupported:
|
||||
names = ", ".join(sorted(unreadable_method_names(unsupported)))
|
||||
raise CommandError(
|
||||
f"This archive uses compression this Python cannot "
|
||||
f"read ({names}). zstd archives require Python 3.14+.",
|
||||
)
|
||||
zf.extractall(tmp_dir)
|
||||
```
|
||||
|
||||
Confirm `CommandError` is imported in `document_importer.py` (it is used elsewhere; if not, add `from django.core.management.base import CommandError`).
|
||||
|
||||
- [ ] **Step 4: Run to verify it passes**
|
||||
|
||||
Run: `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests/test_management_importer.py -v"`
|
||||
Expected: PASS — the new test plus all existing importer tests (normal deflated/stored archives still import).
|
||||
|
||||
- [ ] **Step 5: Lint**
|
||||
|
||||
Run: `ruff check src/documents/management/commands/document_importer.py src/documents/tests/test_management_importer.py && ruff format src/documents/management/commands/document_importer.py src/documents/tests/test_management_importer.py`
|
||||
Expected: no errors.
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add src/documents/management/commands/document_importer.py src/documents/tests/test_management_importer.py
|
||||
git commit -m "Feature: importer rejects archives with unreadable compression"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Document the flags
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `docs/administration.md`
|
||||
|
||||
- [ ] **Step 1: Add the flags to the option list**
|
||||
|
||||
In `docs/administration.md`, update the usage block (around line 257) to include the new flags:
|
||||
|
||||
```
|
||||
document_exporter target [-c] [-d] [-f] [-na] [-nt] [-p] [-sm] [-z]
|
||||
|
||||
optional arguments:
|
||||
-c, --compare-checksums
|
||||
-cj, --compare-json
|
||||
-d, --delete
|
||||
-f, --use-filename-format
|
||||
-na, --no-archive
|
||||
-nt, --no-thumbnail
|
||||
-p, --use-folder-prefix
|
||||
-sm, --split-manifest
|
||||
-z, --zip
|
||||
-zn, --zip-name
|
||||
--zip-compression
|
||||
--zip-compression-level
|
||||
--data-only
|
||||
--no-progress-bar
|
||||
--passphrase
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add the prose**
|
||||
|
||||
After the `-z`/`--zip` paragraph (around line 330), add:
|
||||
|
||||
```markdown
|
||||
The compression method for the zip can be set with `--zip-compression`
|
||||
(`stored`, `deflated` (default), `bzip2`, `lzma`, or `zstd`) and tuned with
|
||||
`--zip-compression-level` (deflated: 0–9, bzip2: 1–9, zstd: -131072–22; ignored
|
||||
for `stored` and `lzma`). Both options require `--zip`.
|
||||
|
||||
!!! warning
|
||||
|
||||
`zstd` compression requires Python 3.14 or newer on **both** the machine
|
||||
creating the export and any machine importing it. An archive compressed with
|
||||
`zstd` (or `lzma`/`bzip2` where those modules are unavailable) cannot be
|
||||
imported on a runtime that lacks the codec; the importer will refuse it with
|
||||
a clear error. The default `deflated` is universally readable.
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Verify the docs build is not broken (lint markdown)**
|
||||
|
||||
Run: `ruff check docs/ 2>/dev/null; echo "docs are markdown; rely on prettier pre-commit"`
|
||||
(No code to test. The prettier pre-commit hook will reformat on commit.)
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add docs/administration.md
|
||||
git commit -m "Docs: document --zip-compression and --zip-compression-level"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 6: Final verification
|
||||
|
||||
**Files:** none (verification only).
|
||||
|
||||
- [ ] **Step 1: Full backend suites on the VM**
|
||||
|
||||
Run: `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests/export/ src/documents/tests/test_management_exporter.py src/documents/tests/test_management_importer.py -v"`
|
||||
Expected: PASS, no failures.
|
||||
|
||||
- [ ] **Step 2: Spot-check the zstd happy path on Python 3.14 (cannot run under Django on the 3.11 VM)**
|
||||
|
||||
The zstd positive round-trip can't run in the 3.11 test env. Confirm the policy module behaves on a real 3.14 interpreter with a standalone check (no Django needed):
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
uv run --python 3.14 --no-project python -c "import sys; sys.path.insert(0,'src'); import django; print('skip')" 2>/dev/null || \
|
||||
uv run --python 3.14 --no-project python -c "
|
||||
import zipfile, io
|
||||
from compression.zstd import CompressionParameter as CP
|
||||
print('zstd const', zipfile.ZIP_ZSTANDARD, 'bounds', CP.compression_level.bounds())
|
||||
buf = io.BytesIO()
|
||||
with zipfile.ZipFile(buf,'w',compression=zipfile.ZIP_ZSTANDARD,compresslevel=19) as zf:
|
||||
zf.writestr('a.txt','x'*1000)
|
||||
with zipfile.ZipFile(buf) as zf:
|
||||
assert zf.getinfo('a.txt').compress_type == zipfile.ZIP_ZSTANDARD
|
||||
assert zf.read('a.txt') == b'x'*1000
|
||||
print('zstd round-trip OK')
|
||||
"
|
||||
```
|
||||
|
||||
Expected: prints `zstd const 93 bounds (-131072, 22)` and `zstd round-trip OK`. This validates the constant, bounds, and that a zstd archive round-trips — the parts the 3.11 CI cannot exercise.
|
||||
|
||||
- [ ] **Step 3: Type-check on the VM (pyrefly)**
|
||||
|
||||
```bash
|
||||
tar czf - src pyproject.toml uv.lock .pyrefly-baseline.json | ssh -o BatchMode=yes -p 2244 trenton@localhost 'tar xzf - -C ~/projects/paperless-ngx'
|
||||
ssh -o BatchMode=yes -p 2244 trenton@localhost 'bash -lc "cd ~/projects/paperless-ngx && uv run pyrefly check"'
|
||||
```
|
||||
|
||||
Expected: no new type errors beyond the baseline. (Note: `import compression.zstd` is guarded behind `importlib.import_module`, so it is never statically resolved on the 3.11 baseline.)
|
||||
|
||||
- [ ] **Step 4: Final lint**
|
||||
|
||||
Run: `ruff check src/documents/export/ src/documents/management/commands/document_exporter.py src/documents/management/commands/document_importer.py && ruff format --check src/documents/export/ src/documents/management/commands/document_exporter.py src/documents/management/commands/document_importer.py`
|
||||
Expected: clean.
|
||||
|
||||
---
|
||||
|
||||
## Notes for the implementer
|
||||
|
||||
- **Default behavior is unchanged:** with no flags, the sink is constructed with `compression=ZIP_DEFLATED, compresslevel=None` — byte-method-identical to today (`shutil.make_archive` used `ZIP_DEFLATED` with no level). `test_default_zip_uses_deflate` pins this.
|
||||
- **zstd availability is gated three ways and never imported statically:** the constant via `getattr`, the codec via `importlib.import_module("compression.zstd")`, and the CLI value rejected with a friendly message on < 3.14. The choices list always contains `zstd` so argparse doesn't hide it.
|
||||
- **The importer pre-check is the safety net** for portability foot-guns — without it an unreadable entry raises a bare `NotImplementedError` mid-`extractall`. The check runs on `infolist()` (metadata only) before any extraction.
|
||||
- **Why `--zip-compression` defaults to `None`, not `"deflated"`:** so `handle()` can detect "user passed it without `--zip`" and fail fast. The effective default is resolved as `zip_compression or "deflated"`.
|
||||
@@ -1,236 +0,0 @@
|
||||
# Export Zip Compression Control — Design
|
||||
|
||||
**Date:** 2026-06-16
|
||||
**Branch base:** `dev`
|
||||
**Status:** Design complete (zstd facts verified on CPython 3.14.3) — **depends on**
|
||||
`2026-06-16-export-sink-architecture-design.md` being implemented first.
|
||||
|
||||
## Prerequisite
|
||||
|
||||
This builds directly on the export sink refactor. It assumes `ZipExportSink`
|
||||
already exists and is the single place that owns `zipfile.ZipFile` creation and
|
||||
entry writes. Do not start this until that refactor has landed; without it, the
|
||||
change would have to touch the command's zip branches again.
|
||||
|
||||
## Problem
|
||||
|
||||
Zip export is hardwired to `ZIP_DEFLATED` at the library default level. Users
|
||||
have no way to trade speed against archive size — a fast `ZIP_STORED` pass for a
|
||||
quick local copy, or a maximal `ZIP_LZMA` pass for the smallest off-site backup.
|
||||
The sink refactor turns "which compression" into a single constructor argument,
|
||||
so exposing it is now a small, isolated change.
|
||||
|
||||
## Goal
|
||||
|
||||
Let the operator choose the zip compression method and level from the CLI, with
|
||||
behavior identical to today when the flags are omitted. All knowledge of
|
||||
compression stays inside `ZipExportSink`; the command only parses flags and maps
|
||||
them to sink arguments.
|
||||
|
||||
## Scope
|
||||
|
||||
In scope:
|
||||
|
||||
- `ZipExportSink` gains `compression: int` and `compresslevel: int | None`
|
||||
constructor parameters (default `ZIP_DEFLATED`, `None` → library default),
|
||||
passed straight to `zipfile.ZipFile(...)`.
|
||||
- New `document_exporter` flags: `--zip-compression` and
|
||||
`--zip-compression-level`, valid only with `--zip`.
|
||||
- Validation: method availability, level range per method, and the
|
||||
requires-`--zip` guard.
|
||||
- Import-side: a pre-extract support check in `document_importer` that turns an
|
||||
unsupported codec into a clear `CommandError` (the importer otherwise decompresses
|
||||
transparently via `ZipFile.extractall`).
|
||||
- Docs: add both flags and the zstd-portability caveat to `docs/administration.md`
|
||||
(the `document_exporter` option list, lines ~257-270 and the `-z`/`-zn` section,
|
||||
lines ~328-330). New flags are long-form only (`--zip-compression`,
|
||||
`--zip-compression-level`) — no short aliases, to avoid `-zc`/`-zl` collisions
|
||||
with the existing `-z`/`-zn`.
|
||||
|
||||
Out of scope:
|
||||
|
||||
- Compression for any non-zip sink (folder has none; a future S3 sink would
|
||||
handle its own object storage compression separately).
|
||||
- Changing the default. Omitting the flags must produce a byte-compatible-method
|
||||
archive to today's (`ZIP_DEFLATED`, default level).
|
||||
|
||||
## Design
|
||||
|
||||
### `ZipExportSink` changes
|
||||
|
||||
The base sink's signature is `ZipExportSink(target, zip_name, *, delete)`; this
|
||||
adds two keyword-only params after `delete`:
|
||||
|
||||
```python
|
||||
def __init__(
|
||||
self,
|
||||
target: Path,
|
||||
zip_name: str,
|
||||
*,
|
||||
delete: bool = False,
|
||||
compression: int = zipfile.ZIP_DEFLATED,
|
||||
compresslevel: int | None = None,
|
||||
) -> None:
|
||||
...
|
||||
# opened in __enter__:
|
||||
self._zip = zipfile.ZipFile(
|
||||
self._tmp_path,
|
||||
"w",
|
||||
compression=compression,
|
||||
compresslevel=compresslevel,
|
||||
allowZip64=True,
|
||||
)
|
||||
```
|
||||
|
||||
`ZipFile` applies `compression`/`compresslevel` as the default for every
|
||||
`write`/`writestr` (verified: a `ZipFile(..., compression=ZIP_BZIP2)` yields
|
||||
entries with `compress_type == ZIP_BZIP2` without per-call args), so `add_file` /
|
||||
`add_json` / the manifest entry need no changes. Directory marker entries are
|
||||
empty so their compressed payload is zero, but they are still _tagged_ with the
|
||||
chosen `compress_type` — harmless, but tests that read `infolist()` should filter
|
||||
or account for marker entries (see Testing).
|
||||
|
||||
### CLI flags (`document_exporter`)
|
||||
|
||||
- `--zip-compression {stored,deflated,bzip2,lzma}` — and `zstd` **when the
|
||||
runtime supports it** (see below). Maps to the matching `zipfile.ZIP_*`
|
||||
constant. Default `deflated`.
|
||||
- `--zip-compression-level N` — integer. Per-method accepted ranges (verified
|
||||
against the [3.14 `zipfile` docs](https://docs.python.org/3.14/library/zipfile.html#zipfile.ZipFile)):
|
||||
- `deflated`: **0–9** (`zlib` also accepts `-1` = "default", identical to
|
||||
omitting the flag / `compresslevel=None`).
|
||||
- `bzip2`: **1–9** (`0` is invalid for bzip2).
|
||||
- `lzma`, `stored`: level has **no effect** — passing `--zip-compression-level`
|
||||
with either is a `CommandError`, not a silent accept (consistent with the
|
||||
base refactor's fail-fast posture).
|
||||
- `zstd`: **-131072 … 22** (the documented commonly-accepted range; the
|
||||
authoritative bounds are
|
||||
`compression.zstd.CompressionParameter.compression_level.bounds()`).
|
||||
|
||||
Default: unset → library default (`compresslevel=None`).
|
||||
|
||||
Both flags require `--zip`; passing either without `--zip` raises a
|
||||
`CommandError`, matching the incremental-flag rule from the base refactor.
|
||||
|
||||
**Why validate up front (not let `zipfile` raise) — verified on 3.14.3:** an
|
||||
invalid level does _not_ fail at `ZipFile(...)` construction — it fails at the
|
||||
**first `write`/`writestr` call**, with an opaque message
|
||||
(`ValueError: Invalid initialization option` for deflated > 9, or
|
||||
`ValueError: compresslevel must be between 1 and 9` for bzip2). Worse, on context
|
||||
exit the half-initialized write handle emits a secondary
|
||||
`AttributeError: '_ZipWriteFile' object has no attribute '_compressor'` during GC
|
||||
finalization, so the user sees stack-trace noise unrelated to the real cause.
|
||||
Up-front validation turns all of that into a single clean `CommandError`.
|
||||
|
||||
### Validation (in `handle()`, before constructing the sink)
|
||||
|
||||
1. **Requires `--zip`.** Either flag without `--zip` → `CommandError`.
|
||||
2. **Method availability — via a named, patchable seam.** Expose a module-level
|
||||
helper `compression_available(method: str) -> bool` that does
|
||||
`try: import bz2 / import lzma / from compression import zstd except ImportError:
|
||||
return False` — **not** `importlib.util.find_spec`, which can report a stdlib
|
||||
C-extension as present when importing it actually fails. `stored`/`deflated`
|
||||
are always available (`zlib` is a hard CPython dependency). For `zstd` the probe
|
||||
must import `compression.zstd` (3.14+), not merely check that
|
||||
`zipfile.ZIP_ZSTANDARD` exists. Making this a named function is also what lets
|
||||
the test patch "method unavailable" with `mocker`. If the chosen method is
|
||||
unavailable, raise a `CommandError` naming the missing capability — `zipfile`
|
||||
itself would otherwise raise a bare `RuntimeError`
|
||||
("Compression requires the (missing) … module").
|
||||
3. **Level range.** Reject an out-of-range `--zip-compression-level` for the
|
||||
chosen method with a clear `CommandError`; reject the flag entirely for
|
||||
`stored`/`lzma` (see above).
|
||||
|
||||
### zstd (Python 3.14+)
|
||||
|
||||
**Verified empirically on CPython 3.14.3** (via `uv run --python 3.14 --no-project`)
|
||||
and against [PEP 784](https://peps.python.org/pep-0784/) +
|
||||
[the 3.14 `zipfile` docs](https://docs.python.org/3.14/library/zipfile.html):
|
||||
|
||||
- The compression-method constant is **`zipfile.ZIP_ZSTANDARD`** (added 3.14; its
|
||||
numeric value is `93`). It does **not** exist on < 3.14.
|
||||
- It is backed by the new **`compression.zstd`** stdlib module (PEP 784 added a
|
||||
`compression` namespace package; legacy `bz2`/`lzma`/`zlib` imports are
|
||||
unchanged). `zipfile` raises `RuntimeError` if `compression.zstd` is
|
||||
unavailable when zstd is requested.
|
||||
- Accepted `compresslevel` is **`-131072 … 22`**, confirmed at runtime via
|
||||
`compression.zstd.CompressionParameter.compression_level.bounds() == (-131072, 22)`.
|
||||
|
||||
Gate everything zstd-related at runtime so nothing is imported or referenced on
|
||||
< 3.14 (the project targets Python ≥ 3.11):
|
||||
|
||||
```python
|
||||
_ZSTD: int | None = getattr(zipfile, "ZIP_ZSTANDARD", None) # None before 3.14
|
||||
```
|
||||
|
||||
Presence of the _constant_ does not guarantee the _codec_ is usable, so the
|
||||
availability probe (validation step 2) imports `compression.zstd`, not merely
|
||||
checks the constant.
|
||||
|
||||
Keep `zstd` in the `--zip-compression` `choices` **always** (even on < 3.14), and
|
||||
reject it in validation with a friendly "zstd requires Python 3.14+" message. If
|
||||
it were dropped from `choices` on older runtimes, argparse would emit a generic
|
||||
"invalid choice" that reads as though the option never existed — worse UX.
|
||||
|
||||
### Import-side compatibility
|
||||
|
||||
`document_importer` reads zips with `ZipFile(self.source).extractall(...)`
|
||||
(`document_importer.py:453`), which decompresses each entry transparently using
|
||||
whatever method it was stored with — **provided the matching module exists on the
|
||||
importing machine.**
|
||||
|
||||
The failure mode when it doesn't is unfriendly and must be handled: a zstd (or
|
||||
otherwise unsupported) entry raises a bare `NotImplementedError` **per-entry,
|
||||
during `extractall`** — _not_ at `ZipFile(self.source)` open, and `is_zipfile()`
|
||||
still returns true (a zstd archive is a valid zip container). So the importer
|
||||
enters the zip branch, creates its temp dir, may partially extract other entries,
|
||||
then blows up mid-extract with no context. **Mitigation (in scope here):** before
|
||||
extracting, inspect `ZipFile(self.source).infolist()` compress types and, if any
|
||||
is unsupported on this runtime, raise a `CommandError` naming the method and the
|
||||
requirement (e.g. "this archive uses zstd, which needs Python 3.14+") instead of
|
||||
letting `NotImplementedError` escape.
|
||||
|
||||
Per-method summary (document in help text + `administration.md`):
|
||||
|
||||
- `deflated`/`stored`: universally importable.
|
||||
- `bzip2`/`lzma`: importable wherever the `bz2`/`lzma` modules are present
|
||||
(essentially always).
|
||||
- `zstd`: importable only on Python 3.14+. An archive compressed with `zstd` is
|
||||
**not** importable on older runtimes.
|
||||
|
||||
## Testing
|
||||
|
||||
New cases in the sink tests and an export→import round-trip
|
||||
(pytest classes, factory-boy, `mocker`, `parametrize`, typed; run on the VM):
|
||||
|
||||
- **Round-trip per method.** Parametrize over the available methods (skip `zstd`
|
||||
below 3.14, skip `bzip2`/`lzma` if the module is somehow absent): export a
|
||||
small library, import it back, assert documents/manifest match.
|
||||
- **Method is applied.** Assert each written _file_ entry's `compress_type`
|
||||
equals the requested method (read back via `ZipFile.infolist()`), filtering out
|
||||
directory marker entries (which are tagged but empty).
|
||||
- **Level affects size — robustly.** Do **not** compare deflate level 9 vs 1
|
||||
(on small or incompressible fixtures level 9 can equal or slightly exceed level
|
||||
1, causing flaky CI). Instead assert that a compressing method on a
|
||||
moderately-compressible fixture yields a total smaller than `stored`
|
||||
(`ZIP_STORED`), which is a stable invariant.
|
||||
- **Validation.** Each flag without `--zip` → `CommandError`; out-of-range level
|
||||
(`--zip-compression-level 99`) → a clean `CommandError` from validation
|
||||
(asserting we never reach the `writestr` that would raise the masked
|
||||
`ValueError`); `--zip-compression-level` with `stored`/`lzma` → `CommandError`;
|
||||
unavailable method (patch the named availability seam with `mocker`) →
|
||||
`CommandError`; on < 3.14, `--zip-compression zstd` → the friendly
|
||||
"requires 3.14+" `CommandError`.
|
||||
- **Import pre-check.** An archive containing an unsupported compress type
|
||||
produces a `CommandError` from the importer naming the method, not a raw
|
||||
`NotImplementedError` (simulate by patching the importer's support probe).
|
||||
- **Default unchanged.** Omitting both flags yields file entries with
|
||||
`compress_type == ZIP_DEFLATED`, identical to pre-feature behavior.
|
||||
|
||||
## Risks
|
||||
|
||||
- **Foot-gun archives.** A user could produce a `zstd`/`lzma` archive their
|
||||
import target can't read. Mitigation: explicit help text and the import-side
|
||||
notes above; the default stays the universally-readable `deflated`.
|
||||
- **Optional-module assumptions.** Don't assume `bz2`/`lzma` are always compiled
|
||||
in; probe and error clearly. Mitigation: the availability validation step.
|
||||
@@ -1,108 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import zipfile
|
||||
|
||||
# ZIP_ZSTANDARD exists only on Python 3.14+ (PEP 784). None elsewhere.
|
||||
ZSTD: int | None = getattr(zipfile, "ZIP_ZSTANDARD", None)
|
||||
|
||||
# CLI choices are fixed across runtimes so argparse never hides zstd; runtime
|
||||
# availability is enforced separately in compression_available().
|
||||
COMPRESSION_CHOICES: tuple[str, ...] = (
|
||||
"stored",
|
||||
"deflated",
|
||||
"bzip2",
|
||||
"lzma",
|
||||
"zstd",
|
||||
)
|
||||
|
||||
# Method name -> zipfile compression constant (zstd only when supported).
|
||||
COMPRESSION_METHODS: dict[str, int] = {
|
||||
"stored": zipfile.ZIP_STORED,
|
||||
"deflated": zipfile.ZIP_DEFLATED,
|
||||
"bzip2": zipfile.ZIP_BZIP2,
|
||||
"lzma": zipfile.ZIP_LZMA,
|
||||
}
|
||||
if ZSTD is not None:
|
||||
COMPRESSION_METHODS["zstd"] = ZSTD
|
||||
|
||||
# Inclusive (min, max) level bounds per method; None => level not applicable.
|
||||
# Verified on CPython 3.14.3.
|
||||
#
|
||||
# zstd's raw library bounds are (-131072, 22)
|
||||
# (compression.zstd.CompressionParameter.compression_level.bounds()) — the
|
||||
# minimum is an internal implementation constant (-ZSTD_TARGETLENGTH_MAX),
|
||||
# not a meaningful distinct "level"; deeper negative values than -22 buy
|
||||
# nothing over -22 in practice. We expose the conventional zstd CLI range
|
||||
# instead of the raw library bounds.
|
||||
LEVEL_BOUNDS: dict[str, tuple[int, int] | None] = {
|
||||
"stored": None,
|
||||
"deflated": (0, 9),
|
||||
"bzip2": (1, 9),
|
||||
"lzma": None,
|
||||
"zstd": (-22, 22),
|
||||
}
|
||||
|
||||
# zipfile compress_type id -> method name. 93 = current zstd id, 20 = legacy
|
||||
# zstd id that zipfile can still read.
|
||||
_COMPRESS_TYPE_TO_METHOD: dict[int, str] = {
|
||||
zipfile.ZIP_STORED: "stored",
|
||||
zipfile.ZIP_DEFLATED: "deflated",
|
||||
zipfile.ZIP_BZIP2: "bzip2",
|
||||
zipfile.ZIP_LZMA: "lzma",
|
||||
93: "zstd",
|
||||
20: "zstd",
|
||||
}
|
||||
|
||||
|
||||
def compression_available(method: str) -> bool:
|
||||
"""Whether the running interpreter can actually use the given method."""
|
||||
if method in ("stored", "deflated"):
|
||||
# zlib is a hard CPython dependency; stored needs nothing.
|
||||
return True
|
||||
if method == "bzip2":
|
||||
return _module_importable("bz2")
|
||||
if method == "lzma":
|
||||
return _module_importable("lzma")
|
||||
if method == "zstd":
|
||||
return ZSTD is not None and _module_importable("compression.zstd")
|
||||
return False
|
||||
|
||||
|
||||
def _module_importable(name: str) -> bool:
|
||||
try:
|
||||
importlib.import_module(name)
|
||||
except ImportError:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def level_error(method: str, level: int | None) -> str | None:
|
||||
"""Return a human message if (method, level) is invalid, else None."""
|
||||
if level is None:
|
||||
return None
|
||||
bounds = LEVEL_BOUNDS[method]
|
||||
if bounds is None:
|
||||
return f"--zip-compression-level has no effect for '{method}'"
|
||||
low, high = bounds
|
||||
if not (low <= level <= high):
|
||||
return (
|
||||
f"--zip-compression-level for '{method}' must be between {low} and {high}"
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
def compress_type_readable(compress_type: int) -> bool:
|
||||
"""Whether this interpreter can decompress an entry of the given type."""
|
||||
method = _COMPRESS_TYPE_TO_METHOD.get(compress_type)
|
||||
if method is None:
|
||||
return False
|
||||
return compression_available(method)
|
||||
|
||||
|
||||
def unreadable_method_names(compress_types: set[int]) -> set[str]:
|
||||
"""Map a set of compress_type ids to human method names for error messages."""
|
||||
names: set[str] = set()
|
||||
for ct in compress_types:
|
||||
names.add(_COMPRESS_TYPE_TO_METHOD.get(ct, f"method {ct}"))
|
||||
return names
|
||||
@@ -239,21 +239,11 @@ class ZipExportSink(ExportSink):
|
||||
added as an entry at finalize (a zip entry cannot be interleaved with others).
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
target: Path,
|
||||
zip_name: str,
|
||||
*,
|
||||
delete: bool = False,
|
||||
compression: int = zipfile.ZIP_DEFLATED,
|
||||
compresslevel: int | None = None,
|
||||
) -> None:
|
||||
def __init__(self, target: Path, zip_name: str, *, delete: bool = False) -> None:
|
||||
self._target = target.resolve()
|
||||
self._zip_path = (self._target / zip_name).with_suffix(".zip")
|
||||
self._tmp_path = self._zip_path.with_name(self._zip_path.name + ".tmp")
|
||||
self._delete = delete
|
||||
self._compression = compression
|
||||
self._compresslevel = compresslevel
|
||||
self._zip: zipfile.ZipFile | None = None
|
||||
self._dirs: set[str] = set()
|
||||
self._pending_manifest: tuple[Path, str] | None = None
|
||||
@@ -264,8 +254,7 @@ class ZipExportSink(ExportSink):
|
||||
self._zip = zipfile.ZipFile(
|
||||
self._tmp_path,
|
||||
"w",
|
||||
compression=self._compression,
|
||||
compresslevel=self._compresslevel,
|
||||
compression=zipfile.ZIP_DEFLATED,
|
||||
allowZip64=True,
|
||||
)
|
||||
|
||||
|
||||
@@ -29,11 +29,6 @@ if TYPE_CHECKING:
|
||||
if settings.AUDIT_LOG_ENABLED:
|
||||
from auditlog.models import LogEntry
|
||||
|
||||
from documents.export.compression import COMPRESSION_CHOICES
|
||||
from documents.export.compression import COMPRESSION_METHODS
|
||||
from documents.export.compression import ZSTD
|
||||
from documents.export.compression import compression_available
|
||||
from documents.export.compression import level_error
|
||||
from documents.export.sinks import DirectoryExportSink
|
||||
from documents.export.sinks import ExportSink
|
||||
from documents.export.sinks import StreamingManifestWriter
|
||||
@@ -196,28 +191,6 @@ class Command(CryptMixin, PaperlessCommand):
|
||||
help="Sets the export zip file name",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--zip-compression",
|
||||
choices=COMPRESSION_CHOICES,
|
||||
default=None,
|
||||
help=(
|
||||
"Compression method for the export zip (requires --zip). "
|
||||
"Default: deflated. 'zstd' requires Python 3.14+ on both the "
|
||||
"exporting and importing machine."
|
||||
),
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--zip-compression-level",
|
||||
type=int,
|
||||
default=None,
|
||||
help=(
|
||||
"Compression level for the export zip (requires --zip). "
|
||||
"deflated: 0-9, bzip2: 1-9, zstd: -22..22; ignored for "
|
||||
"stored/lzma."
|
||||
),
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--data-only",
|
||||
default=False,
|
||||
@@ -273,39 +246,12 @@ class Command(CryptMixin, PaperlessCommand):
|
||||
if not os.access(self.target, os.W_OK):
|
||||
raise CommandError("That path doesn't appear to be writable")
|
||||
|
||||
zip_compression: str | None = options["zip_compression"]
|
||||
zip_compression_level: int | None = options["zip_compression_level"]
|
||||
|
||||
if not self.zip_export and (
|
||||
zip_compression is not None or zip_compression_level is not None
|
||||
):
|
||||
raise CommandError(
|
||||
"--zip-compression and --zip-compression-level require --zip",
|
||||
)
|
||||
|
||||
compression_method = zip_compression or "deflated"
|
||||
if self.zip_export:
|
||||
if not compression_available(compression_method):
|
||||
if compression_method == "zstd" and ZSTD is None:
|
||||
raise CommandError(
|
||||
"zstd compression requires Python 3.14 or newer",
|
||||
)
|
||||
raise CommandError(
|
||||
f"Compression method '{compression_method}' is not "
|
||||
f"available on this Python runtime",
|
||||
)
|
||||
level_msg = level_error(compression_method, zip_compression_level)
|
||||
if level_msg is not None:
|
||||
raise CommandError(level_msg)
|
||||
|
||||
sink: ExportSink
|
||||
if self.zip_export:
|
||||
sink = ZipExportSink(
|
||||
self.target,
|
||||
options["zip_name"],
|
||||
delete=self.delete,
|
||||
compression=COMPRESSION_METHODS[compression_method],
|
||||
compresslevel=zip_compression_level,
|
||||
)
|
||||
else:
|
||||
sink = DirectoryExportSink(
|
||||
|
||||
@@ -32,8 +32,6 @@ from django.db.models.signals import post_save
|
||||
from filelock import FileLock
|
||||
from guardian.shortcuts import clear_ct_cache
|
||||
|
||||
from documents.export.compression import compress_type_readable
|
||||
from documents.export.compression import unreadable_method_names
|
||||
from documents.file_handling import create_source_path_directory
|
||||
from documents.management.commands.base import PaperlessCommand
|
||||
from documents.management.commands.mixins import CryptMixin
|
||||
@@ -453,20 +451,6 @@ class Command(CryptMixin, PaperlessCommand):
|
||||
with tempfile.TemporaryDirectory() as tmp_dir:
|
||||
if is_zipfile(self.source):
|
||||
with ZipFile(self.source) as zf:
|
||||
unsupported = {
|
||||
info.compress_type
|
||||
for info in zf.infolist()
|
||||
if not compress_type_readable(info.compress_type)
|
||||
}
|
||||
if unsupported:
|
||||
names = sorted(unreadable_method_names(unsupported))
|
||||
message = (
|
||||
f"This archive uses compression this Python cannot "
|
||||
f"read ({', '.join(names)})."
|
||||
)
|
||||
if "zstd" in names:
|
||||
message += " zstd archives require Python 3.14+."
|
||||
raise CommandError(message)
|
||||
zf.extractall(tmp_dir)
|
||||
self.source = Path(tmp_dir)
|
||||
self._run_import()
|
||||
|
||||
@@ -1,190 +0,0 @@
|
||||
import sys
|
||||
import zipfile
|
||||
|
||||
import pytest
|
||||
|
||||
from documents.export import compression
|
||||
|
||||
|
||||
class TestCompressionMethods:
|
||||
def test_choices_always_include_zstd(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- The compression policy module's CLI choices list
|
||||
WHEN:
|
||||
- Read on any runtime
|
||||
THEN:
|
||||
- zstd is always present; availability is checked separately so
|
||||
argparse never hides it based on the current Python version
|
||||
"""
|
||||
assert compression.COMPRESSION_CHOICES == (
|
||||
"stored",
|
||||
"deflated",
|
||||
"bzip2",
|
||||
"lzma",
|
||||
"zstd",
|
||||
)
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("name", "constant"),
|
||||
[
|
||||
("stored", zipfile.ZIP_STORED),
|
||||
("deflated", zipfile.ZIP_DEFLATED),
|
||||
("bzip2", zipfile.ZIP_BZIP2),
|
||||
("lzma", zipfile.ZIP_LZMA),
|
||||
],
|
||||
)
|
||||
def test_method_maps_to_zipfile_constant(self, name: str, constant: int) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- A compression method name
|
||||
WHEN:
|
||||
- Looked up in COMPRESSION_METHODS
|
||||
THEN:
|
||||
- It maps to the matching zipfile compression constant
|
||||
"""
|
||||
assert compression.COMPRESSION_METHODS[name] == constant
|
||||
|
||||
def test_stored_and_deflated_always_available(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- The stored and deflated compression methods
|
||||
WHEN:
|
||||
- Checked with compression_available()
|
||||
THEN:
|
||||
- Both are always available (zlib is a hard CPython dependency)
|
||||
"""
|
||||
assert compression.compression_available("stored")
|
||||
assert compression.compression_available("deflated")
|
||||
|
||||
def test_zstd_availability_tracks_runtime(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- The zstd compression method
|
||||
WHEN:
|
||||
- Checked with compression_available() on this runtime
|
||||
THEN:
|
||||
- Availability matches whether Python is 3.14+
|
||||
"""
|
||||
expected: bool = sys.version_info >= (3, 14)
|
||||
assert compression.compression_available("zstd") == expected
|
||||
|
||||
|
||||
class TestLevelError:
|
||||
@pytest.mark.parametrize(
|
||||
("method", "level"),
|
||||
[
|
||||
("deflated", 0),
|
||||
("deflated", 9),
|
||||
("bzip2", 1),
|
||||
("bzip2", 9),
|
||||
("zstd", -22),
|
||||
("zstd", 22),
|
||||
("deflated", None),
|
||||
("stored", None),
|
||||
],
|
||||
)
|
||||
def test_valid_levels_return_none(self, method: str, level: int | None) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- A method and a level within its valid bounds (or no level)
|
||||
WHEN:
|
||||
- Checked with level_error()
|
||||
THEN:
|
||||
- No error message is returned
|
||||
"""
|
||||
assert compression.level_error(method, level) is None
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("method", "level"),
|
||||
[
|
||||
("deflated", 10),
|
||||
("deflated", -1),
|
||||
("bzip2", 0),
|
||||
("bzip2", 10),
|
||||
("zstd", -23),
|
||||
("zstd", 23),
|
||||
],
|
||||
)
|
||||
def test_out_of_range_levels_return_message(
|
||||
self,
|
||||
method: str,
|
||||
level: int,
|
||||
) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- A method and a level outside its valid bounds
|
||||
WHEN:
|
||||
- Checked with level_error()
|
||||
THEN:
|
||||
- An error message naming the valid range is returned
|
||||
"""
|
||||
msg: str | None = compression.level_error(method, level)
|
||||
assert msg is not None
|
||||
assert "between" in msg
|
||||
|
||||
@pytest.mark.parametrize("method", ["stored", "lzma"])
|
||||
def test_level_on_levelless_method_is_rejected(self, method: str) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- A method that ignores compression level (stored, lzma)
|
||||
WHEN:
|
||||
- A level is passed to level_error() anyway
|
||||
THEN:
|
||||
- An error message noting the level has no effect is returned
|
||||
"""
|
||||
msg: str | None = compression.level_error(method, 5)
|
||||
assert msg is not None
|
||||
assert "no effect" in msg
|
||||
|
||||
|
||||
class TestCompressTypeReadable:
|
||||
@pytest.mark.parametrize("ct", [zipfile.ZIP_STORED, zipfile.ZIP_DEFLATED])
|
||||
def test_stored_and_deflated_always_readable(self, ct: int) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- A stored or deflated compress_type id
|
||||
WHEN:
|
||||
- Checked with compress_type_readable()
|
||||
THEN:
|
||||
- It is always readable
|
||||
"""
|
||||
assert compression.compress_type_readable(ct)
|
||||
|
||||
def test_zstd_compress_type_readability_tracks_runtime(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- The current (93) and legacy (20) zstd compress_type ids
|
||||
WHEN:
|
||||
- Checked with compress_type_readable() on this runtime
|
||||
THEN:
|
||||
- Readability matches whether Python is 3.14+
|
||||
"""
|
||||
# 93 = ZIP_ZSTANDARD; 20 = legacy zstd method id (read-only)
|
||||
expected: bool = sys.version_info >= (3, 14)
|
||||
assert compression.compress_type_readable(93) == expected
|
||||
assert compression.compress_type_readable(20) == expected
|
||||
|
||||
def test_unknown_compress_type_is_unreadable(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- An unrecognized compress_type id
|
||||
WHEN:
|
||||
- Checked with compress_type_readable()
|
||||
THEN:
|
||||
- It is reported as unreadable
|
||||
"""
|
||||
assert not compression.compress_type_readable(9999)
|
||||
|
||||
def test_unreadable_method_names_lists_methods(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- A set containing an unknown compress_type id
|
||||
WHEN:
|
||||
- Passed to unreadable_method_names()
|
||||
THEN:
|
||||
- It is reported generically as "method <id>"
|
||||
"""
|
||||
# An unknown method id maps to no name and is reported generically.
|
||||
names: set[str] = compression.unreadable_method_names({9999})
|
||||
assert names == {"method 9999"}
|
||||
@@ -5,7 +5,6 @@ import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import pytest_mock
|
||||
|
||||
from documents.export.sinks import DirectoryExportSink
|
||||
from documents.export.sinks import ExportSink
|
||||
@@ -268,48 +267,6 @@ class TestZipExportSink:
|
||||
assert not (target / "export.zip").exists()
|
||||
|
||||
|
||||
class TestZipExportSinkCompression:
|
||||
@pytest.mark.parametrize(
|
||||
("method", "constant"),
|
||||
[
|
||||
("stored", zipfile.ZIP_STORED),
|
||||
("deflated", zipfile.ZIP_DEFLATED),
|
||||
("bzip2", zipfile.ZIP_BZIP2),
|
||||
("lzma", zipfile.ZIP_LZMA),
|
||||
],
|
||||
)
|
||||
def test_compression_and_level_forwarded_to_zipfile(
|
||||
self,
|
||||
mocker: pytest_mock.MockerFixture,
|
||||
tmp_path: Path,
|
||||
method: str,
|
||||
constant: int,
|
||||
) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- A ZipExportSink constructed with a compression method and level
|
||||
WHEN:
|
||||
- The sink is opened
|
||||
THEN:
|
||||
- zipfile.ZipFile is constructed with those values forwarded
|
||||
unchanged (whether ZipFile actually compresses is Python's own
|
||||
contract, not ours, so this checks the call args, not a real
|
||||
archive)
|
||||
"""
|
||||
target: Path = tmp_path / "out"
|
||||
target.mkdir()
|
||||
zip_cls = mocker.patch("documents.export.sinks.zipfile.ZipFile")
|
||||
sink = ZipExportSink(target, "export", compression=constant, compresslevel=5)
|
||||
sink._open()
|
||||
zip_cls.assert_called_once_with(
|
||||
mocker.ANY,
|
||||
"w",
|
||||
compression=constant,
|
||||
compresslevel=5,
|
||||
allowZip64=True,
|
||||
)
|
||||
|
||||
|
||||
class TestStreamContract:
|
||||
@pytest.fixture(params=["dir", "zip"])
|
||||
def sink(self, request: pytest.FixtureRequest, tmp_path: Path) -> ExportSink:
|
||||
|
||||
@@ -6,8 +6,6 @@ from datetime import timedelta
|
||||
from io import StringIO
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
from zipfile import ZIP_DEFLATED
|
||||
from zipfile import ZIP_LZMA
|
||||
from zipfile import ZipFile
|
||||
|
||||
import pytest
|
||||
@@ -1080,170 +1078,6 @@ class TestExportImport(
|
||||
skip_checks=True,
|
||||
)
|
||||
|
||||
def test_compression_flags_require_zip(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- A request to export without --zip
|
||||
WHEN:
|
||||
- --zip-compression or --zip-compression-level is passed anyway
|
||||
THEN:
|
||||
- A CommandError is raised (the flags are meaningless without --zip)
|
||||
"""
|
||||
for args in (
|
||||
["--zip-compression", "lzma"],
|
||||
["--zip-compression-level", "5"],
|
||||
):
|
||||
with self.assertRaises(CommandError):
|
||||
call_command(
|
||||
"document_exporter",
|
||||
self.target,
|
||||
*args,
|
||||
skip_checks=True,
|
||||
)
|
||||
|
||||
def test_zip_compression_level_out_of_range_raises(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- A request to export to a zip file
|
||||
WHEN:
|
||||
- --zip-compression-level is outside the chosen method's valid range
|
||||
THEN:
|
||||
- A CommandError is raised
|
||||
"""
|
||||
with self.assertRaises(CommandError):
|
||||
call_command(
|
||||
"document_exporter",
|
||||
self.target,
|
||||
"--zip",
|
||||
"--zip-compression",
|
||||
"deflated",
|
||||
"--zip-compression-level",
|
||||
"99",
|
||||
skip_checks=True,
|
||||
)
|
||||
|
||||
def test_zip_compression_level_rejected_for_stored(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- A request to export to a zip file with --zip-compression stored
|
||||
WHEN:
|
||||
- --zip-compression-level is also passed
|
||||
THEN:
|
||||
- A CommandError is raised (stored ignores level entirely)
|
||||
"""
|
||||
with self.assertRaises(CommandError):
|
||||
call_command(
|
||||
"document_exporter",
|
||||
self.target,
|
||||
"--zip",
|
||||
"--zip-compression",
|
||||
"stored",
|
||||
"--zip-compression-level",
|
||||
"5",
|
||||
skip_checks=True,
|
||||
)
|
||||
|
||||
def test_zip_compression_level_rejected_for_lzma(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- A request to export to a zip file with --zip-compression lzma
|
||||
WHEN:
|
||||
- --zip-compression-level is also passed
|
||||
THEN:
|
||||
- A CommandError is raised (lzma ignores level entirely)
|
||||
"""
|
||||
with self.assertRaises(CommandError):
|
||||
call_command(
|
||||
"document_exporter",
|
||||
self.target,
|
||||
"--zip",
|
||||
"--zip-compression",
|
||||
"lzma",
|
||||
"--zip-compression-level",
|
||||
"5",
|
||||
skip_checks=True,
|
||||
)
|
||||
|
||||
def test_zstd_unavailable_raises_friendly_error(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- A Python runtime without zstd support (< 3.14)
|
||||
WHEN:
|
||||
- --zip-compression zstd is requested
|
||||
THEN:
|
||||
- A CommandError naming the Python version requirement is raised
|
||||
"""
|
||||
with self.assertRaises(CommandError) as e:
|
||||
call_command(
|
||||
"document_exporter",
|
||||
self.target,
|
||||
"--zip",
|
||||
"--zip-compression",
|
||||
"zstd",
|
||||
skip_checks=True,
|
||||
)
|
||||
self.assertIn("3.14", str(e.exception))
|
||||
|
||||
def test_zip_compression_flag_resolves_to_sink_constant(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- A request to export to a zip file with --zip-compression lzma
|
||||
WHEN:
|
||||
- The export runs
|
||||
THEN:
|
||||
- ZipExportSink is constructed with the resolved ZIP_LZMA constant
|
||||
(whether zipfile actually compresses with the chosen method is
|
||||
Python's own contract, and ZipExportSink's own tests already
|
||||
cover the forwarding; what this command owns is resolving the
|
||||
CLI string to the right constant, so assert that resolution
|
||||
directly)
|
||||
"""
|
||||
with mock.patch(
|
||||
"documents.management.commands.document_exporter.ZipExportSink",
|
||||
) as sink_cls:
|
||||
call_command(
|
||||
"document_exporter",
|
||||
self.target,
|
||||
"--zip",
|
||||
"--zip-compression",
|
||||
"lzma",
|
||||
skip_checks=True,
|
||||
)
|
||||
sink_cls.assert_called_once_with(
|
||||
mock.ANY,
|
||||
mock.ANY,
|
||||
delete=False,
|
||||
compression=ZIP_LZMA,
|
||||
compresslevel=None,
|
||||
)
|
||||
|
||||
def test_default_zip_compression_resolves_to_deflate(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- A request to export to a zip file with no --zip-compression flag
|
||||
WHEN:
|
||||
- The export runs
|
||||
THEN:
|
||||
- ZipExportSink is constructed with the default ZIP_DEFLATED
|
||||
constant and compresslevel=None, matching pre-existing behavior
|
||||
"""
|
||||
with mock.patch(
|
||||
"documents.management.commands.document_exporter.ZipExportSink",
|
||||
) as sink_cls:
|
||||
call_command(
|
||||
"document_exporter",
|
||||
self.target,
|
||||
"--zip",
|
||||
skip_checks=True,
|
||||
)
|
||||
sink_cls.assert_called_once_with(
|
||||
mock.ANY,
|
||||
mock.ANY,
|
||||
delete=False,
|
||||
compression=ZIP_DEFLATED,
|
||||
compresslevel=None,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.management
|
||||
class TestCryptExportImport(
|
||||
|
||||
@@ -525,35 +525,6 @@ class TestCommandImport(
|
||||
self.assertEqual(doc.tags.count(), 1)
|
||||
self.assertEqual(doc.tags.first().name, "batch-flush-tag")
|
||||
|
||||
def test_import_rejects_unreadable_compression(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- A zip archive with an entry whose compression this Python can't read
|
||||
WHEN:
|
||||
- Import is attempted
|
||||
THEN:
|
||||
- A CommandError naming the issue is raised, before extraction
|
||||
"""
|
||||
import zipfile
|
||||
from unittest import mock
|
||||
|
||||
archive = Path(self.dirs.scratch_dir) / "export.zip"
|
||||
with zipfile.ZipFile(archive, "w") as zf:
|
||||
zf.writestr("manifest.json", "[]")
|
||||
|
||||
with mock.patch(
|
||||
"documents.management.commands.document_importer.compress_type_readable",
|
||||
return_value=False,
|
||||
):
|
||||
with self.assertRaises(CommandError) as e:
|
||||
call_command(
|
||||
"document_importer",
|
||||
str(archive),
|
||||
"--no-progress-bar",
|
||||
skip_checks=True,
|
||||
)
|
||||
self.assertIn("compression", str(e.exception))
|
||||
|
||||
|
||||
@pytest.mark.management
|
||||
@pytest.mark.django_db
|
||||
|
||||
Reference in New Issue
Block a user