chore: add whoosh-compat transition skill

Encodes the settled integration decisions for replacing the
hand-maintained search translation layer with whoosh-compat:
user-typed query surface policy, analyzer seam, diagnostics-before-emit
contract, mandatory date parity audit, test churn, and rollout plan.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

.claude/skills/whoosh-compat-transition/SKILL.md

@@ -0,0 +1,55 @@
+---
+name: whoosh-compat-transition
+description: Use when integrating the whoosh-compat library into paperless-ngx search, replacing src/documents/search/_translate.py or _dates.py, building the search FieldRegistry, or changing user query parsing during the whoosh-to-tantivy transition
+---
+
+# whoosh-compat transition
+
+## Overview
+
+whoosh-compat (github.com/stumpylog/whoosh-compat; local checkout usually at `../whoosh-compat`) replaces the hand-maintained translation layer (`src/documents/search/_translate.py`, `_dates.py`): it parses user queries with a faithful fork of whoosh's real grammar into a typed AST and emits programmatic tantivy queries. Read its README and ARCHITECTURE.md before wiring anything; its DIVERGENCES.md lists intended behavior differences and is the authority on "is this difference a bug".
+
+## Decisions already made (do not re-derive)
+
+- **Queries are user-typed free text.** The advanced search box passes whatever the user types straight to the parser (that is how the issue #13568 queries exist). Do NOT try to infer the supported field surface from frontend code; the frontend only generates a few date filter strings, everything else is typed by users.
+- **The field surface is a policy decision, not `KNOWN_FIELDS`.** Today's `KNOWN_FIELDS` accepts internal ID fields (`tag_id`, `owner_id`, `viewer_id`, other `*_id`) that are undocumented in `docs/usage.md` and were ruled not user-searchable by the maintainer: exclude them from the `FieldRegistry` (they stay as programmatic permission/filter fields in `build_permission_filter`, which never touches user query text). The registry is built from documented syntax in `docs/usage.md` plus the v2-compat aliases (`type`, `path`, `type_id`-style aliases follow their canonical field's fate). Undocumented-but-working fields (`asn`, `page_count`, `num_notes`, `original_filename`, `checksum`) need an explicit maintainer yes/no; since users type freely, silently dropping one breaks any saved view using it, so a drop must be a visible, documented decision.
+- **Analyzer seam:** `FieldSpec.analyzer` binds the live registered tantivy analyzer's `.analyze` (the same Rust analyzer used at index time; language-keyed, so rebuild the registry when `SEARCH_LANGUAGE` changes, on the same trigger as `register_tokenizers`). `pattern_normalizer` is `_tokenizer.ascii_fold`: character-level lowercase+fold only, NEVER stemming.
+- **Diagnostics before emit:** `whoosh_compat.parse()` never raises on bad input. Check `ParseResult.diagnostics` and map to `SearchQueryError`/`InvalidDateQuery` (HTTP 400) BEFORE calling `emit()`; also catch the emitter's `UnsupportedQueryError` into a 400. Never carry forward the legacy raw-string fallback (`except Exception: query_str = raw_query`) into the new path; it masks integration bugs.
+- **`notes` and `custom_fields` are JSON fields** with fixed subpaths (`notes.user`/`notes.note`, `custom_fields.name`/`custom_fields.value`); the registry stays a static, language-keyed singleton, never per-request.
+
+## Mandatory before deleting old code
+
+- Date-grammar parity audit, line by line: every keyword, relative unit, and abbreviation `_dates.py` and `_translate.py` accept today (including the whoosh-era abbreviations kept for old saved views) must have an accepted form in whoosh-compat's dateparse grammar. Silent keyword loss is the saved-view breakage class behind issue #13568.
+- Acceptance corpus compared by matched-document-ID sets, not query strings: the #13568 queries verbatim, real saved-view strings, every date keyword, field aliases, comma lists, date and numeric ranges, wildcards with bracket classes, boosts, JSON subpaths.
+
+## Tests: what goes, what comes
+
+Removed with their modules (do not port their string-level assertions):
+
+- `src/documents/tests/search/test_translate.py`: its subject is deleted; string-translation unit cases are whoosh-compat's own responsibility now. Cases that encode real user-visible behavior get reincarnated as result-level acceptance cases, not string assertions.
+- Date-keyword unit tests tied to `_dates.py` internals: same treatment.
+- `test_query.py` cases asserting `parse_user_query` internals or intermediate query strings: rewritten against the new pipeline, asserting on matched results.
+
+Kept: `test_migration_fulltext_query_field_prefixes.py` (data migration, orthogonal), `test_schema.py`, `test_tokenizer.py`, permission-filter and simple-search tests.
+
+Added:
+
+- A result-level acceptance module (paperless's analogue of whoosh-compat's `test_acceptance_e2e.py`): the corpus above against a real index built from `build_schema()`, asserting document-ID sets. Use `pytest.param(..., id="...")` for every case.
+- Registry unit tests: internal `*_id` names rejected, aliases resolve to canonical fields, JSON subpaths match `docs/usage.md`, construction deterministic per language.
+- One `Multitoken` case nested inside a top-level `OR` (whoosh-compat DIVERGENCES entry on Multitoken.DEFAULT) to prove it does not matter for paperless's data.
+- If acceptance work surfaces a new whoosh-compat divergence, that is a whoosh-compat-repo change (its `differential-triage` skill applies), not a silent paperless workaround.
+
+## Coordination
+
+- whoosh-compat is pre-1.0: pin an exact version or git SHA; upgrades are deliberate, reviewed changes.
+- JSON subpath emission depends on the installed tantivy-py version (fallback until quickwit-oss/tantivy-py#716 ships). The whoosh-compat repo has a `carve-out-retirement` skill; coordinate tantivy pin bumps with it, in a separate PR from the parser migration.
+- Rollout: settings flag defaulting to the legacy path plus shadow-compare logging (log when old and new paths return different ID sets; sample if cost matters) for one release; delete `_translate.py`/`_dates.py` only after the flag defaults to the new path with no material reports.
+
+## Common mistakes
+
+- Inferring the field surface from frontend code (users type queries directly).
+- Copying `KNOWN_FIELDS` into the registry wholesale (resurfaces internal fields).
+- Wiring stemming into `pattern_normalizer`.
+- Calling `emit()` unconditionally, or porting the legacy raw-string fallback.
+- Deleting `_dates.py` without the parity audit.
+- Porting `test_translate.py`'s string assertions instead of writing result-level tests.

docs/administration.md

@@ -299,8 +299,6 @@ optional arguments:
 -sm, --split-manifest
 -z,  --zip
 -zn, --zip-name
---zip-compression
---zip-compression-level
 --data-only
 --no-progress-bar
 --passphrase
@@ -363,19 +361,6 @@ If `-z` or `--zip` is provided, the export will be a zip file
 in the target directory, named according to the current local date or the
 value set in `-zn` or `--zip-name`.
 
-The compression method for the zip can be set with `--zip-compression`
-(`stored`, `deflated` (default), `bzip2`, `lzma`, or `zstd`) and tuned with
-`--zip-compression-level` (deflated: 0–9, bzip2: 1–9, zstd: -22–22; ignored
-for `stored` and `lzma`). Both options require `--zip`.
-
-!!! warning
-
-    `zstd` compression requires Python 3.14 or newer on **both** the machine
-    creating the export and any machine importing it. An archive compressed with
-    `zstd` (or `lzma`/`bzip2` where those modules are unavailable) cannot be
-    imported on a runtime that lacks the codec; the importer will refuse it with
-    a clear error. The default `deflated` is universally readable.
-
 If `--data-only` is provided, only the database will be exported. This option is intended
 to facilitate database upgrades without needing to clean documents and thumbnails from the media directory.
 

src-ui/messages.xlf

@@ -8238,11 +8238,11 @@
         <source>An error occurred loading tiff: <x id="PH" equiv-text="err.toString()"/></source>
         <context-group purpose="location">
           <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">2024</context>
+          <context context-type="linenumber">2026</context>
         </context-group>
         <context-group purpose="location">
           <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">2030</context>
+          <context context-type="linenumber">2032</context>
         </context-group>
       </trans-unit>
       <trans-unit id="4958946940233632319" datatype="html">

src-ui/src/app/components/document-detail/document-detail.component.spec.ts

@@ -2161,8 +2161,14 @@ describe('DocumentDetailComponent', () => {
   it('should support open share links and email modals', () => {
     const modalSpy = jest.spyOn(modalService, 'open')
     initNormally()
+    component.selectedVersionId.set(10)
     component.openShareLinks()
     expect(modalSpy).toHaveBeenCalled()
+    expect(
+      (
+        modalSpy.mock.results[0].value as NgbModalRef
+      ).componentInstance.documentId()
+    ).toBe(10)
     component.openEmailDocument()
     expect(modalSpy).toHaveBeenCalled()
   })

src-ui/src/app/components/document-detail/document-detail.component.ts

@@ -1959,7 +1959,9 @@ export class DocumentDetailComponent
 
   public openShareLinks() {
     const modal = this.modalService.open(ShareLinksDialogComponent)
-    modal.componentInstance.documentId.set(this.document().id)
+    modal.componentInstance.documentId.set(
+      this.selectedVersionId() ?? this.document().id
+    )
     modal.componentInstance.hasArchiveVersion.set(
       this.metadata()?.has_archive_version ??
         !!this.document()?.archived_file_name

src/documents/export/compression.py

@@ -1,108 +0,0 @@
-from __future__ import annotations
-
-import importlib
-import zipfile
-
-# ZIP_ZSTANDARD exists only on Python 3.14+ (PEP 784). None elsewhere.
-ZSTD: int | None = getattr(zipfile, "ZIP_ZSTANDARD", None)
-
-# CLI choices are fixed across runtimes so argparse never hides zstd; runtime
-# availability is enforced separately in compression_available().
-COMPRESSION_CHOICES: tuple[str, ...] = (
-    "stored",
-    "deflated",
-    "bzip2",
-    "lzma",
-    "zstd",
-)
-
-# Method name -> zipfile compression constant (zstd only when supported).
-COMPRESSION_METHODS: dict[str, int] = {
-    "stored": zipfile.ZIP_STORED,
-    "deflated": zipfile.ZIP_DEFLATED,
-    "bzip2": zipfile.ZIP_BZIP2,
-    "lzma": zipfile.ZIP_LZMA,
-}
-if ZSTD is not None:
-    COMPRESSION_METHODS["zstd"] = ZSTD
-
-# Inclusive (min, max) level bounds per method; None => level not applicable.
-# Verified on CPython 3.14.3.
-#
-# zstd's raw library bounds are (-131072, 22)
-# (compression.zstd.CompressionParameter.compression_level.bounds()) — the
-# minimum is an internal implementation constant (-ZSTD_TARGETLENGTH_MAX),
-# not a meaningful distinct "level"; deeper negative values than -22 buy
-# nothing over -22 in practice. We expose the conventional zstd CLI range
-# instead of the raw library bounds.
-LEVEL_BOUNDS: dict[str, tuple[int, int] | None] = {
-    "stored": None,
-    "deflated": (0, 9),
-    "bzip2": (1, 9),
-    "lzma": None,
-    "zstd": (-22, 22),
-}
-
-# zipfile compress_type id -> method name. 93 = current zstd id, 20 = legacy
-# zstd id that zipfile can still read.
-_COMPRESS_TYPE_TO_METHOD: dict[int, str] = {
-    zipfile.ZIP_STORED: "stored",
-    zipfile.ZIP_DEFLATED: "deflated",
-    zipfile.ZIP_BZIP2: "bzip2",
-    zipfile.ZIP_LZMA: "lzma",
-    93: "zstd",
-    20: "zstd",
-}
-
-
-def compression_available(method: str) -> bool:
-    """Whether the running interpreter can actually use the given method."""
-    if method in ("stored", "deflated"):
-        # zlib is a hard CPython dependency; stored needs nothing.
-        return True
-    if method == "bzip2":
-        return _module_importable("bz2")
-    if method == "lzma":
-        return _module_importable("lzma")
-    if method == "zstd":
-        return ZSTD is not None and _module_importable("compression.zstd")
-    return False
-
-
-def _module_importable(name: str) -> bool:
-    try:
-        importlib.import_module(name)
-    except ImportError:
-        return False
-    return True
-
-
-def level_error(method: str, level: int | None) -> str | None:
-    """Return a human message if (method, level) is invalid, else None."""
-    if level is None:
-        return None
-    bounds = LEVEL_BOUNDS[method]
-    if bounds is None:
-        return f"--zip-compression-level has no effect for '{method}'"
-    low, high = bounds
-    if not (low <= level <= high):
-        return (
-            f"--zip-compression-level for '{method}' must be between {low} and {high}"
-        )
-    return None
-
-
-def compress_type_readable(compress_type: int) -> bool:
-    """Whether this interpreter can decompress an entry of the given type."""
-    method = _COMPRESS_TYPE_TO_METHOD.get(compress_type)
-    if method is None:
-        return False
-    return compression_available(method)
-
-
-def unreadable_method_names(compress_types: set[int]) -> set[str]:
-    """Map a set of compress_type ids to human method names for error messages."""
-    names: set[str] = set()
-    for ct in compress_types:
-        names.add(_COMPRESS_TYPE_TO_METHOD.get(ct, f"method {ct}"))
-    return names

src/documents/export/sinks.py

@@ -1,357 +0,0 @@
-from __future__ import annotations
-
-import abc
-import hashlib
-import json
-import os
-import shutil
-import tempfile
-import zipfile
-from contextlib import AbstractContextManager
-from contextlib import contextmanager
-from pathlib import Path
-from pathlib import PurePosixPath
-from typing import TYPE_CHECKING
-
-from django.conf import settings
-from django.core.serializers.json import DjangoJSONEncoder
-
-from documents.file_handling import delete_empty_directories
-from documents.utils import compute_checksum
-from documents.utils import copy_file_with_basic_stats
-
-if TYPE_CHECKING:
-    from collections.abc import Iterator
-    from typing import TextIO
-
-
-def _dumps(content: list | dict) -> str:
-    """Serialize export JSON consistently across all sinks."""
-    return json.dumps(content, cls=DjangoJSONEncoder, indent=2, ensure_ascii=False)
-
-
-class StreamingManifestWriter:
-    """Incrementally writes a JSON array to a text handle, one record at a time.
-
-    Knows nothing about folders or zips: it writes the array framing and records
-    to whatever handle the sink's ``stream()`` yields. The sink owns the handle's
-    lifecycle (atomic rename, compare, spooling).
-    """
-
-    def __init__(self, handle: TextIO) -> None:
-        self._file = handle
-        self._first = True
-        self._file.write("[")
-
-    def write_record(self, record: dict) -> None:
-        if not self._first:
-            self._file.write(",\n")
-        else:
-            self._first = False
-        self._file.write(_dumps(record))
-
-    def write_batch(self, records: list[dict]) -> None:
-        for record in records:
-            self.write_record(record)
-
-    def close(self) -> None:
-        """Write the closing bracket. Does NOT close the handle (the sink owns it)."""
-        self._file.write("\n]")
-
-
-class ExportSink(AbstractContextManager, abc.ABC):
-    """Destination for a document export.
-
-    The command declares export contents via three verbs; the sink decides how to
-    persist each. ``arcname`` is always a relative POSIX path
-    (e.g. ``"manifest.json"``, ``"originals/foo.pdf"``).
-
-    Contract:
-      * At most one ``stream()`` open at a time (it is the manifest);
-        ``add_file``/``add_json`` may be called while it is open.
-      * Context-manager: normal exit finalizes, an exception aborts. No partial or
-        failed run leaves a complete-looking artifact.
-    """
-
-    @abc.abstractmethod
-    def add_file(
-        self,
-        source: Path,
-        arcname: str,
-        *,
-        checksum: str | None = None,
-    ) -> None: ...
-
-    @abc.abstractmethod
-    def add_json(self, content: list | dict, arcname: str) -> None: ...
-
-    @abc.abstractmethod
-    def stream(self, arcname: str) -> AbstractContextManager[TextIO]: ...
-
-    def _open(self) -> None:
-        """Hook called on context entry. Override as needed."""
-
-    @abc.abstractmethod
-    def _finalize(self) -> None:
-        """Commit on clean exit."""
-
-    @abc.abstractmethod
-    def _abort(self) -> None:
-        """Roll back on exception."""
-
-    def __enter__(self) -> ExportSink:
-        self._open()
-        return self
-
-    def __exit__(self, exc_type, exc_val, exc_tb) -> None:
-        if exc_type is not None:
-            self._abort()
-        else:
-            self._finalize()
-
-
-class DirectoryExportSink(ExportSink):
-    """Writes loose files into a target directory, with incremental sync.
-
-    Owns the snapshot/skip/compare/prune machinery that used to live in the
-    command (``files_in_export_dir``, ``check_and_copy``, ``check_and_write_json``,
-    and the ``--delete`` pass).
-    """
-
-    def __init__(
-        self,
-        target: Path,
-        *,
-        compare_checksums: bool,
-        compare_json: bool,
-        delete: bool,
-    ) -> None:
-        self._target = target.resolve()
-        self._compare_checksums = compare_checksums
-        self._compare_json = compare_json
-        self._delete = delete
-        self._snapshot: set[Path] = set()
-        self._stream_open = False
-
-    def _open(self) -> None:
-        for x in self._target.glob("**/*"):
-            if x.is_file():
-                self._snapshot.add(x.resolve())
-
-    def add_file(
-        self,
-        source: Path,
-        arcname: str,
-        *,
-        checksum: str | None = None,
-    ) -> None:
-        target = (self._target / arcname).resolve()
-        self._snapshot.discard(target)
-        perform_copy = False
-        if target.exists():
-            source_stat = source.stat()
-            target_stat = target.stat()
-            if self._compare_checksums and checksum:
-                perform_copy = compute_checksum(target) != checksum
-            elif (
-                source_stat.st_mtime != target_stat.st_mtime
-                or source_stat.st_size != target_stat.st_size
-            ):
-                perform_copy = True
-        else:
-            perform_copy = True
-        if perform_copy:
-            target.parent.mkdir(parents=True, exist_ok=True)
-            copy_file_with_basic_stats(source, target)
-
-    @staticmethod
-    def _content_unchanged(target: Path, new_bytes: bytes) -> bool:
-        """True if ``target`` already holds byte-identical content (BLAKE2b)."""
-        return (
-            hashlib.blake2b(target.read_bytes()).hexdigest()
-            == hashlib.blake2b(new_bytes).hexdigest()
-        )
-
-    def add_json(self, content: list | dict, arcname: str) -> None:
-        target = (self._target / arcname).resolve()
-        json_str = _dumps(content)
-        perform_write = True
-        if target in self._snapshot:
-            self._snapshot.discard(target)
-            if self._compare_json and self._content_unchanged(
-                target,
-                json_str.encode("utf-8"),
-            ):
-                perform_write = False
-        if perform_write:
-            target.parent.mkdir(parents=True, exist_ok=True)
-            target.write_text(json_str, encoding="utf-8")
-
-    @contextmanager
-    def stream(self, arcname: str) -> Iterator[TextIO]:
-        if self._stream_open:
-            raise RuntimeError("A stream is already open on this sink")
-        target = (self._target / arcname).resolve()
-        tmp = target.with_suffix(target.suffix + ".tmp")
-        target.parent.mkdir(parents=True, exist_ok=True)
-        handle = tmp.open("w", encoding="utf-8")
-        self._stream_open = True
-        try:
-            yield handle
-        except BaseException:
-            handle.close()
-            tmp.unlink(missing_ok=True)
-            raise
-        else:
-            handle.close()
-            self._commit_streamed_file(target, tmp)
-        finally:
-            self._stream_open = False
-
-    def _commit_streamed_file(self, target: Path, tmp: Path) -> None:
-        if target in self._snapshot:
-            self._snapshot.discard(target)
-            if self._compare_json and self._content_unchanged(
-                target,
-                tmp.read_bytes(),
-            ):
-                tmp.unlink()
-                return
-        tmp.rename(target)
-
-    def _finalize(self) -> None:
-        if self._delete:
-            for f in self._snapshot:
-                if not f.is_relative_to(self._target):  # pragma: no cover
-                    # Defense in depth: a symlink inside the export dir can
-                    # resolve outside of it; never delete outside the target.
-                    continue
-                f.unlink()
-                delete_empty_directories(f.parent, self._target)
-
-    def _abort(self) -> None:
-        # Folder mode is in-place/incremental: streamed .tmp files are already
-        # cleaned in stream(); leave everything else intact and skip the prune.
-        return None
-
-
-class ZipExportSink(ExportSink):
-    """Writes a single zip archive, produced atomically only on success.
-
-    Builds into ``<target>/<zip_name>.zip.tmp`` and renames to ``.zip`` on clean
-    finalize. The manifest stream is spooled to a temp file in SCRATCH_DIR and
-    added as an entry at finalize (a zip entry cannot be interleaved with others).
-    """
-
-    def __init__(
-        self,
-        target: Path,
-        zip_name: str,
-        *,
-        delete: bool = False,
-        compression: int = zipfile.ZIP_DEFLATED,
-        compresslevel: int | None = None,
-    ) -> None:
-        self._target = target.resolve()
-        self._zip_path = (self._target / zip_name).with_suffix(".zip")
-        self._tmp_path = self._zip_path.with_name(self._zip_path.name + ".tmp")
-        self._delete = delete
-        self._compression = compression
-        self._compresslevel = compresslevel
-        self._zip: zipfile.ZipFile | None = None
-        self._dirs: set[str] = set()
-        self._pending_manifest: tuple[Path, str] | None = None
-        self._stream_open = False
-
-    def _open(self) -> None:
-        settings.SCRATCH_DIR.mkdir(parents=True, exist_ok=True)
-        self._zip = zipfile.ZipFile(
-            self._tmp_path,
-            "w",
-            compression=self._compression,
-            compresslevel=self._compresslevel,
-            allowZip64=True,
-        )
-
-    def _ensure_dirs(self, arcname: str) -> None:
-        assert self._zip is not None
-        dir_arc = ""
-        for part in PurePosixPath(arcname).parts[:-1]:
-            dir_arc += f"{part}/"
-            if dir_arc not in self._dirs:
-                self._dirs.add(dir_arc)
-                self._zip.mkdir(dir_arc)
-
-    def add_file(
-        self,
-        source: Path,
-        arcname: str,
-        *,
-        checksum: str | None = None,
-    ) -> None:
-        assert self._zip is not None
-        self._ensure_dirs(arcname)
-        self._zip.write(source, arcname=arcname)
-
-    def add_json(self, content: list | dict, arcname: str) -> None:
-        assert self._zip is not None
-        self._ensure_dirs(arcname)
-        self._zip.writestr(arcname, _dumps(content))
-
-    @contextmanager
-    def stream(self, arcname: str) -> Iterator[TextIO]:
-        if self._stream_open:
-            raise RuntimeError("A stream is already open on this sink")
-        settings.SCRATCH_DIR.mkdir(parents=True, exist_ok=True)
-        fd, tmp_name = tempfile.mkstemp(
-            dir=settings.SCRATCH_DIR,
-            prefix="export-manifest-",
-            suffix=".json",
-        )
-        tmp = Path(tmp_name)
-        handle = os.fdopen(fd, "w", encoding="utf-8")
-        self._stream_open = True
-        try:
-            yield handle
-        except BaseException:
-            handle.close()
-            tmp.unlink(missing_ok=True)
-            raise
-        else:
-            handle.close()
-            self._pending_manifest = (tmp, arcname)
-        finally:
-            self._stream_open = False
-
-    def _finalize(self) -> None:
-        assert self._zip is not None
-        if self._pending_manifest is not None:
-            tmp, arcname = self._pending_manifest
-            self._ensure_dirs(arcname)
-            self._zip.write(tmp, arcname=arcname)
-            tmp.unlink(missing_ok=True)
-            self._pending_manifest = None
-        self._zip.close()
-        self._zip = None
-        if self._delete:
-            self._wipe_destination()
-        self._tmp_path.replace(self._zip_path)
-
-    def _wipe_destination(self) -> None:
-        skip = {self._zip_path.resolve(), self._tmp_path.resolve()}
-        for item in self._target.glob("*"):
-            if item.resolve() in skip:
-                continue
-            if item.is_dir():
-                shutil.rmtree(item)
-            else:
-                item.unlink()
-
-    def _abort(self) -> None:
-        if self._zip is not None:
-            self._zip.close()
-            self._zip = None
-        self._tmp_path.unlink(missing_ok=True)
-        if self._pending_manifest is not None:
-            self._pending_manifest[0].unlink(missing_ok=True)
-            self._pending_manifest = None

src/documents/management/commands/document_exporter.py

@@ -1,4 +1,8 @@
+import hashlib
+import json
 import os
+import shutil
+import tempfile
 from itertools import islice
 from pathlib import Path
 from typing import TYPE_CHECKING
@@ -15,6 +19,7 @@ from django.contrib.auth.models import User
 from django.contrib.contenttypes.models import ContentType
 from django.core import serializers
 from django.core.management.base import CommandError
+from django.core.serializers.json import DjangoJSONEncoder
 from django.db import transaction
 from django.utils import timezone
 from filelock import FileLock
@@ -29,15 +34,7 @@ if TYPE_CHECKING:
 if settings.AUDIT_LOG_ENABLED:
     from auditlog.models import LogEntry
 
-from documents.export.compression import COMPRESSION_CHOICES
-from documents.export.compression import COMPRESSION_METHODS
-from documents.export.compression import ZSTD
-from documents.export.compression import compression_available
-from documents.export.compression import level_error
-from documents.export.sinks import DirectoryExportSink
-from documents.export.sinks import ExportSink
-from documents.export.sinks import StreamingManifestWriter
-from documents.export.sinks import ZipExportSink
+from documents.file_handling import delete_empty_directories
 from documents.file_handling import generate_filename
 from documents.management.commands.base import PaperlessCommand
 from documents.management.commands.mixins import CryptMixin
@@ -63,7 +60,8 @@ from documents.settings import EXPORTER_ARCHIVE_NAME
 from documents.settings import EXPORTER_FILE_NAME
 from documents.settings import EXPORTER_SHARE_LINK_BUNDLE_NAME
 from documents.settings import EXPORTER_THUMBNAIL_NAME
-from documents.utils import QuerySetStream
+from documents.utils import compute_checksum
+from documents.utils import copy_file_with_basic_stats
 from paperless import version
 from paperless.models import ApplicationConfiguration
 from paperless_mail.models import MailAccount
@@ -86,6 +84,87 @@ def serialize_queryset_batched(
         yield serializers.serialize("python", chunk)
 
 
+class StreamingManifestWriter:
+    """Incrementally writes a JSON array to a file, one record at a time.
+
+    Writes to <target>.tmp first; on close(), optionally BLAKE2b-compares
+    with the existing file (--compare-json) and renames or discards accordingly.
+    On exception, discard() deletes the tmp file and leaves the original intact.
+    """
+
+    def __init__(
+        self,
+        path: Path,
+        *,
+        compare_json: bool = False,
+        files_in_export_dir: "set[Path] | None" = None,
+    ) -> None:
+        self._path = path.resolve()
+        self._tmp_path = self._path.with_suffix(self._path.suffix + ".tmp")
+        self._compare_json = compare_json
+        self._files_in_export_dir: set[Path] = (
+            files_in_export_dir if files_in_export_dir is not None else set()
+        )
+        self._file = None
+        self._first = True
+
+    def open(self) -> None:
+        self._path.parent.mkdir(parents=True, exist_ok=True)
+        self._file = self._tmp_path.open("w", encoding="utf-8")
+        self._file.write("[")
+        self._first = True
+
+    def write_record(self, record: dict) -> None:
+        if not self._first:
+            self._file.write(",\n")
+        else:
+            self._first = False
+        self._file.write(
+            json.dumps(record, cls=DjangoJSONEncoder, indent=2, ensure_ascii=False),
+        )
+
+    def write_batch(self, records: list[dict]) -> None:
+        for record in records:
+            self.write_record(record)
+
+    def close(self) -> None:
+        if self._file is None:
+            return
+        self._file.write("\n]")
+        self._file.close()
+        self._file = None
+        self._finalize()
+
+    def discard(self) -> None:
+        if self._file is not None:
+            self._file.close()
+            self._file = None
+        if self._tmp_path.exists():
+            self._tmp_path.unlink()
+
+    def _finalize(self) -> None:
+        """Compare with existing file (if --compare-json) then rename or discard tmp."""
+        if self._path in self._files_in_export_dir:
+            self._files_in_export_dir.remove(self._path)
+            if self._compare_json:
+                existing_hash = hashlib.blake2b(self._path.read_bytes()).hexdigest()
+                new_hash = hashlib.blake2b(self._tmp_path.read_bytes()).hexdigest()
+                if existing_hash == new_hash:
+                    self._tmp_path.unlink()
+                    return
+        self._tmp_path.rename(self._path)
+
+    def __enter__(self) -> "StreamingManifestWriter":
+        self.open()
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb) -> None:
+        if exc_type is not None:
+            self.discard()
+        else:
+            self.close()
+
+
 class Command(CryptMixin, PaperlessCommand):
     help = (
         "Decrypt and rename all files in our collection into a given target "
@@ -197,28 +276,6 @@ class Command(CryptMixin, PaperlessCommand):
             help="Sets the export zip file name",
         )
 
-        parser.add_argument(
-            "--zip-compression",
-            choices=COMPRESSION_CHOICES,
-            default=None,
-            help=(
-                "Compression method for the export zip (requires --zip). "
-                "Default: deflated. 'zstd' requires Python 3.14+ on both the "
-                "exporting and importing machine."
-            ),
-        )
-
-        parser.add_argument(
-            "--zip-compression-level",
-            type=int,
-            default=None,
-            help=(
-                "Compression level for the export zip (requires --zip). "
-                "deflated: 0-9, bzip2: 1-9, zstd: -22..22; ignored for "
-                "stored/lzma."
-            ),
-        )
-
         parser.add_argument(
             "--data-only",
             default=False,
@@ -257,13 +314,20 @@ class Command(CryptMixin, PaperlessCommand):
         self.passphrase: str | None = options.get("passphrase")
         self.batch_size: int = options["batch_size"]
 
+        self.files_in_export_dir: set[Path] = set()
         self.exported_files: set[str] = set()
 
-        if self.zip_export and (self.compare_checksums or self.compare_json):
-            raise CommandError(
-                "--compare-checksums and --compare-json have no effect when "
-                "used with --zip",
+        # If zipping, save the original target for later and
+        # get a temporary directory for the target instead
+        temp_dir = None
+        self.original_target = self.target
+        if self.zip_export:
+            settings.SCRATCH_DIR.mkdir(parents=True, exist_ok=True)
+            temp_dir = tempfile.TemporaryDirectory(
+                dir=settings.SCRATCH_DIR,
+                prefix="paperless-export",
             )
+            self.target = Path(temp_dir.name).resolve()
 
         if not self.target.exists():
             raise CommandError("That path doesn't exist")
@@ -274,55 +338,33 @@ class Command(CryptMixin, PaperlessCommand):
         if not os.access(self.target, os.W_OK):
             raise CommandError("That path doesn't appear to be writable")
 
-        zip_compression: str | None = options["zip_compression"]
-        zip_compression_level: int | None = options["zip_compression_level"]
+        try:
+            # Prevent any ongoing changes in the documents
+            with FileLock(settings.MEDIA_LOCK):
+                self.dump()
 
-        if not self.zip_export and (
-            zip_compression is not None or zip_compression_level is not None
-        ):
-            raise CommandError(
-                "--zip-compression and --zip-compression-level require --zip",
-            )
-
-        compression_method = zip_compression or "deflated"
-        if self.zip_export:
-            if not compression_available(compression_method):
-                if compression_method == "zstd" and ZSTD is None:
-                    raise CommandError(
-                        "zstd compression requires Python 3.14 or newer",
+                # We've written everything to the temporary directory in this case,
+                # now make an archive in the original target, with all files stored
+                if self.zip_export and temp_dir is not None:
+                    shutil.make_archive(
+                        self.original_target / options["zip_name"],
+                        format="zip",
+                        root_dir=temp_dir.name,
                     )
-                raise CommandError(
-                    f"Compression method '{compression_method}' is not "
-                    f"available on this Python runtime",
-                )
-            level_msg = level_error(compression_method, zip_compression_level)
-            if level_msg is not None:
-                raise CommandError(level_msg)
 
-        sink: ExportSink
-        if self.zip_export:
-            sink = ZipExportSink(
-                self.target,
-                options["zip_name"],
-                delete=self.delete,
-                compression=COMPRESSION_METHODS[compression_method],
-                compresslevel=zip_compression_level,
-            )
-        else:
-            sink = DirectoryExportSink(
-                self.target,
-                compare_checksums=self.compare_checksums,
-                compare_json=self.compare_json,
-                delete=self.delete,
-            )
+        finally:
+            # Always cleanup the temporary directory, if one was created
+            if self.zip_export and temp_dir is not None:
+                temp_dir.cleanup()
 
-        # Prevent any ongoing changes in the documents while exporting
-        with FileLock(settings.MEDIA_LOCK), sink:
-            self.dump(sink)
+    def dump(self) -> None:
+        # 1. Take a snapshot of what files exist in the current export folder
+        for x in self.target.glob("**/*"):
+            if x.is_file():
+                self.files_in_export_dir.add(x.resolve())
 
-    def dump(self, sink: ExportSink) -> None:
-        # 1. Create manifest, containing all correspondents, types, tags, storage
-        #    paths, note, documents and ui_settings
+        # 2. Create manifest, containing all correspondents, types, tags, storage paths
+        # note, documents and ui_settings
         _excluded_usernames = ["consumer", "AnonymousUser"]
         manifest_key_to_object_query: dict[str, QuerySet[Any]] = {
             "correspondents": Correspondent.objects.all(),
@@ -385,9 +427,13 @@ class Command(CryptMixin, PaperlessCommand):
 
         document_manifest: list[dict] = []
         share_link_bundle_manifest: list[dict] = []
+        manifest_path = (self.target / "manifest.json").resolve()
 
-        with sink.stream("manifest.json") as handle:
-            writer = StreamingManifestWriter(handle)
+        with StreamingManifestWriter(
+            manifest_path,
+            compare_json=self.compare_json,
+            files_in_export_dir=self.files_in_export_dir,
+        ) as writer:
             with transaction.atomic():
                 for key, qs in manifest_key_to_object_query.items():
                     if key == "documents":
@@ -423,6 +469,9 @@ class Command(CryptMixin, PaperlessCommand):
                                 self._encrypt_record_inline(record)
                             writer.write_batch(batch)
 
+            document_map: dict[int, Document] = {
+                d.pk: d for d in Document.global_objects.order_by("id")
+            }
             share_link_bundle_map: dict[int, ShareLinkBundle] = {
                 b.pk: b
                 for b in ShareLinkBundle.objects.order_by("id").prefetch_related(
@@ -430,72 +479,84 @@ class Command(CryptMixin, PaperlessCommand):
                 )
             }
 
-            # 2. Export files from each document
-            # document_manifest and this stream are both ordered by id from the
-            # same underlying rows, so zip them in lockstep instead of building
-            # a dict of every Document instance up front (QuerySetStream keeps
-            # only one batch of documents resident at a time).
-            documents_stream = QuerySetStream(
-                Document.global_objects.order_by("id"),
-                chunk_size=self.batch_size,
-            )
-            for document_dict, document in self.track(
-                zip(document_manifest, documents_stream, strict=True),
-                description="Exporting documents...",
-                total=len(document_manifest),
+            # 3. Export files from each document
+            for index, document_dict in enumerate(
+                self.track(
+                    document_manifest,
+                    description="Exporting documents...",
+                    total=len(document_manifest),
+                ),
             ):
-                # Both document_manifest and documents_stream come from the same
-                # Document.global_objects.order_by("id") query, taken while
-                # MEDIA_LOCK is held, so this should be unreachable -- it guards
-                # against silent data corruption if that invariant ever breaks.
-                if document.pk != document_dict["pk"]:  # pragma: no cover
-                    raise CommandError(
-                        "Document export ordering mismatch: expected "
-                        f"pk={document_dict['pk']}, got pk={document.pk}. "
-                        "Documents may have changed during export.",
-                    )
+                document = document_map[document_dict["pk"]]
 
-                # generate a unique filename, then the arcnames for its files
+                # 3.1. generate a unique filename
                 base_name = self.generate_base_name(document)
-                original_arc, thumbnail_arc, archive_arc = (
+
+                # 3.2. write filenames into manifest
+                original_target, thumbnail_target, archive_target = (
                     self.generate_document_targets(document, base_name, document_dict)
                 )
 
+                # 3.3. write files to target folder
                 if not self.data_only:
                     self.copy_document_files(
                         document,
-                        sink,
-                        original_arc,
-                        thumbnail_arc,
-                        archive_arc,
+                        original_target,
+                        thumbnail_target,
+                        archive_target,
                     )
 
                 if self.split_manifest:
-                    self._write_split_manifest(sink, document_dict, document, base_name)
+                    self._write_split_manifest(document_dict, document, base_name)
                 else:
                     writer.write_record(document_dict)
 
             for bundle_dict in share_link_bundle_manifest:
                 bundle = share_link_bundle_map[bundle_dict["pk"]]
-                bundle_arc = self.generate_share_link_bundle_target(
+
+                bundle_target = self.generate_share_link_bundle_target(
                     bundle,
                     bundle_dict,
                 )
-                if not self.data_only and bundle_arc is not None:
-                    self.copy_share_link_bundle_file(bundle, sink, bundle_arc)
+
+                if not self.data_only and bundle_target is not None:
+                    self.copy_share_link_bundle_file(bundle, bundle_target)
+
                 writer.write_record(bundle_dict)
 
-            writer.close()
-
-        # 3. Write version (and crypto params) to metadata.json
-        # Django stores most crypto values in the field itself; we store
-        # them once here for the whole export
+        # 4.2 write version information to target folder
+        extra_metadata_path = (self.target / "metadata.json").resolve()
         metadata: dict[str, str | int | dict[str, str | int]] = {
             "version": version.__full_version_str__,
         }
+
+        # 4.2.1 If needed, write the crypto values into the metadata
+        # Django stores most of these in the field itself, we store them once here
         if self.passphrase:
             metadata.update(self.get_crypt_params())
-        sink.add_json(metadata, "metadata.json")
+
+        self.check_and_write_json(
+            metadata,
+            extra_metadata_path,
+        )
+
+        if self.delete:
+            # 5. Remove files which we did not explicitly export in this run
+            if not self.zip_export:
+                for f in self.files_in_export_dir:
+                    f.unlink()
+
+                    delete_empty_directories(
+                        f.parent,
+                        self.target,
+                    )
+            else:
+                # 5. Remove anything in the original location (before moving the zip)
+                for item in self.original_target.glob("*"):
+                    if item.is_dir():
+                        shutil.rmtree(item)
+                    else:
+                        item.unlink()
 
     def generate_base_name(self, document: Document) -> Path:
         """
@@ -523,69 +584,73 @@ class Command(CryptMixin, PaperlessCommand):
         document: Document,
         base_name: Path,
         document_dict: dict,
-    ) -> tuple[str, str | None, str | None]:
+    ) -> tuple[Path, Path | None, Path | None]:
         """
-        Generates the relative POSIX arcnames for a document's original, thumbnail
-        and archive files (depending on settings), and records them in the manifest.
+        Generates the targets for a given document, including the original file, archive file and thumbnail (depending on settings).
         """
         original_name = base_name
         if self.use_folder_prefix:
             original_name = Path("originals") / original_name
-        original_arc = original_name.as_posix()
-        document_dict[EXPORTER_FILE_NAME] = original_arc
+        original_target = (self.target / original_name).resolve()
+        document_dict[EXPORTER_FILE_NAME] = str(original_name)
 
         if not self.no_thumbnail:
             thumbnail_name = base_name.parent / (base_name.stem + "-thumbnail.webp")
             if self.use_folder_prefix:
                 thumbnail_name = Path("thumbnails") / thumbnail_name
-            thumbnail_arc = thumbnail_name.as_posix()
-            document_dict[EXPORTER_THUMBNAIL_NAME] = thumbnail_arc
+            thumbnail_target = (self.target / thumbnail_name).resolve()
+            document_dict[EXPORTER_THUMBNAIL_NAME] = str(thumbnail_name)
         else:
-            thumbnail_arc = None
+            thumbnail_target = None
 
         if not self.no_archive and document.has_archive_version:
             archive_name = base_name.parent / (base_name.stem + "-archive.pdf")
             if self.use_folder_prefix:
                 archive_name = Path("archive") / archive_name
-            archive_arc = archive_name.as_posix()
-            document_dict[EXPORTER_ARCHIVE_NAME] = archive_arc
+            archive_target = (self.target / archive_name).resolve()
+            document_dict[EXPORTER_ARCHIVE_NAME] = str(archive_name)
         else:
-            archive_arc = None
+            archive_target = None
 
-        return original_arc, thumbnail_arc, archive_arc
+        return original_target, thumbnail_target, archive_target
 
     def copy_document_files(
         self,
         document: Document,
-        sink: ExportSink,
-        original_arc: str,
-        thumbnail_arc: str | None,
-        archive_arc: str | None,
+        original_target: Path,
+        thumbnail_target: Path | None,
+        archive_target: Path | None,
     ) -> None:
         """
-        Hands the document's files to the sink (original, thumbnail, archive).
+        Copies files from the document storage location to the specified target location.
+
+        If the document is encrypted, the files are decrypted before copying them to the target location.
         """
-        sink.add_file(document.source_path, original_arc, checksum=document.checksum)
+        self.check_and_copy(
+            document.source_path,
+            document.checksum,
+            original_target,
+        )
 
-        if thumbnail_arc:
-            sink.add_file(document.thumbnail_path, thumbnail_arc)
+        if thumbnail_target:
+            self.check_and_copy(document.thumbnail_path, None, thumbnail_target)
 
-        if archive_arc:
+        if archive_target:
             if TYPE_CHECKING:
                 assert isinstance(document.archive_path, Path)
-            sink.add_file(
+            self.check_and_copy(
                 document.archive_path,
-                archive_arc,
-                checksum=document.archive_checksum,
+                document.archive_checksum,
+                archive_target,
             )
 
     def generate_share_link_bundle_target(
         self,
         bundle: ShareLinkBundle,
         bundle_dict: dict,
-    ) -> str | None:
+    ) -> Path | None:
         """
-        Generates the relative POSIX arcname for a share link bundle file, if any.
+        Generates the export target for a share link bundle file, when present.
         """
         if not bundle.file_path:
             return None
@@ -601,22 +666,25 @@ class Command(CryptMixin, PaperlessCommand):
         bundle_dict["fields"]["file_path"] = portable_bundle_path.as_posix()
         bundle_dict[EXPORTER_SHARE_LINK_BUNDLE_NAME] = export_bundle_path.as_posix()
 
-        return export_bundle_path.as_posix()
+        return (self.target / export_bundle_path).resolve()
 
     def copy_share_link_bundle_file(
         self,
         bundle: ShareLinkBundle,
-        sink: ExportSink,
-        bundle_arc: str,
+        bundle_target: Path,
     ) -> None:
         """
-        Hands a share link bundle ZIP to the sink.
+        Copies a share link bundle ZIP into the export directory.
         """
         bundle_source_path = bundle.absolute_file_path
         if bundle_source_path is None:
             raise FileNotFoundError(f"Share link bundle {bundle.pk} has no file path")
 
-        sink.add_file(bundle_source_path, bundle_arc)
+        self.check_and_copy(
+            bundle_source_path,
+            None,
+            bundle_target,
+        )
 
     def _encrypt_record_inline(self, record: dict) -> None:
         """Encrypt sensitive fields in a single record, if passphrase is set."""
@@ -632,7 +700,6 @@ class Command(CryptMixin, PaperlessCommand):
 
     def _write_split_manifest(
         self,
-        sink: ExportSink,
         document_dict: dict,
         document: Document,
         base_name: Path,
@@ -654,4 +721,81 @@ class Command(CryptMixin, PaperlessCommand):
         manifest_name = base_name.with_name(f"{base_name.stem}-manifest.json")
         if self.use_folder_prefix:
             manifest_name = Path("json") / manifest_name
-        sink.add_json(content, manifest_name.as_posix())
+        manifest_name = (self.target / manifest_name).resolve()
+        manifest_name.parent.mkdir(parents=True, exist_ok=True)
+        self.check_and_write_json(content, manifest_name)
+
+    def check_and_write_json(
+        self,
+        content: list[dict] | dict,
+        target: Path,
+    ) -> None:
+        """
+        Writes the source content to the target json file.
+        If --compare-json arg was used, don't write to target file if
+        the file exists and checksum is identical to content checksum.
+        This preserves the file timestamps when no changes are made.
+        """
+
+        target = target.resolve()
+        perform_write = True
+        if target in self.files_in_export_dir:
+            self.files_in_export_dir.remove(target)
+            if self.compare_json:
+                target_checksum = hashlib.blake2b(target.read_bytes()).hexdigest()
+                src_str = json.dumps(
+                    content,
+                    cls=DjangoJSONEncoder,
+                    indent=2,
+                    ensure_ascii=False,
+                )
+                src_checksum = hashlib.blake2b(src_str.encode("utf-8")).hexdigest()
+                if src_checksum == target_checksum:
+                    perform_write = False
+
+        if perform_write:
+            target.write_text(
+                json.dumps(
+                    content,
+                    cls=DjangoJSONEncoder,
+                    indent=2,
+                    ensure_ascii=False,
+                ),
+                encoding="utf-8",
+            )
+
+    def check_and_copy(
+        self,
+        source: Path,
+        source_checksum: str | None,
+        target: Path,
+    ) -> None:
+        """
+        Copies the source to the target, if target doesn't exist or the target doesn't seem to match
+        the source attributes
+        """
+
+        target = target.resolve()
+        if target in self.files_in_export_dir:
+            self.files_in_export_dir.remove(target)
+
+        perform_copy = False
+
+        if target.exists():
+            source_stat = source.stat()
+            target_stat = target.stat()
+            if self.compare_checksums and source_checksum:
+                target_checksum = compute_checksum(target)
+                perform_copy = target_checksum != source_checksum
+            elif (
+                source_stat.st_mtime != target_stat.st_mtime
+                or source_stat.st_size != target_stat.st_size
+            ):
+                perform_copy = True
+        else:
+            # Copy if it does not exist
+            perform_copy = True
+
+        if perform_copy:
+            target.parent.mkdir(parents=True, exist_ok=True)
+            copy_file_with_basic_stats(source, target)

src/documents/management/commands/document_importer.py

@@ -32,8 +32,6 @@ from django.db.models.signals import post_save
 from filelock import FileLock
 from guardian.shortcuts import clear_ct_cache
 
-from documents.export.compression import compress_type_readable
-from documents.export.compression import unreadable_method_names
 from documents.file_handling import create_source_path_directory
 from documents.management.commands.base import PaperlessCommand
 from documents.management.commands.mixins import CryptMixin
@@ -462,20 +460,6 @@ class Command(CryptMixin, PaperlessCommand):
         with tempfile.TemporaryDirectory() as tmp_dir:
             if is_zipfile(self.source):
                 with ZipFile(self.source) as zf:
-                    unsupported = {
-                        info.compress_type
-                        for info in zf.infolist()
-                        if not compress_type_readable(info.compress_type)
-                    }
-                    if unsupported:
-                        names = sorted(unreadable_method_names(unsupported))
-                        message = (
-                            f"This archive uses compression this Python cannot "
-                            f"read ({', '.join(names)})."
-                        )
-                        if "zstd" in names:
-                            message += " zstd archives require Python 3.14+."
-                        raise CommandError(message)
                     zf.extractall(tmp_dir)
                 self.source = Path(tmp_dir)
             self._run_import()

src/documents/tests/export/test_compression.py

@@ -1,190 +0,0 @@
-import sys
-import zipfile
-
-import pytest
-
-from documents.export import compression
-
-
-class TestCompressionMethods:
-    def test_choices_always_include_zstd(self) -> None:
-        """
-        GIVEN:
-            - The compression policy module's CLI choices list
-        WHEN:
-            - Read on any runtime
-        THEN:
-            - zstd is always present; availability is checked separately so
-              argparse never hides it based on the current Python version
-        """
-        assert compression.COMPRESSION_CHOICES == (
-            "stored",
-            "deflated",
-            "bzip2",
-            "lzma",
-            "zstd",
-        )
-
-    @pytest.mark.parametrize(
-        ("name", "constant"),
-        [
-            ("stored", zipfile.ZIP_STORED),
-            ("deflated", zipfile.ZIP_DEFLATED),
-            ("bzip2", zipfile.ZIP_BZIP2),
-            ("lzma", zipfile.ZIP_LZMA),
-        ],
-    )
-    def test_method_maps_to_zipfile_constant(self, name: str, constant: int) -> None:
-        """
-        GIVEN:
-            - A compression method name
-        WHEN:
-            - Looked up in COMPRESSION_METHODS
-        THEN:
-            - It maps to the matching zipfile compression constant
-        """
-        assert compression.COMPRESSION_METHODS[name] == constant
-
-    def test_stored_and_deflated_always_available(self) -> None:
-        """
-        GIVEN:
-            - The stored and deflated compression methods
-        WHEN:
-            - Checked with compression_available()
-        THEN:
-            - Both are always available (zlib is a hard CPython dependency)
-        """
-        assert compression.compression_available("stored")
-        assert compression.compression_available("deflated")
-
-    def test_zstd_availability_tracks_runtime(self) -> None:
-        """
-        GIVEN:
-            - The zstd compression method
-        WHEN:
-            - Checked with compression_available() on this runtime
-        THEN:
-            - Availability matches whether Python is 3.14+
-        """
-        expected: bool = sys.version_info >= (3, 14)
-        assert compression.compression_available("zstd") == expected
-
-
-class TestLevelError:
-    @pytest.mark.parametrize(
-        ("method", "level"),
-        [
-            ("deflated", 0),
-            ("deflated", 9),
-            ("bzip2", 1),
-            ("bzip2", 9),
-            ("zstd", -22),
-            ("zstd", 22),
-            ("deflated", None),
-            ("stored", None),
-        ],
-    )
-    def test_valid_levels_return_none(self, method: str, level: int | None) -> None:
-        """
-        GIVEN:
-            - A method and a level within its valid bounds (or no level)
-        WHEN:
-            - Checked with level_error()
-        THEN:
-            - No error message is returned
-        """
-        assert compression.level_error(method, level) is None
-
-    @pytest.mark.parametrize(
-        ("method", "level"),
-        [
-            ("deflated", 10),
-            ("deflated", -1),
-            ("bzip2", 0),
-            ("bzip2", 10),
-            ("zstd", -23),
-            ("zstd", 23),
-        ],
-    )
-    def test_out_of_range_levels_return_message(
-        self,
-        method: str,
-        level: int,
-    ) -> None:
-        """
-        GIVEN:
-            - A method and a level outside its valid bounds
-        WHEN:
-            - Checked with level_error()
-        THEN:
-            - An error message naming the valid range is returned
-        """
-        msg: str | None = compression.level_error(method, level)
-        assert msg is not None
-        assert "between" in msg
-
-    @pytest.mark.parametrize("method", ["stored", "lzma"])
-    def test_level_on_levelless_method_is_rejected(self, method: str) -> None:
-        """
-        GIVEN:
-            - A method that ignores compression level (stored, lzma)
-        WHEN:
-            - A level is passed to level_error() anyway
-        THEN:
-            - An error message noting the level has no effect is returned
-        """
-        msg: str | None = compression.level_error(method, 5)
-        assert msg is not None
-        assert "no effect" in msg
-
-
-class TestCompressTypeReadable:
-    @pytest.mark.parametrize("ct", [zipfile.ZIP_STORED, zipfile.ZIP_DEFLATED])
-    def test_stored_and_deflated_always_readable(self, ct: int) -> None:
-        """
-        GIVEN:
-            - A stored or deflated compress_type id
-        WHEN:
-            - Checked with compress_type_readable()
-        THEN:
-            - It is always readable
-        """
-        assert compression.compress_type_readable(ct)
-
-    def test_zstd_compress_type_readability_tracks_runtime(self) -> None:
-        """
-        GIVEN:
-            - The current (93) and legacy (20) zstd compress_type ids
-        WHEN:
-            - Checked with compress_type_readable() on this runtime
-        THEN:
-            - Readability matches whether Python is 3.14+
-        """
-        # 93 = ZIP_ZSTANDARD; 20 = legacy zstd method id (read-only)
-        expected: bool = sys.version_info >= (3, 14)
-        assert compression.compress_type_readable(93) == expected
-        assert compression.compress_type_readable(20) == expected
-
-    def test_unknown_compress_type_is_unreadable(self) -> None:
-        """
-        GIVEN:
-            - An unrecognized compress_type id
-        WHEN:
-            - Checked with compress_type_readable()
-        THEN:
-            - It is reported as unreadable
-        """
-        assert not compression.compress_type_readable(9999)
-
-    def test_unreadable_method_names_lists_methods(self) -> None:
-        """
-        GIVEN:
-            - A set containing an unknown compress_type id
-        WHEN:
-            - Passed to unreadable_method_names()
-        THEN:
-            - It is reported generically as "method <id>"
-        """
-        # An unknown method id maps to no name and is reported generically.
-        names: set[str] = compression.unreadable_method_names({9999})
-        assert names == {"method 9999"}

src/documents/tests/export/test_sinks.py

@@ -1,370 +0,0 @@
-import io
-import json
-import os
-import zipfile
-from pathlib import Path
-
-import pytest
-import pytest_mock
-from pytest_django.fixtures import SettingsWrapper
-
-from documents.export.sinks import DirectoryExportSink
-from documents.export.sinks import ExportSink
-from documents.export.sinks import StreamingManifestWriter
-from documents.export.sinks import ZipExportSink
-from documents.export.sinks import _dumps
-
-
-@pytest.fixture()
-def source_file(tmp_path: Path) -> Path:
-    src: Path = tmp_path / "src" / "doc.pdf"
-    src.parent.mkdir(parents=True)
-    src.write_bytes(b"PDF-CONTENT")
-    return src
-
-
-class TestDumps:
-    def test_dumps_is_indented_unicode_json(self) -> None:
-        result: str = _dumps({"a": "é", "b": 1})
-        assert '"é"' in result  # ensure_ascii=False keeps unicode literal
-        assert "\n" in result  # indent=2 produces newlines
-        assert json.loads(result) == {"a": "é", "b": 1}
-
-
-class TestStreamingManifestWriter:
-    def test_writes_json_array_of_records(self) -> None:
-        handle: io.StringIO = io.StringIO()
-        writer: StreamingManifestWriter = StreamingManifestWriter(handle)
-        writer.write_batch([{"pk": 1}, {"pk": 2}])
-        writer.write_record({"pk": 3})
-        writer.close()
-        assert json.loads(handle.getvalue()) == [{"pk": 1}, {"pk": 2}, {"pk": 3}]
-
-    def test_empty_manifest_is_valid_empty_array(self) -> None:
-        handle: io.StringIO = io.StringIO()
-        writer: StreamingManifestWriter = StreamingManifestWriter(handle)
-        writer.close()
-        assert json.loads(handle.getvalue()) == []
-
-
-class TestDirectoryExportSink:
-    def test_add_file_copies_to_relative_arcname(
-        self,
-        tmp_path: Path,
-        source_file: Path,
-    ) -> None:
-        target: Path = tmp_path / "out"
-        target.mkdir()
-        with DirectoryExportSink(
-            target,
-            compare_checksums=False,
-            compare_json=False,
-            delete=False,
-        ) as sink:
-            sink.add_file(source_file, "originals/doc.pdf")
-        assert (target / "originals" / "doc.pdf").read_bytes() == b"PDF-CONTENT"
-
-    def test_add_json_writes_file(self, tmp_path: Path) -> None:
-        target: Path = tmp_path / "out"
-        target.mkdir()
-        with DirectoryExportSink(
-            target,
-            compare_checksums=False,
-            compare_json=False,
-            delete=False,
-        ) as sink:
-            sink.add_json({"version": "x"}, "metadata.json")
-        assert json.loads((target / "metadata.json").read_text()) == {"version": "x"}
-
-    def test_stream_writes_manifest(self, tmp_path: Path) -> None:
-        target: Path = tmp_path / "out"
-        target.mkdir()
-        with DirectoryExportSink(
-            target,
-            compare_checksums=False,
-            compare_json=False,
-            delete=False,
-        ) as sink:
-            with sink.stream("manifest.json") as handle:
-                writer: StreamingManifestWriter = StreamingManifestWriter(handle)
-                writer.write_record({"pk": 1})
-                writer.close()
-        assert json.loads((target / "manifest.json").read_text()) == [{"pk": 1}]
-
-    def test_add_file_skips_when_size_and_mtime_match(
-        self,
-        tmp_path: Path,
-        source_file: Path,
-    ) -> None:
-        # Pre-existing target with identical size+mtime but DIFFERENT content:
-        # if add_file skips (no compare-checksums), the old content survives.
-        target: Path = tmp_path / "out"
-        target.mkdir()
-        existing: Path = target / "originals" / "doc.pdf"
-        existing.parent.mkdir(parents=True)
-        # Same byte length as the source but different content + matching mtime,
-        # so a size/mtime comparison treats it as unchanged and skips the copy.
-        existing.write_bytes(b"X" * len(b"PDF-CONTENT"))
-        stat = source_file.stat()
-        os.utime(existing, (stat.st_atime, stat.st_mtime))
-        with DirectoryExportSink(
-            target,
-            compare_checksums=False,
-            compare_json=False,
-            delete=False,
-        ) as sink:
-            sink.add_file(source_file, "originals/doc.pdf", checksum="abc")
-        assert existing.read_bytes() == b"X" * len(b"PDF-CONTENT")  # skipped
-
-    def test_add_file_recopies_when_compare_checksums_differ(
-        self,
-        tmp_path: Path,
-        source_file: Path,
-    ) -> None:
-        target: Path = tmp_path / "out"
-        target.mkdir()
-        existing: Path = target / "originals" / "doc.pdf"
-        existing.parent.mkdir(parents=True)
-        existing.write_bytes(b"X" * len(b"PDF-CONTENT"))
-        stat = source_file.stat()
-        os.utime(existing, (stat.st_atime, stat.st_mtime))
-        with DirectoryExportSink(
-            target,
-            compare_checksums=True,
-            compare_json=False,
-            delete=False,
-        ) as sink:
-            # wrong checksum forces recopy despite matching size/mtime
-            sink.add_file(source_file, "originals/doc.pdf", checksum="not-the-real-sum")
-        assert existing.read_bytes() == b"PDF-CONTENT"  # recopied
-
-    def test_delete_prunes_unwritten_snapshot_files(
-        self,
-        tmp_path: Path,
-        source_file: Path,
-    ) -> None:
-        target: Path = tmp_path / "out"
-        target.mkdir()
-        stale: Path = target / "stale.pdf"
-        stale.write_bytes(b"STALE")
-        with DirectoryExportSink(
-            target,
-            compare_checksums=False,
-            compare_json=False,
-            delete=True,
-        ) as sink:
-            sink.add_file(source_file, "originals/doc.pdf")
-        assert not stale.exists()
-        assert (target / "originals" / "doc.pdf").exists()
-
-    def test_no_delete_keeps_unwritten_files(
-        self,
-        tmp_path: Path,
-        source_file: Path,
-    ) -> None:
-        target: Path = tmp_path / "out"
-        target.mkdir()
-        stale: Path = target / "stale.pdf"
-        stale.write_bytes(b"STALE")
-        with DirectoryExportSink(
-            target,
-            compare_checksums=False,
-            compare_json=False,
-            delete=False,
-        ) as sink:
-            sink.add_file(source_file, "originals/doc.pdf")
-        assert stale.exists()
-
-
-class TestZipExportSink:
-    def test_round_trip_files_json_and_stream(
-        self,
-        tmp_path: Path,
-        source_file: Path,
-    ) -> None:
-        target: Path = tmp_path / "out"
-        target.mkdir()
-        with ZipExportSink(target, "export", delete=False) as sink:
-            sink.add_file(source_file, "originals/doc.pdf")
-            sink.add_json({"version": "x"}, "metadata.json")
-            with sink.stream("manifest.json") as handle:
-                writer = StreamingManifestWriter(handle)
-                writer.write_record({"pk": 1})
-                writer.close()
-        zip_path: Path = target / "export.zip"
-        assert zip_path.exists()
-        assert not (target / "export.zip.tmp").exists()
-        with zipfile.ZipFile(zip_path) as zf:
-            names = set(zf.namelist())
-            assert {"originals/doc.pdf", "metadata.json", "manifest.json"} <= names
-            assert zf.read("originals/doc.pdf") == b"PDF-CONTENT"
-            assert json.loads(zf.read("manifest.json")) == [{"pk": 1}]
-
-    def test_nested_arcname_emits_directory_marker(
-        self,
-        tmp_path: Path,
-        source_file: Path,
-    ) -> None:
-        target: Path = tmp_path / "out"
-        target.mkdir()
-        with ZipExportSink(target, "export", delete=False) as sink:
-            sink.add_file(source_file, "originals/doc.pdf")
-        with zipfile.ZipFile(target / "export.zip") as zf:
-            assert "originals/" in zf.namelist()
-
-    def test_flat_arcname_has_no_directory_markers(
-        self,
-        tmp_path: Path,
-        source_file: Path,
-    ) -> None:
-        target: Path = tmp_path / "out"
-        target.mkdir()
-        with ZipExportSink(target, "export", delete=False) as sink:
-            sink.add_file(source_file, "doc.pdf")
-        with zipfile.ZipFile(target / "export.zip") as zf:
-            assert all(not n.endswith("/") for n in zf.namelist())
-
-    def test_exception_leaves_no_zip_and_no_tmp(
-        self,
-        tmp_path: Path,
-        source_file: Path,
-    ) -> None:
-        target: Path = tmp_path / "out"
-        target.mkdir()
-        with pytest.raises(RuntimeError):
-            with ZipExportSink(target, "export", delete=False) as sink:
-                sink.add_file(source_file, "doc.pdf")
-                raise RuntimeError("boom")
-        assert not (target / "export.zip").exists()
-        assert not (target / "export.zip.tmp").exists()
-
-    def test_exception_inside_stream_cleans_up_manifest_tmp(
-        self,
-        tmp_path: Path,
-        source_file: Path,
-        settings: SettingsWrapper,
-    ) -> None:
-        scratch_dir = tmp_path / "scratch"
-        settings.SCRATCH_DIR = scratch_dir
-        target: Path = tmp_path / "out"
-        target.mkdir()
-        with pytest.raises(RuntimeError):
-            with ZipExportSink(target, "export", delete=False) as sink:
-                sink.add_file(source_file, "doc.pdf")
-                with sink.stream("manifest.json") as handle:
-                    handle.write("[")
-                    raise RuntimeError("boom")
-        assert list(scratch_dir.glob("export-manifest-*")) == []
-        assert not (target / "export.zip").exists()
-        assert not (target / "export.zip.tmp").exists()
-
-    def test_abort_after_manifest_written_cleans_up_pending_tmp(
-        self,
-        tmp_path: Path,
-        settings: SettingsWrapper,
-    ) -> None:
-        scratch_dir = tmp_path / "scratch"
-        settings.SCRATCH_DIR = scratch_dir
-        target: Path = tmp_path / "out"
-        target.mkdir()
-        with pytest.raises(RuntimeError):
-            with ZipExportSink(target, "export", delete=False) as sink:
-                with sink.stream("manifest.json") as handle:
-                    handle.write("[]")
-                raise RuntimeError("boom")
-        assert list(scratch_dir.glob("export-manifest-*")) == []
-        assert not (target / "export.zip").exists()
-
-    def test_delete_wipes_destination_on_success(
-        self,
-        tmp_path: Path,
-        source_file: Path,
-    ) -> None:
-        target: Path = tmp_path / "out"
-        target.mkdir()
-        (target / "preexisting.txt").write_text("old")
-        (target / "olddir").mkdir()
-        with ZipExportSink(target, "export", delete=True) as sink:
-            sink.add_file(source_file, "doc.pdf")
-        assert (target / "export.zip").exists()
-        assert not (target / "preexisting.txt").exists()
-        assert not (target / "olddir").exists()
-
-    def test_abort_with_delete_does_not_wipe_destination(
-        self,
-        tmp_path: Path,
-        source_file: Path,
-    ) -> None:
-        target: Path = tmp_path / "out"
-        target.mkdir()
-        (target / "preexisting.txt").write_text("old")
-        with pytest.raises(RuntimeError):
-            with ZipExportSink(target, "export", delete=True) as sink:
-                sink.add_file(source_file, "doc.pdf")
-                raise RuntimeError("boom")
-        assert (target / "preexisting.txt").exists()
-        assert not (target / "export.zip").exists()
-
-
-class TestZipExportSinkCompression:
-    @pytest.mark.parametrize(
-        ("method", "constant"),
-        [
-            ("stored", zipfile.ZIP_STORED),
-            ("deflated", zipfile.ZIP_DEFLATED),
-            ("bzip2", zipfile.ZIP_BZIP2),
-            ("lzma", zipfile.ZIP_LZMA),
-        ],
-    )
-    def test_compression_and_level_forwarded_to_zipfile(
-        self,
-        mocker: pytest_mock.MockerFixture,
-        tmp_path: Path,
-        method: str,
-        constant: int,
-    ) -> None:
-        """
-        GIVEN:
-            - A ZipExportSink constructed with a compression method and level
-        WHEN:
-            - The sink is opened
-        THEN:
-            - zipfile.ZipFile is constructed with those values forwarded
-              unchanged (whether ZipFile actually compresses is Python's own
-              contract, not ours, so this checks the call args, not a real
-              archive)
-        """
-        target: Path = tmp_path / "out"
-        target.mkdir()
-        zip_cls = mocker.patch("documents.export.sinks.zipfile.ZipFile")
-        sink = ZipExportSink(target, "export", compression=constant, compresslevel=5)
-        sink._open()
-        zip_cls.assert_called_once_with(
-            mocker.ANY,
-            "w",
-            compression=constant,
-            compresslevel=5,
-            allowZip64=True,
-        )
-
-
-class TestStreamContract:
-    @pytest.fixture(params=["dir", "zip"])
-    def sink(self, request: pytest.FixtureRequest, tmp_path: Path) -> ExportSink:
-        target: Path = tmp_path / "out"
-        target.mkdir()
-        if request.param == "dir":
-            return DirectoryExportSink(
-                target,
-                compare_checksums=False,
-                compare_json=False,
-                delete=False,
-            )
-        return ZipExportSink(target, "export", delete=False)
-
-    def test_second_concurrent_stream_is_rejected(self, sink: ExportSink) -> None:
-        with sink:
-            with sink.stream("manifest.json"):
-                with pytest.raises(RuntimeError, match="already open"):
-                    with sink.stream("other.json"):
-                        pass

src/documents/tests/test_management_exporter.py

@@ -6,8 +6,6 @@ from datetime import timedelta
 from io import StringIO
 from pathlib import Path
 from unittest import mock
-from zipfile import ZIP_DEFLATED
-from zipfile import ZIP_LZMA
 from zipfile import ZipFile
 
 import pytest
@@ -428,7 +426,7 @@ class TestExportImport(
         st_mtime_1 = (self.target / "manifest.json").stat().st_mtime
 
         with mock.patch(
-            "documents.export.sinks.copy_file_with_basic_stats",
+            "documents.management.commands.document_exporter.copy_file_with_basic_stats",
         ) as m:
             self._do_export()
             m.assert_not_called()
@@ -439,7 +437,7 @@ class TestExportImport(
         Path(self.d1.source_path).touch()
 
         with mock.patch(
-            "documents.export.sinks.copy_file_with_basic_stats",
+            "documents.management.commands.document_exporter.copy_file_with_basic_stats",
         ) as m:
             self._do_export()
             self.assertEqual(m.call_count, 1)
@@ -466,7 +464,7 @@ class TestExportImport(
         self.assertIsFile(self.target / "manifest.json")
 
         with mock.patch(
-            "documents.export.sinks.copy_file_with_basic_stats",
+            "documents.management.commands.document_exporter.copy_file_with_basic_stats",
         ) as m:
             self._do_export()
             m.assert_not_called()
@@ -477,7 +475,7 @@ class TestExportImport(
         self.d2.save()
 
         with mock.patch(
-            "documents.export.sinks.copy_file_with_basic_stats",
+            "documents.management.commands.document_exporter.copy_file_with_basic_stats",
         ) as m:
             self._do_export(compare_checksums=True)
             self.assertEqual(m.call_count, 1)
@@ -1060,206 +1058,6 @@ class TestExportImport(
 
         self.assertEqual(Document.objects.all().count(), 4)
 
-    def test_zip_with_compare_flags_raises(self) -> None:
-        """
-        GIVEN:
-            - A request to export to a zip file
-        WHEN:
-            - --compare-checksums or --compare-json is also passed
-        THEN:
-            - A CommandError is raised (the flags are no-ops in zip mode)
-        """
-        for flag in ("--compare-checksums", "--compare-json"):
-            with self.subTest(flag=flag):
-                with self.assertRaises(CommandError):
-                    call_command(
-                        "document_exporter",
-                        self.target,
-                        "--zip",
-                        flag,
-                        skip_checks=True,
-                    )
-
-    def test_compression_flags_require_zip(self) -> None:
-        """
-        GIVEN:
-            - A request to export without --zip
-        WHEN:
-            - --zip-compression or --zip-compression-level is passed anyway
-        THEN:
-            - A CommandError is raised (the flags are meaningless without --zip)
-        """
-        for args in (
-            ["--zip-compression", "lzma"],
-            ["--zip-compression-level", "5"],
-        ):
-            with self.assertRaises(CommandError):
-                call_command(
-                    "document_exporter",
-                    self.target,
-                    *args,
-                    skip_checks=True,
-                )
-
-    def test_zip_compression_level_out_of_range_raises(self) -> None:
-        """
-        GIVEN:
-            - A request to export to a zip file
-        WHEN:
-            - --zip-compression-level is outside the chosen method's valid range
-        THEN:
-            - A CommandError is raised
-        """
-        with self.assertRaises(CommandError):
-            call_command(
-                "document_exporter",
-                self.target,
-                "--zip",
-                "--zip-compression",
-                "deflated",
-                "--zip-compression-level",
-                "99",
-                skip_checks=True,
-            )
-
-    def test_zip_compression_level_rejected_for_stored(self) -> None:
-        """
-        GIVEN:
-            - A request to export to a zip file with --zip-compression stored
-        WHEN:
-            - --zip-compression-level is also passed
-        THEN:
-            - A CommandError is raised (stored ignores level entirely)
-        """
-        with self.assertRaises(CommandError):
-            call_command(
-                "document_exporter",
-                self.target,
-                "--zip",
-                "--zip-compression",
-                "stored",
-                "--zip-compression-level",
-                "5",
-                skip_checks=True,
-            )
-
-    def test_zip_compression_level_rejected_for_lzma(self) -> None:
-        """
-        GIVEN:
-            - A request to export to a zip file with --zip-compression lzma
-        WHEN:
-            - --zip-compression-level is also passed
-        THEN:
-            - A CommandError is raised (lzma ignores level entirely)
-        """
-        with self.assertRaises(CommandError):
-            call_command(
-                "document_exporter",
-                self.target,
-                "--zip",
-                "--zip-compression",
-                "lzma",
-                "--zip-compression-level",
-                "5",
-                skip_checks=True,
-            )
-
-    def test_zstd_unavailable_raises_friendly_error(self) -> None:
-        """
-        GIVEN:
-            - A Python runtime without zstd support (< 3.14)
-        WHEN:
-            - --zip-compression zstd is requested
-        THEN:
-            - A CommandError naming the Python version requirement is raised
-
-        zstd availability is mocked rather than relying on the actual
-        runtime: on a Python 3.14+ CI leg, ZSTD is not None, so without the
-        mock this check is skipped and the command falls through into the
-        real export, which fails on missing document files instead of
-        raising the expected CommandError.
-        """
-        with (
-            mock.patch(
-                "documents.management.commands.document_exporter.ZSTD",
-                None,
-            ),
-            mock.patch(
-                "documents.management.commands.document_exporter.compression_available",
-                return_value=False,
-            ),
-            self.assertRaises(CommandError) as e,
-        ):
-            call_command(
-                "document_exporter",
-                self.target,
-                "--zip",
-                "--zip-compression",
-                "zstd",
-                skip_checks=True,
-            )
-        self.assertIn("3.14", str(e.exception))
-
-    def test_zip_compression_flag_resolves_to_sink_constant(self) -> None:
-        """
-        GIVEN:
-            - A request to export to a zip file with --zip-compression lzma
-        WHEN:
-            - The export runs
-        THEN:
-            - ZipExportSink is constructed with the resolved ZIP_LZMA constant
-              (whether zipfile actually compresses with the chosen method is
-              Python's own contract, and ZipExportSink's own tests already
-              cover the forwarding; what this command owns is resolving the
-              CLI string to the right constant, so assert that resolution
-              directly)
-        """
-        with mock.patch(
-            "documents.management.commands.document_exporter.ZipExportSink",
-        ) as sink_cls:
-            call_command(
-                "document_exporter",
-                self.target,
-                "--zip",
-                "--zip-compression",
-                "lzma",
-                skip_checks=True,
-            )
-        sink_cls.assert_called_once_with(
-            mock.ANY,
-            mock.ANY,
-            delete=False,
-            compression=ZIP_LZMA,
-            compresslevel=None,
-        )
-
-    def test_default_zip_compression_resolves_to_deflate(self) -> None:
-        """
-        GIVEN:
-            - A request to export to a zip file with no --zip-compression flag
-        WHEN:
-            - The export runs
-        THEN:
-            - ZipExportSink is constructed with the default ZIP_DEFLATED
-              constant and compresslevel=None, matching pre-existing behavior
-        """
-        with mock.patch(
-            "documents.management.commands.document_exporter.ZipExportSink",
-        ) as sink_cls:
-            call_command(
-                "document_exporter",
-                self.target,
-                "--zip",
-                skip_checks=True,
-            )
-        sink_cls.assert_called_once_with(
-            mock.ANY,
-            mock.ANY,
-            delete=False,
-            compression=ZIP_DEFLATED,
-            compresslevel=None,
-        )
-
 
 @pytest.mark.management
 class TestCryptExportImport(

src/documents/tests/test_management_importer.py

@@ -525,35 +525,6 @@ class TestCommandImport(
         self.assertEqual(doc.tags.count(), 1)
         self.assertEqual(doc.tags.first().name, "batch-flush-tag")
 
-    def test_import_rejects_unreadable_compression(self) -> None:
-        """
-        GIVEN:
-            - A zip archive with an entry whose compression this Python can't read
-        WHEN:
-            - Import is attempted
-        THEN:
-            - A CommandError naming the issue is raised, before extraction
-        """
-        import zipfile
-        from unittest import mock
-
-        archive = Path(self.dirs.scratch_dir) / "export.zip"
-        with zipfile.ZipFile(archive, "w") as zf:
-            zf.writestr("manifest.json", "[]")
-
-        with mock.patch(
-            "documents.management.commands.document_importer.compress_type_readable",
-            return_value=False,
-        ):
-            with self.assertRaises(CommandError) as e:
-                call_command(
-                    "document_importer",
-                    str(archive),
-                    "--no-progress-bar",
-                    skip_checks=True,
-                )
-        self.assertIn("compression", str(e.exception))
-
 
 @pytest.mark.management
 @pytest.mark.django_db

src/paperless/parsers/remote.py

@@ -21,6 +21,7 @@ from typing import Self
 
 from django.conf import settings
 
+from documents.parsers import ParseError
 from paperless.version import __full_version_str__
 
 if TYPE_CHECKING:
@@ -366,8 +367,7 @@ class RemoteDocumentParser:
         """Send ``file`` to Azure AI Document Intelligence and return text.
 
         Downloads the searchable PDF output from Azure and stores it at
-        ``self._archive_path``.  Returns the extracted text content, or
-        ``None`` on failure (the error is logged).
+        ``self._archive_path``.
 
         Parameters
         ----------
@@ -379,7 +379,14 @@ class RemoteDocumentParser:
         Returns
         -------
         str | None
-            Extracted text, or None if the Azure call failed.
+            Extracted text.
+
+        Raises
+        ------
+        ParseError
+            If the Azure call fails for any reason. The error is logged
+            and re-raised so consumption fails loudly instead of silently
+            producing a document with no content.
         """
         if TYPE_CHECKING:
             # Callers must have already validated config via engine_is_valid():
@@ -426,8 +433,7 @@ class RemoteDocumentParser:
 
         except Exception as e:
             logger.exception("Azure AI Vision parsing failed: %s", e)
+            raise ParseError(f"Azure AI Vision parsing failed: {e}") from e
 
         finally:
             client.close()
-
-        return None

src/paperless/tests/parsers/test_remote_parser.py

@@ -20,6 +20,7 @@ from unittest.mock import Mock
 
 import pytest
 
+from documents.parsers import ParseError
 from paperless.parsers import ParserContext
 from paperless.parsers import ParserProtocol
 from paperless.parsers.remote import RemoteDocumentParser
@@ -342,15 +343,14 @@ class TestRemoteParserParse:
 
 
 class TestRemoteParserParseError:
-    def test_parse_returns_empty_on_azure_error(
+    def test_parse_raises_parse_error_on_azure_error(
         self,
         remote_parser: RemoteDocumentParser,
         simple_digital_pdf_file: Path,
         failing_azure_client: Mock,
     ) -> None:
-        remote_parser.parse(simple_digital_pdf_file, "application/pdf")
-
-        assert remote_parser.get_text() == ""
+        with pytest.raises(ParseError, match="Azure AI Vision parsing failed"):
+            remote_parser.parse(simple_digital_pdf_file, "application/pdf")
 
     def test_parse_closes_client_on_error(
         self,
@@ -358,7 +358,8 @@ class TestRemoteParserParseError:
         simple_digital_pdf_file: Path,
         failing_azure_client: Mock,
     ) -> None:
-        remote_parser.parse(simple_digital_pdf_file, "application/pdf")
+        with pytest.raises(ParseError):
+            remote_parser.parse(simple_digital_pdf_file, "application/pdf")
 
         failing_azure_client.close.assert_called_once()
 
@@ -371,7 +372,8 @@ class TestRemoteParserParseError:
     ) -> None:
         mock_log = mocker.patch("paperless.parsers.remote.logger")
 
-        remote_parser.parse(simple_digital_pdf_file, "application/pdf")
+        with pytest.raises(ParseError):
+            remote_parser.parse(simple_digital_pdf_file, "application/pdf")
 
         mock_log.exception.assert_called_once()
         assert "Azure AI Vision parsing failed" in mock_log.exception.call_args[0][0]