test: move container test suites to pytest (#112)

* test: scaffold pytest suite with mise, uv and Task Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * ci: run the pytest suite via mise and task Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix: make test cleanup survive repeated SIGINT A group Ctrl-C reaches pytest a second time via task and uv while the session fixture tears down, aborting the volume cleanup. Ignore SIGINT and SIGTERM during cleanup, remove volumes in finally, and tolerate already-removed objects. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix: pin Python 3.13 for uv without mise Add .python-version (read by mise and uv) and cap requires-python at <3.14 so bare uv uses 3.13 or fails instead of picking 3.14. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix: clean up test containers on SIGTERM Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * test: port compat tests to pytest Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * test: assert the success message first in the corruption test Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * test: assert the empty-tag snapshot carries no tags Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * test: remove anonymous volumes with short-lived test containers Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * test: port feature-gated tests to pytest Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * test: port sabotage self-test to task test:sabotage Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix: create parent directories in put_file Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * test: move structure.yml to tests and remove cst volume leak container-structure-test left one anonymous volume per test container. tests/structure.py runs it with --save and removes the new containers with their volumes afterwards. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * test: remove the Bash suite Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * ci: run only the Python suite in the test job Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * docs: describe the Python test suite Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * test: isolate scenario steps, share helpers, use restore_latest, HEAD dir probe Scenario fixtures record each step; a failing step fails only the tests that assert on it. running() and raise_interrupt() live in helpers.py. put_file probes parent directories with a HEAD request instead of a get_archive stream. stream_run uses restore_latest and asserts its result. Docstrings no longer refer to the removed Bash suite. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * test: sabotage check rejects errors and malformed reports Only <failure> of the expected tests counts as red; errors in them and a malformed JUnit report exit 2 (invalid) instead of passing or exiting 1. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * test: structure wrapper removes only its own containers Restrict cleanup to containers of the image under test without the rbd-test-run label and ignore NotFound. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * build: run all test suites and aggregate the exit code task test no longer stops at the first failing suite. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix: sabotage check treats failed scenario steps as INVALID StepFailed is no AssertionError any more; sabotage.py counts a JUnit failure named StepFailed as error, so an expected-red test only counts when its own assertion fails. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix: hard timeout for container-structure-test CST_TIMEOUT (default 300 s) kills a hanging run; existing cleanup removes the containers; exit code 124 with a clear message. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix: shorten sabotage message for lint Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix: handle JUnit failures without message in sabotage check Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-13 15:45:22 +00:00 · 2026-10-20 20:47:03 +02:00
1 parent 31f06b5d9b
commit be7d426f78
@@ -0,0 +1,171 @@
+"""Shared fixtures and feature gating for the container test suite."""
+
+from __future__ import annotations
+
+import contextlib
+import os
+import signal
+import subprocess
+import uuid
+from collections.abc import Callable, Iterator
+from pathlib import Path
+
+import docker
+import pytest
+from helpers import Container, raise_interrupt, remove_quietly, run_once, start_container
+
+ROOT = Path(__file__).resolve().parent.parent
+IMAGE = os.environ.get("IMAGE", "restic-backup-docker:test")
+REQUIRED_FEATURES_FILE = Path(
+    os.environ.get("REQUIRED_FEATURES_FILE", Path(__file__).with_name("required-features.txt"))
+)
+RUN_ID = uuid.uuid4().hex[:8]
+
+# Shell probe run once in a throwaway container; one line per detected feature.
+FEATURE_PROBE = """
+grep -q RESTIC_STREAM /bin/backup && echo stream
+command -v docker >/dev/null && echo docker-cli
+command -v bash >/dev/null && echo bash
+command -v msmtp >/dev/null && echo msmtp
+true
+"""
+KNOWN_FEATURES = ("stream", "docker-cli", "bash", "msmtp")
+
+
+def required_features() -> set[str]:
+    if not REQUIRED_FEATURES_FILE.is_file():
+        return set()
+    lines = (line.strip() for line in REQUIRED_FEATURES_FILE.read_text().splitlines())
+    return {line for line in lines if line and not line.startswith("#")}
+
+
+def pytest_configure(config: pytest.Config) -> None:
+    """Turn SIGTERM into KeyboardInterrupt so session teardown (cleanup) runs."""
+    signal.signal(signal.SIGTERM, raise_interrupt)
+
+
+def pytest_addoption(parser: pytest.Parser) -> None:
+    parser.addoption("--build", action="store_true", help="build the image before testing")
+
+
+@pytest.fixture(scope="session")
+def docker_client() -> Iterator[docker.DockerClient]:
+    try:
+        client = docker.from_env()
+        client.ping()
+    except docker.errors.DockerException as exc:
+        pytest.exit(f"docker is not available: {exc}", returncode=2)
+    yield client
+    client.close()
+
+
+@pytest.fixture(scope="session")
+def image(request: pytest.FixtureRequest, docker_client: docker.DockerClient) -> str:
+    """Tag of the image under test; built on demand with --build."""
+    if request.config.getoption("--build"):
+        subprocess.run(["docker", "build", "--quiet", "-t", IMAGE, str(ROOT)], check=True)
+    try:
+        docker_client.images.get(IMAGE)
+    except docker.errors.ImageNotFound:
+        pytest.exit(f"image {IMAGE} not found, run `task build` or pass --build", returncode=2)
+    return IMAGE
+
+
+def _remove_leftovers(client: docker.DockerClient) -> None:
+    """Remove containers, then volumes, labelled with this run; safe to call repeatedly."""
+    flt = {"label": f"rbd-test-run={RUN_ID}"}
+    try:
+        for container in client.containers.list(all=True, filters=flt):
+            with contextlib.suppress(docker.errors.NotFound):
+                container.remove(force=True, v=True)
+    finally:
+        for volume in client.volumes.list(filters=flt):
+            with contextlib.suppress(docker.errors.NotFound):
+                volume.remove(force=True)
+
+
+@pytest.fixture(scope="session", autouse=True)
+def _cleanup_leftovers(docker_client: docker.DockerClient) -> Iterator[None]:
+    """Remove every container/volume this run created, even after failures or Ctrl-C.
+
+    A Ctrl-C in `task`/`uv` is forwarded as a second SIGINT while teardown runs, so
+    SIGINT/SIGTERM are ignored until cleanup is finished.
+    """
+    try:
+        yield
+    finally:
+        old = {sig: signal.signal(sig, signal.SIG_IGN) for sig in (signal.SIGINT, signal.SIGTERM)}
+        try:
+            _remove_leftovers(docker_client)
+        finally:
+            for sig, handler in old.items():
+                signal.signal(sig, handler)
+
+
+@pytest.fixture(scope="session")
+def detected_features(docker_client: docker.DockerClient, image: str) -> set[str]:
+    out = run_once(
+        docker_client,
+        image,
+        entrypoint=["sh", "-c", FEATURE_PROBE],
+        labels={"rbd-test-run": RUN_ID},
+    )
+    return set(out.split())
+
+
+def pytest_collection_modifyitems(items: list[pytest.Item]) -> None:
+    for item in items:
+        for marker in item.iter_markers("feature"):
+            item.fixturenames.append("_feature_gate")
+            if marker.args[0] not in KNOWN_FEATURES:
+                raise pytest.UsageError(f"unknown feature gate: {marker.args[0]}")
+
+
+def _apply_feature_gate(node: pytest.Item | pytest.Class, detected: set[str]) -> None:
+    for marker in node.iter_markers("feature"):
+        name = marker.args[0]
+        if name in detected:
+            continue
+        if name in required_features():
+            pytest.fail(
+                f"feature '{name}' listed in {REQUIRED_FEATURES_FILE.name} "
+                "but not present in the image",
+                pytrace=False,
+            )
+        pytest.skip(f"feature '{name}' not in image yet")
+
+
+@pytest.fixture
+def _feature_gate(request: pytest.FixtureRequest, detected_features: set[str]) -> None:
+    """Skip a feature test when the image lacks it; fail when it is required."""
+    _apply_feature_gate(request.node, detected_features)
+
+
+@pytest.fixture(scope="class")
+def feature_gate_class(request: pytest.FixtureRequest, detected_features: set[str]) -> None:
+    """Same gate for class-scoped fixtures: request it first so no container starts when gated."""
+    _apply_feature_gate(request.node, detected_features)
+
+
+LABELS = {"rbd-test-run": RUN_ID}
+
+
+@pytest.fixture(scope="session")
+def run_container(
+    docker_client: docker.DockerClient, image: str
+) -> Callable[..., contextlib.AbstractContextManager[Container]]:
+    """Factory: `with run_container("name", env={...}) as c:` starts, waits, always removes."""
+
+    @contextlib.contextmanager
+    def _run(
+        suffix: str, env: dict[str, str] | None = None, files: dict[str, str] | None = None
+    ) -> Iterator[Container]:
+        container = start_container(
+            docker_client, image, f"rbd-test-{suffix}-{RUN_ID}", LABELS, env, files
+        )
+        try:
+            yield container
+        finally:
+            remove_quietly(container)
+
+    return _run
@@ -0,0 +1,253 @@
+"""Helpers around a running test container.
+
+Rule for everything in here: a failing command is never turned into an empty or
+zero result. Helpers that return parsed data raise on a non-zero exit, so a broken
+restic call fails the test instead of looking like "no snapshots".
+"""
+
+from __future__ import annotations
+
+import contextlib
+import io
+import json
+import signal
+import tarfile
+import time
+from collections.abc import Callable
+from dataclasses import dataclass
+from pathlib import PurePosixPath
+from typing import Any
+
+import docker
+import pytest
+from docker.models.containers import Container as DockerContainer
+
+START_TIMEOUT = 60
+START_MARKER = "Container started."
+
+
+@dataclass(frozen=True)
+class ExecResult:
+    rc: int
+    out: str  # stdout and stderr, interleaved like `2>&1`
+
+
+class CommandError(AssertionError):
+    pass
+
+
+def raise_interrupt(signum: int, _frame: object) -> None:
+    """Signal handler: turn SIGTERM into KeyboardInterrupt so cleanup code runs."""
+    raise KeyboardInterrupt(f"received {signal.Signals(signum).name}")
+
+
+class StepFailed(Exception):
+    """A scenario step could not run; not an assertion about the behaviour under test.
+
+    Deliberately no AssertionError: sabotage.py tells it apart from a real assertion
+    failure by the exception name in the JUnit report.
+    """
+
+
+class Scenario:
+    """Results of the actions of one scenario fixture.
+
+    `record(name, fn)` runs one step and stores its result or its error. Reading an
+    attribute of a failed step raises, so only the tests that assert on that step fail;
+    the other tests of the class still run against what the scenario did produce.
+    """
+
+    def __init__(self) -> None:
+        self._steps: dict[str, tuple[Any, BaseException | None]] = {}
+
+    def record(self, name: str, step: Callable[[], Any]) -> None:
+        try:
+            self._steps[name] = (step(), None)
+        except Exception as exc:  # noqa: BLE001 - the error is re-raised on access
+            self._steps[name] = (None, exc)
+
+    def __getattr__(self, name: str) -> Any:
+        try:
+            value, error = self.__dict__["_steps"][name]
+        except KeyError:
+            raise AttributeError(name) from None
+        if error is not None:
+            raise StepFailed(f"scenario step '{name}' failed: {error}") from error
+        return value
+
+
+def running(container: Container) -> Container:
+    """Scenarios that act on the container need it started; the start tests do not."""
+    if not container.started:
+        pytest.fail(f"container did not start:\n{container.start_log}", pytrace=False)
+    return container
+
+
+class Container:
+    """A started image container with exec/file helpers."""
+
+    def __init__(self, raw: DockerContainer, started: bool, start_log: str) -> None:
+        self.raw = raw
+        self.started = started
+        self.start_log = start_log
+
+    def exec(self, *cmd: str, env: dict[str, str] | None = None) -> ExecResult:
+        """Run a command; the exit code is returned, not checked."""
+        code, out = self.raw.exec_run(list(cmd), environment=env)
+        return ExecResult(code, out.decode(errors="replace").replace("\r", ""))
+
+    def run(self, *cmd: str, env: dict[str, str] | None = None) -> str:
+        """Run a command that must succeed; return its output."""
+        res = self.exec(*cmd, env=env)
+        if res.rc != 0:
+            raise CommandError(f"{' '.join(cmd)} exited {res.rc}:\n{res.out}")
+        return res.out
+
+    def write_file(self, path: str, content: str, *, executable: bool = False) -> None:
+        """Create a file (and its directory) without a bind mount."""
+        self.run("mkdir", "-p", str(PurePosixPath(path).parent))
+        put_file(self.raw, path, content, executable=executable)
+
+    def read_file(self, path: str) -> str:
+        return self.run("cat", path)
+
+    def file_exists(self, path: str) -> bool:
+        res = self.exec("test", "-f", path)
+        if res.rc not in (0, 1):
+            raise CommandError(f"test -f {path} exited {res.rc}:\n{res.out}")
+        return res.rc == 0
+
+    def logs(self) -> str:
+        return self.raw.logs().decode(errors="replace").replace("\r", "")
+
+    # restic -------------------------------------------------------------
+
+    def restic_json(self, *args: str) -> Any:
+        """Run restic with --json and parse stdout; any failure raises."""
+        code, (out, err) = self.raw.exec_run(["restic", *args, "--json"], demux=True)
+        if code != 0:
+            raise CommandError(f"restic {' '.join(args)} exited {code}:\n{(err or b'').decode()}")
+        return json.loads((out or b"").decode())
+
+    def snapshots(self) -> list[dict[str, Any]]:
+        return self.restic_json("snapshots") or []
+
+    def backed_up_paths(self, snapshot: str = "latest") -> set[str]:
+        """Paths contained in a snapshot (`restic ls --json` emits one JSON object per line)."""
+        code, (out, err) = self.raw.exec_run(["restic", "ls", snapshot, "--json"], demux=True)
+        if code != 0:
+            raise CommandError(f"restic ls {snapshot} exited {code}:\n{(err or b'').decode()}")
+        lines = (out or b"").decode().splitlines()
+        return {n["path"] for n in map(json.loads, lines) if n.get("struct_type") == "node"}
+
+    def restore_latest(self, target: str) -> ExecResult:
+        return self.exec("restic", "restore", "latest", "--target", target)
+
+
+def _dir_exists(raw: DockerContainer, path: str) -> bool:
+    """Ask the daemon with a HEAD request on the archive endpoint: stat only, no tar stream.
+
+    A container that is not started cannot exec `test -d`, and `get_archive` would leave
+    its streaming response open.
+    """
+    api = raw.client.api
+    response = api.head(api._url("/containers/{0}/archive", raw.id), params={"path": path})
+    try:
+        if response.status_code == 404:
+            return False
+        response.raise_for_status()
+        return True
+    finally:
+        response.close()
+
+
+def put_file(raw: DockerContainer, path: str, content: str, *, executable: bool = False) -> None:
+    """Copy a file into an existing (also not yet started) container via a tar archive.
+
+    Missing parent directories are created (like `mkdir -p`); existing ones are left
+    untouched, so their mode and owner stay as the image defines them.
+    """
+    target = PurePosixPath(path)
+    missing = []
+    for parent in target.parents:
+        if _dir_exists(raw, str(parent)):
+            break
+        missing.append(parent)
+    data = content.encode()
+    buf = io.BytesIO()
+    with tarfile.open(fileobj=buf, mode="w") as tar:
+        for directory in reversed(missing):
+            dir_info = tarfile.TarInfo(str(directory).lstrip("/"))
+            dir_info.type = tarfile.DIRTYPE
+            dir_info.mode = 0o755
+            tar.addfile(dir_info)
+        info = tarfile.TarInfo(str(target).lstrip("/"))
+        info.size = len(data)
+        info.mode = 0o755 if executable else 0o644
+        tar.addfile(info, io.BytesIO(data))
+    buf.seek(0)
+    if not raw.put_archive("/", buf):
+        raise CommandError(f"could not write {path}")
+
+
+def start_container(
+    client: docker.DockerClient,
+    image: str,
+    name: str,
+    labels: dict[str, str],
+    env: dict[str, str] | None = None,
+    files: dict[str, str] | None = None,
+) -> Container:
+    """Start the image with a cron far in the future, so only explicit backups run.
+
+    `files` (path in container -> content) are placed before entry.sh runs, without a
+    bind mount, so tests can cover what the entrypoint does with them at startup.
+    """
+    environment = {
+        "RESTIC_PASSWORD": "testpassword",
+        "RESTIC_TAG": "testtag",
+        "BACKUP_CRON": "0 0 31 2 *",
+        **(env or {}),
+    }
+    if files:
+        raw = client.containers.create(image, name=name, environment=environment, labels=labels)
+        for path, content in files.items():
+            put_file(raw, path, content)
+        raw.start()
+    else:
+        raw = client.containers.run(
+            image, detach=True, name=name, environment=environment, labels=labels
+        )
+    deadline = time.monotonic() + START_TIMEOUT
+    while time.monotonic() < deadline:
+        log = raw.logs().decode(errors="replace")
+        if START_MARKER in log:
+            return Container(raw, True, log)
+        raw.reload()
+        if raw.status != "running":
+            return Container(raw, False, f"container exited early:\n{log}")
+        time.sleep(0.5)
+    return Container(raw, False, f"timeout waiting for start:\n{raw.logs().decode()}")
+
+
+def remove_quietly(container: Container) -> None:
+    with contextlib.suppress(docker.errors.NotFound):
+        container.raw.remove(force=True, v=True)
+
+
+def run_once(client: docker.DockerClient, image: str, **kwargs: Any) -> str:
+    """Run a short-lived container and return its output.
+
+    `containers.run(remove=True)` keeps the image's anonymous volumes (VOLUME /data),
+    so run detached and remove the container together with its volumes.
+    """
+    container = client.containers.run(image, detach=True, **kwargs)
+    try:
+        result = container.wait()
+        output = container.logs(stdout=True, stderr=False).decode()
+        if result["StatusCode"] != 0:
+            raise CommandError(f"container exited {result['StatusCode']}:\n{output}")
+        return output
+    finally:
+        with contextlib.suppress(docker.errors.NotFound):
+            container.remove(force=True, v=True)
@@ -0,0 +1,17 @@
+# Optional features that MUST be present in the built image.
+#
+# One gate name per line. Known gates and the pull request that introduces
+# them (see tests/test_features.py):
+#
+#   stream       stream backups via RESTIC_STREAM            PR #109
+#   docker-cli   docker cli inside the container             PR #99
+#   bash         bash for pre/post hooks                     PR #99
+#   msmtp        mail relay via msmtp                        PR #84
+#
+# While a gate is NOT listed here its tests are reported as SKIP when the
+# feature is absent, so an open pull request does not turn master red.
+#
+# When you merge a feature, add its gate name below. From then on the feature
+# tests are mandatory and a feature that silently disappears from the image
+# fails the build.
+#
@@ -0,0 +1,146 @@
+"""Self test of the suite: prove that it goes red when the backup is broken.
+
+Run with `task test:sabotage`.
+
+A throwaway copy of the tree gets a backup.sh whose `restic backup` call is a no-op, the
+nastiest kind of regression (the script still reports success). The compat tests run
+against an image built from that copy. The check passes only if the assertions about the
+backup itself fail by name. Red for another reason (an image that cannot start, a
+non-executable /bin/backup) proves nothing and is reported as INVALID.
+
+Exit codes: 0 sabotage detected, 1 suite did not detect it, 2 check invalid or broken.
+Tracked files are never modified.
+"""
+
+from __future__ import annotations
+
+import os
+import shutil
+import signal
+import subprocess
+import sys
+import tempfile
+import uuid
+import xml.etree.ElementTree as ET
+from pathlib import Path
+
+from helpers import raise_interrupt
+
+ROOT = Path(__file__).resolve().parent.parent
+COPIED = ("Dockerfile", "backup.sh", "check.sh", "entry.sh")
+# Tests about the backup itself; each must fail on the sabotaged image.
+EXPECTED_RED = (
+    "test_one_snapshot_after_one_backup",
+    "test_second_backup_adds_snapshot",
+)
+
+
+STEP_FAILED = "StepFailed"  # name of helpers.StepFailed in the JUnit report
+
+
+class Invalid(Exception):
+    """The check itself is broken; the suite result means nothing."""
+
+
+def sabotage_tree(tmp: Path) -> None:
+    for name in COPIED:
+        shutil.copy2(ROOT / name, tmp / name)
+    script = tmp / "backup.sh"
+    lines = script.read_text().splitlines(keepends=True)
+    patched = [
+        ("true " + ln[len("restic backup ") :]) if ln.startswith("restic backup ") else ln
+        for ln in lines
+    ]
+    if patched == lines:
+        raise Invalid("sabotage patch did not apply, backup.sh changed shape")
+    script.write_text("".join(patched))
+    # Dockerfile COPY has no chmod: the image only works when the file is executable.
+    script.chmod(script.stat().st_mode | 0o111)
+    if not os.access(script, os.X_OK):
+        raise Invalid("sabotaged backup.sh is not executable")
+
+
+def red_tests(report: Path) -> tuple[set[str], set[str]]:
+    """Names of tests with an assertion failure, and of tests with an error.
+
+    Errors are setup errors and failed scenario steps (StepFailed): neither is an assertion
+    about the behaviour under test.
+    """
+    try:
+        cases = list(ET.parse(report).getroot().iter("testcase"))
+    except ET.ParseError as exc:
+        raise Invalid(f"pytest report is malformed or truncated: {exc}") from exc
+    errors = {c.get("name", "") for c in cases if c.find("error") is not None}
+    failures = set()
+    for case in cases:
+        failure = case.find("failure")
+        if failure is None:
+            continue
+        if STEP_FAILED in (failure.get("message") or "") or STEP_FAILED in (failure.text or ""):
+            errors.add(case.get("name", ""))
+        else:
+            failures.add(case.get("name", ""))
+    return failures, errors
+
+
+def main() -> int:
+    tag = f"rbd-sabotage:{uuid.uuid4().hex[:8]}"
+    tmp = Path(tempfile.mkdtemp(prefix="rbd-sabotage-"))
+    try:
+        sabotage_tree(tmp)
+        print(f">>> building {tag} from a sabotaged copy")
+        build = subprocess.run(["docker", "build", "--quiet", "-t", tag, str(tmp)])
+        if build.returncode != 0:
+            raise Invalid("image build failed")
+        print(">>> running the compat tests against the sabotaged image (failure expected)")
+        report = tmp / "report.xml"
+        proc = subprocess.run(
+            [
+                sys.executable,
+                "-m",
+                "pytest",
+                "tests/test_compat.py",
+                "-p",
+                "no:cacheprovider",
+                f"--junitxml={report}",
+            ],
+            cwd=ROOT,
+            env={**os.environ, "IMAGE": tag},
+            stdout=subprocess.PIPE,
+            stderr=subprocess.STDOUT,
+            text=True,
+        )
+        print(proc.stdout)
+        if proc.returncode == 0:
+            print(
+                "SABOTAGE CHECK FAILED: the compat tests passed on a broken image", file=sys.stderr
+            )
+            return 1
+        if "permission denied" in proc.stdout.lower():
+            raise Invalid("log contains 'permission denied', suite failed for the wrong reason")
+        if not report.is_file():
+            raise Invalid(f"pytest produced no report (exit code {proc.returncode})")
+        failures, errors = red_tests(report)
+        broken = [name for name in EXPECTED_RED if name in errors]
+        if broken:
+            raise Invalid(f"expected tests hit an error or failed step, not an assertion: {broken}")
+        missing = [name for name in EXPECTED_RED if name not in failures]
+        for name in missing:
+            print(
+                f"SABOTAGE CHECK FAILED: expected assertion did not fail: {name}", file=sys.stderr
+            )
+        if missing:
+            return 1
+        print("\nsabotage check ok: the compat tests detect a broken backup")
+        return 0
+    except Invalid as exc:
+        print(f"SABOTAGE CHECK INVALID: {exc}", file=sys.stderr)
+        return 2
+    finally:
+        shutil.rmtree(tmp, ignore_errors=True)
+        subprocess.run(["docker", "rmi", "-f", tag], capture_output=True)
+
+
+if __name__ == "__main__":
+    signal.signal(signal.SIGTERM, raise_interrupt)
+    sys.exit(main())
@@ -0,0 +1,77 @@
+"""Run container-structure-test without leaking Docker volumes.
+
+container-structure-test starts one container per test and never removes it. The image
+declares `VOLUME /data`, so every container drags an anonymous volume along that outlives
+`docker rm`. With `--save` the containers are kept, and this wrapper removes the ones of
+the image under test that the run created, together with their volumes (`remove(v=True)`);
+containers of other images or of the compat suite (a parallel run, a compose stack) are left
+alone.
+Run with `task test:structure`.
+
+A run longer than CST_TIMEOUT seconds (default 300) is killed: container-structure-test has
+no timeout of its own and can block forever waiting for a container.
+
+Exit code: the one of container-structure-test; 124 on timeout.
+"""
+
+from __future__ import annotations
+
+import contextlib
+import os
+import signal
+import subprocess
+import sys
+from pathlib import Path
+
+import docker
+from helpers import raise_interrupt
+
+CONFIG = Path(__file__).with_name("structure.yml")
+TIMEOUT_EXIT = 124
+
+
+def _containers_of(client: docker.DockerClient, image_id: str) -> list:
+    """Containers of the image that container-structure-test could have created.
+
+    Containers of the compat suite carry the `rbd-test-run` label; their owner removes them.
+    """
+    found = client.containers.list(all=True, filters={"ancestor": image_id})
+    return [c for c in found if "rbd-test-run" not in c.labels]
+
+
+def main() -> int:
+    image = os.environ.get("IMAGE", "restic-backup-docker:test")
+    signal.signal(signal.SIGTERM, raise_interrupt)
+    client = docker.from_env()
+    image_id = client.images.get(image).id
+    before = {c.id for c in _containers_of(client, image_id)}
+    try:
+        timeout = float(os.environ.get("CST_TIMEOUT", "300"))
+        try:
+            return subprocess.run(
+                [
+                    "container-structure-test",
+                    "test",
+                    "--save",
+                    "--image",
+                    image,
+                    "--config",
+                    os.environ.get("CST_CONFIG", str(CONFIG)),
+                ],
+                check=False,
+                timeout=timeout,
+            ).returncode
+        except subprocess.TimeoutExpired:
+            print(f"container-structure-test timed out after {timeout:g} s", file=sys.stderr)
+            return TIMEOUT_EXIT
+    finally:
+        signal.signal(signal.SIGINT, signal.SIG_IGN)
+        for container in _containers_of(client, image_id):
+            if container.id not in before:
+                with contextlib.suppress(docker.errors.NotFound):
+                    container.remove(v=True, force=True)
+        signal.signal(signal.SIGINT, signal.default_int_handler)
+
+
+if __name__ == "__main__":
+    sys.exit(main())
@@ -0,0 +1,64 @@
+schemaVersion: '2.0.0'
+
+# Structural contract of the image. Everything asserted here is something a
+# user can reach from outside: a binary they call from a hook, a path they
+# mount, or the way the container is started.
+
+commandTests:
+  - name: 'restic is installed'
+    command: 'restic'
+    args: ['version']
+    expectedOutput: ['restic \d+\.\d+\.\d+.*']
+  - name: 'rclone is installed'
+    command: 'rclone'
+    args: ['--version']
+    expectedOutput: ['rclone v.*']
+  - name: 'curl is installed, used for Teams notifications'
+    command: 'curl'
+    args: ['--version']
+    expectedOutput: ['curl \d+\..*']
+  - name: 'mail is installed, used for mail notifications'
+    command: 'sh'
+    args: ['-c', 'command -v mail']
+    expectedOutput: ['/usr/bin/mail']
+  - name: 'crond is available'
+    command: 'sh'
+    args: ['-c', 'command -v crond']
+    expectedOutput: ['/usr/sbin/crond']
+  - name: 'flock is available, the cron jobs depend on it'
+    command: 'sh'
+    args: ['-c', 'test -x /usr/bin/flock && echo ok']
+    expectedOutput: ['ok']
+
+fileExistenceTests:
+  - name: 'backup script'
+    path: '/bin/backup'
+    shouldExist: true
+    permissions: '-rwxr-xr-x'
+  - name: 'check script'
+    path: '/bin/check'
+    shouldExist: true
+    permissions: '-rwxr-xr-x'
+  - name: 'entrypoint'
+    path: '/entry.sh'
+    shouldExist: true
+    permissions: '-rwxr-xr-x'
+  - name: 'default repository mount point'
+    path: '/mnt/restic'
+    shouldExist: true
+  - name: 'cron log file'
+    path: '/var/log/cron.log'
+    shouldExist: true
+  - name: 'crontab directory'
+    path: '/var/spool/cron/crontabs'
+    shouldExist: true
+
+metadataTest:
+  volumes: ['/data']
+  entrypoint: ['/entry.sh']
+  cmd: ['tail', '-fn0', '/var/log/cron.log']
+  envVars:
+    - key: 'RESTIC_REPOSITORY'
+      value: '/mnt/restic'
+    - key: 'BACKUP_CRON'
+      value: '0 */6 * * *'
@@ -0,0 +1,469 @@
+"""Backwards-compatibility suite.
+
+Everything here describes behaviour the image has today and that users depend on.
+A change that turns one of these red is a breaking change, whether or not intended.
+
+Layout: every scenario fixture starts one throwaway container, performs the actions
+once and records the results; the tests only assert on what was recorded, so they do
+not depend on each other. A scenario step that fails fails only the tests that assert
+on its result. Each docstring states the behaviour the test pins.
+Tests run in a container with the image defaults: repository /mnt/restic, tag
+`testtag`, cron far in the future.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Iterator
+from dataclasses import dataclass
+
+import docker
+import pytest
+from conftest import LABELS
+from helpers import Container, Scenario, run_once, running
+
+HELLO = "backup me"
+PRE_BACKUP_HOOK = '#!/bin/sh\necho "PRE-HOOK-RAN"\n'
+POST_BACKUP_HOOK = '#!/bin/sh\necho "POST-HOOK-RAN with rc=$1"\n'
+PRE_CHECK_HOOK = '#!/bin/sh\necho "PRE-CHECK-HOOK-RAN"\n'
+POST_CHECK_HOOK = '#!/bin/sh\necho "POST-CHECK-HOOK-RAN with rc=$1"\n'
+
+
+@pytest.fixture(scope="class")
+def started(request: pytest.FixtureRequest, run_container) -> Iterator[Container]:
+    """Container for the class, configured by `@pytest.mark.container(suffix, **env)`."""
+    marker = request.node.get_closest_marker("container")
+    with run_container(marker.args[0], env=marker.kwargs) as container:
+        yield container
+
+
+def _crontab_lines(container: Container) -> list[str]:
+    return container.read_file("/var/spool/cron/crontabs/root").splitlines()
+
+
+# ---------------------------------------------------------------------------
+# repository bootstrap and default backup
+# ---------------------------------------------------------------------------
+
+
+@pytest.fixture(scope="class")
+def default_run(started: Container) -> Scenario:
+    c = running(started)
+    c.write_file("/data/hello.txt", HELLO)
+    run = Scenario()
+    run.record("backup", lambda: c.exec("/bin/backup"))
+    run.record("snapshots_after_first", c.snapshots)
+    run.record("restore", lambda: c.restore_latest("/tmp/restore"))
+    run.record("restored", lambda: c.exec("cat", "/tmp/restore/data/hello.txt"))
+    run.record("backup_last_log", lambda: c.read_file("/var/log/backup-last.log"))
+    run.record("second_backup", lambda: c.run("/bin/backup"))
+    run.record("snapshots_after_second", c.snapshots)
+    run.record("check", lambda: c.exec("/bin/check"))
+    c.write_file("/hooks/pre-backup.sh", PRE_BACKUP_HOOK, executable=True)
+    c.write_file("/hooks/post-backup.sh", POST_BACKUP_HOOK, executable=True)
+    run.record("hook_backup", lambda: c.exec("/bin/backup"))
+    return run
+
+
+@pytest.mark.container("default")
+class TestDefaultBackup:
+    def test_container_starts(self, started):
+        """container starts with default configuration"""
+        assert started.started, started.start_log
+
+    def test_repository_is_initialised(self, started):
+        """empty repository is initialised on first start"""
+        assert "Running restic init" in started.logs()
+
+    def test_repository_init_succeeds(self, started):
+        """repository initialisation succeeds"""
+        assert "Repo init status 0" in started.logs()
+
+    def test_backup_cron_in_crontab(self, started):
+        """BACKUP_CRON is written to the crontab"""
+        lines = _crontab_lines(started)
+        assert any(line.startswith("0 0 31 2 *") and "/bin/backup" in line for line in lines)
+
+    def test_backup_cron_guarded_by_flock(self, started):
+        """backup cron job is guarded by flock"""
+        line = next(line for line in _crontab_lines(started) if "/bin/backup" in line)
+        assert "flock -n /var/run/backup.lock" in line
+
+    def test_no_check_cron_by_default(self, started):
+        """no check cron job is installed while CHECK_CRON is unset"""
+        lines = _crontab_lines(started)
+        assert any("/bin/backup" in line for line in lines)  # the crontab we looked at is real
+        assert not any("/bin/check" in line for line in lines)
+
+    def test_backup_succeeds(self, default_run):
+        """backup of /data succeeds"""
+        assert default_run.backup.rc == 0
+        assert "Backup Successful" in default_run.backup.out
+
+    def test_missing_pre_backup_hook_reported(self, default_run):
+        """missing pre-backup hook is reported, not fatal"""
+        assert "Pre-backup script not found" in default_run.backup.out
+        assert default_run.backup.rc == 0
+
+    def test_missing_post_backup_hook_reported(self, default_run):
+        """missing post-backup hook is reported, not fatal"""
+        assert "Post-backup script not found" in default_run.backup.out
+        assert default_run.backup.rc == 0
+
+    def test_no_notification_without_config(self, default_run):
+        """no mail or Teams notification is attempted without configuration"""
+        assert "Backup Successful" in default_run.backup.out
+        assert "notification" not in default_run.backup.out
+
+    def test_one_snapshot_after_one_backup(self, default_run):
+        """exactly one snapshot exists after one backup"""
+        assert len(default_run.snapshots_after_first) == 1
+
+    def test_snapshot_carries_restic_tag(self, default_run):
+        """snapshot carries the configured RESTIC_TAG"""
+        assert [s["tags"] for s in default_run.snapshots_after_first] == [["testtag"]]
+
+    def test_restore_exits_zero(self, default_run):
+        """restic restore of the latest snapshot exits 0"""
+        assert default_run.restore.rc == 0, default_run.restore.out
+
+    def test_restored_content_matches(self, default_run):
+        """restored file content matches the original"""
+        assert default_run.restored.rc == 0, default_run.restored.out
+        assert default_run.restored.out == HELLO
+
+    def test_backup_last_log_records_parameters(self, default_run):
+        """backup-last.log records the run parameters"""
+        assert "RESTIC_REPOSITORY: /mnt/restic" in default_run.backup_last_log
+
+    def test_second_backup_adds_snapshot(self, default_run):
+        """a second backup adds a second snapshot"""
+        ids = {s["id"] for s in default_run.snapshots_after_second}
+        assert len(default_run.snapshots_after_second) == 2
+        assert {s["id"] for s in default_run.snapshots_after_first} < ids
+
+    def test_check_passes_on_healthy_repository(self, default_run):
+        """restic check passes on a healthy repository"""
+        assert default_run.check.rc == 0
+        assert "Check Successful" in default_run.check.out
+
+    def test_missing_pre_check_hook_reported(self, default_run):
+        """missing pre-check hook is reported, not fatal"""
+        assert "Pre-check script not found" in default_run.check.out
+        assert "Check Successful" in default_run.check.out
+
+    def test_pre_backup_hook_executed(self, default_run):
+        """pre-backup hook is executed"""
+        assert "PRE-HOOK-RAN" in default_run.hook_backup.out
+
+    def test_post_backup_hook_gets_return_code(self, default_run):
+        """post-backup hook receives the backup return code"""
+        assert "POST-HOOK-RAN with rc=0" in default_run.hook_backup.out
+
+
+# ---------------------------------------------------------------------------
+# CHECK_CRON and check hooks
+# ---------------------------------------------------------------------------
+
+
+@pytest.fixture(scope="class")
+def checkcron_run(started: Container) -> Scenario:
+    c = running(started)
+    run = Scenario()
+    run.record("crontab", lambda: _crontab_lines(c))
+    c.write_file("/data/hello.txt", HELLO)
+    run.record("backup", lambda: c.run("/bin/backup"))
+    c.write_file("/hooks/pre-check.sh", PRE_CHECK_HOOK, executable=True)
+    c.write_file("/hooks/post-check.sh", POST_CHECK_HOOK, executable=True)
+    run.record("check", lambda: c.exec("/bin/check"))
+    return run
+
+
+@pytest.mark.container("checkcron", CHECK_CRON="0 0 31 2 *")
+class TestCheckCron:
+    def test_check_cron_guarded_by_shared_lock(self, checkcron_run):
+        """check cron job is guarded by the shared backup lock"""
+        check_lines = [line for line in checkcron_run.crontab if "/bin/check" in line]
+        assert len(check_lines) == 1
+        assert "flock -n /var/run/backup.lock" in check_lines[0]
+
+    def test_check_cron_adds_cron_job(self, checkcron_run):
+        """CHECK_CRON adds a check cron job to the crontab"""
+        assert any(
+            line.startswith("0 0 31 2 *") and "/bin/check" in line for line in checkcron_run.crontab
+        )
+
+    def test_crontab_has_backup_and_check_line(self, checkcron_run):
+        """crontab has a backup line and a check line"""
+        assert len([line for line in checkcron_run.crontab if line.strip()]) == 2
+
+    def test_check_cron_keeps_backup_cron(self, checkcron_run):
+        """CHECK_CRON does not replace the backup cron job"""
+        assert "/bin/backup" in checkcron_run.crontab[0]
+
+    def test_pre_check_hook_executed(self, checkcron_run):
+        """pre-check hook is executed"""
+        assert "PRE-CHECK-HOOK-RAN" in checkcron_run.check.out
+
+    def test_post_check_hook_gets_return_code(self, checkcron_run):
+        """post-check hook receives the check return code"""
+        assert "POST-CHECK-HOOK-RAN with rc=0" in checkcron_run.check.out
+
+
+# ---------------------------------------------------------------------------
+# a corrupted repository is reported as failed
+#
+# check.sh prints "Check Successful" whenever restic exits 0, so a `restic check`
+# that does nothing would keep the success path green. Damage the repository and
+# require the failure path. The container is throwaway; the repo stays broken.
+# ---------------------------------------------------------------------------
+
+CORRUPT_CMD = 'f=$(find /mnt/restic/data -type f | head -n1); chmod u+w "$f"; printf garbage > "$f"'
+
+
+@pytest.fixture(scope="class")
+def corrupt_run(started: Container) -> Scenario:
+    c = running(started)
+    c.write_file("/data/hello.txt", HELLO)
+    run = Scenario()
+    run.record("backup", lambda: c.run("/bin/backup"))
+    run.record("snapshots_before", c.snapshots)
+    run.record("corrupt", lambda: c.run("sh", "-c", CORRUPT_CMD))
+    run.record("check", lambda: c.exec("/bin/check"))
+    run.record("error_log_exists", lambda: c.file_exists("/var/log/check-error-last.log"))
+    return run
+
+
+@pytest.mark.container("corrupt")
+class TestCorruptedRepository:
+    def test_repository_to_corrupt_has_snapshot(self, corrupt_run):
+        """the repository to be corrupted has a snapshot"""
+        assert len(corrupt_run.snapshots_before) == 1
+
+    def test_check_detects_corruption(self, corrupt_run):
+        """restic check detects a corrupted repository"""
+        assert "Check Failed with Status" in corrupt_run.check.out
+
+    def test_corruption_not_reported_successful(self, corrupt_run):
+        """a corrupted repository is not reported as successful"""
+        assert "Check Successful" not in corrupt_run.check.out
+        assert "Check Failed with Status" in corrupt_run.check.out  # the check did run and report
+
+    def test_failing_check_writes_error_log(self, corrupt_run):
+        """failing check writes check-error-last.log"""
+        assert corrupt_run.error_log_exists
+
+
+# ---------------------------------------------------------------------------
+# RESTIC_FORGET_ARGS
+# ---------------------------------------------------------------------------
+
+
+@pytest.fixture(scope="class")
+def forget_run(started: Container) -> Scenario:
+    c = running(started)
+    c.write_file("/data/hello.txt", HELLO)
+    run = Scenario()
+    run.record("first_backup", lambda: c.run("/bin/backup"))
+    run.record("first", c.snapshots)
+    run.record("second_backup", lambda: c.exec("/bin/backup"))
+    run.record("snapshots", c.snapshots)
+    return run
+
+
+@pytest.mark.container("forget", RESTIC_FORGET_ARGS="--keep-last 1")
+class TestForgetArgs:
+    def test_forget_runs_after_backup(self, forget_run):
+        """forget runs after a successful backup when configured"""
+        assert "Forget Successful" in forget_run.second_backup.out
+
+    def test_keep_last_leaves_one_snapshot(self, forget_run):
+        """--keep-last 1 leaves exactly one snapshot"""
+        # two backups ran: one snapshot left that is not the first one proves forget pruned
+        assert len(forget_run.first) == 1
+        assert len(forget_run.snapshots) == 1
+        assert forget_run.snapshots[0]["id"] != forget_run.first[0]["id"]
+
+
+# ---------------------------------------------------------------------------
+# RESTIC_JOB_ARGS
+#
+# backup-last.log only records the variable, it does not prove restic ever saw it.
+# The arguments need a visible effect, otherwise a backup that silently drops
+# RESTIC_JOB_ARGS still looks healthy.
+# ---------------------------------------------------------------------------
+
+
+@pytest.fixture(scope="class")
+def jobargs_run(started: Container) -> Scenario:
+    c = running(started)
+    c.write_file("/data/keep.txt", "keep")
+    c.write_file("/data/skipme.txt", "skip")
+    run = Scenario()
+    run.record("backup", lambda: c.exec("/bin/backup"))
+    run.record("restore", lambda: c.restore_latest("/tmp/restore"))
+    run.record("restored_keep", lambda: c.exec("cat", "/tmp/restore/data/keep.txt"))
+    run.record("paths", c.backed_up_paths)
+    return run
+
+
+@pytest.mark.container("jobargs", RESTIC_JOB_ARGS="--exclude=/data/skipme.txt")
+class TestJobArgs:
+    def test_backup_with_job_args_succeeds(self, jobargs_run):
+        """backup with RESTIC_JOB_ARGS succeeds"""
+        assert "Backup Successful" in jobargs_run.backup.out
+
+    def test_job_args_do_not_break_backup(self, jobargs_run):
+        """RESTIC_JOB_ARGS does not break a normal backup"""
+        assert jobargs_run.restore.rc == 0, jobargs_run.restore.out
+        assert jobargs_run.restored_keep.out == "keep"
+
+    def test_job_args_passed_to_restic(self, jobargs_run):
+        """RESTIC_JOB_ARGS is passed to restic backup"""
+        # keep.txt shows the snapshot is the one of this scenario, skipme.txt must be missing
+        assert "/data/keep.txt" in jobargs_run.paths
+        assert "/data/skipme.txt" not in jobargs_run.paths
+
+
+# ---------------------------------------------------------------------------
+# BACKUP_SOURCES
+# ---------------------------------------------------------------------------
+
+
+@pytest.fixture(scope="class")
+def sources_run(started: Container) -> Scenario:
+    c = running(started)
+    c.write_file("/opt/custom/custom.txt", "custom source")
+    c.write_file("/data/ignored.txt", "should not be backed up")
+    run = Scenario()
+    run.record("backup", lambda: c.run("/bin/backup"))
+    run.record("restore", lambda: c.restore_latest("/tmp/restore"))
+    run.record("restored", lambda: c.exec("cat", "/tmp/restore/opt/custom/custom.txt"))
+    run.record("paths", c.backed_up_paths)
+    return run
+
+
+@pytest.mark.container("sources", BACKUP_SOURCES="/opt/custom")
+class TestBackupSources:
+    def test_sources_override_default(self, sources_run):
+        """BACKUP_SOURCES overrides the default /data source"""
+        assert sources_run.restore.rc == 0, sources_run.restore.out
+        assert sources_run.restored.out == "custom source"
+
+    def test_sources_exclude_data(self, sources_run):
+        """BACKUP_SOURCES excludes /data"""
+        assert "/opt/custom/custom.txt" in sources_run.paths
+        assert not [p for p in sources_run.paths if p.startswith("/data")]
+
+
+# ---------------------------------------------------------------------------
+# failure handling
+# ---------------------------------------------------------------------------
+
+
+@pytest.fixture(scope="class")
+def failure_run(started: Container) -> Scenario:
+    c = running(started)
+    c.write_file("/data/hello.txt", HELLO)
+    # Back up a source that does not exist. The repository stays the initialised
+    # default one, so the snapshot list below is of the repo the run really used.
+    run = Scenario()
+    run.record("backup", lambda: c.exec("/bin/backup", env={"BACKUP_SOURCES": "/does-not-exist"}))
+    run.record("error_log_exists", lambda: c.file_exists("/var/log/backup-error-last.log"))
+    run.record("snapshots", c.snapshots)
+    return run
+
+
+@pytest.mark.container("failure")
+class TestFailureHandling:
+    def test_failing_backup_reported(self, failure_run):
+        """a failing backup is reported as failed"""
+        assert "Backup Failed with Status" in failure_run.backup.out
+
+    def test_failing_backup_writes_error_log(self, failure_run):
+        """failing backup writes backup-error-last.log"""
+        assert failure_run.error_log_exists
+
+    def test_failed_backup_leaves_no_snapshot(self, failure_run):
+        """a failed backup leaves no snapshot behind"""
+        # snapshots() raises on a restic error, so [] means a readable, empty repository
+        assert "Backup Failed with Status" in failure_run.backup.out
+        assert failure_run.snapshots == []
+
+
+# ---------------------------------------------------------------------------
+# timezone
+#
+# The base image already ships tzdata, so TZ works today. Pinned here so that
+# neither a base image change nor a slimming effort silently breaks cron times
+# for users who already rely on it.
+# ---------------------------------------------------------------------------
+
+
+@dataclass
+class TimezoneRunner:
+    client: docker.DockerClient
+    image: str
+
+    def zone(self, env: dict[str, str] | None = None) -> str:
+        out = run_once(
+            self.client, self.image, entrypoint=["date", "+%Z"], environment=env, labels=LABELS
+        )
+        return out.strip()
+
+
+@pytest.fixture(scope="module")
+def timezone(docker_client, image) -> TimezoneRunner:
+    return TimezoneRunner(docker_client, image)
+
+
+class TestTimezone:
+    def test_default_timezone_is_utc(self, timezone):
+        """the default timezone is UTC"""
+        assert timezone.zone() == "UTC"
+
+    def test_tz_variable_changes_timezone(self, timezone):
+        """TZ=Europe/Berlin changes the container timezone"""
+        assert timezone.zone({"TZ": "Europe/Berlin"}) in ("CET", "CEST")
+
+
+# ---------------------------------------------------------------------------
+# RESTIC_TAG handling
+#
+# The Dockerfile declares RESTIC_TAG="", so in a normal container the variable is
+# always set and an empty value means "no tag". The `${RESTIC_TAG?...}` guard in
+# backup.sh only triggers when the variable is truly unset.
+# ---------------------------------------------------------------------------
+
+
+@pytest.fixture(scope="class")
+def tag_run(started: Container) -> Scenario:
+    c = running(started)
+    c.write_file("/data/hello.txt", HELLO)
+    run = Scenario()
+    run.record("empty", lambda: c.exec("/bin/backup", env={"RESTIC_TAG": ""}))
+    run.record("snapshots_after_empty", c.snapshots)
+    run.record("unset", lambda: c.exec("env", "-u", "RESTIC_TAG", "/bin/backup"))
+    run.record("snapshots_after_unset", c.snapshots)
+    return run
+
+
+@pytest.mark.container("tag")
+class TestResticTag:
+    def test_empty_tag_creates_untagged_backup(self, tag_run):
+        """an empty RESTIC_TAG creates an untagged backup"""
+        assert "Backup Successful" in tag_run.empty.out
+        [snapshot] = tag_run.snapshots_after_empty
+        assert not snapshot.get("tags")
+
+    def test_empty_tag_still_produces_snapshot(self, tag_run):
+        """an empty RESTIC_TAG still produces a snapshot"""
+        assert len(tag_run.snapshots_after_empty) == 1
+
+    def test_unset_tag_refuses_backup(self, tag_run):
+        """backup refuses to run when RESTIC_TAG is unset entirely"""
+        assert "Missing environment variable RESTIC_TAG" in tag_run.unset.out
+
+    def test_unset_tag_creates_no_snapshot(self, tag_run):
+        """no extra snapshot is created when RESTIC_TAG is unset"""
+        assert len(tag_run.snapshots_after_empty) == 1  # there was something to not add to
+        assert len(tag_run.snapshots_after_unset) == 1
@@ -0,0 +1,213 @@
+"""Feature-gated tests.
+
+Each class is guarded by `@pytest.mark.feature(<gate>)`: it runs as soon as the built
+image ships the feature (detected in conftest.py) and is skipped before that. A gate
+listed in tests/required-features.txt turns the skip into a failure. A scenario step
+that fails fails only the tests that assert on its result.
+
+Scenario fixtures request `feature_gate_class` first, so a gated class never starts a
+container on an image without the feature.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Iterator
+
+import pytest
+from conftest import LABELS
+from helpers import Container, Scenario, run_once, running
+
+STREAM_ENV = {
+    "RESTIC_STREAM": "1",
+    "RESTIC_STREAM_CMD": "echo hello-stream",
+    "RESTIC_STREAM_FILENAME": "stream.txt",
+}
+BASH_HOOK = """#!/bin/bash
+arr=(one two)
+if [[ "${arr[1]}" == "two" ]]; then echo "BASH-HOOK-RAN"; fi
+"""
+MSMTP_CONFIG = "defaults\naccount default\nhost localhost\n"
+
+
+# ---------------------------------------------------------------------------
+# stream backup (PR #109)
+# ---------------------------------------------------------------------------
+
+
+@pytest.fixture(scope="class")
+def stream_run(feature_gate_class, run_container) -> Iterator[Scenario]:
+    with run_container("stream", env=STREAM_ENV) as c:
+        running(c)
+        run = Scenario()
+        run.record("backup", lambda: c.exec("/bin/backup"))
+        run.record("snapshots", c.snapshots)
+        run.record("restore", lambda: c.restore_latest("/tmp/restore"))
+        run.record("restored", lambda: c.exec("cat", "/tmp/restore/stream.txt"))
+        yield run
+
+
+@pytest.fixture(scope="class")
+def stream_jobargs_run(feature_gate_class, run_container) -> Iterator[Scenario]:
+    # --host is used because it is visible in the snapshot metadata of a stdin backup.
+    with run_container(
+        "stream-jobargs", env={**STREAM_ENV, "RESTIC_JOB_ARGS": "--host=jobargshost"}
+    ) as c:
+        running(c)
+        run = Scenario()
+        run.record("backup", lambda: c.run("/bin/backup"))
+        # restic's own --host filter only returns snapshots whose hostname equals the
+        # value; both directions are checked so the filter is known to discriminate.
+        run.record("matching", lambda: c.restic_json("snapshots", "--host", "jobargshost") or [])
+        run.record(
+            "unrelated", lambda: c.restic_json("snapshots", "--host", "some-other-host") or []
+        )
+        yield run
+
+
+@pytest.fixture(scope="class")
+def stream_misconfigured_run(feature_gate_class, run_container) -> Iterator[Scenario]:
+    with run_container("stream-misconfigured", env={"RESTIC_STREAM": "1"}) as c:
+        running(c)
+        run = Scenario()
+        run.record("backup", lambda: c.exec("/bin/backup"))
+        run.record("snapshots", c.snapshots)
+        yield run
+
+
+@pytest.fixture(scope="class")
+def stream_disabled_run(feature_gate_class, run_container) -> Iterator[Scenario]:
+    with run_container("stream-disabled") as c:
+        running(c)
+        c.write_file("/data/hello.txt", "backup me")
+        run = Scenario()
+        run.record("backup", lambda: c.run("/bin/backup"))
+        run.record("snapshots", c.snapshots)
+        yield run
+
+
+@pytest.mark.feature("stream")
+class TestStream:
+    def test_stream_backup_succeeds(self, stream_run):
+        """stream backup succeeds"""
+        assert "Backup Successful" in stream_run.backup.out
+
+    def test_stream_backup_creates_one_snapshot(self, stream_run):
+        """stream backup creates exactly one snapshot"""
+        assert len(stream_run.snapshots) == 1
+
+    def test_stream_content_stored_under_filename(self, stream_run):
+        """stream content is stored under RESTIC_STREAM_FILENAME"""
+        assert stream_run.restore.rc == 0, stream_run.restore.out
+        assert stream_run.restored.rc == 0, stream_run.restored.out
+        assert stream_run.restored.out == "hello-stream\n"
+
+    def test_job_args_set_host_of_stream_snapshot(self, stream_jobargs_run):
+        """RESTIC_JOB_ARGS sets the host of the stream backup snapshot"""
+        assert len(stream_jobargs_run.matching) == 1
+
+    def test_host_filter_does_not_match_unrelated_host(self, stream_jobargs_run):
+        """the host filter does not match an unrelated hostname"""
+        assert stream_jobargs_run.unrelated == []
+
+    def test_incomplete_stream_config_does_not_report_success(self, stream_misconfigured_run):
+        """incomplete stream configuration does not report success"""
+        assert "Backup Successful" not in stream_misconfigured_run.backup.out
+
+    def test_incomplete_stream_config_creates_no_snapshot(self, stream_misconfigured_run):
+        """incomplete stream configuration creates no snapshot"""
+        assert stream_misconfigured_run.snapshots == []
+
+    def test_file_backup_still_works_with_stream_feature(self, stream_disabled_run):
+        """file backup still works while the stream feature is present"""
+        assert len(stream_disabled_run.snapshots) == 1
+
+
+# ---------------------------------------------------------------------------
+# docker cli (PR #99)
+# ---------------------------------------------------------------------------
+
+
+@pytest.mark.feature("docker-cli")
+def test_docker_cli_usable(docker_client, image):
+    """docker cli is usable inside the image"""
+    out = run_once(
+        docker_client, image, entrypoint=["docker"], command=["--version"], labels=LABELS
+    )
+    assert "Docker version" in out
+
+
+# ---------------------------------------------------------------------------
+# bash in hooks (PR #99)
+# ---------------------------------------------------------------------------
+
+
+@pytest.fixture(scope="class")
+def bash_hook_run(feature_gate_class, run_container) -> Iterator[Scenario]:
+    with run_container("bash-hooks") as c:
+        running(c)
+        c.write_file("/hooks/pre-backup.sh", BASH_HOOK, executable=True)
+        c.write_file("/data/hello.txt", "backup me")
+        run = Scenario()
+        run.record("backup", lambda: c.exec("/bin/backup"))
+        yield run
+
+
+@pytest.mark.feature("bash")
+class TestBashHooks:
+    def test_bash_hook_runs(self, bash_hook_run):
+        """a hook with a bash shebang and bash syntax runs"""
+        assert "BASH-HOOK-RAN" in bash_hook_run.backup.out
+
+
+# ---------------------------------------------------------------------------
+# msmtp mail relay (PR #84)
+# ---------------------------------------------------------------------------
+
+
+@pytest.fixture(scope="class")
+def msmtp_config_run(feature_gate_class, run_container) -> Iterator[Container]:
+    # entry.sh copies the config at startup, so it must exist before the container starts.
+    with run_container("msmtp-config", files={"/config/msmtprc": MSMTP_CONFIG}) as c:
+        yield running(c)
+
+
+@pytest.fixture(scope="class")
+def msmtp_missing_config_run(feature_gate_class, run_container) -> Iterator[Scenario]:
+    # Users who configured mail before msmtp existed must not lose their backups.
+    env = {"MAILX_ARGS": "-r from@example.com -s subject to@example.com"}
+    with run_container("msmtp-missing-config", env=env) as c:
+        running(c)
+        c.write_file("/data/hello.txt", "backup me")
+        run = Scenario()
+        run.record("backup", lambda: c.exec("/bin/backup"))
+        run.record("snapshots", c.snapshots)
+        yield run
+
+
+@pytest.mark.feature("msmtp")
+class TestMsmtp:
+    def test_sendmail_is_msmtp(self, docker_client, image):
+        """sendmail is provided by msmtp"""
+        target = run_once(
+            docker_client,
+            image,
+            entrypoint=["sh", "-c", "readlink -f /usr/sbin/sendmail"],
+            labels=LABELS,
+        )
+        assert "msmtp" in target
+
+    def test_config_copied_to_etc(self, msmtp_config_run):
+        """msmtp config from /config is copied to /etc/msmtprc"""
+        assert msmtp_config_run.file_exists("/etc/msmtprc")
+
+    def test_copied_config_not_world_readable(self, msmtp_config_run):
+        """copied msmtp config is not world readable"""
+        assert msmtp_config_run.run("stat", "-c", "%a", "/etc/msmtprc").strip() == "600"
+
+    def test_backup_succeeds_without_msmtp_config(self, msmtp_missing_config_run):
+        """backup succeeds with MAILX_ARGS set but no msmtp config"""
+        assert "Backup Successful" in msmtp_missing_config_run.backup.out
+
+    def test_snapshot_created_without_msmtp_config(self, msmtp_missing_config_run):
+        """snapshot is created even when the mail relay is unconfigured"""
+        assert len(msmtp_missing_config_run.snapshots) == 1