test: move container test suites to pytest (#112)

* test: scaffold pytest suite with mise, uv and Task

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* ci: run the pytest suite via mise and task

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* fix: make test cleanup survive repeated SIGINT

A group Ctrl-C reaches pytest a second time via task and uv while the
session fixture tears down, aborting the volume cleanup. Ignore SIGINT
and SIGTERM during cleanup, remove volumes in finally, and tolerate
already-removed objects.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* fix: pin Python 3.13 for uv without mise

Add .python-version (read by mise and uv) and cap requires-python at
<3.14 so bare uv uses 3.13 or fails instead of picking 3.14.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* fix: clean up test containers on SIGTERM

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* test: port compat tests to pytest

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* test: assert the success message first in the corruption test

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* test: assert the empty-tag snapshot carries no tags

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* test: remove anonymous volumes with short-lived test containers

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* test: port feature-gated tests to pytest

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* test: port sabotage self-test to task test:sabotage

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* fix: create parent directories in put_file

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* test: move structure.yml to tests and remove cst volume leak

container-structure-test left one anonymous volume per test container.
tests/structure.py runs it with --save and removes the new containers
with their volumes afterwards.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* test: remove the Bash suite

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* ci: run only the Python suite in the test job

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* docs: describe the Python test suite

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* test: isolate scenario steps, share helpers, use restore_latest, HEAD dir probe

Scenario fixtures record each step; a failing step fails only the tests
that assert on it. running() and raise_interrupt() live in helpers.py.
put_file probes parent directories with a HEAD request instead of a
get_archive stream. stream_run uses restore_latest and asserts its result.
Docstrings no longer refer to the removed Bash suite.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* test: sabotage check rejects errors and malformed reports

Only <failure> of the expected tests counts as red; errors in them and a
malformed JUnit report exit 2 (invalid) instead of passing or exiting 1.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* test: structure wrapper removes only its own containers

Restrict cleanup to containers of the image under test without the
rbd-test-run label and ignore NotFound.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* build: run all test suites and aggregate the exit code

task test no longer stops at the first failing suite.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* fix: sabotage check treats failed scenario steps as INVALID

StepFailed is no AssertionError any more; sabotage.py counts a JUnit failure
named StepFailed as error, so an expected-red test only counts when its own
assertion fails.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* fix: hard timeout for container-structure-test

CST_TIMEOUT (default 300 s) kills a hanging run; existing cleanup removes the
containers; exit code 124 with a clear message.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* fix: shorten sabotage message for lint

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* fix: handle JUnit failures without message in sabotage check

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Tobias KaupatandClaude Sonnet 5.5 authored and GitHub committed 2026-10-20 20:47:03 +02:00
1 parent 31f06b5d9b
commit be7d426f78
21 files changed
+1692 -958

No files matched your search

+27 -25
View File
@@ -28,43 +28,45 @@ When behaviour of the image changes (Features, Bugfixes, Changes in the API) ple
## Tests
The test suites build the image and exercise it as a real container. They need
`bash` and `docker`, nothing else:
The tests build the image and exercise it as a real container. Tools are pinned
in `mise.toml` (Python, [uv](https://docs.astral.sh/uv/),
[Task](https://taskfile.dev/), container-structure-test), Python dependencies in
`uv.lock`. You need [mise](https://mise.jdx.dev/) and Docker, nothing else:
```sh
bash test/run.sh # build the image and run every suite
bash test/run.sh --no-build # reuse an already built image
bash test/run.sh --suite compat # run a single suite
mise install # install the pinned tools
task test # build the image and run every suite
task test:sabotage # self test, see below
task lint # ruff check and format check of tests/
```
They run unchanged on Linux, macOS and Windows/Git-Bash. On Git-Bash the
scripts set `MSYS_NO_PATHCONV=1` and `MSYS2_ARG_CONV_EXCL='*'` themselves,
because MSYS would otherwise rewrite container paths such as
`/usr/bin/backup` into `C:/Program Files/Git/usr/bin/backup` before `docker`
sees them. Nothing has to be exported by hand.
Single suites: `task test:structure`, `task test:compat`, `task test:features`.
Arguments after `--` go to pytest, e.g. `task test:compat -- -k hooks`. Set
`IMAGE` to test an already built image instead of `restic-backup-docker:test`.
There are three suites:
The suites live in `tests/`:
* **compat** (`test/compat.sh`) - backup, restore, forget, check, hooks and
* **compat** (`tests/test_compat.py`) - backup, restore, forget, check, hooks and
failure handling against the default configuration. These tests describe the
behaviour users rely on. A change that turns one of them red is a breaking
change.
* **features** (`test/features.sh`) - tests for optional features. Each block
is guarded by a detection check against the built image, so a feature that is
not part of the image is reported as `SKIP` instead of failing. Pull requests
that add such a feature are therefore covered as soon as they are rebased,
without the author having to write the basic tests. After merging a feature,
add its gate name to `test/required-features.txt` so that a missing feature
turns into a failure.
* **structure** (`test/structure.yml`) - image layout checks via
* **features** (`tests/test_features.py`) - tests for optional features. Each
test is marked `@pytest.mark.feature("<gate>")` and guarded by a detection
check against the built image, so a feature that is not part of the image is
reported as skipped instead of failing. Pull requests that add such a feature
are therefore covered as soon as they are rebased, without the author having
to write the basic tests. After merging a feature, add its gate name to
`tests/required-features.txt` so that a missing feature turns into a failure.
* **structure** (`tests/structure.yml`) - image layout checks via
[container-structure-test](https://github.com/GoogleContainerTools/container-structure-test).
When the tool is not installed, its official container image is used instead.
`tests/structure.py` runs it and removes the containers and anonymous volumes
it leaves behind.
`bash test/run.sh --sabotage` is a self test: it breaks the backup call on a
throwaway copy of the tree and expects the compat suite to fail. CI runs it so
that the suites cannot be watered down into always passing.
`task test:sabotage` is a self test: it breaks the backup call on a throwaway
copy of the tree and expects the compat tests to fail. CI runs it so that the
tests cannot be watered down into always passing.
When you add a feature, please add a block to `test/features.sh` for it.
When you add a feature, please add tests with a `feature` marker for it.
## Hooks