mirror of
https://github.com/lobaro/restic-backup-docker.git
synced 2026-11-21 00:56:52 +00:00
test: move container test suites to pytest (#112)
* test: scaffold pytest suite with mise, uv and Task Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * ci: run the pytest suite via mise and task Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix: make test cleanup survive repeated SIGINT A group Ctrl-C reaches pytest a second time via task and uv while the session fixture tears down, aborting the volume cleanup. Ignore SIGINT and SIGTERM during cleanup, remove volumes in finally, and tolerate already-removed objects. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix: pin Python 3.13 for uv without mise Add .python-version (read by mise and uv) and cap requires-python at <3.14 so bare uv uses 3.13 or fails instead of picking 3.14. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix: clean up test containers on SIGTERM Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * test: port compat tests to pytest Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * test: assert the success message first in the corruption test Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * test: assert the empty-tag snapshot carries no tags Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * test: remove anonymous volumes with short-lived test containers Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * test: port feature-gated tests to pytest Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * test: port sabotage self-test to task test:sabotage Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix: create parent directories in put_file Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * test: move structure.yml to tests and remove cst volume leak container-structure-test left one anonymous volume per test container. tests/structure.py runs it with --save and removes the new containers with their volumes afterwards. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * test: remove the Bash suite Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * ci: run only the Python suite in the test job Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * docs: describe the Python test suite Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * test: isolate scenario steps, share helpers, use restore_latest, HEAD dir probe Scenario fixtures record each step; a failing step fails only the tests that assert on it. running() and raise_interrupt() live in helpers.py. put_file probes parent directories with a HEAD request instead of a get_archive stream. stream_run uses restore_latest and asserts its result. Docstrings no longer refer to the removed Bash suite. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * test: sabotage check rejects errors and malformed reports Only <failure> of the expected tests counts as red; errors in them and a malformed JUnit report exit 2 (invalid) instead of passing or exiting 1. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * test: structure wrapper removes only its own containers Restrict cleanup to containers of the image under test without the rbd-test-run label and ignore NotFound. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * build: run all test suites and aggregate the exit code task test no longer stops at the first failing suite. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix: sabotage check treats failed scenario steps as INVALID StepFailed is no AssertionError any more; sabotage.py counts a JUnit failure named StepFailed as error, so an expected-red test only counts when its own assertion fails. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix: hard timeout for container-structure-test CST_TIMEOUT (default 300 s) kills a hanging run; existing cleanup removes the containers; exit code 124 with a clear message. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix: shorten sabotage message for lint Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix: handle JUnit failures without message in sabotage check Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
31f06b5d9b
commit
be7d426f78
21 files changed
+1692
-958
No files matched your search
@@ -28,43 +28,45 @@ When behaviour of the image changes (Features, Bugfixes, Changes in the API) ple
|
||||
|
||||
## Tests
|
||||
|
||||
The test suites build the image and exercise it as a real container. They need
|
||||
`bash` and `docker`, nothing else:
|
||||
The tests build the image and exercise it as a real container. Tools are pinned
|
||||
in `mise.toml` (Python, [uv](https://docs.astral.sh/uv/),
|
||||
[Task](https://taskfile.dev/), container-structure-test), Python dependencies in
|
||||
`uv.lock`. You need [mise](https://mise.jdx.dev/) and Docker, nothing else:
|
||||
|
||||
```sh
|
||||
bash test/run.sh # build the image and run every suite
|
||||
bash test/run.sh --no-build # reuse an already built image
|
||||
bash test/run.sh --suite compat # run a single suite
|
||||
mise install # install the pinned tools
|
||||
task test # build the image and run every suite
|
||||
task test:sabotage # self test, see below
|
||||
task lint # ruff check and format check of tests/
|
||||
```
|
||||
|
||||
They run unchanged on Linux, macOS and Windows/Git-Bash. On Git-Bash the
|
||||
scripts set `MSYS_NO_PATHCONV=1` and `MSYS2_ARG_CONV_EXCL='*'` themselves,
|
||||
because MSYS would otherwise rewrite container paths such as
|
||||
`/usr/bin/backup` into `C:/Program Files/Git/usr/bin/backup` before `docker`
|
||||
sees them. Nothing has to be exported by hand.
|
||||
Single suites: `task test:structure`, `task test:compat`, `task test:features`.
|
||||
Arguments after `--` go to pytest, e.g. `task test:compat -- -k hooks`. Set
|
||||
`IMAGE` to test an already built image instead of `restic-backup-docker:test`.
|
||||
|
||||
There are three suites:
|
||||
The suites live in `tests/`:
|
||||
|
||||
* **compat** (`test/compat.sh`) - backup, restore, forget, check, hooks and
|
||||
* **compat** (`tests/test_compat.py`) - backup, restore, forget, check, hooks and
|
||||
failure handling against the default configuration. These tests describe the
|
||||
behaviour users rely on. A change that turns one of them red is a breaking
|
||||
change.
|
||||
* **features** (`test/features.sh`) - tests for optional features. Each block
|
||||
is guarded by a detection check against the built image, so a feature that is
|
||||
not part of the image is reported as `SKIP` instead of failing. Pull requests
|
||||
that add such a feature are therefore covered as soon as they are rebased,
|
||||
without the author having to write the basic tests. After merging a feature,
|
||||
add its gate name to `test/required-features.txt` so that a missing feature
|
||||
turns into a failure.
|
||||
* **structure** (`test/structure.yml`) - image layout checks via
|
||||
* **features** (`tests/test_features.py`) - tests for optional features. Each
|
||||
test is marked `@pytest.mark.feature("<gate>")` and guarded by a detection
|
||||
check against the built image, so a feature that is not part of the image is
|
||||
reported as skipped instead of failing. Pull requests that add such a feature
|
||||
are therefore covered as soon as they are rebased, without the author having
|
||||
to write the basic tests. After merging a feature, add its gate name to
|
||||
`tests/required-features.txt` so that a missing feature turns into a failure.
|
||||
* **structure** (`tests/structure.yml`) - image layout checks via
|
||||
[container-structure-test](https://github.com/GoogleContainerTools/container-structure-test).
|
||||
When the tool is not installed, its official container image is used instead.
|
||||
`tests/structure.py` runs it and removes the containers and anonymous volumes
|
||||
it leaves behind.
|
||||
|
||||
`bash test/run.sh --sabotage` is a self test: it breaks the backup call on a
|
||||
throwaway copy of the tree and expects the compat suite to fail. CI runs it so
|
||||
that the suites cannot be watered down into always passing.
|
||||
`task test:sabotage` is a self test: it breaks the backup call on a throwaway
|
||||
copy of the tree and expects the compat tests to fail. CI runs it so that the
|
||||
tests cannot be watered down into always passing.
|
||||
|
||||
When you add a feature, please add a block to `test/features.sh` for it.
|
||||
When you add a feature, please add tests with a `feature` marker for it.
|
||||
|
||||
## Hooks
|
||||
|
||||
|
||||
Reference in new issue
Block a user