docs: describe the Python test suite

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Tobias KaupatandClaude Sonnet 5.5 committed 2026-10-19 17:49:47 +02:00
1 parent d1a6744b8b
commit 5c03a2e31a
2 files changed
+29 -27

No files matched your search

+2 -2
View File
@@ -3,11 +3,11 @@
## Unreleased ## Unreleased
### Added ### Added
* Test suites in `test/` (compat, features, structure) that run the image as a real container, plus a `Test` GitHub Actions workflow running them on every pull request * Test suites in `tests/` (compat, features, structure) that run the image as a real container, plus a `Test` GitHub Actions workflow running them on every pull request. The suites are written in Python (pytest) and run via `mise install` and `task test`; `task test:sabotage` proves they detect a broken backup
* `.gitattributes` pinning shell scripts, the Dockerfile and YAML files to LF line endings. With a Windows checkout (`core.autocrlf=true`) the scripts were checked out with CRLF, which made the built image fail to start with `exec /entry.sh: no such file or directory` * `.gitattributes` pinning shell scripts, the Dockerfile and YAML files to LF line endings. With a Windows checkout (`core.autocrlf=true`) the scripts were checked out with CRLF, which made the built image fail to start with `exec /entry.sh: no such file or directory`
### Removed ### Removed
* `config-broken.yml` and `docker-compose.test.broken.yml`, replaced by `test/structure.yml` and the test workflow * `config-broken.yml` and `docker-compose.test.broken.yml`, replaced by `tests/structure.yml` and the test workflow
## v1.4.2 ## v1.4.2
### Removed ### Removed
+27 -25
View File
@@ -28,43 +28,45 @@ When behaviour of the image changes (Features, Bugfixes, Changes in the API) ple
## Tests ## Tests
The test suites build the image and exercise it as a real container. They need The tests build the image and exercise it as a real container. Tools are pinned
`bash` and `docker`, nothing else: in `mise.toml` (Python, [uv](https://docs.astral.sh/uv/),
[Task](https://taskfile.dev/), container-structure-test), Python dependencies in
`uv.lock`. You need [mise](https://mise.jdx.dev/) and Docker, nothing else:
```sh ```sh
bash test/run.sh # build the image and run every suite mise install # install the pinned tools
bash test/run.sh --no-build # reuse an already built image task test # build the image and run every suite
bash test/run.sh --suite compat # run a single suite task test:sabotage # self test, see below
task lint # ruff check and format check of tests/
``` ```
They run unchanged on Linux, macOS and Windows/Git-Bash. On Git-Bash the Single suites: `task test:structure`, `task test:compat`, `task test:features`.
scripts set `MSYS_NO_PATHCONV=1` and `MSYS2_ARG_CONV_EXCL='*'` themselves, Arguments after `--` go to pytest, e.g. `task test:compat -- -k hooks`. Set
because MSYS would otherwise rewrite container paths such as `IMAGE` to test an already built image instead of `restic-backup-docker:test`.
`/usr/bin/backup` into `C:/Program Files/Git/usr/bin/backup` before `docker`
sees them. Nothing has to be exported by hand.
There are three suites: The suites live in `tests/`:
* **compat** (`test/compat.sh`) - backup, restore, forget, check, hooks and * **compat** (`tests/test_compat.py`) - backup, restore, forget, check, hooks and
failure handling against the default configuration. These tests describe the failure handling against the default configuration. These tests describe the
behaviour users rely on. A change that turns one of them red is a breaking behaviour users rely on. A change that turns one of them red is a breaking
change. change.
* **features** (`test/features.sh`) - tests for optional features. Each block * **features** (`tests/test_features.py`) - tests for optional features. Each
is guarded by a detection check against the built image, so a feature that is test is marked `@pytest.mark.feature("<gate>")` and guarded by a detection
not part of the image is reported as `SKIP` instead of failing. Pull requests check against the built image, so a feature that is not part of the image is
that add such a feature are therefore covered as soon as they are rebased, reported as skipped instead of failing. Pull requests that add such a feature
without the author having to write the basic tests. After merging a feature, are therefore covered as soon as they are rebased, without the author having
add its gate name to `test/required-features.txt` so that a missing feature to write the basic tests. After merging a feature, add its gate name to
turns into a failure. `tests/required-features.txt` so that a missing feature turns into a failure.
* **structure** (`test/structure.yml`) - image layout checks via * **structure** (`tests/structure.yml`) - image layout checks via
[container-structure-test](https://github.com/GoogleContainerTools/container-structure-test). [container-structure-test](https://github.com/GoogleContainerTools/container-structure-test).
When the tool is not installed, its official container image is used instead. `tests/structure.py` runs it and removes the containers and anonymous volumes
it leaves behind.
`bash test/run.sh --sabotage` is a self test: it breaks the backup call on a `task test:sabotage` is a self test: it breaks the backup call on a throwaway
throwaway copy of the tree and expects the compat suite to fail. CI runs it so copy of the tree and expects the compat tests to fail. CI runs it so that the
that the suites cannot be watered down into always passing. tests cannot be watered down into always passing.
When you add a feature, please add a block to `test/features.sh` for it. When you add a feature, please add tests with a `feature` marker for it.
## Hooks ## Hooks