diff --git a/CHANGELOG.md b/CHANGELOG.md index d2b902f..fb022ca 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,11 +3,11 @@ ## Unreleased ### Added -* Test suites in `test/` (compat, features, structure) that run the image as a real container, plus a `Test` GitHub Actions workflow running them on every pull request +* Test suites in `tests/` (compat, features, structure) that run the image as a real container, plus a `Test` GitHub Actions workflow running them on every pull request. The suites are written in Python (pytest) and run via `mise install` and `task test`; `task test:sabotage` proves they detect a broken backup * `.gitattributes` pinning shell scripts, the Dockerfile and YAML files to LF line endings. With a Windows checkout (`core.autocrlf=true`) the scripts were checked out with CRLF, which made the built image fail to start with `exec /entry.sh: no such file or directory` ### Removed -* `config-broken.yml` and `docker-compose.test.broken.yml`, replaced by `test/structure.yml` and the test workflow +* `config-broken.yml` and `docker-compose.test.broken.yml`, replaced by `tests/structure.yml` and the test workflow ## v1.4.2 ### Removed diff --git a/README.md b/README.md index 2ff88d0..f985d46 100644 --- a/README.md +++ b/README.md @@ -28,43 +28,45 @@ When behaviour of the image changes (Features, Bugfixes, Changes in the API) ple ## Tests -The test suites build the image and exercise it as a real container. They need -`bash` and `docker`, nothing else: +The tests build the image and exercise it as a real container. Tools are pinned +in `mise.toml` (Python, [uv](https://docs.astral.sh/uv/), +[Task](https://taskfile.dev/), container-structure-test), Python dependencies in +`uv.lock`. You need [mise](https://mise.jdx.dev/) and Docker, nothing else: ```sh -bash test/run.sh # build the image and run every suite -bash test/run.sh --no-build # reuse an already built image -bash test/run.sh --suite compat # run a single suite +mise install # install the pinned tools +task test # build the image and run every suite +task test:sabotage # self test, see below +task lint # ruff check and format check of tests/ ``` -They run unchanged on Linux, macOS and Windows/Git-Bash. On Git-Bash the -scripts set `MSYS_NO_PATHCONV=1` and `MSYS2_ARG_CONV_EXCL='*'` themselves, -because MSYS would otherwise rewrite container paths such as -`/usr/bin/backup` into `C:/Program Files/Git/usr/bin/backup` before `docker` -sees them. Nothing has to be exported by hand. +Single suites: `task test:structure`, `task test:compat`, `task test:features`. +Arguments after `--` go to pytest, e.g. `task test:compat -- -k hooks`. Set +`IMAGE` to test an already built image instead of `restic-backup-docker:test`. -There are three suites: +The suites live in `tests/`: -* **compat** (`test/compat.sh`) - backup, restore, forget, check, hooks and +* **compat** (`tests/test_compat.py`) - backup, restore, forget, check, hooks and failure handling against the default configuration. These tests describe the behaviour users rely on. A change that turns one of them red is a breaking change. -* **features** (`test/features.sh`) - tests for optional features. Each block - is guarded by a detection check against the built image, so a feature that is - not part of the image is reported as `SKIP` instead of failing. Pull requests - that add such a feature are therefore covered as soon as they are rebased, - without the author having to write the basic tests. After merging a feature, - add its gate name to `test/required-features.txt` so that a missing feature - turns into a failure. -* **structure** (`test/structure.yml`) - image layout checks via +* **features** (`tests/test_features.py`) - tests for optional features. Each + test is marked `@pytest.mark.feature("")` and guarded by a detection + check against the built image, so a feature that is not part of the image is + reported as skipped instead of failing. Pull requests that add such a feature + are therefore covered as soon as they are rebased, without the author having + to write the basic tests. After merging a feature, add its gate name to + `tests/required-features.txt` so that a missing feature turns into a failure. +* **structure** (`tests/structure.yml`) - image layout checks via [container-structure-test](https://github.com/GoogleContainerTools/container-structure-test). - When the tool is not installed, its official container image is used instead. + `tests/structure.py` runs it and removes the containers and anonymous volumes + it leaves behind. -`bash test/run.sh --sabotage` is a self test: it breaks the backup call on a -throwaway copy of the tree and expects the compat suite to fail. CI runs it so -that the suites cannot be watered down into always passing. +`task test:sabotage` is a self test: it breaks the backup call on a throwaway +copy of the tree and expects the compat tests to fail. CI runs it so that the +tests cannot be watered down into always passing. -When you add a feature, please add a block to `test/features.sh` for it. +When you add a feature, please add tests with a `feature` marker for it. ## Hooks