Document the setup-only and fulltest development and usage (#4362)

docs/misc/contributing/development.md

@@ -21,6 +21,40 @@ docker compose -f compose-dev.yml run --rm -it [-e key=value]  mc-dev
 
     To speed up the development cycle, it is recommended to set `SETUP_ONLY` to `true` as part of the run command above.
 
+## Testing changes
+
+Most changes should be covered by a test under `tests/setuponlytests`. Each test scenario needs its own `compose.yml` or `docker-compose.yml`, which defines the container configuration and inputs for that case. These tests run server setup with `SETUP_ONLY=true`, so they can verify configuration, downloads, and generated files without starting Minecraft.
+
+A scenario can include a `require.sh` to check whether its prerequisites are available in the test environment, such as a required environment variable. It runs before the scenario; a non-zero exit status skips that test. For example, this skips a test when `CF_API_KEY` is not set:
+
+```bash title="require.sh"
+[[ -n "${CF_API_KEY:-}" ]] || exit 1
+```
+
+Since `require.sh` runs on the host, it sees the variables provided by the CI workflow. These are the ones typically checked:
+
+- `EXTENDED_TESTS`: opt-in flag for slower or network-heavy scenarios; the CI workflows don't set it, so those scenarios only run when it is exported locally
+- `CF_API_KEY`: CurseForge API key, which is only available to trusted runs
+- `MINECRAFT_VERSION`: the Minecraft version of the CI matrix entry, such as `latest` or `1.12.2`
+- `VARIANT`: the image variant of the CI matrix entry, such as `java21-alpine`
+- `GH_TOKEN`: GitHub token used for API calls that would otherwise be rate limited
+
+A `require_inside.sh` works the same way, but runs inside a container of the image under test, which is useful when a test depends on the image's contents or Java variant. Checking for JVM runtime files is a good use of it. For example, this skips a test that needs the JDWP agent, which some Java variants such as GraalVM do not include:
+
+```bash title="require_inside.sh"
+[[ -f /opt/java/openjdk/lib/libjdwp.so ]] || exit 1
+```
+
+Add a `verify.sh` to assert that setup produced the expected files or contents. It runs only after setup succeeds, with the scenario's `data` directory mounted at `/data`; a non-zero exit status fails the test. See the [`mc-image-helper assert` command documentation](https://github.com/itzg/mc-image-helper#assert) for available assertions. For example, use it to check that setup downloaded an expected mod:
+
+```bash title="verify.sh"
+mc-image-helper assert fileExists "/data/mods/ExplorersCompass-1.16.5-1.1.2-forge.jar"
+```
+
+Run the test suite with `tests/test.sh`. For a focused setup-only check, run `bash tests/setuponlytests/test.sh <scenario>` from the repository root, where `<scenario>` is the directory name under `tests/setuponlytests` (for example, `defaults`). The pull request workflow runs the suite, so prefer setup-only tests for new scenarios.
+
+Tests under `tests/fulltests` start Minecraft and wait for the monitor to confirm that the server is healthy. Keep this suite limited to maintained tests that provide essential full-startup coverage; adding every scenario there would increase pull request validation time.
+
 ## Building the image with a new release of a tool
 
 In this exapmle, let's say that [mc-image-helper](https://github.com/itzg/mc-image-helper) has been [released](https://github.com/itzg/mc-image-helper/releases) at 1.56.0, but the corresponding changes in the image [scripts](https://github.com/itzg/docker-minecraft-server/tree/23205471db9814cff9c6602361dbc6cdd6c4230a/scripts) need to be tested against that version while updating [the Dockerfile](https://github.com/itzg/docker-minecraft-server/blob/23205471db9814cff9c6602361dbc6cdd6c4230a/Dockerfile#L58).

tests/fulltests/test.sh

@@ -28,8 +28,8 @@ fullMinecraftUpTest(){
 }
 
 # go through each folder in fulltests and run fullbuilds
-files=$(ls */docker-compose.yml)
+readarray -t files < <(find . -maxdepth 2 -mindepth 2 \( -name compose.yml -o -name docker-compose.yml \) | sort)
-for file in $files; do
+for file in "${files[@]}"; do
 
     echo "Starting Tests on $(dirname "$file")"
     if ! fullMinecraftUpTest "$file"; then

tests/setuponlytests/test.sh

@@ -41,6 +41,16 @@ setupOnlyMinecraftTest(){
     fi
   fi
 
+  if [ -f require_inside.sh ]; then
+    # require_inside.sh scripts run inside the image under test to check for things like a particular
+    # JVM file or feature. A non-zero exit status indicates the test should be skipped.
+    if ! docker run --rm --entrypoint bash -v "${PWD}/require_inside.sh":/require_inside:ro "${IMAGE_TO_TEST}" -e /require_inside; then
+      echo "${folder} SKIP (require_inside)"
+      cd ..
+      return 0
+    fi
+  fi
+
   # false positive since it's used in delta calculations below
   # shellcheck disable=SC2034
   start=$(date +%s)
@@ -71,7 +81,7 @@ image=""
 
 # Go through each folder in setuponly and test setups
 if (( $# == 0 )); then
-  readarray -t folders < <(find . -maxdepth 2 -mindepth 2 -name docker-compose.yml -exec dirname "{}" \;)
+  readarray -t folders < <(find . -maxdepth 2 -mindepth 2 \( -name compose.yml -o -name docker-compose.yml \) -exec dirname "{}" \; | sort -u)
   foldersList=("${folders[@]}")
   image=" using $IMAGE_TO_TEST"
 fi