diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 9faeb3a9d..032c74c76 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -36,7 +36,7 @@ Please always follow these steps: - Format all commit messages in the same style as [the other commits in the repository](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#git-commits). --> -- [ ] I have added tests for all code changes. +- [ ] I have added tests for all code changes, see [writing tests](https://restic.readthedocs.io/en/stable/090_participating.html#writing-tests) - [ ] I have added documentation for relevant changes (in the manual). - [ ] There's a new file in `changelog/unreleased/` that describes the changes for our users (see [template](https://github.com/restic/restic/blob/master/changelog/TEMPLATE)). - [ ] I'm done! This pull request is ready for review. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index dc278fa3a..d1900fa16 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -202,6 +202,9 @@ we'll be glad to assist. Having a PR with failing integration tests is nothing to be ashamed of. In contrast, that happens regularly for all of us. That's what the tests are there for. +More details of how to structure tests can be found here at +[writing tests](https://restic.readthedocs.io/en/stable/090_participating.html#writing-tests). + Git Commits ----------- diff --git a/doc/047_tuning_parameters.rst b/doc/047_tuning_parameters.rst index eba958211..1cd6b9329 100644 --- a/doc/047_tuning_parameters.rst +++ b/doc/047_tuning_parameters.rst @@ -41,7 +41,7 @@ most cases. For high-latency backends it can be beneficial to increase the numbe connections. Please be aware that this increases the resource consumption of restic and that a too high connection count *will degrade performance*. This can also result in longer upload times for single temporary packs, which can lead to more disk wear on SSDs (see -:ref:`Pack Size`). +:ref:`pack_size`). CPU Usage @@ -87,7 +87,7 @@ by reading more files in parallel. You can specify the concurrency of file reads the ``backup`` command. -.. Pack Size: +.. _pack_size: Pack Size ========= diff --git a/doc/090_participating.rst b/doc/090_participating.rst index 2a3964cac..7e84327bc 100644 --- a/doc/090_participating.rst +++ b/doc/090_participating.rst @@ -105,6 +105,129 @@ project `_. + +************* +Writing tests +************* + +In case you want or need to create tests for an enhancement or a new feature of restic, +here is a brief description of how to write tests. + +Tests are typically falling into two categories: functional tests (unit tests) and integration tests. +Functional tests will verify the correct workings of a function or a set of functions. +See more on integration tests below. + +The restic test package located in ``internal/test``, here named ``rtest``, +provides the following very basic test functions: +:: + + rtest.Equals(t, a, b, msg) compares two values, fails test if a != b, optional ``msg`` string + for detailed message + rtest.Assert(t, a == b, "msg", more variables a, b) checks for a condition to be true, + is a format string to represent the values of + rtest.OK(t, err) expects err to be ``nil``, otherwise fails test + rtest.OKs(t, errs) expects a slice of errs to be ``nil`` + + +Functional tests +================ + +The packages in ``internal/...`` often provide ``Test*(...)`` functions and structs or +have a dedicated test package like ``internal/backend/test``. +A good starting points are also the `testing.go` files that exist in several places. +Functional tests are stored in the same directory as their function, e.g. +``internal//_test.go``. + +Tests in ``internal`` that need a full repository should just create one using the +memory backend by calling ``repository.TestRepository(t)``. + +``checker.TestCheckRepo()`` can be used to verify the repository integrity. + +In general, in-memory operation should be preferred over creating temporary files. +If necessary, temporary files can be stored in ``t.TempDir()``. However, in most cases test code +only requires a few of the methods provided by a ``Repository``. +Then some helper like ``data.TestTreeMap`` can be used or just create a basic mock yourself. + +For backends the test suite in ``internal/backend/test`` is mandatory. + +To temporarily enable feature flags in tests, use +``defer feature.TestSetFlag(t, feature.Flag, feature.DeviceIDForHardlinks, true)()``. +Such tests must not run in parallel as this changes global state. + + +Integration tests +================= + +The classical helpers for integration tests are, amongst others: + +- ``env, cleanup := withTestEnvironment(t)``: build an environment for tests +- ``datafile := testSetupBackupData(t, env)``: initialize a repo, unpack standard backup tree structure +- ``testRunBackup(t, "", []string{env.testdata}, BackupOptions{}, env.gopts)``: backup all of the standard tree structure +- ``testListSnapshots(t, env.gopts, )``: check that there are snapshots in the repository +- ``testRunCheck(t, env.gopts)``: check that the repository is sound and happy +- the above mentioned ``rtest.OK()``, ``rtest.Equals()``, ``rtest.Assert()`` helpers +- ``withCaptureStdout()`` and ``withTermStatus()`` wrappers: both functions are found in ``cmd/restic/integration_helpers_test.go`` for creating an enviroment where one can analyze the output created by the ``testRunXXX()`` command, particularly when checking JSON output + +Integration tests test the overall workings of a command. Integration tests are used for commands and +are stored in the same directory ``cmd/restic``. The recommended naming convention is +``cmd__integration_test.go``. +See the ``cmd/restic/*_integration_test.go`` files for further details. +A lot of the base helpers are found in ``cmd/restic/integration_helpers_test.go``. + +This is a typical setting for an integration test: + +- run a ``backup``, compare number of files backup with the expected number of files +- run a ``backup``, run the ``ls`` command with a ``sort`` option and compare actual output with the expected output. + +For all backup related functions there is a directory tree which can be used for a +default backup, to be found at ``cmd/restic/testdata/backup-data.tar.gz``. +In this compressed archive you will find files, hardlinked files, +symlinked files, an empty directory and a simple directory structure which is good for testing purposes. + +Commands that require a ``progress.Printer`` should either be wrapped in ``withTermStatus`` or ``withCaptureStdout``. +If you want to analyze JSON output, you use ``withCaptureStdout()``. +It returns the generated output in a ``*bytes.Buffer``. +JSON output can be unmarshalled to produce the approriate go structures; see +``cmd/restic/cmd_find_integration_test.go`` as an example. + +Example: this is a typical setup for a backup / find scenario +:: + + import ( + ... // all your other imports here + + rtest "github.com/restic/restic/internal/test" + ) + + // setup test + env, cleanup := withTestEnvironment(t) + defer cleanup() + + // init repository and expand compressed archive into a tree structure + testSetupBackupData(t, env) + + // run one backup + opts := BackupOptions{} + testRunBackup(t, env.testdata+"/0", []string{"."}, opts, env.gopts) + + // make sure we have exactly one snapshot + testListSnapshots(t, env.gopts, 1) + + // run command ``restic XXX`` + // notice that we use the existing wrapper 'testRunXXXX', here 'testRunFind'. + // whenever possible use the existing wrapper or modify an existing wrapper + // to suit your extra needs. + // any remotely complex assertion should be extracted into a reusable helper function. + // ``testRunFind()`` uses ``withCaptureStdout()`` to capture output text (in ``results``) + results := testRunFind(t, false, FindOptions{}, env.gopts, "testfile") + + // there is always a ``\n`` at the end of the output! + lines := strings.Split(string(results), "\n") + + // make sure that we have correct output + rtest.Assert(t, len(lines) == 2, "expected one file, found (%v) in repo", len(lines)-1) + + ******** Security ********