docs: expand documentation about testing (#5346)

.github/PULL_REQUEST_TEMPLATE.md

@@ -36,7 +36,7 @@ Please always follow these steps:
 - Format all commit messages in the same style as [the other commits in the repository](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#git-commits).
 -->
 
-- [ ] I have added tests for all code changes.
+- [ ] I have added tests for all code changes, see [writing tests](https://restic.readthedocs.io/en/stable/090_participating.html#writing-tests)
 - [ ] I have added documentation for relevant changes (in the manual).
 - [ ] There's a new file in `changelog/unreleased/` that describes the changes for our users (see [template](https://github.com/restic/restic/blob/master/changelog/TEMPLATE)).
 - [ ] I'm done! This pull request is ready for review.

CONTRIBUTING.md

@@ -202,6 +202,9 @@ we'll be glad to assist. Having a PR with failing integration tests is nothing
 to be ashamed of. In contrast, that happens regularly for all of us. That's
 what the tests are there for.
 
+More details of how to structure tests can be found here at
+[writing tests](https://restic.readthedocs.io/en/stable/090_participating.html#writing-tests).
+
 Git Commits
 -----------
 

doc/047_tuning_parameters.rst

@@ -41,7 +41,7 @@ most cases. For high-latency backends it can be beneficial to increase the numbe
 connections. Please be aware that this increases the resource consumption of restic and
 that a too high connection count *will degrade performance*. This can also result in longer
 upload times for single temporary packs, which can lead to more disk wear on SSDs (see
-:ref:`Pack Size`).
+:ref:`pack_size`).
 
 
 CPU Usage
@@ -87,7 +87,7 @@ by reading more files in parallel. You can specify the concurrency of file reads
 the ``backup`` command.
 
 
-.. Pack Size:
+.. _pack_size:
 
 Pack Size
 =========

doc/090_participating.rst

@@ -105,6 +105,129 @@ project <https://dave.cheney.net/2016/03/12/suggestions-for-contributing-to-an-o
 A few issues have been tagged with the label ``help wanted``, you can
 start looking at `those <https://github.com/restic/restic/labels/help%3A%20wanted>`_.
 
+
+*************
+Writing tests
+*************
+
+In case you want or need to create tests for an enhancement or a new feature of restic,
+here is a brief description of how to write tests.
+
+Tests are typically falling into two categories: functional tests (unit tests) and integration tests.
+Functional tests will verify the correct workings of a function or a set of functions.
+See more on integration tests below.
+
+The restic test package located in ``internal/test``, here named ``rtest``,
+provides the following very basic test functions:
+::
+
+ rtest.Equals(t, a, b, msg)         compares two values, fails test if a != b, optional ``msg`` string
+                                    for detailed message
+ rtest.Assert(t, a == b, "msg", more variables a, b) checks for a condition to be true,
+                                    <msg> is a format string to represent the values of <a and b>
+ rtest.OK(t, err)                   expects err to be ``nil``, otherwise fails test
+ rtest.OKs(t, errs)                 expects a slice of errs to be ``nil``
+
+
+Functional tests
+================
+
+The packages in ``internal/...`` often provide ``Test*(...)`` functions and structs or
+have a dedicated test package like ``internal/backend/test``.
+A good starting points are also the `testing.go` files that exist in several places.
+Functional tests are stored in the same directory as their function, e.g.
+``internal/<sub-component>/<function>_test.go``.
+
+Tests in ``internal`` that need a full repository should just create one using the
+memory backend by calling ``repository.TestRepository(t)``.
+
+``checker.TestCheckRepo()`` can be used to verify the repository integrity.
+
+In general, in-memory operation should be preferred over creating temporary files.
+If necessary, temporary files can be stored in ``t.TempDir()``. However, in most cases test code
+only requires a few of the methods provided by a ``Repository``.
+Then some helper like ``data.TestTreeMap`` can be used or just create a basic mock yourself.
+
+For backends the test suite in ``internal/backend/test`` is mandatory.
+
+To temporarily enable feature flags in tests, use
+``defer feature.TestSetFlag(t, feature.Flag, feature.DeviceIDForHardlinks, true)()``.
+Such tests must not run in parallel as this changes global state.
+
+
+Integration tests
+=================
+
+The classical helpers for integration tests are, amongst others:
+
+- ``env, cleanup := withTestEnvironment(t)``: build an environment for tests
+- ``datafile := testSetupBackupData(t, env)``: initialize a repo, unpack standard backup tree structure
+- ``testRunBackup(t, "", []string{env.testdata}, BackupOptions{}, env.gopts)``: backup all of the standard tree structure
+- ``testListSnapshots(t, env.gopts, <n>)``: check that there are <n> snapshots in the repository
+- ``testRunCheck(t, env.gopts)``: check that the repository is sound and happy
+- the above mentioned ``rtest.OK()``, ``rtest.Equals()``, ``rtest.Assert()`` helpers
+- ``withCaptureStdout()`` and ``withTermStatus()`` wrappers: both functions are found in ``cmd/restic/integration_helpers_test.go`` for creating an enviroment where one can analyze the output created by the ``testRunXXX()`` command, particularly when checking JSON output
+
+Integration tests test the overall workings of a command. Integration tests are used for commands and
+are stored in the same directory ``cmd/restic``. The recommended naming convention is
+``cmd_<command>_integration_test.go``.
+See the ``cmd/restic/*_integration_test.go`` files for further details.
+A lot of the base helpers are found in ``cmd/restic/integration_helpers_test.go``.
+
+This is a typical setting for an integration test:
+
+- run a ``backup``, compare number of files backup with the expected number of files
+- run a ``backup``, run the ``ls`` command with a ``sort`` option and compare actual output with the expected output.
+
+For all backup related functions there is a directory tree which can be used for a
+default backup, to be found at ``cmd/restic/testdata/backup-data.tar.gz``.
+In this compressed archive you will find files, hardlinked files,
+symlinked files, an empty directory and a simple directory structure which is good for testing purposes.
+
+Commands that require a ``progress.Printer`` should either be wrapped in ``withTermStatus`` or ``withCaptureStdout``.
+If you want to analyze JSON output, you use ``withCaptureStdout()``.
+It returns the  generated output in a ``*bytes.Buffer``.
+JSON output can be unmarshalled to produce the approriate go structures; see
+``cmd/restic/cmd_find_integration_test.go`` as an example.
+
+Example: this is a typical setup for a backup / find scenario
+::
+
+ import (
+   ... // all your other imports here
+
+   rtest "github.com/restic/restic/internal/test"
+ )
+
+ // setup test
+ env, cleanup := withTestEnvironment(t)
+ defer cleanup()
+
+ // init repository and expand compressed archive into a tree structure
+ testSetupBackupData(t, env)
+
+ // run one backup
+ opts := BackupOptions{}
+ testRunBackup(t, env.testdata+"/0", []string{"."}, opts, env.gopts)
+
+ // make sure we have exactly one snapshot
+ testListSnapshots(t, env.gopts, 1)
+
+ // run command ``restic XXX``
+ // notice that we use the existing wrapper 'testRunXXXX', here 'testRunFind'.
+ // whenever possible use the existing wrapper or modify an existing wrapper
+ // to suit your extra needs.
+ // any remotely complex assertion should be extracted into a reusable helper function.
+ // ``testRunFind()`` uses ``withCaptureStdout()`` to capture output text (in ``results``)
+ results := testRunFind(t, false, FindOptions{}, env.gopts, "testfile")
+
+ // there is always a ``\n`` at  the end of the output!
+ lines := strings.Split(string(results), "\n")
+
+ // make sure that we have correct output
+ rtest.Assert(t, len(lines) == 2, "expected one file, found (%v) in repo", len(lines)-1)
+
+
 ********
 Security
 ********