.. Normally, there are no heading levels assigned to certain characters as the structure is determined from the succession of headings. However, this convention is used in Python’s Style Guide for documenting which you may follow: # with overline, for parts * for chapters = for sections - for subsections ^ for subsubsections " for paragraphs ########### Development ########### This section covers how to develop, debug, profile and write tests for restic. For opening issues and pull requests, see the next section. ************ Contributing ************ Contributions are welcome! More information can be found in `CONTRIBUTING.md `__. A document describing the design of restic and the data structures stored on the back end is contained in :ref:`repository-format`. If you'd like to start contributing to restic, but don't know exactly what to do, have a look at this great article by Dave Cheney: `Suggestions for contributing to an Open Source project `__. *********************** Development Environment *********************** The repository contains the code written for restic in the directories ``cmd/`` and ``internal/``. Make sure you have the `minimum required Go version installed `__. Clone the repo and ``cd`` into the directory: .. code-block:: console $ git clone https://github.com/restic/restic $ cd restic Then use the ``go`` tool to build restic: .. code-block:: console $ go run build.go $ ./restic version restic 0.19.1-dev (compiled manually) compiled with go1.26.10 on linux/amd64 To create a debug build use: .. code-block:: console $ go run build.go -tags debug You can run all tests with the following command: .. code-block:: console $ go test ./... ********** Debug Logs ********** Set the environment variable ``DEBUG_LOG`` to let restic write extensive debug messages to the specified file, e.g.: .. code-block:: console $ DEBUG_LOG=/tmp/restic-debug.log restic backup ~/work If you suspect that there is a bug, you can have a look at the debug log. Please be aware that the debug log might contain sensitive information such as file and directory names. The debug log will always contain all log messages restic generates. You can also instruct restic to print some or all debug messages to stderr. These can also be limited to e.g. a list of source files or a list of patterns for function names. The patterns are globbing patterns (see the documentation for `filepath.Match `__). Multiple patterns are separated by commas. Patterns are case sensitive. Printing all log messages to the console can be achieved by setting the file filter to ``*``: .. code-block:: console $ DEBUG_FILES=* restic check If you want restic to just print all debug log messages from the files ``main.go`` and ``lock.go``, set the environment variable ``DEBUG_FILES`` like this: .. code-block:: console $ DEBUG_FILES=main.go,lock.go restic check The following command line instructs restic to only print debug statements originating in functions that match the pattern ``*unlock*`` (case sensitive): .. code-block:: console $ DEBUG_FUNCS=*unlock* restic check *********** Debug Hangs *********** If restic gets stuck, you can generate a stacktrace as follows. On non-Windows systems, you can send a ``SIGQUIT`` signal to restic or press ``Ctrl-\`` to achieve the same result. This causes restic to print a stacktrace and then exit immediately. This will not damage your repository, however, it might be necessary to manually clean up stale lock files using ``restic unlock``. On Windows, please run restic with the environment variable ``RESTIC_DEBUG_STACKTRACE_SIGINT`` set to ``true`` and press ``Ctrl-C`` to create a stacktrace. ********* Debugging ********* The program can be built with debug support like this: .. code-block:: console $ go run build.go -tags debug This will make the ``restic debug `` available which can be used to inspect internal data structures. *********************************** Performance and Memory Usage Issues *********************************** Debug builds of restic support the ``--block-profile``, ``--cpu-profile``, ``--mem-profile``, and ``--trace-profile`` options which collect performance data that later on can be analyzed using the go tools: .. code-block:: console $ restic --cpu-profile . [...] $ go tool pprof -http localhost:12345 cpu.pprof To analyze a trace profile use ``go tool trace -http=localhost:12345 trace.out``. As the memory usage of restic changes over time, it may be useful to capture a snapshot of the current heap. This is possible using the ``--listen-profile`` option. Then while restic runs you can query and afterwards analyze the heap statistics. .. code-block:: console $ restic --listen-profile localhost:12345 [...] $ curl http://localhost:12345/debug/pprof/heap -o heap.pprof $ go tool pprof -http localhost:12345 heap.pprof The memory usage of restic primarily depends on the index size. To get an idea of the number of tree and data blobs in the repository, you can use the ``restic stats --mode debug`` command. Setting ``GODEBUG=gctrace=1`` prints garbage-collector runs. For a graphical variant combine this with `gcvis `__. ************* Writing tests ************* In case you want or need to create tests for an enhancement or a new feature of restic, here is a brief description of how to write tests. Tests typically fall into two categories: functional tests (unit tests) and integration tests. Functional tests will verify the correct workings of a function or a set of functions. See more on integration tests below. The restic test package located in ``internal/test``, here named ``rtest``, provides the following very basic test functions: :: rtest.Equals(t, a, b, msg) compares two values, fails test if a != b, optional ``msg`` string for detailed message rtest.Assert(t, a == b, "msg", more variables a, b) checks for a condition to be true, is a format string to represent the values of rtest.OK(t, err) expects err to be ``nil``, otherwise fails test rtest.OKs(t, errs) expects a slice of errs to be ``nil`` Functional tests ================ The packages in ``internal/...`` often provide ``Test*(...)`` functions and structs or have a dedicated test package like ``internal/backend/test``. A good starting point is to look at the `testing.go` files that exist in several places. Functional tests are stored in the same directory as their function, e.g. ``internal//_test.go``. Tests in ``internal`` that need a full repository should just create one using the memory backend by calling ``repository.TestRepository(t)``. ``checker.TestCheckRepo()`` can be used to verify the repository integrity. In general, in-memory operation should be preferred over creating temporary files. If necessary, temporary files can be stored in ``t.TempDir()``. However, in most cases test code only requires a few of the methods provided by a ``Repository``. Then some helper like ``data.TestTreeMap`` can be used or just create a basic mock yourself. For backends the test suite in ``internal/backend/test`` is mandatory. To temporarily enable feature flags in tests, use ``defer feature.TestSetFlag(t, feature.Flag, feature.DeviceIDForHardlinks, true)()``. Such tests must not run in parallel as this changes global state. Integration tests ================= The classical helpers for integration tests are, amongst others: - ``env, cleanup := withTestEnvironment(t)``: build an environment for tests - ``datafile := testSetupBackupData(t, env)``: initialize a repo, unpack standard backup tree structure - ``testRunBackup(t, "", []string{env.testdata}, BackupOptions{}, env.gopts)``: backup all of the standard tree structure - ``testListSnapshots(t, env.gopts, )``: check that there are snapshots in the repository - ``testRunCheck(t, env.gopts)``: check that the repository is sound and happy - the above mentioned ``rtest.OK()``, ``rtest.Equals()``, ``rtest.Assert()`` helpers - ``withCaptureStdout()`` and ``withTermStatus()`` wrappers: both functions are found in ``cmd/restic/integration_helpers_test.go`` for creating an environment where you can analyze the output created by the ``testRunXXX()`` command, particularly when checking JSON output Integration tests test the overall workings of a command. Integration tests are used for commands and are stored in the same directory ``cmd/restic``. The recommended naming convention is ``cmd__integration_test.go``. See the ``cmd/restic/*_integration_test.go`` files for further details. A lot of the base helpers are found in ``cmd/restic/integration_helpers_test.go``. This is a typical setting for an integration test: - run a ``backup``, compare number of files backed up with the expected number of files - run a ``backup``, run the ``ls`` command with a ``sort`` option and compare actual output with the expected output. For all backup related functions there is a directory tree which can be used for a default backup, to be found at ``cmd/restic/testdata/backup-data.tar.gz``. In this compressed archive you will find files, hardlinked files, symlinked files, an empty directory and a simple directory structure which is good for testing purposes. Commands that require a ``restic.Printer`` should either be wrapped in ``withTermStatus`` or ``withCaptureStdout``. The interface is defined in ``internal/restic``; ``progress.NewTerminalPrinter`` returns a ``restic.Printer``. If you want to analyze JSON output, you use ``withCaptureStdout()``. It returns the generated output in a ``*bytes.Buffer``. JSON output can be unmarshalled to produce the appropriate go structures; see ``cmd/restic/cmd_find_integration_test.go`` as an example. Example: this is a typical setup for a backup / find scenario :: import ( ... // all your other imports here rtest "github.com/restic/restic/internal/test" ) // setup test env, cleanup := withTestEnvironment(t) defer cleanup() // init repository and expand compressed archive into a tree structure testSetupBackupData(t, env) // run one backup opts := BackupOptions{} testRunBackup(t, env.testdata+"/0", []string{"."}, opts, env.gopts) // make sure we have exactly one snapshot testListSnapshots(t, env.gopts, 1) // run command ``restic XXX`` // notice that we use the existing wrapper 'testRunXXXX', here 'testRunFind'. // whenever possible use the existing wrapper or modify an existing wrapper // to suit your extra needs. // any remotely complex assertion should be extracted into a reusable helper function. // ``testRunFind()`` uses ``withCaptureStdout()`` to capture output text (in ``results``) results := testRunFind(t, false, FindOptions{}, env.gopts, "testfile") // there is always a ``\n`` at the end of the output! lines := strings.Split(string(results), "\n") // make sure that we have correct output rtest.Assert(t, len(lines) == 2, "expected one file, found (%v) in repo", len(lines)-1) ************* Compatibility ************* Backward compatibility for backups is important so that our users are always able to restore saved data. Therefore restic follows `Semantic Versioning `__ to clearly define which versions are compatible. The repository and data structures contained therein are considered the "Public API" in the sense of Semantic Versioning. Once version 1.0.0 is released, we guarantee backward compatibility of all repositories within one major version; as long as we do not increment the major version, data can be read and restored. We strive to be fully backward compatible to all prior versions. During initial development (versions prior to 1.0.0), maintainers and developers will do their utmost to keep backwards compatibility and stability, although there might be breaking changes without increasing the major version. ********************** Building documentation ********************** The restic documentation is built with `Sphinx `__, therefore building it locally requires a recent Python version and requirements listed in ``doc/requirements.txt``. This example will guide you through the process using `virtualenv `__: :: $ virtualenv venv # create virtual python environment $ source venv/bin/activate # activate the virtual environment $ cd doc $ pip install -r requirements.txt # install dependencies $ make html # build html documentation $ # open _build/html/index.html with your favorite browser