Add VERSION for 0.9.2

minor-fixes_examples

.github/ISSUE_TEMPLATE.md

@@ -1,63 +0,0 @@
-<!--
-NOTE: Not filling out the issue template needs a good reason, otherwise it may
-take a lot longer to find the problem! Please take the time to help us
-debugging the problem by collecting information, even if it seems irrelevant to
-you. Thanks!
-
-If you have a question, the forum at https://forum.restic.net is a better place.
-Please do not create issues for usage or documentation questions! We're using
-the GitHub issue tracker mainly for tracking bugs and feature requests.
--->
-
-## Output of `restic version`
-
-
-## How did you run restic exactly?
-
-<!--
-This section should include at least:
-
- * The complete command line and any environment variables you used to
-   configure restic's backend access. Make sure to replace sensitive values!
-
- * The output of the commands, what restic prints gives may give us much
-   information to diagnose the problem!
--->
-
-
-## What backend/server/service did you use to store the repository?
-
-
-## Expected behavior
-
-<!--
-Describe what you'd like restic to do differently.
--->
-
-## Actual behavior
-
-<!--
-In this section, please try to concentrate on observations, so only describe
-what you observed directly.
--->
-
-## Steps to reproduce the behavior
-
-<!--
-The more time you spend describing an easy way to reproduce the behavior (if
-this is possible), the easier it is for the project developers to fix it!
--->
-
-
-## Do you have any idea what may have caused this?
-
-
-## Do you have an idea how to solve the issue?
-
-## Did restic help you or made you happy in any way?
-
-<!--
-Answering this question is not required, but if you have anything positive to share, please do so here!
-Sometimes we get tired of reading bug reports all day and a little positive end note does wonders.
-Idea by Joey Hess, https://joeyh.name/blog/entry/two_holiday_stories/
--->

.github/ISSUE_TEMPLATE/Bug.md

@@ -0,0 +1,93 @@
+---
+name: Bug report
+about: Report a problem with restic to help us resolve it and improve
+---
+
+<!--
+
+Welcome! - We kindly ask that you:
+
+  1. Fill out the issue template below - not doing so needs a good reason.
+  2. Use the forum if you have a question rather than a bug or feature request.
+
+The forum is at: https://forum.restic.net
+
+NOTE: Not filling out the issue template needs a good reason, as otherwise it
+may take a lot longer to find the problem, not to mention it can take up a lot
+more time which can otherwise be spent on development. Please also take the
+time to help us debug the issue by collecting relevant information, even if
+it doesn't seem to be relevant to you. Thanks!
+
+The forum is a better place for questions about restic or general suggestions
+and topics, e.g. usage or documentation questions! This issue tracker is mainly
+for tracking bugs and feature requests directly relating to the development of
+the software itself, rather than the project.
+
+Thanks for understanding, and for contributing to the project!
+
+-->
+
+
+Output of `restic version`
+--------------------------
+
+
+How did you run restic exactly?
+-------------------------------
+
+<!--
+This section should include at least:
+
+ * The complete command line and any environment variables you used to
+   configure restic's backend access. Make sure to replace sensitive values!
+
+ * The output of the commands, what restic prints gives may give us much
+   information to diagnose the problem!
+-->
+
+What backend/server/service did you use to store the repository?
+----------------------------------------------------------------
+
+
+
+Expected behavior
+-----------------
+
+<!--
+Describe what you'd like restic to do differently.
+-->
+
+Actual behavior
+---------------
+
+<!--
+In this section, please try to concentrate on observations, so only describe
+what you observed directly.
+-->
+
+Steps to reproduce the behavior
+-------------------------------
+
+<!--
+The more time you spend describing an easy way to reproduce the behavior (if
+this is possible), the easier it is for the project developers to fix it!
+-->
+
+Do you have any idea what may have caused this?
+-----------------------------------------------
+
+
+
+Do you have an idea how to solve the issue?
+-------------------------------------------
+
+
+
+Did restic help you or made you happy in any way?
+-------------------------------------------------
+
+<!--
+Answering this question is not required, but if you have anything positive to share, please do so here!
+Sometimes we get tired of reading bug reports all day and a little positive end note does wonders.
+Idea by Joey Hess, https://joeyh.name/blog/entry/two_holiday_stories/
+-->

.github/ISSUE_TEMPLATE/Feature.md

@@ -0,0 +1,57 @@
+---
+name: Feature request
+about: Suggest a new feature or enhancement for restic
+---
+
+<!--
+
+Welcome! - We kindly ask that you:
+
+  1. Fill out the issue template below - not doing so needs a good reason.
+  2. Use the forum if you have a question rather than a bug or feature request.
+
+The forum is at: https://forum.restic.net
+
+The forum is a better place for questions about restic or general suggestions
+and topics, e.g. usage or documentation questions! This issue tracker is mainly
+for tracking bugs and feature requests directly relating to the development of
+the software itself, rather than the project.
+
+Thanks for understanding, and for contributing to the project!
+
+-->
+
+
+Output of `restic version`
+--------------------------
+
+<!--
+Please add the version of restic you're currently using here, this helps us
+later to see what has changed in restic when we revisit this issue after some
+time.
+-->
+
+What should restic do differently? Which functionality do you think we should add?
+----------------------------------------------------------------------------------
+
+<!--
+Please describe the feature you'd like us to add here.
+-->
+
+
+What are you trying to do?
+--------------------------
+
+<!--
+This section should contain a brief description what you're trying to do, which
+would be possible after implementing the new feature.
+-->
+
+Did restic help you or made you happy in any way?
+-------------------------------------------------
+
+<!--
+Answering this question is not required, but if you have anything positive to share, please do so here!
+Sometimes we get tired of reading bug reports all day and a little positive end note does wonders.
+Idea by Joey Hess, https://joeyh.name/blog/entry/two_holiday_stories/
+-->

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,3 +1,5 @@
+
+
 <!--
 Thank you very much for contributing code or documentation to restic! Please
 fill out the following questions to make it easier for us to review your
@@ -8,24 +10,27 @@ your time and add more commits. If you're done and ready for review, please
 check the last box.
 -->
 
-### What is the purpose of this change? What does it change?
+What is the purpose of this change? What does it change?
+--------------------------------------------------------
 
 <!--
 Describe the changes here, as detailed as needed.
 -->
 
-### Was the change discussed in an issue or in the forum before?
+Was the change discussed in an issue or in the forum before?
+------------------------------------------------------------
 
 <!--
 Link issues and relevant forum posts here.
 -->
 
-### Checklist
+Checklist
+---------
 
 - [ ] I have read the [Contribution Guidelines](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#providing-patches)
 - [ ] I have added tests for all changes in this PR
 - [ ] I have added documentation for the changes (in the manual)
-- [ ] There's a new file in a subdir of `changelog/x.y.z` that describe the changes for our users (template [here](https://github.com/restic/restic/blob/master/changelog/changelog-entry.tmpl))
+- [ ] There's a new file in `changelog/unreleased/` that describes the changes for our users (template [here](https://github.com/restic/restic/blob/master/changelog/TEMPLATE))
 - [ ] I have run `gofmt` on the code in all commits
 - [ ] All commit messages are formatted in the same style as [the other commits in the repo](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#git-commits)
 - [ ] I'm done, this Pull Request is ready for review

.travis.yml

@@ -1,33 +1,20 @@
 language: go
 sudo: false
 
-go:
-  - "1.8.x"
-  - "1.9.x"
-  - "1.10"
-
-os:
-  - linux
-  - osx
-
-env:
-  matrix:
-    RESTIC_TEST_FUSE=0
-
 matrix:
-  exclude:
-    - os: osx
-      go: "1.8.x"
-    - os: osx
-      go: "1.9.x"
-    - os: linux
-      go: "1.10"
   include:
     - os: linux
-      go: "1.10"
+      go: "1.9.x"
+      env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0 RESTIC_BUILD_SOLARIS=0
+
+    # only run fuse and cloud backends tests on Travis for the latest Go on Linux
+    - os: linux
+      go: "1.10.x"
       sudo: true
-      env:
-        RESTIC_TEST_FUSE=1
+
+    - os: osx
+      go: "1.10.x"
+      env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0
 
 branches:
   only:

CHANGELOG.md

@@ -1,3 +1,532 @@
+Changelog for restic 0.9.2 (2018-08-06)
+=======================================
+
+The following sections list the changes in restic 0.9.2 relevant to
+restic users. The changes are ordered by importance.
+
+Summary
+-------
+
+ * Fix #1854: Allow saving files/dirs on different fs with `--one-file-system`
+ * Fix #1870: Fix restore with --include
+ * Fix #1880: Use `--cache-dir` argument for `check` command
+ * Fix #1893: Return error when exclude file cannot be read
+ * Fix #1861: Fix case-insensitive search with restic find
+ * Enh #1906: Add support for B2 application keys
+ * Enh #874: Add stats command to get information about a repository
+ * Enh #1772: Add restore --verify to verify restored file content
+ * Enh #1853: Add JSON output support to `restic key list`
+ * Enh #1477: S3 backend: accept AWS_SESSION_TOKEN
+ * Enh #1901: Update the Backblaze B2 library
+
+Details
+-------
+
+ * Bugfix #1854: Allow saving files/dirs on different fs with `--one-file-system`
+
+   Restic now allows saving files/dirs on a different file system in a subdir correctly even when
+   `--one-file-system` is specified.
+
+   The first thing the restic archiver code does is to build a tree of the target
+   files/directories. If it detects that a parent directory is already included (e.g. `restic
+   backup /foo /foo/bar/baz`), it'll ignore the latter argument.
+
+   Without `--one-file-system`, that's perfectly valid: If `/foo` is to be archived, it will
+   include `/foo/bar/baz`. But with `--one-file-system`, `/foo/bar/baz` may reside on a
+   different file system, so it won't be included with `/foo`.
+
+   https://github.com/restic/restic/issues/1854
+   https://github.com/restic/restic/pull/1855
+
+ * Bugfix #1870: Fix restore with --include
+
+   We fixed a bug which prevented restic to restore files with an include filter.
+
+   https://github.com/restic/restic/issues/1870
+   https://github.com/restic/restic/pull/1900
+
+ * Bugfix #1880: Use `--cache-dir` argument for `check` command
+
+   `check` command now uses a temporary sub-directory of the specified directory if set using the
+   `--cache-dir` argument. If not set, the cache directory is created in the default temporary
+   directory as before. In either case a temporary cache is used to ensure the actual repository is
+   checked (rather than a local copy).
+
+   The `--cache-dir` argument was not used by the `check` command, instead a cache directory was
+   created in the temporary directory.
+
+   https://github.com/restic/restic/issues/1880
+
+ * Bugfix #1893: Return error when exclude file cannot be read
+
+   A bug was found: when multiple exclude files were passed to restic and one of them could not be
+   read, an error was printed and restic continued, ignoring even the existing exclude files.
+   Now, an error message is printed and restic aborts when an exclude file cannot be read.
+
+   https://github.com/restic/restic/issues/1893
+
+ * Bugfix #1861: Fix case-insensitive search with restic find
+
+   We've fixed the behavior for `restic find -i PATTERN`, which was broken in v0.9.1.
+
+   https://github.com/restic/restic/pull/1861
+
+ * Enhancement #1906: Add support for B2 application keys
+
+   Restic can now use so-called "application keys" which can be created in the B2 dashboard and
+   were only introduced recently. In contrast to the "master key", such keys can be restricted to a
+   specific bucket and/or path.
+
+   https://github.com/restic/restic/issues/1906
+   https://github.com/restic/restic/pull/1914
+
+ * Enhancement #874: Add stats command to get information about a repository
+
+   https://github.com/restic/restic/issues/874
+   https://github.com/restic/restic/pull/1729
+
+ * Enhancement #1772: Add restore --verify to verify restored file content
+
+   Restore will print error message if restored file content does not match expected SHA256
+   checksum
+
+   https://github.com/restic/restic/pull/1772
+
+ * Enhancement #1853: Add JSON output support to `restic key list`
+
+   This PR enables users to get the output of `restic key list` in JSON in addition to the existing
+   table format.
+
+   https://github.com/restic/restic/pull/1853
+
+ * Enhancement #1477: S3 backend: accept AWS_SESSION_TOKEN
+
+   Before, it was not possible to use s3 backend with AWS temporary security credentials(with
+   AWS_SESSION_TOKEN). This change gives higher priority to credentials.EnvAWS credentials
+   provider.
+
+   https://github.com/restic/restic/issues/1477
+   https://github.com/restic/restic/pull/1479
+   https://github.com/restic/restic/pull/1647
+
+ * Enhancement #1901: Update the Backblaze B2 library
+
+   We've updated the library we're using for accessing the Backblaze B2 service to 0.5.0 to
+   include support for upcoming so-called "application keys". With this feature, you can create
+   access credentials for B2 which are restricted to e.g. a single bucket or even a sub-directory
+   of a bucket.
+
+   https://github.com/restic/restic/pull/1901
+   https://github.com/kurin/blazer
+
+
+Changelog for restic 0.9.1 (2018-06-10)
+=======================================
+
+The following sections list the changes in restic 0.9.1 relevant to
+restic users. The changes are ordered by importance.
+
+Summary
+-------
+
+ * Fix #1801: Add limiting bandwidth to the rclone backend
+ * Fix #1822: Allow uploading large files to MS Azure
+ * Fix #1825: Correct `find` to not skip snapshots
+ * Fix #1833: Fix caching files on error
+ * Fix #1834: Resolve deadlock
+
+Details
+-------
+
+ * Bugfix #1801: Add limiting bandwidth to the rclone backend
+
+   The rclone backend did not respect `--limit-upload` or `--limit-download`. Oftentimes it's
+   not necessary to use this, as the limiting in rclone itself should be used because it gives much
+   better results, but in case a remote instance of rclone is used (e.g. called via ssh), it is still
+   relevant to limit the bandwidth from restic to rclone.
+
+   https://github.com/restic/restic/issues/1801
+
+ * Bugfix #1822: Allow uploading large files to MS Azure
+
+   Sometimes, restic creates files to be uploaded to the repository which are quite large, e.g.
+   when saving directories with many entries or very large files. The MS Azure API does not allow
+   uploading files larger that 256MiB directly, rather restic needs to upload them in blocks of
+   100MiB. This is now implemented.
+
+   https://github.com/restic/restic/issues/1822
+
+ * Bugfix #1825: Correct `find` to not skip snapshots
+
+   Under certain circumstances, the `find` command was found to skip snapshots containing
+   directories with files to look for when the directories haven't been modified at all, and were
+   already printed as part of a different snapshot. This is now corrected.
+
+   In addition, we've switched to our own matching/pattern implementation, so now things like
+   `restic find "/home/user/foo/**/main.go"` are possible.
+
+   https://github.com/restic/restic/issues/1825
+   https://github.com/restic/restic/issues/1823
+
+ * Bugfix #1833: Fix caching files on error
+
+   During `check` it may happen that different threads access the same file in the backend, which
+   is then downloaded into the cache only once. When that fails, only the thread which is
+   responsible for downloading the file signals the correct error. The other threads just assume
+   that the file has been downloaded successfully and then get an error when they try to access the
+   cached file.
+
+   https://github.com/restic/restic/issues/1833
+
+ * Bugfix #1834: Resolve deadlock
+
+   When the "scanning" process restic runs to find out how much data there is does not finish before
+   the backup itself is done, restic stops doing anything. This is resolved now.
+
+   https://github.com/restic/restic/issues/1834
+   https://github.com/restic/restic/pull/1835
+
+
+Changelog for restic 0.9.0 (2018-05-21)
+=======================================
+
+The following sections list the changes in restic 0.9.0 relevant to
+restic users. The changes are ordered by importance.
+
+Summary
+-------
+
+ * Fix #1608: Respect time stamp for new backup when reading from stdin
+ * Fix #1652: Ignore/remove invalid lock files
+ * Fix #1730: Ignore sockets for restore
+ * Fix #1684: Fix backend tests for rest-server
+ * Fix #1745: Correctly parse the argument to --tls-client-cert
+ * Enh #1433: Support UTF-16 encoding and process Byte Order Mark
+ * Enh #1561: Allow using rclone to access other services
+ * Enh #1665: Improve cache handling for `restic check`
+ * Enh #1721: Add `cache` command to list cache dirs
+ * Enh #1758: Allow saving OneDrive folders in Windows
+ * Enh #549: Rework archiver code
+ * Enh #1552: Use Google Application Default credentials
+ * Enh #1477: Accept AWS_SESSION_TOKEN for the s3 backend
+ * Enh #1648: Ignore AWS permission denied error when creating a repository
+ * Enh #1649: Add illumos/Solaris support
+ * Enh #1709: Improve messages `restic check` prints
+ * Enh #827: Add --new-password-file flag for non-interactive password changes
+ * Enh #1735: Allow keeping a time range of snaphots
+ * Enh #1782: Use default AWS credentials chain for S3 backend
+
+Details
+-------
+
+ * Bugfix #1608: Respect time stamp for new backup when reading from stdin
+
+   When reading backups from stdin (via `restic backup --stdin`), restic now uses the time stamp
+   for the new backup passed in `--time`.
+
+   https://github.com/restic/restic/issues/1608
+   https://github.com/restic/restic/pull/1703
+
+ * Bugfix #1652: Ignore/remove invalid lock files
+
+   This corrects a bug introduced recently: When an invalid lock file in the repo is encountered
+   (e.g. if the file is empty), the code used to ignore that, but now returns the error. Now, invalid
+   files are ignored for the normal lock check, and removed when `restic unlock --remove-all` is
+   run.
+
+   https://github.com/restic/restic/issues/1652
+   https://github.com/restic/restic/pull/1653
+
+ * Bugfix #1730: Ignore sockets for restore
+
+   We've received a report and correct the behavior in which the restore code aborted restoring a
+   directory when a socket was encountered. Unix domain socket files cannot be restored (they are
+   created on the fly once a process starts listening). The error handling was corrected, and in
+   addition we're now ignoring sockets during restore.
+
+   https://github.com/restic/restic/issues/1730
+   https://github.com/restic/restic/pull/1731
+
+ * Bugfix #1684: Fix backend tests for rest-server
+
+   The REST server for restic now requires an explicit parameter (`--no-auth`) if no
+   authentication should be allowed. This is fixed in the tests.
+
+   https://github.com/restic/restic/pull/1684
+
+ * Bugfix #1745: Correctly parse the argument to --tls-client-cert
+
+   Previously, the --tls-client-cert method attempt to read ARGV[1] (hardcoded) instead of the
+   argument that was passed to it. This has been corrected.
+
+   https://github.com/restic/restic/issues/1745
+   https://github.com/restic/restic/pull/1746
+
+ * Enhancement #1433: Support UTF-16 encoding and process Byte Order Mark
+
+   On Windows, text editors commonly leave a Byte Order Mark at the beginning of the file to define
+   which encoding is used (oftentimes UTF-16). We've added code to support processing the BOMs in
+   text files, like the exclude files, the password file and the file passed via `--files-from`.
+   This does not apply to any file being saved in a backup, those are not touched and archived as they
+   are.
+
+   https://github.com/restic/restic/issues/1433
+   https://github.com/restic/restic/issues/1738
+   https://github.com/restic/restic/pull/1748
+
+ * Enhancement #1561: Allow using rclone to access other services
+
+   We've added the ability to use rclone to store backup data on all backends that it supports. This
+   was done in collaboration with Nick, the author of rclone. You can now use it to first configure a
+   service, then restic manages the rest (starting and stopping rclone). For details, please see
+   the manual.
+
+   https://github.com/restic/restic/issues/1561
+   https://github.com/restic/restic/pull/1657
+   https://rclone.org
+
+ * Enhancement #1665: Improve cache handling for `restic check`
+
+   For safety reasons, restic does not use a local metadata cache for the `restic check` command,
+   so that data is loaded from the repository and restic can check it's in good condition. When the
+   cache is disabled, restic will fetch each tiny blob needed for checking the integrity using a
+   separate backend request. For non-local backends, that will take a long time, and depending on
+   the backend (e.g. B2) may also be much more expensive.
+
+   This PR adds a few commits which will change the behavior as follows:
+
+   * When `restic check` is called without any additional parameters, it will build a new cache in a
+   temporary directory, which is removed at the end of the check. This way, we'll get readahead for
+   metadata files (so restic will fetch the whole file when the first blob from the file is
+   requested), but all data is freshly fetched from the storage backend. This is the default
+   behavior and will work for almost all users.
+
+   * When `restic check` is called with `--with-cache`, the default on-disc cache is used. This
+   behavior hasn't changed since the cache was introduced.
+
+   * When `--no-cache` is specified, restic falls back to the old behavior, and read all tiny blobs
+   in separate requests.
+
+   https://github.com/restic/restic/issues/1665
+   https://github.com/restic/restic/issues/1694
+   https://github.com/restic/restic/pull/1696
+
+ * Enhancement #1721: Add `cache` command to list cache dirs
+
+   The command `cache` was added, it allows listing restic's cache directoriers together with
+   the last usage. It also allows removing old cache dirs without having to access a repo, via
+   `restic cache --cleanup`
+
+   https://github.com/restic/restic/issues/1721
+   https://github.com/restic/restic/pull/1749
+
+ * Enhancement #1758: Allow saving OneDrive folders in Windows
+
+   Restic now contains a bugfix to two libraries, which allows saving OneDrive folders in
+   Windows. In order to use the newer versions of the libraries, the minimal version required to
+   compile restic is now Go 1.9.
+
+   https://github.com/restic/restic/issues/1758
+   https://github.com/restic/restic/pull/1765
+
+ * Enhancement #549: Rework archiver code
+
+   The core archiver code and the complementary code for the `backup` command was rewritten
+   completely. This resolves very annoying issues such as 549. The first backup with this release
+   of restic will likely result in all files being re-read locally, so it will take a lot longer. The
+   next backup after that will be fast again.
+
+   Basically, with the old code, restic took the last path component of each to-be-saved file or
+   directory as the top-level file/directory within the snapshot. This meant that when called as
+   `restic backup /home/user/foo`, the snapshot would contain the files in the directory
+   `/home/user/foo` as `/foo`.
+
+   This is not the case any more with the new archiver code. Now, restic works very similar to what
+   `tar` does: When restic is called with an absolute path to save, then it'll preserve the
+   directory structure within the snapshot. For the example above, the snapshot would contain
+   the files in the directory within `/home/user/foo` in the snapshot. For relative
+   directories, it only preserves the relative path components. So `restic backup user/foo`
+   will save the files as `/user/foo` in the snapshot.
+
+   While we were at it, the status display and notification system was completely rewritten. By
+   default, restic now shows which files are currently read (unless `--quiet` is specified) in a
+   multi-line status display.
+
+   The `backup` command also gained a new option: `--verbose`. It can be specified once (which
+   prints a bit more detail what restic is doing) or twice (which prints a line for each
+   file/directory restic encountered, together with some statistics).
+
+   Another issue that was resolved is the new code only reads two files at most. The old code would
+   read way too many files in parallel, thereby slowing down the backup process on spinning discs a
+   lot.
+
+   https://github.com/restic/restic/issues/549
+   https://github.com/restic/restic/issues/1286
+   https://github.com/restic/restic/issues/446
+   https://github.com/restic/restic/issues/1344
+   https://github.com/restic/restic/issues/1416
+   https://github.com/restic/restic/issues/1456
+   https://github.com/restic/restic/issues/1145
+   https://github.com/restic/restic/issues/1160
+   https://github.com/restic/restic/pull/1494
+
+ * Enhancement #1552: Use Google Application Default credentials
+
+   Google provide libraries to generate appropriate credentials with various fallback
+   sources. This change uses the library to generate our GCS client, which allows us to make use of
+   these extra methods.
+
+   This should be backward compatible with previous restic behaviour while adding the
+   additional capabilities to auth from Google's internal metadata endpoints. For users
+   running restic in GCP this can make authentication far easier than it was before.
+
+   https://github.com/restic/restic/pull/1552
+   https://developers.google.com/identity/protocols/application-default-credentials
+
+ * Enhancement #1477: Accept AWS_SESSION_TOKEN for the s3 backend
+
+   Before, it was not possible to use s3 backend with AWS temporary security credentials(with
+   AWS_SESSION_TOKEN). This change gives higher priority to credentials.EnvAWS credentials
+   provider.
+
+   https://github.com/restic/restic/issues/1477
+   https://github.com/restic/restic/pull/1479
+   https://github.com/restic/restic/pull/1647
+
+ * Enhancement #1648: Ignore AWS permission denied error when creating a repository
+
+   It's not possible to use s3 backend scoped to a subdirectory(with specific permissions).
+   Restic doesn't try to create repository in a subdirectory, when 'bucket exists' of parent
+   directory check fails due to permission issues.
+
+   https://github.com/restic/restic/pull/1648
+
+ * Enhancement #1649: Add illumos/Solaris support
+
+   https://github.com/restic/restic/pull/1649
+
+ * Enhancement #1709: Improve messages `restic check` prints
+
+   Some messages `restic check` prints are not really errors, so from now on restic does not treat
+   them as errors any more and exits cleanly.
+
+   https://github.com/restic/restic/pull/1709
+   https://forum.restic.net/t/what-is-the-standard-procedure-to-follow-if-a-backup-or-restore-is-interrupted/571/2
+
+ * Enhancement #827: Add --new-password-file flag for non-interactive password changes
+
+   This makes it possible to change a repository password without being prompted.
+
+   https://github.com/restic/restic/issues/827
+   https://github.com/restic/restic/pull/1720
+   https://forum.restic.net/t/changing-repo-password-without-prompt/591
+
+ * Enhancement #1735: Allow keeping a time range of snaphots
+
+   We've added the `--keep-within` option to the `forget` command. It instructs restic to keep
+   all snapshots within the given duration since the newest snapshot. For example, running
+   `restic forget --keep-within 5m7d` will keep all snapshots which have been made in the five
+   months and seven days since the latest snapshot.
+
+   https://github.com/restic/restic/pull/1735
+
+ * Enhancement #1782: Use default AWS credentials chain for S3 backend
+
+   Adds support for file credentials to the S3 backend (e.g. ~/.aws/credentials), and reorders
+   the credentials chain for the S3 backend to match AWS's standard, which is static credentials,
+   env vars, credentials file, and finally remote.
+
+   https://github.com/restic/restic/pull/1782
+
+
+Changelog for restic 0.8.3 (2018-02-26)
+=======================================
+
+The following sections list the changes in restic 0.8.3 relevant to
+restic users. The changes are ordered by importance.
+
+Summary
+-------
+
+ * Fix #1633: Fixed unexpected 'pack file cannot be listed' error
+ * Fix #1641: Ignore files with invalid names in the repo
+ * Fix #1638: Handle errors listing files in the backend
+ * Enh #1497: Add --read-data-subset flag to check command
+ * Enh #1560: Retry all repository file download errors
+ * Enh #1623: Don't check for presence of files in the backend before writing
+ * Enh #1634: Upgrade B2 client library, reduce HTTP requests
+
+Details
+-------
+
+ * Bugfix #1633: Fixed unexpected 'pack file cannot be listed' error
+
+   Due to a regression introduced in 0.8.2, the `rebuild-index` and `prune` commands failed to
+   read pack files with size of 587, 588, 589 or 590 bytes.
+
+   https://github.com/restic/restic/issues/1633
+   https://github.com/restic/restic/pull/1635
+
+ * Bugfix #1641: Ignore files with invalid names in the repo
+
+   The release 0.8.2 introduced a bug: when restic encounters files in the repo which do not have a
+   valid name, it tries to load a file with a name of lots of zeroes instead of ignoring it. This is now
+   resolved, invalid file names are just ignored.
+
+   https://github.com/restic/restic/issues/1641
+   https://github.com/restic/restic/pull/1643
+   https://forum.restic.net/t/help-fixing-repo-no-such-file/485/3
+
+ * Bugfix #1638: Handle errors listing files in the backend
+
+   A user reported in the forum that restic completes a backup although a concurrent `prune`
+   operation was running. A few error messages were printed, but the backup was attempted and
+   completed successfully. No error code was returned.
+
+   This should not happen: The repository is exclusively locked during `prune`, so when `restic
+   backup` is run in parallel, it should abort and return an error code instead.
+
+   It was found that the bug was in the code introduced only recently, which retries a List()
+   operation on the backend should that fail. It is now corrected.
+
+   https://github.com/restic/restic/pull/1638
+   https://forum.restic.net/t/restic-backup-returns-0-exit-code-when-already-locked/484
+
+ * Enhancement #1497: Add --read-data-subset flag to check command
+
+   This change introduces ability to check integrity of a subset of repository data packs. This
+   can be used to spread integrity check of larger repositories over a period of time.
+
+   https://github.com/restic/restic/issues/1497
+   https://github.com/restic/restic/pull/1556
+
+ * Enhancement #1560: Retry all repository file download errors
+
+   Restic will now retry failed downloads, similar to other operations.
+
+   https://github.com/restic/restic/pull/1560
+
+ * Enhancement #1623: Don't check for presence of files in the backend before writing
+
+   Before, all backend implementations were required to return an error if the file that is to be
+   written already exists in the backend. For most backends, that means making a request (e.g. via
+   HTTP) and returning an error when the file already exists.
+
+   This is not accurate, the file could have been created between the HTTP request testing for it,
+   and when writing starts, so we've relaxed this requeriment, which saves one additional HTTP
+   request per newly added file.
+
+   https://github.com/restic/restic/pull/1623
+
+ * Enhancement #1634: Upgrade B2 client library, reduce HTTP requests
+
+   We've upgraded the B2 client library restic uses to access BackBlaze B2. This reduces the
+   number of HTTP requests needed to upload a new file from two to one, which should improve
+   throughput to B2.
+
+   https://github.com/restic/restic/pull/1634
+
+
 Changelog for restic 0.8.2 (2018-02-17)
 =======================================
 
@@ -69,6 +598,7 @@ Details
    of data loss, just minor inconvenience for our users.
 
    https://github.com/restic/restic/pull/1589
+   https://forum.restic.net/t/error-loading-tree-check-prune-and-forget-gives-error-b2-backend/406
 
  * Bugfix #1594: Google Cloud Storage: Use generic HTTP transport
 
@@ -607,7 +1137,7 @@ Details
  * Enhancement #1203: Print stats on all BSD systems when SIGINFO (ctrl+t) is received
 
    https://github.com/restic/restic/pull/1203
-   https://github.com/restic/restic/pull/1082
+   https://github.com/restic/restic/pull/1082#issuecomment-326279920
 
  * Enhancement #1205: Allow specifying time/date for a backup with `--time`
 
@@ -647,12 +1177,12 @@ Details
  * Enhancement #1055: Create subdirs below `data/` for local/sftp backends
 
    The local and sftp backends now create the subdirs below `data/` on open/init. This way, restic
-   makes sure that they always exist. This is connected to an issue for the sftp server:
+   makes sure that they always exist. This is connected to an issue for the sftp server.
 
    https://github.com/restic/restic/issues/1055
+   https://github.com/restic/rest-server/pull/11#issuecomment-309879710
    https://github.com/restic/restic/pull/1077
    https://github.com/restic/restic/pull/1105
-   https://github.com/restic/rest-server/pull/11#issuecomment-309879710
 
  * Enhancement #1067: Allow loading credentials for s3 from IAM
 
@@ -664,7 +1194,7 @@ Details
 
  * Enhancement #1073: Add `migrate` cmd to migrate from `s3legacy` to `default` layout
 
-   The `migrate` command for chaning the `s3legacy` layout to the `default` layout for s3
+   The `migrate` command for changing the `s3legacy` layout to the `default` layout for s3
    backends has been improved: It can now be restarted with `restic migrate --force s3_layout`
    and automatically retries operations on error.
 

CONTRIBUTING.md

@@ -164,7 +164,7 @@ history and triaging bugs much easier.
 Git commit messages have a very terse summary in the first line of the commit
 message, followed by an empty line, followed by a more verbose description or a
 List of changed things. For examples, please refer to the excellent [How to
-Write a Git Commit Message](http://chris.beams.io/posts/git-commit/).
+Write a Git Commit Message](https://chris.beams.io/posts/git-commit/).
 
 If you change/add multiple different things that aren't related at all, try to
 make several smaller commits. This is much easier to review. Using `git add -p`

GOVERNANCE.md

@@ -0,0 +1,27 @@
+# restic project governance
+
+## Overview
+
+The restic project uses a governance model commonly described as Benevolent
+Dictator For Life (BDFL). This document outlines our understanding of what this
+means. It is derived from the [i3 window manager project
+governance](https://raw.githubusercontent.com/i3/i3/next/.github/GOVERNANCE.md).
+
+## Roles
+
+* user: anyone who interacts with the restic project
+* core contributor: a handful of people who have contributed significantly to
+  the project by any means (issue triage, support, documentation, code, etc.).
+  Core contributors are recognizable via GitHub’s "Member" badge.
+* Benevolent Dictator For Life (BDFL): a single individual who makes decisions
+  when consensus cannot be reached. restic's current BDFL is [@fd0](https://github.com/fd0).
+
+## Decision making process
+
+In general, we try to reach consensus in discussions. In case consensus cannot
+be reached, the BDFL makes a decision.
+
+## Contribution process
+
+The contribution process is described in a separate document called
+[CONTRIBUTING](CONTRIBUTING.md).

Gopkg.lock

@@ -3,234 +3,472 @@
 
 [[projects]]
   branch = "master"
+  digest = "1:94e9caf404409a2990cfd22aca37d758494c098eff3e2c37fda1abed862e74dd"
   name = "bazil.org/fuse"
-  packages = [".","fs","fuseutil"]
-  revision = "371fbbdaa8987b715bdd21d6adc4c9b20155f748"
+  packages = [
+    ".",
+    "fs",
+    "fuseutil",
+  ]
+  pruneopts = "UT"
+  revision = "65cc252bf6691cb3c7014bcb2c8dc29de91e3a7e"
 
 [[projects]]
+  digest = "1:5c3894b2aa4d6bead0ceeea6831b305d62879c871780e7b76296ded1b004bc57"
   name = "cloud.google.com/go"
   packages = ["compute/metadata"]
-  revision = "767c40d6a2e058483c25fa193e963a22da17236d"
-  version = "v0.18.0"
+  pruneopts = "UT"
+  revision = "aad3f485ee528456e0768f20397b4d9dd941e755"
+  version = "v0.25.0"
 
 [[projects]]
+  digest = "1:46ea9487304f4b3c787f54483ecb13a338d686dcd670db0ab1a112ed0ae2128e"
   name = "github.com/Azure/azure-sdk-for-go"
-  packages = ["storage"]
-  revision = "eae258195456be76b2ec9ad2ee2ab63cdda365d9"
-  version = "v12.2.0-beta"
+  packages = [
+    "storage",
+    "version",
+  ]
+  pruneopts = "UT"
+  revision = "4e8cbbfb1aeab140cd0fa97fd16b64ee18c3ca6a"
+  version = "v19.1.0"
 
 [[projects]]
+  digest = "1:27d0cd1a78fc836f7c0f07749d029a5f7895c84ad066187b08b70e9d1830098e"
   name = "github.com/Azure/go-autorest"
-  packages = ["autorest","autorest/adal","autorest/azure","autorest/date"]
-  revision = "c2a68353555b68de3ee8455a4fd3e890a0ac6d99"
-  version = "v9.8.1"
+  packages = [
+    "autorest",
+    "autorest/adal",
+    "autorest/azure",
+    "autorest/date",
+    "logger",
+    "version",
+  ]
+  pruneopts = "UT"
+  revision = "dd94e014aaf16d1df746762e392aa201c1b4c461"
+  version = "v10.15.0"
 
 [[projects]]
+  digest = "1:2209584c0f7c9b68c23374e659357ab546e1b70eec2761f03280f69a8fd23d77"
   name = "github.com/cenkalti/backoff"
   packages = ["."]
-  revision = "61153c768f31ee5f130071d08fc82b85208528de"
+  pruneopts = "UT"
+  revision = "2ea60e5f094469f9e65adb9cd103795b73ae743e"
+  version = "v2.0.0"
+
+[[projects]]
+  digest = "1:7cb4fdca4c251b3ef8027c90ea35f70c7b661a593b9eeae34753c65499098bb1"
+  name = "github.com/cpuguy83/go-md2man"
+  packages = ["md2man"]
+  pruneopts = "UT"
+  revision = "20f5889cbdc3c73dbd2862796665e7c465ade7d1"
+  version = "v1.0.8"
+
+[[projects]]
+  digest = "1:76dc72490af7174349349838f2fe118996381b31ea83243812a97e5a0fd5ed55"
+  name = "github.com/dgrijalva/jwt-go"
+  packages = ["."]
+  pruneopts = "UT"
+  revision = "06ea1031745cb8b3dab3f6a236daf2b0aa468b7e"
+  version = "v3.2.0"
+
+[[projects]]
+  branch = "master"
+  digest = "1:6f9339c912bbdda81302633ad7e99a28dfa5a639c864061f1929510a9a64aa74"
+  name = "github.com/dustin/go-humanize"
+  packages = ["."]
+  pruneopts = "UT"
+  revision = "9f541cc9db5d55bce703bd99987c9d5cb8eea45e"
+
+[[projects]]
+  digest = "1:c7edfbb6320d6a93240d663dc52bca92bed4c116abe54c35679eec4e7cc2bd77"
+  name = "github.com/elithrar/simple-scrypt"
+  packages = ["."]
+  pruneopts = "UT"
+  revision = "d150773194090feb6c897805a7bcea8d49544e2c"
+  version = "v1.3.0"
+
+[[projects]]
+  digest = "1:fe8a03a8222d5b913f256972933d26d24ad7c8286692a42943bc01633cc8fce3"
+  name = "github.com/go-ini/ini"
+  packages = ["."]
+  pruneopts = "UT"
+  revision = "358ee7663966325963d4e8b2e1fbd570c5195153"
+  version = "v1.38.1"
+
+[[projects]]
+  digest = "1:15042ad3498153684d09f393bbaec6b216c8eec6d61f63dff711de7d64ed8861"
+  name = "github.com/golang/protobuf"
+  packages = ["proto"]
+  pruneopts = "UT"
+  revision = "b4deda0973fb4c70b50d226b1af49f3da59f5265"
   version = "v1.1.0"
 
 [[projects]]
-  name = "github.com/cpuguy83/go-md2man"
-  packages = ["md2man"]
-  revision = "1d903dcb749992f3741d744c0f8376b4bd7eb3e1"
-  version = "v1.0.7"
-
-[[projects]]
-  name = "github.com/dgrijalva/jwt-go"
-  packages = ["."]
-  revision = "dbeaa9332f19a944acb5736b4456cfcc02140e29"
-  version = "v3.1.0"
-
-[[projects]]
-  branch = "master"
-  name = "github.com/dustin/go-humanize"
-  packages = ["."]
-  revision = "bb3d318650d48840a39aa21a027c6630e198e626"
-
-[[projects]]
-  name = "github.com/elithrar/simple-scrypt"
-  packages = ["."]
-  revision = "2325946f714c95de4a6088202c402fbdfa64163b"
-  version = "v1.2.0"
-
-[[projects]]
-  name = "github.com/go-ini/ini"
-  packages = ["."]
-  revision = "32e4c1e6bc4e7d0d8451aa6b75200d19e37a536a"
-  version = "v1.32.0"
-
-[[projects]]
-  branch = "master"
-  name = "github.com/golang/protobuf"
-  packages = ["proto"]
-  revision = "c65a0412e71e8b9b3bfd22925720d23c0f054237"
+  digest = "1:2e3c336fc7fde5c984d2841455a658a6d626450b1754a854b3b32e7a8f49a07a"
+  name = "github.com/google/go-cmp"
+  packages = [
+    "cmp",
+    "cmp/internal/diff",
+    "cmp/internal/function",
+    "cmp/internal/value",
+  ]
+  pruneopts = "UT"
+  revision = "3af367b6b30c263d47e8895973edcca9a49cf029"
+  version = "v0.2.0"
 
 [[projects]]
+  digest = "1:870d441fe217b8e689d7949fef6e43efbc787e50f200cb1e70dbca9204a1d6be"
   name = "github.com/inconshreveable/mousetrap"
   packages = ["."]
+  pruneopts = "UT"
   revision = "76626ae9c91c4f2a10f34cad8ce83ea42c93bb75"
   version = "v1.0"
 
 [[projects]]
+  digest = "1:190ff84d9b2ed6589088f178cba8edb4b8ecb334df4572421fb016be1ac20463"
   name = "github.com/juju/ratelimit"
   packages = ["."]
+  pruneopts = "UT"
   revision = "59fac5042749a5afb9af70e813da1dd5474f0167"
   version = "1.0.1"
 
 [[projects]]
-  branch = "master"
+  digest = "1:9cedee824c21326bd26950bd9e1ffe9dc4e7ca03dc8634d0e6f954ee6a383172"
   name = "github.com/kr/fs"
   packages = ["."]
-  revision = "2788f0dbd16903de03cb8186e5c7d97b69ad387b"
+  pruneopts = "UT"
+  revision = "1455def202f6e05b95cc7bfc7e8ae67ae5141eba"
+  version = "v0.1.0"
 
 [[projects]]
+  digest = "1:1faa76bd9bffce9c25eaca0597afb67bd05a21ae57fe4378154ce8855ef163d1"
   name = "github.com/kurin/blazer"
-  packages = ["b2","base","internal/b2types","internal/blog"]
-  revision = "e269a1a17bb6aec278c06a57cb7e8f8d0d333e04"
-  version = "v0.2.1"
+  packages = [
+    "b2",
+    "base",
+    "internal/b2assets",
+    "internal/b2types",
+    "internal/blog",
+    "x/window",
+  ]
+  pruneopts = "UT"
+  revision = "caf65aa76491dc533bac68ad3243ce72fa4e0a0a"
+  version = "v0.5.1"
 
 [[projects]]
+  digest = "1:4e878df5f4e9fd625bf9c9aac77ef7cbfa4a74c01265505527c23470c0e40300"
   name = "github.com/marstr/guid"
   packages = ["."]
-  revision = "8bdf7d1a087ccc975cf37dd6507da50698fd19ca"
+  pruneopts = "UT"
+  revision = "8bd9a64bf37eb297b492a4101fb28e80ac0b290f"
+  version = "v1.1.0"
 
 [[projects]]
+  digest = "1:d4d17353dbd05cb52a2a52b7fe1771883b682806f68db442b436294926bbfafb"
+  name = "github.com/mattn/go-isatty"
+  packages = ["."]
+  pruneopts = "UT"
+  revision = "0360b2af4f38e8d38c7fce2a9f4e702702d73a39"
+  version = "v0.0.3"
+
+[[projects]]
+  digest = "1:95c73c666919be2843b955eafc83f58c136312b74f79c703152f4c4a95fd64dc"
   name = "github.com/minio/minio-go"
-  packages = [".","pkg/credentials","pkg/encrypt","pkg/policy","pkg/s3signer","pkg/s3utils","pkg/set"]
-  revision = "14f1d472d115bac5ca4804094aa87484a72ced61"
-  version = "4.0.6"
+  packages = [
+    ".",
+    "pkg/credentials",
+    "pkg/encrypt",
+    "pkg/s3signer",
+    "pkg/s3utils",
+    "pkg/set",
+  ]
+  pruneopts = "UT"
+  revision = "70799fe8dae6ecfb6c7d7e9e048fce27f23a1992"
+  version = "v6.0.5"
 
 [[projects]]
   branch = "master"
+  digest = "1:8eb17c2ec4df79193ae65b621cd1c0c4697db3bc317fe6afdc76d7f2746abd05"
   name = "github.com/mitchellh/go-homedir"
   packages = ["."]
-  revision = "b8bc1bf767474819792c23f32d8286a45736f1c6"
+  pruneopts = "UT"
+  revision = "3864e76763d94a6df2f9960b16a20a33da9f9a66"
 
 [[projects]]
-  branch = "master"
+  digest = "1:928de5172dd3563964d1b88a4ee3775cf72e16f1efabb482ab6d0e0bab86ee69"
   name = "github.com/ncw/swift"
   packages = ["."]
-  revision = "ae9f0ea1605b9aa6434ed5c731ca35d83ba67c55"
+  pruneopts = "UT"
+  revision = "b2a7479cf26fa841ff90dd932d0221cb5c50782d"
+  version = "v1.0.39"
 
 [[projects]]
+  digest = "1:40e195917a951a8bf867cd05de2a46aaf1806c50cf92eebf4c16f78cd196f747"
   name = "github.com/pkg/errors"
   packages = ["."]
+  pruneopts = "UT"
   revision = "645ef00459ed84a119197bfb8d8205042c6df63d"
   version = "v0.8.0"
 
 [[projects]]
+  digest = "1:cfa0d7741863a0e1d30e0ccdd4b48a96a471cdb47892303de8b92c3713af3e77"
   name = "github.com/pkg/profile"
   packages = ["."]
+  pruneopts = "UT"
   revision = "5b67d428864e92711fcbd2f8629456121a56d91f"
   version = "v1.2.1"
 
 [[projects]]
+  digest = "1:23ed92ba5d90a2dfe817f3895027ccef796e79c30be5125d48e17afdcc395d73"
   name = "github.com/pkg/sftp"
   packages = ["."]
-  revision = "f6a9258a0f570c3a76681b897b6ded57cb0dfa88"
-  version = "1.2.0"
+  pruneopts = "UT"
+  revision = "57673e38ea946592a59c26592b7e6fbda646975b"
+  version = "v1.8.0"
 
 [[projects]]
+  digest = "1:0d67664e93e366f072ac9672feea29bfc63c9f90f005e9e8a0df1954153f5a14"
   name = "github.com/pkg/xattr"
   packages = ["."]
-  revision = "23c75e3f6c1d8b13b3dd905b011a7f38a06044b7"
-  version = "v0.2.1"
+  pruneopts = "UT"
+  revision = "ae385d07bb53f092fcc7daaf738d8513df084931"
+  version = "v0.3.1"
 
 [[projects]]
+  digest = "1:13ecc4000f49cf0aa3ee56fffcc93119c8edffacfff08674c80d2757d8c33a83"
   name = "github.com/restic/chunker"
   packages = ["."]
+  pruneopts = "UT"
   revision = "db83917be3b88cc307464b7d8a221c173e34a0db"
   version = "v0.2.0"
 
 [[projects]]
+  digest = "1:8bc629776d035c003c7814d4369521afe67fdb8efc4b5f66540d29343b98cf23"
   name = "github.com/russross/blackfriday"
   packages = ["."]
-  revision = "4048872b16cc0fc2c5fd9eacf0ed2c2fedaa0c8c"
-  version = "v1.5"
+  pruneopts = "UT"
+  revision = "55d61fa8aa702f59229e6cff85793c22e580eaf5"
+  version = "v1.5.1"
 
 [[projects]]
+  digest = "1:274f67cb6fed9588ea2521ecdac05a6d62a8c51c074c1fccc6a49a40ba80e925"
   name = "github.com/satori/go.uuid"
   packages = ["."]
+  pruneopts = "UT"
   revision = "f58768cc1a7a7e77a3bd49e98cdd21419399b6a3"
   version = "v1.2.0"
 
 [[projects]]
+  digest = "1:d867dfa6751c8d7a435821ad3b736310c2ed68945d05b50fb9d23aee0540c8cc"
   name = "github.com/sirupsen/logrus"
   packages = ["."]
-  revision = "d682213848ed68c0a260ca37d6dd5ace8423f5ba"
-  version = "v1.0.4"
+  pruneopts = "UT"
+  revision = "3e01752db0189b9157070a0e1668a620f9a85da2"
+  version = "v1.0.6"
 
 [[projects]]
+  digest = "1:e01b05ba901239c783dfe56450bcde607fc858908529868259c9a8765dc176d0"
   name = "github.com/spf13/cobra"
-  packages = [".","doc"]
-  revision = "7b2c5ac9fc04fc5efafb60700713d4fa609b777b"
-  version = "v0.0.1"
+  packages = [
+    ".",
+    "doc",
+  ]
+  pruneopts = "UT"
+  revision = "ef82de70bb3f60c65fb8eebacbb2d122ef517385"
+  version = "v0.0.3"
 
 [[projects]]
+  digest = "1:9424f440bba8f7508b69414634aef3b2b3a877e522d8a4624692412805407bb7"
   name = "github.com/spf13/pflag"
   packages = ["."]
-  revision = "e57e3eeb33f795204c1ca35f56c44f83227c6e66"
-  version = "v1.0.0"
+  pruneopts = "UT"
+  revision = "583c0c0531f06d5278b7d917446061adc344b5cd"
+  version = "v1.0.1"
 
 [[projects]]
   branch = "master"
+  digest = "1:eefb1f49ec07e71206d4c9ea1a3e634cad331c2180733e4121b8ae39e8e92ecb"
   name = "golang.org/x/crypto"
-  packages = ["curve25519","ed25519","ed25519/internal/edwards25519","internal/chacha20","pbkdf2","poly1305","scrypt","ssh","ssh/terminal"]
-  revision = "3d37316aaa6bd9929127ac9a527abf408178ea7b"
+  packages = [
+    "argon2",
+    "blake2b",
+    "curve25519",
+    "ed25519",
+    "ed25519/internal/edwards25519",
+    "internal/chacha20",
+    "internal/subtle",
+    "pbkdf2",
+    "poly1305",
+    "scrypt",
+    "ssh",
+    "ssh/terminal",
+  ]
+  pruneopts = "UT"
+  revision = "c126467f60eb25f8f27e5a981f32a87e3965053f"
 
 [[projects]]
   branch = "master"
+  digest = "1:8356aa7bdcb10a210b814b64ff76d61de7c36ac4cb6263de3af5e3e2e546956d"
   name = "golang.org/x/net"
-  packages = ["context","context/ctxhttp","idna","lex/httplex"]
-  revision = "5ccada7d0a7ba9aeb5d3aca8d3501b4c2a509fec"
+  packages = [
+    "context",
+    "context/ctxhttp",
+    "http/httpguts",
+    "http2",
+    "http2/hpack",
+    "idna",
+  ]
+  pruneopts = "UT"
+  revision = "32f9bdbd7df18e8641d215e7ea68be88b971feb0"
 
 [[projects]]
   branch = "master"
+  digest = "1:bea0314c10bd362ab623af4880d853b5bad3b63d0ab9945c47e461b8d04203ed"
   name = "golang.org/x/oauth2"
-  packages = [".","google","internal","jws","jwt"]
-  revision = "b28fcf2b08a19742b43084fb40ab78ac6c3d8067"
+  packages = [
+    ".",
+    "google",
+    "internal",
+    "jws",
+    "jwt",
+  ]
+  pruneopts = "UT"
+  revision = "3d292e4d0cdc3a0113e6d207bb137145ef1de42f"
 
 [[projects]]
   branch = "master"
+  digest = "1:39ebcc2b11457b703ae9ee2e8cca0f68df21969c6102cb3b705f76cca0ea0239"
   name = "golang.org/x/sync"
   packages = ["errgroup"]
-  revision = "fd80eb99c8f653c847d294a001bdf2a3a6f768f5"
+  pruneopts = "UT"
+  revision = "1d60e4601c6fd243af51cc01ddf169918a5407ca"
 
 [[projects]]
   branch = "master"
+  digest = "1:a220a85c72a6cb7339c412cb2b117019a7fd94007cdfffb3b5b1d058227a2bf8"
   name = "golang.org/x/sys"
-  packages = ["unix","windows"]
-  revision = "af50095a40f9041b3b38960738837185c26e9419"
+  packages = [
+    "cpu",
+    "unix",
+    "windows",
+  ]
+  pruneopts = "UT"
+  revision = "bd9dbc187b6e1dacfdd2722a87e83093c2d7bd6e"
 
 [[projects]]
-  branch = "master"
+  digest = "1:e5a8511f063c38c51ab9ab80e718e9149f692652aeb4e393a8c020dd1bf38ca2"
   name = "golang.org/x/text"
-  packages = ["collate","collate/build","internal/colltab","internal/gen","internal/tag","internal/triegen","internal/ucd","language","secure/bidirule","transform","unicode/bidi","unicode/cldr","unicode/norm","unicode/rangetable"]
-  revision = "e19ae1496984b1c655b8044a65c0300a3c878dd3"
+  packages = [
+    "collate",
+    "collate/build",
+    "encoding",
+    "encoding/internal",
+    "encoding/internal/identifier",
+    "encoding/unicode",
+    "internal/colltab",
+    "internal/gen",
+    "internal/tag",
+    "internal/triegen",
+    "internal/ucd",
+    "internal/utf8internal",
+    "language",
+    "runes",
+    "secure/bidirule",
+    "transform",
+    "unicode/bidi",
+    "unicode/cldr",
+    "unicode/norm",
+    "unicode/rangetable",
+  ]
+  pruneopts = "UT"
+  revision = "f21a4dfb5e38f5895301dc265a8def02365cc3d0"
+  version = "v0.3.0"
 
 [[projects]]
   branch = "master"
+  digest = "1:fb983acae7bd9c3ed9aadc1b1241d9e559ed21dbf84c17a0dda663ca169ccd69"
   name = "google.golang.org/api"
-  packages = ["gensupport","googleapi","googleapi/internal/uritemplates","storage/v1"]
-  revision = "65b0d8655182691ad23b4fac11e6f7b897d9b634"
+  packages = [
+    "gensupport",
+    "googleapi",
+    "googleapi/internal/uritemplates",
+    "storage/v1",
+  ]
+  pruneopts = "UT"
+  revision = "31ca0e01cd791f07750cb23fc99327721f753290"
 
 [[projects]]
+  digest = "1:c8907869850adaa8bd7631887948d0684f3787d0912f1c01ab72581a6c34432e"
   name = "google.golang.org/appengine"
-  packages = [".","internal","internal/app_identity","internal/base","internal/datastore","internal/log","internal/modules","internal/remote_api","internal/urlfetch","urlfetch"]
-  revision = "150dc57a1b433e64154302bdc40b6bb8aefa313a"
-  version = "v1.0.0"
+  packages = [
+    ".",
+    "internal",
+    "internal/app_identity",
+    "internal/base",
+    "internal/datastore",
+    "internal/log",
+    "internal/modules",
+    "internal/remote_api",
+    "internal/urlfetch",
+    "urlfetch",
+  ]
+  pruneopts = "UT"
+  revision = "b1f26356af11148e710935ed1ac8a7f5702c7612"
+  version = "v1.1.0"
 
 [[projects]]
   branch = "v2"
+  digest = "1:5bb148b78468350091db2ffbb2370f35cc6dcd74d9378a31b1c7b86ff7528f08"
+  name = "gopkg.in/tomb.v2"
+  packages = ["."]
+  pruneopts = "UT"
+  revision = "d5d1b5820637886def9eef33e03a27a9f166942c"
+
+[[projects]]
+  digest = "1:342378ac4dcb378a5448dd723f0784ae519383532f5e70ade24132c4c8693202"
   name = "gopkg.in/yaml.v2"
   packages = ["."]
-  revision = "d670f9405373e636a5a2765eea47fac0c9bc91a4"
+  pruneopts = "UT"
+  revision = "5420a8b6744d3b0345ab293f6fcba19c978f1183"
+  version = "v2.2.1"
 
 [solve-meta]
   analyzer-name = "dep"
   analyzer-version = 1
-  inputs-digest = "336ac5c261c174cac89f9a7102b493f08edfbd51fd61d1673d1d2ec4132d80ab"
+  input-imports = [
+    "bazil.org/fuse",
+    "bazil.org/fuse/fs",
+    "github.com/Azure/azure-sdk-for-go/storage",
+    "github.com/cenkalti/backoff",
+    "github.com/elithrar/simple-scrypt",
+    "github.com/google/go-cmp/cmp",
+    "github.com/juju/ratelimit",
+    "github.com/kurin/blazer/b2",
+    "github.com/mattn/go-isatty",
+    "github.com/minio/minio-go",
+    "github.com/minio/minio-go/pkg/credentials",
+    "github.com/ncw/swift",
+    "github.com/pkg/errors",
+    "github.com/pkg/profile",
+    "github.com/pkg/sftp",
+    "github.com/pkg/xattr",
+    "github.com/restic/chunker",
+    "github.com/spf13/cobra",
+    "github.com/spf13/cobra/doc",
+    "github.com/spf13/pflag",
+    "golang.org/x/crypto/poly1305",
+    "golang.org/x/crypto/scrypt",
+    "golang.org/x/crypto/ssh/terminal",
+    "golang.org/x/net/context",
+    "golang.org/x/net/context/ctxhttp",
+    "golang.org/x/net/http2",
+    "golang.org/x/oauth2/google",
+    "golang.org/x/sync/errgroup",
+    "golang.org/x/sys/unix",
+    "golang.org/x/text/encoding/unicode",
+    "google.golang.org/api/googleapi",
+    "google.golang.org/api/storage/v1",
+    "gopkg.in/tomb.v2",
+  ]
   solver-name = "gps-cdcl"
   solver-version = 1

Gopkg.toml

@@ -19,3 +19,7 @@
 # [[override]]
 #  name = "github.com/x/y"
 #  version = "2.4.0"
+
+[prune]
+  unused-packages = true
+  go-tests = true

README.rst

@@ -1,4 +1,4 @@
-|Documentation| |Build Status| |Build status| |Report Card| |Say Thanks| |TestCoverage|
+|Documentation| |Build Status| |Build status| |Report Card| |Say Thanks| |TestCoverage| |Reviewed by Hound|
 
 Introduction
 ------------
@@ -29,7 +29,7 @@ and add some data:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup backup ~/work
+    $ restic --repo /tmp/backup backup ~/work
     enter password for repository:
     scan [/home/user/work]
     scanned 764 directories, 1816 files in 0:00
@@ -57,6 +57,7 @@ Therefore, restic supports the following backends for storing backups natively:
 - `BackBlaze B2 <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#backblaze-b2>`__
 - `Microsoft Azure Blob Storage <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#microsoft-azure-blob-storage>`__
 - `Google Cloud Storage <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#google-cloud-storage>`__
+- And many other services via the `rclone <https://rclone.org>`__ `Backend <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#other-services-via-rclone>`__
 
 Design Principles
 -----------------
@@ -110,10 +111,18 @@ License
 Restic is licensed under `BSD 2-Clause License <https://opensource.org/licenses/BSD-2-Clause>`__. You can find the
 complete text in ``LICENSE``.
 
+Sponsorship
+-----------
+
+Backend integration tests for Google Cloud Storage and Microsoft Azure Blob
+Storage are sponsored by `AppsCode <https://appscode.com>`__!
+
+|AppsCode|
+
 .. |Documentation| image:: https://readthedocs.org/projects/restic/badge/?version=latest
    :target: https://restic.readthedocs.io/en/latest/?badge=latest
-.. |Build Status| image:: https://travis-ci.org/restic/restic.svg?branch=master
-   :target: https://travis-ci.org/restic/restic
+.. |Build Status| image:: https://travis-ci.com/restic/restic.svg?branch=master
+   :target: https://travis-ci.com/restic/restic
 .. |Build status| image:: https://ci.appveyor.com/api/projects/status/nuy4lfbgfbytw92q/branch/master?svg=true
    :target: https://ci.appveyor.com/project/fd0/restic/branch/master
 .. |Report Card| image:: https://goreportcard.com/badge/github.com/restic/restic
@@ -122,3 +131,7 @@ complete text in ``LICENSE``.
    :target: https://saythanks.io/to/restic
 .. |TestCoverage| image:: https://codecov.io/gh/restic/restic/branch/master/graph/badge.svg
    :target: https://codecov.io/gh/restic/restic
+.. |AppsCode| image:: https://cdn.appscode.com/images/logo/appscode/ac-logo-color.png
+   :target: https://appscode.com
+.. |Reviewed by Hound| image:: https://img.shields.io/badge/Reviewed_by-Hound-8E64B0.svg
+   :target: https://houndci.com

VERSION

@@ -1 +1 @@
-0.8.2
+0.9.2

build.go

@@ -53,7 +53,7 @@ var config = Config{
 		"github.com/restic/restic/internal/...",
 		"github.com/restic/restic/cmd/...",
 	},
-	MinVersion: GoVersion{Major: 1, Minor: 8, Patch: 0}, // minimum Go version supported
+	MinVersion: GoVersion{Major: 1, Minor: 9, Patch: 0}, // minimum Go version supported
 }
 
 // Config configures the build.
@@ -70,6 +70,7 @@ var (
 	keepGopath bool
 	runTests   bool
 	enableCGO  bool
+	enablePIE  bool
 )
 
 // specialDir returns true if the file begins with a special character ('.' or '_').
@@ -225,9 +226,11 @@ func showUsage(output io.Writer) {
 	fmt.Fprintf(output, "  -T     --test          run tests\n")
 	fmt.Fprintf(output, "  -o     --output        set output file name\n")
 	fmt.Fprintf(output, "         --enable-cgo    use CGO to link against libc\n")
+	fmt.Fprintf(output, "         --enable-pie    use PIE buildmode\n")
 	fmt.Fprintf(output, "         --goos value    set GOOS for cross-compilation\n")
 	fmt.Fprintf(output, "         --goarch value  set GOARCH for cross-compilation\n")
-	fmt.Fprintf(output, "         --goarm value  set GOARM for cross-compilation\n")
+	fmt.Fprintf(output, "         --goarm value   set GOARM for cross-compilation\n")
+	fmt.Fprintf(output, "         --tempdir dir   use a specific directory for compilation\n")
 }
 
 func verbosePrintf(message string, args ...interface{}) {
@@ -253,10 +256,22 @@ func cleanEnv() (env []string) {
 }
 
 // build runs "go build args..." with GOPATH set to gopath.
-func build(cwd, goos, goarch, goarm, gopath string, args ...string) error {
+func build(cwd string, ver GoVersion, goos, goarch, goarm, gopath string, args ...string) error {
 	a := []string{"build"}
-	a = append(a, "-asmflags", fmt.Sprintf("-trimpath=%s", gopath))
-	a = append(a, "-gcflags", fmt.Sprintf("-trimpath=%s", gopath))
+
+	if ver.AtLeast(GoVersion{1, 10, 0}) {
+		verbosePrintf("Go version is at least 1.10, using new syntax for -gcflags\n")
+		// use new prefix
+		a = append(a, "-asmflags", fmt.Sprintf("all=-trimpath=%s", gopath))
+		a = append(a, "-gcflags", fmt.Sprintf("all=-trimpath=%s", gopath))
+	} else {
+		a = append(a, "-asmflags", fmt.Sprintf("-trimpath=%s", gopath))
+		a = append(a, "-gcflags", fmt.Sprintf("-trimpath=%s", gopath))
+	}
+	if enablePIE {
+		a = append(a, "-buildmode=pie")
+	}
+
 	a = append(a, args...)
 	cmd := exec.Command("go", a...)
 	cmd.Env = append(cleanEnv(), "GOPATH="+gopath, "GOARCH="+goarch, "GOOS="+goos)
@@ -270,7 +285,7 @@ func build(cwd, goos, goarch, goarm, gopath string, args ...string) error {
 	cmd.Dir = cwd
 	cmd.Stdout = os.Stdout
 	cmd.Stderr = os.Stderr
-	verbosePrintf("go %s\n", args)
+	verbosePrintf("go %s\n", a)
 
 	return cmd.Run()
 }
@@ -366,30 +381,39 @@ func ParseGoVersion(s string) (v GoVersion) {
 
 	s = s[2:]
 	data := strings.Split(s, ".")
-	if len(data) != 3 {
-		return
+	if len(data) < 2 || len(data) > 3 {
+		// invalid version
+		return GoVersion{}
 	}
 
-	major, err := strconv.Atoi(data[0])
+	var err error
+
+	v.Major, err = strconv.Atoi(data[0])
 	if err != nil {
-		return
+		return GoVersion{}
 	}
 
-	minor, err := strconv.Atoi(data[1])
-	if err != nil {
-		return
+	// try to parse the minor version while removing an eventual suffix (like
+	// "rc2" or so)
+	for s := data[1]; s != ""; s = s[:len(s)-1] {
+		v.Minor, err = strconv.Atoi(s)
+		if err == nil {
+			break
+		}
 	}
 
-	patch, err := strconv.Atoi(data[2])
-	if err != nil {
-		return
+	if v.Minor == 0 {
+		// no minor version found
+		return GoVersion{}
 	}
 
-	v = GoVersion{
-		Major: major,
-		Minor: minor,
-		Patch: patch,
+	if len(data) >= 3 {
+		v.Patch, err = strconv.Atoi(data[2])
+		if err != nil {
+			return GoVersion{}
+		}
 	}
+
 	return
 }
 
@@ -437,6 +461,8 @@ func main() {
 	targetGOARCH := runtime.GOARCH
 	targetGOARM := ""
 
+	gopath := ""
+
 	var outputFilename string
 
 	for i, arg := range params {
@@ -459,10 +485,15 @@ func main() {
 		case "-o", "--output":
 			skipNext = true
 			outputFilename = params[i+1]
+		case "--tempdir":
+			skipNext = true
+			gopath = params[i+1]
 		case "-T", "--test":
 			runTests = true
 		case "--enable-cgo":
 			enableCGO = true
+		case "--enable-pie":
+			enablePIE = true
 		case "--goos":
 			skipNext = true
 			targetGOOS = params[i+1]
@@ -482,6 +513,8 @@ func main() {
 		}
 	}
 
+	verbosePrintf("detected Go version %v\n", ver)
+
 	if len(buildTags) == 0 {
 		verbosePrintf("adding build-tag release\n")
 		buildTags = []string{"release"}
@@ -498,9 +531,11 @@ func main() {
 		die("Getwd(): %v\n", err)
 	}
 
-	gopath, err := ioutil.TempDir("", fmt.Sprintf("%v-build-", config.Name))
-	if err != nil {
-		die("TempDir(): %v\n", err)
+	if gopath == "" {
+		gopath, err = ioutil.TempDir("", fmt.Sprintf("%v-build-", config.Name))
+		if err != nil {
+			die("TempDir(): %v\n", err)
+		}
 	}
 
 	verbosePrintf("create GOPATH at %v\n", gopath)
@@ -556,7 +591,7 @@ func main() {
 		"-o", output, config.Main,
 	}
 
-	err = build(filepath.Join(gopath, "src"), targetGOOS, targetGOARCH, targetGOARM, gopath, args...)
+	err = build(filepath.Join(gopath, "src"), ver, targetGOOS, targetGOARCH, targetGOARM, gopath, args...)
 	if err != nil {
 		die("build failed: %v\n", err)
 	}

changelog/0.7.1_2017-07-22/issue-1055

@@ -2,9 +2,9 @@ Enhancement: Create subdirs below `data/` for local/sftp backends
 
 The local and sftp backends now create the subdirs below `data/` on
 open/init. This way, restic makes sure that they always exist. This is
-connected to an issue for the sftp server:
+connected to an issue for the sftp server.
 
-https://github.com/restic/rest-server/pull/11#issuecomment-309879710
 https://github.com/restic/restic/issues/1055
+https://github.com/restic/rest-server/pull/11#issuecomment-309879710
 https://github.com/restic/restic/pull/1077
 https://github.com/restic/restic/pull/1105

changelog/0.7.1_2017-07-22/issue-1073

@@ -1,6 +1,6 @@
 Enhancement: Add `migrate` cmd to migrate from `s3legacy` to `default` layout
 
-The `migrate` command for chaning the `s3legacy` layout to the `default` layout
+The `migrate` command for changing the `s3legacy` layout to the `default` layout
 for s3 backends has been improved: It can now be restarted with `restic migrate
 --force s3_layout` and automatically retries operations on error.
 

changelog/0.8.3_2018-02-26/issue-1497

@@ -0,0 +1,8 @@
+Enhancement: Add --read-data-subset flag to check command
+
+This change introduces ability to check integrity of a subset of repository 
+data packs. This can be used to spread integrity check of larger repositories
+over a period of time.
+
+https://github.com/restic/restic/issues/1497
+https://github.com/restic/restic/pull/1556

changelog/0.8.3_2018-02-26/issue-1633

@@ -0,0 +1,7 @@
+Bugfix: Fixed unexpected 'pack file cannot be listed' error
+
+Due to a regression introduced in 0.8.2, the `rebuild-index` and `prune`
+commands failed to read pack files with size of 587, 588, 589 or 590 bytes.
+
+https://github.com/restic/restic/issues/1633
+https://github.com/restic/restic/pull/1635

changelog/0.8.3_2018-02-26/issue-1641

@@ -0,0 +1,10 @@
+Bugfix: Ignore files with invalid names in the repo
+
+The release 0.8.2 introduced a bug: when restic encounters files in the repo
+which do not have a valid name, it tries to load a file with a name of lots of
+zeroes instead of ignoring it. This is now resolved, invalid file names are
+just ignored.
+
+https://github.com/restic/restic/issues/1641
+https://github.com/restic/restic/pull/1643
+https://forum.restic.net/t/help-fixing-repo-no-such-file/485/3

changelog/0.8.3_2018-02-26/pull-1560

@@ -0,0 +1,5 @@
+Enhancement: Retry all repository file download errors
+
+Restic will now retry failed downloads, similar to other operations.
+
+https://github.com/restic/restic/pull/1560

changelog/0.8.3_2018-02-26/pull-1623

@@ -0,0 +1,12 @@
+Enhancement: Don't check for presence of files in the backend before writing
+
+Before, all backend implementations were required to return an error if the
+file that is to be written already exists in the backend. For most backends,
+that means making a request (e.g. via HTTP) and returning an error when the
+file already exists.
+
+This is not accurate, the file could have been created between the HTTP request
+testing for it, and when writing starts, so we've relaxed this requeriment,
+which saves one additional HTTP request per newly added file.
+
+https://github.com/restic/restic/pull/1623

changelog/0.8.3_2018-02-26/pull-1634

@@ -0,0 +1,7 @@
+Enhancement: Upgrade B2 client library, reduce HTTP requests
+
+We've upgraded the B2 client library restic uses to access BackBlaze B2. This
+reduces the number of HTTP requests needed to upload a new file from two to
+one, which should improve throughput to B2.
+
+https://github.com/restic/restic/pull/1634

changelog/0.8.3_2018-02-26/pull-1638

@@ -0,0 +1,16 @@
+Bugfix: Handle errors listing files in the backend
+
+A user reported in the forum that restic completes a backup although a
+concurrent `prune` operation was running. A few error messages were printed,
+but the backup was attempted and completed successfully. No error code was
+returned.
+
+This should not happen: The repository is exclusively locked during `prune`, so
+when `restic backup` is run in parallel, it should abort and return an error
+code instead.
+
+It was found that the bug was in the code introduced only recently, which
+retries a List() operation on the backend should that fail. It is now corrected.
+
+https://github.com/restic/restic/pull/1638
+https://forum.restic.net/t/restic-backup-returns-0-exit-code-when-already-locked/484

changelog/0.9.0_2018-05-21/issue-1433

@@ -0,0 +1,12 @@
+Enhancement: Support UTF-16 encoding and process Byte Order Mark
+
+On Windows, text editors commonly leave a Byte Order Mark at the beginning of
+the file to define which encoding is used (oftentimes UTF-16). We've added code
+to support processing the BOMs in text files, like the exclude files, the
+password file and the file passed via `--files-from`. This does not apply to
+any file being saved in a backup, those are not touched and archived as they
+are.
+
+https://github.com/restic/restic/issues/1433
+https://github.com/restic/restic/issues/1738
+https://github.com/restic/restic/pull/1748

changelog/0.9.0_2018-05-21/issue-1561

@@ -0,0 +1,10 @@
+Enhancement: Allow using rclone to access other services
+
+We've added the ability to use rclone to store backup data on all backends that
+it supports. This was done in collaboration with Nick, the author of rclone.
+You can now use it to first configure a service, then restic manages the rest
+(starting and stopping rclone). For details, please see the manual.
+
+https://github.com/restic/restic/issues/1561
+https://github.com/restic/restic/pull/1657
+https://rclone.org

changelog/0.9.0_2018-05-21/issue-1608

@@ -0,0 +1,7 @@
+Bugfix: Respect time stamp for new backup when reading from stdin
+
+When reading backups from stdin (via `restic backup --stdin`), restic now uses
+the time stamp for the new backup passed in `--time`.
+
+https://github.com/restic/restic/issues/1608
+https://github.com/restic/restic/pull/1703

changelog/0.9.0_2018-05-21/issue-1652

@@ -0,0 +1,9 @@
+Bugfix: Ignore/remove invalid lock files
+
+This corrects a bug introduced recently: When an invalid lock file in the repo
+is encountered (e.g. if the file is empty), the code used to ignore that, but
+now returns the error. Now, invalid files are ignored for the normal lock
+check, and removed when `restic unlock --remove-all` is run.
+
+https://github.com/restic/restic/issues/1652
+https://github.com/restic/restic/pull/1653

changelog/0.9.0_2018-05-21/issue-1665

@@ -0,0 +1,27 @@
+Enhancement: Improve cache handling for `restic check`
+
+For safety reasons, restic does not use a local metadata cache for the `restic
+check` command, so that data is loaded from the repository and restic can check
+it's in good condition. When the cache is disabled, restic will fetch each tiny
+blob needed for checking the integrity using a separate backend request. For
+non-local backends, that will take a long time, and depending on the backend
+(e.g. B2) may also be much more expensive.
+
+This PR adds a few commits which will change the behavior as follows:
+
+ * When `restic check` is called without any additional parameters, it will
+   build a new cache in a temporary directory, which is removed at the end of
+   the check. This way, we'll get readahead for metadata files (so restic will
+   fetch the whole file when the first blob from the file is requested), but
+   all data is freshly fetched from the storage backend. This is the default
+   behavior and will work for almost all users.
+
+ * When `restic check` is called with `--with-cache`, the default on-disc cache
+   is used. This behavior hasn't changed since the cache was introduced.
+
+ * When `--no-cache` is specified, restic falls back to the old behavior, and
+   read all tiny blobs in separate requests.
+
+https://github.com/restic/restic/issues/1665
+https://github.com/restic/restic/issues/1694
+https://github.com/restic/restic/pull/1696

changelog/0.9.0_2018-05-21/issue-1721

@@ -0,0 +1,8 @@
+Enhancement: Add `cache` command to list cache dirs
+
+The command `cache` was added, it allows listing restic's cache directoriers
+together with the last usage. It also allows removing old cache dirs without
+having to access a repo, via `restic cache --cleanup`
+
+https://github.com/restic/restic/issues/1721
+https://github.com/restic/restic/pull/1749

changelog/0.9.0_2018-05-21/issue-1730

@@ -0,0 +1,11 @@
+Bugfix: Ignore sockets for restore
+
+We've received a report and correct the behavior in which the restore code
+aborted restoring a directory when a socket was encountered. Unix domain socket
+files cannot be restored (they are created on the fly once a process starts
+listening). The error handling was corrected, and in addition we're now
+ignoring sockets during restore.
+
+https://github.com/restic/restic/issues/1730
+https://github.com/restic/restic/pull/1731
+

changelog/0.9.0_2018-05-21/issue-1758

@@ -0,0 +1,8 @@
+Enhancement: Allow saving OneDrive folders in Windows
+
+Restic now contains a bugfix to two libraries, which allows saving OneDrive
+folders in Windows. In order to use the newer versions of the libraries, the
+minimal version required to compile restic is now Go 1.9.
+
+https://github.com/restic/restic/issues/1758
+https://github.com/restic/restic/pull/1765

changelog/0.9.0_2018-05-21/issue-549

@@ -0,0 +1,43 @@
+Enhancement: Rework archiver code
+
+The core archiver code and the complementary code for the `backup` command was
+rewritten completely. This resolves very annoying issues such as 549. The first
+backup with this release of restic will likely result in all files being
+re-read locally, so it will take a lot longer. The next backup after that will
+be fast again.
+
+Basically, with the old code, restic took the last path component of each
+to-be-saved file or directory as the top-level file/directory within the
+snapshot. This meant that when called as `restic backup /home/user/foo`, the
+snapshot would contain the files in the directory `/home/user/foo` as `/foo`.
+
+This is not the case any more with the new archiver code. Now, restic works
+very similar to what `tar` does: When restic is called with an absolute path to
+save, then it'll preserve the directory structure within the snapshot. For the
+example above, the snapshot would contain the files in the directory within
+`/home/user/foo` in the snapshot. For relative directories, it only preserves
+the relative path components. So `restic backup user/foo` will save the files
+as `/user/foo` in the snapshot.
+
+While we were at it, the status display and notification system was completely
+rewritten. By default, restic now shows which files are currently read (unless
+`--quiet` is specified) in a multi-line status display.
+
+The `backup` command also gained a new option: `--verbose`. It can be specified
+once (which prints a bit more detail what restic is doing) or twice (which
+prints a line for each file/directory restic encountered, together with some
+statistics).
+
+Another issue that was resolved is the new code only reads two files at most.
+The old code would read way too many files in parallel, thereby slowing down
+the backup process on spinning discs a lot.
+
+https://github.com/restic/restic/issues/549
+https://github.com/restic/restic/issues/1286
+https://github.com/restic/restic/issues/446
+https://github.com/restic/restic/issues/1344
+https://github.com/restic/restic/issues/1416
+https://github.com/restic/restic/issues/1456
+https://github.com/restic/restic/issues/1145
+https://github.com/restic/restic/issues/1160
+https://github.com/restic/restic/pull/1494

changelog/0.9.0_2018-05-21/pull-1552

@@ -0,0 +1,13 @@
+Enhancement: Use Google Application Default credentials
+
+Google provide libraries to generate appropriate credentials with various
+fallback sources. This change uses the library to generate our GCS client, which
+allows us to make use of these extra methods.
+
+This should be backward compatible with previous restic behaviour while adding
+the additional capabilities to auth from Google's internal metadata endpoints.
+For users running restic in GCP this can make authentication far easier than it
+was before.
+
+https://github.com/restic/restic/pull/1552
+https://developers.google.com/identity/protocols/application-default-credentials

changelog/0.9.0_2018-05-21/pull-1647

@@ -0,0 +1,9 @@
+Enhancement: Accept AWS_SESSION_TOKEN for the s3 backend
+
+Before, it was not possible to use s3 backend with AWS temporary security
+credentials(with AWS_SESSION_TOKEN). This change gives higher priority to
+credentials.EnvAWS credentials provider.
+
+https://github.com/restic/restic/issues/1477
+https://github.com/restic/restic/pull/1479
+https://github.com/restic/restic/pull/1647

changelog/0.9.0_2018-05-21/pull-1648

@@ -0,0 +1,6 @@
+Enhancement: Ignore AWS permission denied error when creating a repository
+
+It's not possible to use s3 backend scoped to a subdirectory(with specific permissions).
+Restic doesn't try to create repository in a subdirectory, when 'bucket exists' of parent directory check fails due to permission issues.
+
+https://github.com/restic/restic/pull/1648

changelog/0.9.0_2018-05-21/pull-1649

@@ -0,0 +1,3 @@
+Enhancement: Add illumos/Solaris support
+
+https://github.com/restic/restic/pull/1649

changelog/0.9.0_2018-05-21/pull-1684

@@ -0,0 +1,6 @@
+Bugfix: Fix backend tests for rest-server
+
+The REST server for restic now requires an explicit parameter (`--no-auth`) if
+no authentication should be allowed. This is fixed in the tests.
+
+https://github.com/restic/restic/pull/1684

changelog/0.9.0_2018-05-21/pull-1709

@@ -0,0 +1,7 @@
+Enhancement: Improve messages `restic check` prints
+
+Some messages `restic check` prints are not really errors, so from now on
+restic does not treat them as errors any more and exits cleanly.
+
+https://github.com/restic/restic/pull/1709
+https://forum.restic.net/t/what-is-the-standard-procedure-to-follow-if-a-backup-or-restore-is-interrupted/571/2

changelog/0.9.0_2018-05-21/pull-1720

@@ -0,0 +1,7 @@
+Enhancement: Add --new-password-file flag for non-interactive password changes
+
+This makes it possible to change a repository password without being prompted.
+
+https://github.com/restic/restic/issues/827
+https://github.com/restic/restic/pull/1720
+https://forum.restic.net/t/changing-repo-password-without-prompt/591

changelog/0.9.0_2018-05-21/pull-1735

@@ -0,0 +1,9 @@
+Enhancement: Allow keeping a time range of snaphots
+
+We've added the `--keep-within` option to the `forget` command. It instructs
+restic to keep all snapshots within the given duration since the newest
+snapshot. For example, running `restic forget --keep-within 5m7d` will keep all
+snapshots which have been made in the five months and seven days since the
+latest snapshot.
+
+https://github.com/restic/restic/pull/1735

changelog/0.9.0_2018-05-21/pull-1746

@@ -0,0 +1,7 @@
+Bugfix: Correctly parse the argument to --tls-client-cert
+
+Previously, the --tls-client-cert method attempt to read ARGV[1] (hardcoded)
+instead of the argument that was passed to it. This has been corrected.
+
+https://github.com/restic/restic/issues/1745
+https://github.com/restic/restic/pull/1746

changelog/0.9.0_2018-05-21/pull-1782

@@ -0,0 +1,7 @@
+Enhancement: Use default AWS credentials chain for S3 backend
+
+Adds support for file credentials to the S3 backend (e.g. ~/.aws/credentials),
+and reorders the credentials chain for the S3 backend to match AWS's standard,
+which is static credentials, env vars, credentials file, and finally remote.
+
+https://github.com/restic/restic/pull/1782

changelog/0.9.1_2018-06-10/issue-1801

@@ -0,0 +1,9 @@
+Bugfix: Add limiting bandwidth to the rclone backend
+
+The rclone backend did not respect `--limit-upload` or `--limit-download`.
+Oftentimes it's not necessary to use this, as the limiting in rclone itself
+should be used because it gives much better results, but in case a remote
+instance of rclone is used (e.g. called via ssh), it is still relevant to limit
+the bandwidth from restic to rclone.
+
+https://github.com/restic/restic/issues/1801

changelog/0.9.1_2018-06-10/issue-1822

@@ -0,0 +1,9 @@
+Bugfix: Allow uploading large files to MS Azure
+
+Sometimes, restic creates files to be uploaded to the repository which are
+quite large, e.g. when saving directories with many entries or very large
+files. The MS Azure API does not allow uploading files larger that 256MiB
+directly, rather restic needs to upload them in blocks of 100MiB. This is now
+implemented.
+
+https://github.com/restic/restic/issues/1822

changelog/0.9.1_2018-06-10/issue-1825

@@ -0,0 +1,12 @@
+Bugfix: Correct `find` to not skip snapshots
+
+Under certain circumstances, the `find` command was found to skip snapshots
+containing directories with files to look for when the directories haven't been
+modified at all, and were already printed as part of a different snapshot. This
+is now corrected.
+
+In addition, we've switched to our own matching/pattern implementation, so now
+things like `restic find "/home/user/foo/**/main.go"` are possible.
+
+https://github.com/restic/restic/issues/1825
+https://github.com/restic/restic/issues/1823

changelog/0.9.1_2018-06-10/issue-1833

@@ -0,0 +1,9 @@
+Bugfix: Fix caching files on error
+
+During `check` it may happen that different threads access the same file in the
+backend, which is then downloaded into the cache only once. When that fails,
+only the thread which is responsible for downloading the file signals the
+correct error. The other threads just assume that the file has been downloaded
+successfully and then get an error when they try to access the cached file.
+
+https://github.com/restic/restic/issues/1833

changelog/0.9.1_2018-06-10/issue-1834

@@ -0,0 +1,8 @@
+Bugfix: Resolve deadlock
+
+When the "scanning" process restic runs to find out how much data there is does
+not finish before the backup itself is done, restic stops doing anything. This
+is resolved now.
+
+https://github.com/restic/restic/issues/1834
+https://github.com/restic/restic/pull/1835

changelog/0.9.2_2018-08-06/issue-1854

@@ -0,0 +1,16 @@
+Bugfix: Allow saving files/dirs on different fs with `--one-file-system`
+
+restic now allows saving files/dirs on a different file system in a subdir
+correctly even when `--one-file-system` is specified. 
+
+The first thing the restic archiver code does is to build a tree of the target
+files/directories. If it detects that a parent directory is already included
+(e.g. `restic backup /foo /foo/bar/baz`), it'll ignore the latter argument.
+
+Without `--one-file-system`, that's perfectly valid: If `/foo` is to be
+archived, it will include `/foo/bar/baz`. But with `--one-file-system`,
+`/foo/bar/baz` may reside on a different file system, so it won't be included
+with `/foo`.
+
+https://github.com/restic/restic/issues/1854
+https://github.com/restic/restic/pull/1855

changelog/0.9.2_2018-08-06/issue-1870

@@ -0,0 +1,6 @@
+Bugfix: Fix restore with --include
+
+We fixed a bug which prevented restic to restore files with an include filter.
+
+https://github.com/restic/restic/issues/1870
+https://github.com/restic/restic/pull/1900

changelog/0.9.2_2018-08-06/issue-1880

@@ -0,0 +1,12 @@
+Bugfix: Use `--cache-dir` argument for `check` command
+
+`check` command now uses a temporary sub-directory of the specified directory
+if set using the `--cache-dir` argument. If not set, the cache directory is
+created in the default temporary directory as before.
+In either case a temporary cache is used to ensure the actual repository is
+checked (rather than a local copy).
+
+The `--cache-dir` argument was not used by the `check` command, instead a
+cache directory was created in the temporary directory.
+
+https://github.com/restic/restic/issues/1880

changelog/0.9.2_2018-08-06/issue-1893

@@ -0,0 +1,8 @@
+Bugfix: Return error when exclude file cannot be read
+
+A bug was found: when multiple exclude files were passed to restic and one of
+them could not be read, an error was printed and restic continued, ignoring
+even the existing exclude files. Now, an error message is printed and restic
+aborts when an exclude file cannot be read.
+
+https://github.com/restic/restic/issues/1893

changelog/0.9.2_2018-08-06/issue-1906

@@ -0,0 +1,8 @@
+Enhancement: Add support for B2 application keys
+
+Restic can now use so-called "application keys" which can be created in the B2
+dashboard and were only introduced recently. In contrast to the "master key",
+such keys can be restricted to a specific bucket and/or path.
+
+https://github.com/restic/restic/issues/1906
+https://github.com/restic/restic/pull/1914

changelog/0.9.2_2018-08-06/pull-1729

@@ -0,0 +1,4 @@
+Enhancement: Add stats command to get information about a repository
+
+https://github.com/restic/restic/issues/874
+https://github.com/restic/restic/pull/1729

changelog/0.9.2_2018-08-06/pull-1772

@@ -0,0 +1,6 @@
+Enhancement: Add restore --verify to verify restored file content
+
+Restore will print error message if restored file content does not match
+expected SHA256 checksum
+
+https://github.com/restic/restic/pull/1772

changelog/0.9.2_2018-08-06/pull-1853

@@ -0,0 +1,6 @@
+Enhancement: Add JSON output support to `restic key list`
+
+This PR enables users to get the output of `restic key list` in JSON in addition
+to the existing table format.
+
+https://github.com/restic/restic/pull/1853

changelog/0.9.2_2018-08-06/pull-1861

@@ -0,0 +1,6 @@
+Bugfix: Fix case-insensitive search with restic find
+
+We've fixed the behavior for `restic find -i PATTERN`, which was
+broken in v0.9.1.
+
+https://github.com/restic/restic/pull/1861

changelog/0.9.2_2018-08-06/pull-1882

@@ -0,0 +1,8 @@
+Enhancement: S3 backend: accept AWS_SESSION_TOKEN
+
+Before, it was not possible to use s3 backend with AWS temporary security credentials(with AWS_SESSION_TOKEN).
+This change gives higher priority to credentials.EnvAWS credentials provider.
+
+https://github.com/restic/restic/issues/1477
+https://github.com/restic/restic/pull/1479
+https://github.com/restic/restic/pull/1647

changelog/0.9.2_2018-08-06/pull-1901

@@ -0,0 +1,9 @@
+Enhancement: Update the Backblaze B2 library
+
+We've updated the library we're using for accessing the Backblaze B2 service to
+0.5.0 to include support for upcoming so-called "application keys". With this
+feature, you can create access credentials for B2 which are restricted to e.g.
+a single bucket or even a sub-directory of a bucket.
+
+https://github.com/restic/restic/pull/1901
+https://github.com/kurin/blazer

changelog/CHANGELOG.tmpl

@@ -18,11 +18,11 @@ Details
 {{ range $par := .Paragraphs }}
    {{ wrap $par 80 3 }}
 {{ end -}}
-{{ range $id := .Issues }}
-   https://github.com/restic/restic/issues/{{ $id -}}
+{{ range $url := .IssueURLs }}
+   {{ $url -}}
 {{ end -}}
-{{ range $id := .PRs }}
-   https://github.com/restic/restic/pull/{{ $id -}}
+{{ range $url := .PRURLs }}
+   {{ $url -}}
 {{ end -}}
 {{ range $url := .OtherURLs }}
    {{ $url -}}

changelog/changelog-github.tmpl

@@ -2,8 +2,7 @@
 Changelog for restic {{ .Version }} ({{ .Date }})
 =======================================
 
-The following sections list the changes in restic {{ .Version }} relevant to
-restic users. The changes are ordered by importance.
+The following sections list the changes in restic {{ .Version }} relevant to restic users. The changes are ordered by importance.
 
 Summary
 -------

cmd/restic/background.go

@@ -1,9 +0,0 @@
-// +build !linux
-
-package main
-
-// IsProcessBackground should return true if it is running in the background or false if not
-func IsProcessBackground() bool {
-	//TODO: Check if the process are running in the background in other OS than linux
-	return false
-}

cmd/restic/cleanup.go

@@ -64,7 +64,10 @@ func CleanupHandler(c <-chan os.Signal) {
 		fmt.Fprintf(stderr, "%ssignal %v received, cleaning up\n", ClearLine(), s)
 
 		code := 0
-		if s != syscall.SIGINT {
+
+		if s == syscall.SIGINT {
+			code = 130
+		} else {
 			code = 1
 		}
 

cmd/restic/cmd_backup.go

@@ -2,21 +2,26 @@ package main
 
 import (
 	"bufio"
-	"fmt"
-	"io"
+	"bytes"
+	"context"
+	"io/ioutil"
 	"os"
-	"path"
-	"path/filepath"
+	"strconv"
 	"strings"
 	"time"
 
 	"github.com/spf13/cobra"
+	tomb "gopkg.in/tomb.v2"
 
 	"github.com/restic/restic/internal/archiver"
 	"github.com/restic/restic/internal/debug"
 	"github.com/restic/restic/internal/errors"
 	"github.com/restic/restic/internal/fs"
+	"github.com/restic/restic/internal/repository"
 	"github.com/restic/restic/internal/restic"
+	"github.com/restic/restic/internal/textfile"
+	"github.com/restic/restic/internal/ui"
+	"github.com/restic/restic/internal/ui/termstatus"
 )
 
 var cmdBackup = &cobra.Command{
@@ -42,11 +47,16 @@ given as the arguments.
 			return errors.Fatal("cannot use both `--stdin` and `--files-from -`")
 		}
 
-		if backupOptions.Stdin {
-			return readBackupFromStdin(backupOptions, globalOptions, args)
-		}
+		var t tomb.Tomb
+		term := termstatus.New(globalOptions.stdout, globalOptions.stderr, globalOptions.Quiet)
+		t.Go(func() error { term.Run(t.Context(globalOptions.ctx)); return nil })
 
-		return runBackup(backupOptions, globalOptions, args)
+		err := runBackup(backupOptions, globalOptions, term, args)
+		if err != nil {
+			return err
+		}
+		t.Kill(nil)
+		return t.Wait()
 	},
 }
 
@@ -90,127 +100,6 @@ func init() {
 	f.BoolVar(&backupOptions.WithAtime, "with-atime", false, "store the atime for all files and directories")
 }
 
-func newScanProgress(gopts GlobalOptions) *restic.Progress {
-	if gopts.Quiet {
-		return nil
-	}
-
-	p := restic.NewProgress()
-	p.OnUpdate = func(s restic.Stat, d time.Duration, ticker bool) {
-		if IsProcessBackground() {
-			return
-		}
-
-		PrintProgress("[%s] %d directories, %d files, %s", formatDuration(d), s.Dirs, s.Files, formatBytes(s.Bytes))
-	}
-
-	p.OnDone = func(s restic.Stat, d time.Duration, ticker bool) {
-		PrintProgress("scanned %d directories, %d files in %s\n", s.Dirs, s.Files, formatDuration(d))
-	}
-
-	return p
-}
-
-func newArchiveProgress(gopts GlobalOptions, todo restic.Stat) *restic.Progress {
-	if gopts.Quiet {
-		return nil
-	}
-
-	archiveProgress := restic.NewProgress()
-
-	var bps, eta uint64
-	itemsTodo := todo.Files + todo.Dirs
-
-	archiveProgress.OnUpdate = func(s restic.Stat, d time.Duration, ticker bool) {
-		if IsProcessBackground() {
-			return
-		}
-
-		sec := uint64(d / time.Second)
-		if todo.Bytes > 0 && sec > 0 && ticker {
-			bps = s.Bytes / sec
-			if s.Bytes >= todo.Bytes {
-				eta = 0
-			} else if bps > 0 {
-				eta = (todo.Bytes - s.Bytes) / bps
-			}
-		}
-
-		itemsDone := s.Files + s.Dirs
-
-		status1 := fmt.Sprintf("[%s] %s  %s / %s  %d / %d items  %d errors  ",
-			formatDuration(d),
-			formatPercent(s.Bytes, todo.Bytes),
-			formatBytes(s.Bytes), formatBytes(todo.Bytes),
-			itemsDone, itemsTodo,
-			s.Errors)
-		status2 := fmt.Sprintf("ETA %s ", formatSeconds(eta))
-
-		if w := stdoutTerminalWidth(); w > 0 {
-			maxlen := w - len(status2) - 1
-
-			if maxlen < 4 {
-				status1 = ""
-			} else if len(status1) > maxlen {
-				status1 = status1[:maxlen-4]
-				status1 += "... "
-			}
-		}
-
-		PrintProgress("%s%s", status1, status2)
-	}
-
-	archiveProgress.OnDone = func(s restic.Stat, d time.Duration, ticker bool) {
-		fmt.Printf("\nduration: %s\n", formatDuration(d))
-	}
-
-	return archiveProgress
-}
-
-func newArchiveStdinProgress(gopts GlobalOptions) *restic.Progress {
-	if gopts.Quiet {
-		return nil
-	}
-
-	archiveProgress := restic.NewProgress()
-
-	var bps uint64
-
-	archiveProgress.OnUpdate = func(s restic.Stat, d time.Duration, ticker bool) {
-		if IsProcessBackground() {
-			return
-		}
-
-		sec := uint64(d / time.Second)
-		if s.Bytes > 0 && sec > 0 && ticker {
-			bps = s.Bytes / sec
-		}
-
-		status1 := fmt.Sprintf("[%s] %s  %s/s", formatDuration(d),
-			formatBytes(s.Bytes),
-			formatBytes(bps))
-
-		if w := stdoutTerminalWidth(); w > 0 {
-			maxlen := w - len(status1)
-
-			if maxlen < 4 {
-				status1 = ""
-			} else if len(status1) > maxlen {
-				status1 = status1[:maxlen-4]
-				status1 += "... "
-			}
-		}
-
-		PrintProgress("%s", status1)
-	}
-
-	archiveProgress.OnDone = func(s restic.Stat, d time.Duration, ticker bool) {
-		fmt.Printf("\nduration: %s\n", formatDuration(d))
-	}
-
-	return archiveProgress
-}
-
 // filterExisting returns a slice of all existing items, or an error if no
 // items exist at all.
 func filterExisting(items []string) (result []string, err error) {
@@ -231,78 +120,33 @@ func filterExisting(items []string) (result []string, err error) {
 	return
 }
 
-func readBackupFromStdin(opts BackupOptions, gopts GlobalOptions, args []string) error {
-	if len(args) != 0 {
-		return errors.Fatal("when reading from stdin, no additional files can be specified")
-	}
-
-	fn := opts.StdinFilename
-
-	if fn == "" {
-		return errors.Fatal("filename for backup from stdin must not be empty")
-	}
-
-	if filepath.Base(fn) != fn || path.Base(fn) != fn {
-		return errors.Fatal("filename is invalid (may not contain a directory, slash or backslash)")
-	}
-
-	if gopts.password == "" {
-		return errors.Fatal("unable to read password from stdin when data is to be read from stdin, use --password-file or $RESTIC_PASSWORD")
-	}
-
-	repo, err := OpenRepository(gopts)
-	if err != nil {
-		return err
-	}
-
-	lock, err := lockRepo(repo)
-	defer unlockRepo(lock)
-	if err != nil {
-		return err
-	}
-
-	err = repo.LoadIndex(gopts.ctx)
-	if err != nil {
-		return err
-	}
-
-	r := &archiver.Reader{
-		Repository: repo,
-		Tags:       opts.Tags,
-		Hostname:   opts.Hostname,
-	}
-
-	_, id, err := r.Archive(gopts.ctx, fn, os.Stdin, newArchiveStdinProgress(gopts))
-	if err != nil {
-		return err
-	}
-
-	Verbosef("archived as %v\n", id.Str())
-	return nil
-}
-
-// readFromFile will read all lines from the given filename and write them to a
-// string array, if filename is empty readFromFile returns and empty string
-// array. If filename is a dash (-), readFromFile will read the lines from
-// the standard input.
+// readFromFile will read all lines from the given filename and return them as
+// a string array, if filename is empty readFromFile returns and empty string
+// array. If filename is a dash (-), readFromFile will read the lines from the
+// standard input.
 func readLinesFromFile(filename string) ([]string, error) {
 	if filename == "" {
 		return nil, nil
 	}
 
-	var r io.Reader = os.Stdin
-	if filename != "-" {
-		f, err := os.Open(filename)
-		if err != nil {
-			return nil, err
-		}
-		defer f.Close()
-		r = f
+	var (
+		data []byte
+		err  error
+	)
+
+	if filename == "-" {
+		data, err = ioutil.ReadAll(os.Stdin)
+	} else {
+		data, err = textfile.Read(filename)
+	}
+
+	if err != nil {
+		return nil, err
 	}
 
 	var lines []string
 
-	scanner := bufio.NewScanner(r)
+	scanner := bufio.NewScanner(bytes.NewReader(data))
 	for scanner.Scan() {
 		line := strings.TrimSpace(scanner.Text())
 		// ignore empty lines
@@ -323,56 +167,58 @@ func readLinesFromFile(filename string) ([]string, error) {
 	return lines, nil
 }
 
-func runBackup(opts BackupOptions, gopts GlobalOptions, args []string) error {
+// Check returns an error when an invalid combination of options was set.
+func (opts BackupOptions) Check(gopts GlobalOptions, args []string) error {
 	if opts.FilesFrom == "-" && gopts.password == "" {
 		return errors.Fatal("unable to read password from stdin when data is to be read from stdin, use --password-file or $RESTIC_PASSWORD")
 	}
 
-	fromfile, err := readLinesFromFile(opts.FilesFrom)
-	if err != nil {
-		return err
-	}
-
-	// merge files from files-from into normal args so we can reuse the normal
-	// args checks and have the ability to use both files-from and args at the
-	// same time
-	args = append(args, fromfile...)
-	if len(args) == 0 {
-		return errors.Fatal("nothing to backup, please specify target files/dirs")
-	}
-
-	target := make([]string, 0, len(args))
-	for _, d := range args {
-		if a, err := filepath.Abs(d); err == nil {
-			d = a
+	if opts.Stdin {
+		if opts.FilesFrom != "" {
+			return errors.Fatal("--stdin and --files-from cannot be used together")
+		}
+
+		if len(args) > 0 {
+			return errors.Fatal("--stdin was specified and files/dirs were listed as arguments")
 		}
-		target = append(target, d)
 	}
 
-	target, err = filterExisting(target)
-	if err != nil {
-		return err
-	}
-
-	// rejectFuncs collect functions that can reject items from the backup
-	var rejectFuncs []RejectFunc
+	return nil
+}
 
+// collectRejectFuncs returns a list of all functions which may reject data
+// from being saved in a snapshot
+func collectRejectFuncs(opts BackupOptions, repo *repository.Repository, targets []string) (fs []RejectFunc, err error) {
 	// allowed devices
-	if opts.ExcludeOtherFS {
-		f, err := rejectByDevice(target)
+	if opts.ExcludeOtherFS && !opts.Stdin {
+		f, err := rejectByDevice(targets)
 		if err != nil {
-			return err
+			return nil, err
 		}
-		rejectFuncs = append(rejectFuncs, f)
+		fs = append(fs, f)
+	}
+
+	// exclude restic cache
+	if repo.Cache != nil {
+		f, err := rejectResticCache(repo)
+		if err != nil {
+			return nil, err
+		}
+
+		fs = append(fs, f)
 	}
 
 	// add patterns from file
 	if len(opts.ExcludeFiles) > 0 {
-		opts.Excludes = append(opts.Excludes, readExcludePatternsFromFiles(opts.ExcludeFiles)...)
+		excludes, err := readExcludePatternsFromFiles(opts.ExcludeFiles)
+		if err != nil {
+			return nil, err
+		}
+		opts.Excludes = append(opts.Excludes, excludes...)
 	}
 
 	if len(opts.Excludes) > 0 {
-		rejectFuncs = append(rejectFuncs, rejectByPattern(opts.Excludes))
+		fs = append(fs, rejectByPattern(opts.Excludes))
 	}
 
 	if opts.ExcludeCaches {
@@ -382,124 +228,37 @@ func runBackup(opts BackupOptions, gopts GlobalOptions, args []string) error {
 	for _, spec := range opts.ExcludeIfPresent {
 		f, err := rejectIfPresent(spec)
 		if err != nil {
-			return err
+			return nil, err
 		}
 
-		rejectFuncs = append(rejectFuncs, f)
+		fs = append(fs, f)
 	}
 
-	repo, err := OpenRepository(gopts)
-	if err != nil {
-		return err
-	}
-
-	lock, err := lockRepo(repo)
-	defer unlockRepo(lock)
-	if err != nil {
-		return err
-	}
-
-	// exclude restic cache
-	if repo.Cache != nil {
-		f, err := rejectResticCache(repo)
-		if err != nil {
-			return err
-		}
-
-		rejectFuncs = append(rejectFuncs, f)
-	}
-
-	err = repo.LoadIndex(gopts.ctx)
-	if err != nil {
-		return err
-	}
-
-	var parentSnapshotID *restic.ID
-
-	// Force using a parent
-	if !opts.Force && opts.Parent != "" {
-		id, err := restic.FindSnapshot(repo, opts.Parent)
-		if err != nil {
-			return errors.Fatalf("invalid id %q: %v", opts.Parent, err)
-		}
-
-		parentSnapshotID = &id
-	}
-
-	// Find last snapshot to set it as parent, if not already set
-	if !opts.Force && parentSnapshotID == nil {
-		id, err := restic.FindLatestSnapshot(gopts.ctx, repo, target, []restic.TagList{}, opts.Hostname)
-		if err == nil {
-			parentSnapshotID = &id
-		} else if err != restic.ErrNoSnapshotFound {
-			return err
-		}
-	}
-
-	if parentSnapshotID != nil {
-		Verbosef("using parent snapshot %v\n", parentSnapshotID.Str())
-	}
-
-	Verbosef("scan %v\n", target)
-
-	selectFilter := func(item string, fi os.FileInfo) bool {
-		for _, reject := range rejectFuncs {
-			if reject(item, fi) {
-				return false
-			}
-		}
-		return true
-	}
-
-	stat, err := archiver.Scan(target, selectFilter, newScanProgress(gopts))
-	if err != nil {
-		return err
-	}
-
-	arch := archiver.New(repo)
-	arch.Excludes = opts.Excludes
-	arch.SelectFilter = selectFilter
-	arch.WithAccessTime = opts.WithAtime
-
-	arch.Warn = func(dir string, fi os.FileInfo, err error) {
-		// TODO: make ignoring errors configurable
-		Warnf("%s\rwarning for %s: %v\n", ClearLine(), dir, err)
-	}
-
-	timeStamp := time.Now()
-	if opts.TimeStamp != "" {
-		timeStamp, err = time.Parse(TimeFormat, opts.TimeStamp)
-		if err != nil {
-			return errors.Fatalf("error in time option: %v\n", err)
-		}
-	}
-
-	_, id, err := arch.Snapshot(gopts.ctx, newArchiveProgress(gopts, stat), target, opts.Tags, opts.Hostname, parentSnapshotID, timeStamp)
-	if err != nil {
-		return err
-	}
-
-	Verbosef("snapshot %s saved\n", id.Str())
-
-	return nil
+	return fs, nil
 }
 
-func readExcludePatternsFromFiles(excludeFiles []string) []string {
+// readExcludePatternsFromFiles reads all exclude files and returns the list of
+// exclude patterns. For each line, leading and trailing white space is removed
+// and comment lines are ignored. For each remaining pattern, environment
+// variables are resolved. For adding a literal dollar sign ($), write $$ to
+// the file.
+func readExcludePatternsFromFiles(excludeFiles []string) ([]string, error) {
+	getenvOrDollar := func(s string) string {
+		if s == "$" {
+			return "$"
+		}
+		return os.Getenv(s)
+	}
+
 	var excludes []string
 	for _, filename := range excludeFiles {
 		err := func() (err error) {
-			file, err := fs.Open(filename)
+			data, err := textfile.Read(filename)
 			if err != nil {
 				return err
 			}
-			defer func() {
-				// return pre-close error if there was one
-				if errClose := file.Close(); err == nil {
-					err = errClose
-				}
-			}()
 
-			scanner := bufio.NewScanner(file)
+			scanner := bufio.NewScanner(bytes.NewReader(data))
 			for scanner.Scan() {
 				line := strings.TrimSpace(scanner.Text())
 
@@ -513,15 +272,228 @@ func readExcludePatternsFromFiles(excludeFiles []string) []string {
 					continue
 				}
 
-				line = os.ExpandEnv(line)
+				line = os.Expand(line, getenvOrDollar)
 				excludes = append(excludes, line)
 			}
 			return scanner.Err()
 		}()
 		if err != nil {
-			Warnf("error reading exclude patterns: %v:", err)
-			return nil
+			return nil, err
 		}
 	}
-	return excludes
+	return excludes, nil
+}
+
+// collectTargets returns a list of target files/dirs from several sources.
+func collectTargets(opts BackupOptions, args []string) (targets []string, err error) {
+	if opts.Stdin {
+		return nil, nil
+	}
+
+	fromfile, err := readLinesFromFile(opts.FilesFrom)
+	if err != nil {
+		return nil, err
+	}
+
+	// merge files from files-from into normal args so we can reuse the normal
+	// args checks and have the ability to use both files-from and args at the
+	// same time
+	args = append(args, fromfile...)
+	if len(args) == 0 && !opts.Stdin {
+		return nil, errors.Fatal("nothing to backup, please specify target files/dirs")
+	}
+
+	targets = args
+	targets, err = filterExisting(targets)
+	if err != nil {
+		return nil, err
+	}
+
+	return targets, nil
+}
+
+// parent returns the ID of the parent snapshot. If there is none, nil is
+// returned.
+func findParentSnapshot(ctx context.Context, repo restic.Repository, opts BackupOptions, targets []string) (parentID *restic.ID, err error) {
+	// Force using a parent
+	if !opts.Force && opts.Parent != "" {
+		id, err := restic.FindSnapshot(repo, opts.Parent)
+		if err != nil {
+			return nil, errors.Fatalf("invalid id %q: %v", opts.Parent, err)
+		}
+
+		parentID = &id
+	}
+
+	// Find last snapshot to set it as parent, if not already set
+	if !opts.Force && parentID == nil {
+		id, err := restic.FindLatestSnapshot(ctx, repo, targets, []restic.TagList{}, opts.Hostname)
+		if err == nil {
+			parentID = &id
+		} else if err != restic.ErrNoSnapshotFound {
+			return nil, err
+		}
+	}
+
+	return parentID, nil
+}
+
+func runBackup(opts BackupOptions, gopts GlobalOptions, term *termstatus.Terminal, args []string) error {
+	err := opts.Check(gopts, args)
+	if err != nil {
+		return err
+	}
+
+	targets, err := collectTargets(opts, args)
+	if err != nil {
+		return err
+	}
+
+	timeStamp := time.Now()
+	if opts.TimeStamp != "" {
+		timeStamp, err = time.Parse(TimeFormat, opts.TimeStamp)
+		if err != nil {
+			return errors.Fatalf("error in time option: %v\n", err)
+		}
+	}
+
+	var t tomb.Tomb
+
+	p := ui.NewBackup(term, gopts.verbosity)
+
+	// use the terminal for stdout/stderr
+	prevStdout, prevStderr := gopts.stdout, gopts.stderr
+	defer func() {
+		gopts.stdout, gopts.stderr = prevStdout, prevStderr
+	}()
+	gopts.stdout, gopts.stderr = p.Stdout(), p.Stderr()
+
+	if s, ok := os.LookupEnv("RESTIC_PROGRESS_FPS"); ok {
+		fps, err := strconv.Atoi(s)
+		if err == nil && fps >= 1 {
+			if fps > 60 {
+				fps = 60
+			}
+			p.MinUpdatePause = time.Second / time.Duration(fps)
+		}
+	}
+
+	t.Go(func() error { return p.Run(t.Context(gopts.ctx)) })
+
+	p.V("open repository")
+	repo, err := OpenRepository(gopts)
+	if err != nil {
+		return err
+	}
+
+	p.V("lock repository")
+	lock, err := lockRepo(repo)
+	defer unlockRepo(lock)
+	if err != nil {
+		return err
+	}
+
+	// rejectFuncs collect functions that can reject items from the backup
+	rejectFuncs, err := collectRejectFuncs(opts, repo, targets)
+	if err != nil {
+		return err
+	}
+
+	p.V("load index files")
+	err = repo.LoadIndex(gopts.ctx)
+	if err != nil {
+		return err
+	}
+
+	parentSnapshotID, err := findParentSnapshot(gopts.ctx, repo, opts, targets)
+	if err != nil {
+		return err
+	}
+
+	if parentSnapshotID != nil {
+		p.V("using parent snapshot %v\n", parentSnapshotID.Str())
+	}
+
+	selectFilter := func(item string, fi os.FileInfo) bool {
+		for _, reject := range rejectFuncs {
+			if reject(item, fi) {
+				return false
+			}
+		}
+		return true
+	}
+
+	var targetFS fs.FS = fs.Local{}
+	if opts.Stdin {
+		p.V("read data from stdin")
+		targetFS = &fs.Reader{
+			ModTime:    timeStamp,
+			Name:       opts.StdinFilename,
+			Mode:       0644,
+			ReadCloser: os.Stdin,
+		}
+		targets = []string{opts.StdinFilename}
+	}
+
+	sc := archiver.NewScanner(targetFS)
+	sc.Select = selectFilter
+	sc.Error = p.ScannerError
+	sc.Result = p.ReportTotal
+
+	p.V("start scan on %v", targets)
+	t.Go(func() error { return sc.Scan(t.Context(gopts.ctx), targets) })
+
+	arch := archiver.New(repo, targetFS, archiver.Options{})
+	arch.Select = selectFilter
+	arch.WithAtime = opts.WithAtime
+	arch.Error = p.Error
+	arch.CompleteItem = p.CompleteItemFn
+	arch.StartFile = p.StartFile
+	arch.CompleteBlob = p.CompleteBlob
+
+	if parentSnapshotID == nil {
+		parentSnapshotID = &restic.ID{}
+	}
+
+	snapshotOpts := archiver.SnapshotOptions{
+		Excludes:       opts.Excludes,
+		Tags:           opts.Tags,
+		Time:           timeStamp,
+		Hostname:       opts.Hostname,
+		ParentSnapshot: *parentSnapshotID,
+	}
+
+	uploader := archiver.IndexUploader{
+		Repository: repo,
+		Start: func() {
+			p.VV("uploading intermediate index")
+		},
+		Complete: func(id restic.ID) {
+			p.V("uploaded intermediate index %v", id.Str())
+		},
+	}
+
+	t.Go(func() error {
+		return uploader.Upload(gopts.ctx, t.Context(gopts.ctx), 30*time.Second)
+	})
+
+	p.V("start backup on %v", targets)
+	_, id, err := arch.Snapshot(gopts.ctx, targets, snapshotOpts)
+	if err != nil {
+		return errors.Fatalf("unable to save snapshot: %v", err)
+	}
+
+	p.Finish()
+	p.P("snapshot %s saved\n", id.Str())
+
+	// cleanly shutdown all running goroutines
+	t.Kill(nil)
+
+	// let's see if one returned an error
+	err = t.Wait()
+	if err != nil {
+		return err
+	}
+
+	return nil
 }

cmd/restic/cmd_cache.go

@@ -0,0 +1,122 @@
+package main
+
+import (
+	"fmt"
+	"path/filepath"
+	"sort"
+	"time"
+
+	"github.com/restic/restic/internal/cache"
+	"github.com/restic/restic/internal/errors"
+	"github.com/restic/restic/internal/fs"
+	"github.com/spf13/cobra"
+)
+
+var cmdCache = &cobra.Command{
+	Use:   "cache",
+	Short: "Operate on local cache directories",
+	Long: `
+The "cache" command allows listing and cleaning local cache directories.
+`,
+	DisableAutoGenTag: true,
+	RunE: func(cmd *cobra.Command, args []string) error {
+		return runCache(cacheOptions, globalOptions, args)
+	},
+}
+
+// CacheOptions bundles all options for the snapshots command.
+type CacheOptions struct {
+	Cleanup bool
+	MaxAge  uint
+}
+
+var cacheOptions CacheOptions
+
+func init() {
+	cmdRoot.AddCommand(cmdCache)
+
+	f := cmdCache.Flags()
+	f.BoolVar(&cacheOptions.Cleanup, "cleanup", false, "remove old cache directories")
+	f.UintVar(&cacheOptions.MaxAge, "max-age", 30, "max age in `days` for cache directories to be considered old")
+}
+
+func runCache(opts CacheOptions, gopts GlobalOptions, args []string) error {
+	if len(args) > 0 {
+		return errors.Fatal("the cache command has no arguments")
+	}
+
+	if gopts.NoCache {
+		return errors.Fatal("Refusing to do anything, the cache is disabled")
+	}
+
+	var (
+		cachedir = gopts.CacheDir
+		err      error
+	)
+
+	if cachedir == "" {
+		cachedir, err = cache.DefaultDir()
+		if err != nil {
+			return err
+		}
+	}
+
+	if opts.Cleanup || gopts.CleanupCache {
+		oldDirs, err := cache.OlderThan(cachedir, time.Duration(opts.MaxAge)*24*time.Hour)
+		if err != nil {
+			return err
+		}
+
+		if len(oldDirs) == 0 {
+			Verbosef("no old cache dirs found\n")
+			return nil
+		}
+
+		Verbosef("remove %d old cache directories\n", len(oldDirs))
+
+		for _, item := range oldDirs {
+			dir := filepath.Join(cachedir, item.Name())
+			err = fs.RemoveAll(dir)
+			if err != nil {
+				Warnf("unable to remove %v: %v\n", dir, err)
+			}
+		}
+
+		return nil
+	}
+
+	tab := NewTable()
+	tab.Header = fmt.Sprintf("%-14s   %-16s %s", "Repository ID", "Last Used", "Old")
+	tab.RowFormat = "%-14s   %-16s %s"
+
+	dirs, err := cache.All(cachedir)
+	if err != nil {
+		return err
+	}
+
+	if len(dirs) == 0 {
+		Printf("no cache dirs found, basedir is %v\n", cachedir)
+		return nil
+	}
+
+	sort.Slice(dirs, func(i, j int) bool {
+		return dirs[i].ModTime().Before(dirs[j].ModTime())
+	})
+
+	for _, entry := range dirs {
+		var old string
+		if cache.IsOld(entry.ModTime(), time.Duration(opts.MaxAge)*24*time.Hour) {
+			old = "yes"
+		}
+
+		tab.Rows = append(tab.Rows, []interface{}{
+			entry.Name()[:10],
+			fmt.Sprintf("%d days ago", uint(time.Since(entry.ModTime()).Hours()/24)),
+			old,
+		})
+	}
+
+	tab.Write(gopts.stdout)
+
+	return nil
+}

cmd/restic/cmd_cat.go

@@ -58,7 +58,7 @@ func runCat(gopts GlobalOptions, args []string) error {
 			// find snapshot id with prefix
 			id, err = restic.FindSnapshot(repo, args[1])
 			if err != nil {
-				return err
+				return errors.Fatalf("could not find snapshot: %v\n", err)
 			}
 		}
 	}

cmd/restic/cmd_check.go

@@ -2,13 +2,17 @@ package main
 
 import (
 	"fmt"
+	"io/ioutil"
 	"os"
+	"strconv"
+	"strings"
 	"time"
 
 	"github.com/spf13/cobra"
 
 	"github.com/restic/restic/internal/checker"
 	"github.com/restic/restic/internal/errors"
+	"github.com/restic/restic/internal/fs"
 	"github.com/restic/restic/internal/restic"
 )
 
@@ -26,13 +30,17 @@ repository and not use a local cache.
 	RunE: func(cmd *cobra.Command, args []string) error {
 		return runCheck(checkOptions, globalOptions, args)
 	},
+	PreRunE: func(cmd *cobra.Command, args []string) error {
+		return checkFlags(checkOptions)
+	},
 }
 
 // CheckOptions bundles all options for the 'check' command.
 type CheckOptions struct {
-	ReadData    bool
-	CheckUnused bool
-	WithCache   bool
+	ReadData       bool
+	ReadDataSubset string
+	CheckUnused    bool
+	WithCache      bool
 }
 
 var checkOptions CheckOptions
@@ -42,10 +50,45 @@ func init() {
 
 	f := cmdCheck.Flags()
 	f.BoolVar(&checkOptions.ReadData, "read-data", false, "read all data blobs")
+	f.StringVar(&checkOptions.ReadDataSubset, "read-data-subset", "", "read subset n of m data packs (format: `n/m`)")
 	f.BoolVar(&checkOptions.CheckUnused, "check-unused", false, "find unused blobs")
 	f.BoolVar(&checkOptions.WithCache, "with-cache", false, "use the cache")
 }
 
+func checkFlags(opts CheckOptions) error {
+	if opts.ReadData && opts.ReadDataSubset != "" {
+		return errors.Fatalf("check flags --read-data and --read-data-subset cannot be used together")
+	}
+	if opts.ReadDataSubset != "" {
+		dataSubset, err := stringToIntSlice(opts.ReadDataSubset)
+		if err != nil || len(dataSubset) != 2 {
+			return errors.Fatalf("check flag --read-data-subset must have two positive integer values, e.g. --read-data-subset=1/2")
+		}
+		if dataSubset[0] == 0 || dataSubset[1] == 0 || dataSubset[0] > dataSubset[1] {
+			return errors.Fatalf("check flag --read-data-subset=n/t values must be positive integers, and n <= t, e.g. --read-data-subset=1/2")
+		}
+	}
+
+	return nil
+}
+
+// stringToIntSlice converts string to []uint, using '/' as element separator
+func stringToIntSlice(param string) (split []uint, err error) {
+	if param == "" {
+		return nil, nil
+	}
+	parts := strings.Split(param, "/")
+	result := make([]uint, len(parts))
+	for idx, part := range parts {
+		uintval, err := strconv.ParseUint(part, 10, 0)
+		if err != nil {
+			return nil, err
+		}
+		result[idx] = uint(uintval)
+	}
+	return result, nil
+}
+
 func newReadProgress(gopts GlobalOptions, todo restic.Stat) *restic.Progress {
 	if gopts.Quiet {
 		return nil
@@ -76,15 +119,58 @@ func newReadProgress(gopts GlobalOptions, todo restic.Stat) *restic.Progress {
 	return readProgress
 }
 
+// prepareCheckCache configures a special cache directory for check.
+//
+//  * if --with-cache is specified, the default cache is used
+//  * if the user explicitly requested --no-cache, we don't use any cache
+//  * if the user provides --cache-dir, we use a cache in a temporary sub-directory of the specified directory and the sub-directory is deleted after the check
+//  * by default, we use a cache in a temporary directory that is deleted after the check
+func prepareCheckCache(opts CheckOptions, gopts *GlobalOptions) (cleanup func()) {
+	cleanup = func() {}
+	if opts.WithCache {
+		// use the default cache, no setup needed
+		return cleanup
+	}
+
+	if gopts.NoCache {
+		// don't use any cache, no setup needed
+		return cleanup
+	}
+
+	cachedir := gopts.CacheDir
+
+	// use a cache in a temporary directory
+	tempdir, err := ioutil.TempDir(cachedir, "restic-check-cache-")
+	if err != nil {
+		// if an error occurs, don't use any cache
+		Warnf("unable to create temporary directory for cache during check, disabling cache: %v\n", err)
+		gopts.NoCache = true
+		return cleanup
+	}
+
+	gopts.CacheDir = tempdir
+	Verbosef("using temporary cache in %v\n", tempdir)
+
+	cleanup = func() {
+		err := fs.RemoveAll(tempdir)
+		if err != nil {
+			Warnf("error removing temporary cache directory: %v\n", err)
+		}
+	}
+
+	return cleanup
+}
+
 func runCheck(opts CheckOptions, gopts GlobalOptions, args []string) error {
 	if len(args) != 0 {
 		return errors.Fatal("check has no arguments")
 	}
 
-	if !opts.WithCache {
-		// do not use a cache for the checker
-		gopts.NoCache = true
-	}
+	cleanup := prepareCheckCache(opts, &gopts)
+	AddCleanupHandler(func() error {
+		cleanup()
+		return nil
+	})
 
 	repo, err := OpenRepository(gopts)
 	if err != nil {
@@ -114,7 +200,7 @@ func runCheck(opts CheckOptions, gopts GlobalOptions, args []string) error {
 	}
 
 	if dupFound {
-		Printf("\nrun `restic rebuild-index' to correct this\n")
+		Printf("This is non-critical, you can run `restic rebuild-index' to correct this\n")
 	}
 
 	if len(errs) > 0 {
@@ -125,16 +211,26 @@ func runCheck(opts CheckOptions, gopts GlobalOptions, args []string) error {
 	}
 
 	errorsFound := false
+	orphanedPacks := 0
 	errChan := make(chan error)
 
 	Verbosef("check all packs\n")
 	go chkr.Packs(gopts.ctx, errChan)
 
 	for err := range errChan {
+		if checker.IsOrphanedPack(err) {
+			orphanedPacks++
+			Verbosef("%v\n", err)
+			continue
+		}
 		errorsFound = true
 		fmt.Fprintf(os.Stderr, "%v\n", err)
 	}
 
+	if orphanedPacks > 0 {
+		Verbosef("%d additional files were found in the repo, which likely contain duplicate data.\nYou can run `restic prune` to correct this.\n", orphanedPacks)
+	}
+
 	Verbosef("check snapshots, trees and blobs\n")
 	errChan = make(chan error)
 	go chkr.Structure(gopts.ctx, errChan)
@@ -158,13 +254,25 @@ func runCheck(opts CheckOptions, gopts GlobalOptions, args []string) error {
 		}
 	}
 
-	if opts.ReadData {
-		Verbosef("read all data\n")
+	doReadData := func(bucket, totalBuckets uint) {
+		packs := restic.IDSet{}
+		for pack := range chkr.GetPacks() {
+			if (uint(pack[0]) % totalBuckets) == (bucket - 1) {
+				packs.Insert(pack)
+			}
+		}
+		packCount := uint64(len(packs))
 
-		p := newReadProgress(gopts, restic.Stat{Blobs: chkr.CountPacks()})
+		if packCount < chkr.CountPacks() {
+			Verbosef(fmt.Sprintf("read group #%d of %d data packs (out of total %d packs in %d groups)\n", bucket, packCount, chkr.CountPacks(), totalBuckets))
+		} else {
+			Verbosef("read all data\n")
+		}
+
+		p := newReadProgress(gopts, restic.Stat{Blobs: packCount})
 		errChan := make(chan error)
 
-		go chkr.ReadData(gopts.ctx, p, errChan)
+		go chkr.ReadPacks(gopts.ctx, packs, p, errChan)
 
 		for err := range errChan {
 			errorsFound = true
@@ -172,6 +280,14 @@ func runCheck(opts CheckOptions, gopts GlobalOptions, args []string) error {
 		}
 	}
 
+	switch {
+	case opts.ReadData:
+		doReadData(1, 1)
+	case opts.ReadDataSubset != "":
+		dataSubset, _ := stringToIntSlice(opts.ReadDataSubset)
+		doReadData(dataSubset[0], dataSubset[1])
+	}
+
 	if errorsFound {
 		return errors.Fatal("repository contains errors")
 	}

cmd/restic/cmd_find.go

@@ -3,7 +3,6 @@ package main
 import (
 	"context"
 	"encoding/json"
-	"path/filepath"
 	"strings"
 	"time"
 
@@ -11,7 +10,9 @@ import (
 
 	"github.com/restic/restic/internal/debug"
 	"github.com/restic/restic/internal/errors"
+	"github.com/restic/restic/internal/filter"
 	"github.com/restic/restic/internal/restic"
+	"github.com/restic/restic/internal/walker"
 )
 
 var cmdFind = &cobra.Command{
@@ -94,7 +95,7 @@ type statefulOutput struct {
 	hits     int
 }
 
-func (s *statefulOutput) PrintJSON(prefix string, node *restic.Node) {
+func (s *statefulOutput) PrintJSON(path string, node *restic.Node) {
 	type findNode restic.Node
 	b, err := json.Marshal(struct {
 		// Add these attributes
@@ -111,7 +112,7 @@ func (s *statefulOutput) PrintJSON(prefix string, node *restic.Node) {
 		Content            byte `json:"content,omitempty"`
 		Subtree            byte `json:"subtree,omitempty"`
 	}{
-		Path:        filepath.Join(prefix, node.Name),
+		Path:        path,
 		Permissions: node.Mode.String(),
 		findNode:    (*findNode)(node),
 	})
@@ -138,22 +139,22 @@ func (s *statefulOutput) PrintJSON(prefix string, node *restic.Node) {
 	s.hits++
 }
 
-func (s *statefulOutput) PrintNormal(prefix string, node *restic.Node) {
+func (s *statefulOutput) PrintNormal(path string, node *restic.Node) {
 	if s.newsn != s.oldsn {
 		if s.oldsn != nil {
 			Verbosef("\n")
 		}
 		s.oldsn = s.newsn
-		Verbosef("Found matching entries in snapshot %s\n", s.oldsn.ID())
+		Verbosef("Found matching entries in snapshot %s\n", s.oldsn.ID().Str())
 	}
-	Printf(formatNode(prefix, node, s.ListLong) + "\n")
+	Printf(formatNode(path, node, s.ListLong) + "\n")
 }
 
-func (s *statefulOutput) Print(prefix string, node *restic.Node) {
+func (s *statefulOutput) Print(path string, node *restic.Node) {
 	if s.JSON {
-		s.PrintJSON(prefix, node)
+		s.PrintJSON(path, node)
 	} else {
-		s.PrintNormal(prefix, node)
+		s.PrintNormal(path, node)
 	}
 }
 
@@ -174,74 +175,75 @@ func (s *statefulOutput) Finish() {
 
 // Finder bundles information needed to find a file or directory.
 type Finder struct {
-	repo     restic.Repository
-	pat      findPattern
-	out      statefulOutput
-	notfound restic.IDSet
-}
-
-func (f *Finder) findInTree(ctx context.Context, treeID restic.ID, prefix string) error {
-	if f.notfound.Has(treeID) {
-		debug.Log("%v skipping tree %v, has already been checked", prefix, treeID)
-		return nil
-	}
-
-	debug.Log("%v checking tree %v\n", prefix, treeID)
-
-	tree, err := f.repo.LoadTree(ctx, treeID)
-	if err != nil {
-		return err
-	}
-
-	var found bool
-	for _, node := range tree.Nodes {
-		debug.Log("  testing entry %q\n", node.Name)
-
-		name := node.Name
-		if f.pat.ignoreCase {
-			name = strings.ToLower(name)
-		}
-
-		m, err := filepath.Match(f.pat.pattern, name)
-		if err != nil {
-			return err
-		}
-
-		if m {
-			if !f.pat.oldest.IsZero() && node.ModTime.Before(f.pat.oldest) {
-				debug.Log("    ModTime is older than %s\n", f.pat.oldest)
-				continue
-			}
-
-			if !f.pat.newest.IsZero() && node.ModTime.After(f.pat.newest) {
-				debug.Log("    ModTime is newer than %s\n", f.pat.newest)
-				continue
-			}
-
-			debug.Log("    found match\n")
-			found = true
-			f.out.Print(prefix, node)
-		}
-
-		if node.Type == "dir" {
-			if err := f.findInTree(ctx, *node.Subtree, filepath.Join(prefix, node.Name)); err != nil {
-				return err
-			}
-		}
-	}
-
-	if !found {
-		f.notfound.Insert(treeID)
-	}
-
-	return nil
+	repo        restic.Repository
+	pat         findPattern
+	out         statefulOutput
+	ignoreTrees restic.IDSet
 }
 
 func (f *Finder) findInSnapshot(ctx context.Context, sn *restic.Snapshot) error {
 	debug.Log("searching in snapshot %s\n  for entries within [%s %s]", sn.ID(), f.pat.oldest, f.pat.newest)
 
+	if sn.Tree == nil {
+		return errors.Errorf("snapshot %v has no tree", sn.ID().Str())
+	}
+
 	f.out.newsn = sn
-	return f.findInTree(ctx, *sn.Tree, string(filepath.Separator))
+	return walker.Walk(ctx, f.repo, *sn.Tree, f.ignoreTrees, func(nodepath string, node *restic.Node, err error) (bool, error) {
+		if err != nil {
+			return false, err
+		}
+
+		if node == nil {
+			return false, nil
+		}
+
+		normalizedNodepath := nodepath
+		if f.pat.ignoreCase {
+			normalizedNodepath = strings.ToLower(nodepath)
+		}
+
+		foundMatch, err := filter.Match(f.pat.pattern, normalizedNodepath)
+		if err != nil {
+			return false, err
+		}
+
+		var (
+			ignoreIfNoMatch = true
+			errIfNoMatch    error
+		)
+		if node.Type == "dir" {
+			childMayMatch, err := filter.ChildMatch(f.pat.pattern, normalizedNodepath)
+			if err != nil {
+				return false, err
+			}
+
+			if !childMayMatch {
+				ignoreIfNoMatch = true
+				errIfNoMatch = walker.SkipNode
+			} else {
+				ignoreIfNoMatch = false
+			}
+		}
+
+		if !foundMatch {
+			return ignoreIfNoMatch, errIfNoMatch
+		}
+
+		if !f.pat.oldest.IsZero() && node.ModTime.Before(f.pat.oldest) {
+			debug.Log("    ModTime is older than %s\n", f.pat.oldest)
+			return ignoreIfNoMatch, errIfNoMatch
+		}
+
+		if !f.pat.newest.IsZero() && node.ModTime.After(f.pat.newest) {
+			debug.Log("    ModTime is newer than %s\n", f.pat.newest)
+			return ignoreIfNoMatch, errIfNoMatch
+		}
+
+		debug.Log("    found match\n")
+		f.out.Print(nodepath, node)
+		return false, nil
+	})
 }
 
 func runFind(opts FindOptions, gopts GlobalOptions, args []string) error {
@@ -289,10 +291,10 @@ func runFind(opts FindOptions, gopts GlobalOptions, args []string) error {
 	defer cancel()
 
 	f := &Finder{
-		repo:     repo,
-		pat:      pat,
-		out:      statefulOutput{ListLong: opts.ListLong, JSON: globalOptions.JSON},
-		notfound: restic.NewIDSet(),
+		repo:        repo,
+		pat:         pat,
+		out:         statefulOutput{ListLong: opts.ListLong, JSON: globalOptions.JSON},
+		ignoreTrees: restic.NewIDSet(),
 	}
 	for sn := range FindFilteredSnapshots(ctx, repo, opts.Host, opts.Tags, opts.Paths, opts.Snapshots) {
 		if err = f.findInSnapshot(ctx, sn); err != nil {

cmd/restic/cmd_forget.go

@@ -33,6 +33,7 @@ type ForgetOptions struct {
 	Weekly   int
 	Monthly  int
 	Yearly   int
+	Within   restic.Duration
 	KeepTags restic.TagLists
 
 	Host    string
@@ -58,6 +59,7 @@ func init() {
 	f.IntVarP(&forgetOptions.Weekly, "keep-weekly", "w", 0, "keep the last `n` weekly snapshots")
 	f.IntVarP(&forgetOptions.Monthly, "keep-monthly", "m", 0, "keep the last `n` monthly snapshots")
 	f.IntVarP(&forgetOptions.Yearly, "keep-yearly", "y", 0, "keep the last `n` yearly snapshots")
+	f.VarP(&forgetOptions.Within, "keep-within", "", "keep snapshots that were created within `duration` before the newest (e.g. 1y5m7d)")
 
 	f.Var(&forgetOptions.KeepTags, "keep-tag", "keep snapshots with this `taglist` (can be specified multiple times)")
 	// Sadly the commonly used shortcut `H` is already used.
@@ -170,6 +172,7 @@ func runForget(opts ForgetOptions, gopts GlobalOptions, args []string) error {
 		Weekly:  opts.Weekly,
 		Monthly: opts.Monthly,
 		Yearly:  opts.Yearly,
+		Within:  opts.Within,
 		Tags:    opts.KeepTags,
 	}
 
@@ -178,6 +181,8 @@ func runForget(opts ForgetOptions, gopts GlobalOptions, args []string) error {
 	}
 
 	if !policy.Empty() {
+		Verbosef("Applying Policy: %v\n", policy)
+
 		for k, snapshotGroup := range snapshotGroups {
 			var key key
 			if json.Unmarshal([]byte(k), &key) != nil {

cmd/restic/cmd_key.go

@@ -2,7 +2,11 @@ package main
 
 import (
 	"context"
+	"encoding/json"
 	"fmt"
+	"io/ioutil"
+	"os"
+	"strings"
 
 	"github.com/restic/restic/internal/errors"
 	"github.com/restic/restic/internal/repository"
@@ -23,37 +27,84 @@ The "key" command manages keys (passwords) for accessing the repository.
 	},
 }
 
+var newPasswordFile string
+
 func init() {
 	cmdRoot.AddCommand(cmdKey)
+
+	flags := cmdKey.Flags()
+	flags.StringVarP(&newPasswordFile, "new-password-file", "", "", "the file from which to load a new password")
 }
 
-func listKeys(ctx context.Context, s *repository.Repository) error {
-	tab := NewTable()
-	tab.Header = fmt.Sprintf(" %-10s  %-10s  %-10s  %s", "ID", "User", "Host", "Created")
-	tab.RowFormat = "%s%-10s  %-10s  %-10s  %s"
+type keyInfo struct {
+	Current  bool   `json:"current"`
+	ID       string `json:"id"`
+	UserName string `json:"userName"`
+	HostName string `json:"hostName"`
+	Created  string `json:"created"`
+}
 
-	err := s.List(ctx, restic.KeyFile, func(id restic.ID, size int64) error {
+func (ki keyInfo) CurrentStr() string {
+	if ki.Current {
+		return "*"
+	}
+	return " "
+}
+
+func listKeys(ctx context.Context, s *repository.Repository, gopts GlobalOptions) error {
+	var (
+		appendKey func(keyInfo)
+		printKeys func() error
+	)
+
+	switch gopts.JSON {
+	case true:
+		var keys []keyInfo
+
+		appendKey = func(key keyInfo) {
+			keys = append(keys, key)
+		}
+
+		printKeys = func() error {
+			return json.NewEncoder(gopts.stdout).Encode(keys)
+		}
+	default:
+		tab := NewTable()
+		tab.Header = fmt.Sprintf(" %-10s  %-10s  %-10s  %s", "ID", "User", "Host", "Created")
+		tab.RowFormat = "%s%-10s  %-10s  %-10s  %s"
+
+		appendKey = func(key keyInfo) {
+			tab.Rows = append(tab.Rows, []interface{}{key.CurrentStr(), key.ID, key.UserName, key.HostName, key.Created})
+		}
+
+		printKeys = func() error {
+			return tab.Write(globalOptions.stdout)
+		}
+	}
+
+	if err := s.List(ctx, restic.KeyFile, func(id restic.ID, size int64) error {
 		k, err := repository.LoadKey(ctx, s, id.String())
 		if err != nil {
 			Warnf("LoadKey() failed: %v\n", err)
 			return nil
 		}
 
-		var current string
-		if id.String() == s.KeyName() {
-			current = "*"
-		} else {
-			current = " "
+		key := keyInfo{
+			Current:  id.String() == s.KeyName(),
+			ID:       id.Str(),
+			UserName: k.Username,
+			HostName: k.Hostname,
+			Created:  k.Created.Format(TimeFormat),
 		}
-		tab.Rows = append(tab.Rows, []interface{}{current, id.Str(),
-			k.Username, k.Hostname, k.Created.Format(TimeFormat)})
+
+		appendKey(key)
+
 		return nil
-	})
-	if err != nil {
+	}); err != nil {
 		return err
 	}
 
-	return tab.Write(globalOptions.stdout)
+	return printKeys()
 }
 
 // testKeyNewPassword is used to set a new password during integration testing.
@@ -64,6 +115,10 @@ func getNewPassword(gopts GlobalOptions) (string, error) {
 		return testKeyNewPassword, nil
 	}
 
+	if newPasswordFile != "" {
+		return loadPasswordFromFile(newPasswordFile)
+	}
+
 	// Since we already have an open repository, temporary remove the password
 	// to prompt the user for the passwd.
 	newopts := gopts
@@ -148,7 +203,7 @@ func runKey(gopts GlobalOptions, args []string) error {
 			return err
 		}
 
-		return listKeys(ctx, repo)
+		return listKeys(ctx, repo, gopts)
 	case "add":
 		lock, err := lockRepo(repo)
 		defer unlockRepo(lock)
@@ -182,3 +237,11 @@ func runKey(gopts GlobalOptions, args []string) error {
 
 	return nil
 }
+
+func loadPasswordFromFile(pwdFile string) (string, error) {
+	s, err := ioutil.ReadFile(pwdFile)
+	if os.IsNotExist(err) {
+		return "", errors.Fatalf("%s does not exist", pwdFile)
+	}
+	return strings.TrimSpace(string(s)), errors.Wrap(err, "Readfile")
+}

cmd/restic/cmd_list.go

@@ -18,7 +18,7 @@ The "list" command allows listing objects in the repository based on type.
 `,
 	DisableAutoGenTag: true,
 	RunE: func(cmd *cobra.Command, args []string) error {
-		return runList(globalOptions, args)
+		return runList(cmd, globalOptions, args)
 	},
 }
 
@@ -26,9 +26,9 @@ func init() {
 	cmdRoot.AddCommand(cmdList)
 }
 
-func runList(opts GlobalOptions, args []string) error {
+func runList(cmd *cobra.Command, opts GlobalOptions, args []string) error {
 	if len(args) != 1 {
-		return errors.Fatal("type not specified")
+		return errors.Fatal("type not specified, usage: " + cmd.Use)
 	}
 
 	repo, err := OpenRepository(opts)

cmd/restic/cmd_ls.go

@@ -2,13 +2,12 @@ package main
 
 import (
 	"context"
-	"path/filepath"
 
 	"github.com/spf13/cobra"
 
 	"github.com/restic/restic/internal/errors"
-	"github.com/restic/restic/internal/repository"
 	"github.com/restic/restic/internal/restic"
+	"github.com/restic/restic/internal/walker"
 )
 
 var cmdLs = &cobra.Command{
@@ -46,25 +45,6 @@ func init() {
 	flags.StringArrayVar(&lsOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path`, when no snapshot ID is given")
 }
 
-func printTree(ctx context.Context, repo *repository.Repository, id *restic.ID, prefix string) error {
-	tree, err := repo.LoadTree(ctx, *id)
-	if err != nil {
-		return err
-	}
-
-	for _, entry := range tree.Nodes {
-		Printf("%s\n", formatNode(prefix, entry, lsOptions.ListLong))
-
-		if entry.Type == "dir" && entry.Subtree != nil {
-			if err = printTree(ctx, repo, entry.Subtree, filepath.Join(prefix, entry.Name)); err != nil {
-				return err
-			}
-		}
-	}
-
-	return nil
-}
-
 func runLs(opts LsOptions, gopts GlobalOptions, args []string) error {
 	if len(args) == 0 && opts.Host == "" && len(opts.Tags) == 0 && len(opts.Paths) == 0 {
 		return errors.Fatal("Invalid arguments, either give one or more snapshot IDs or set filters.")
@@ -84,7 +64,18 @@ func runLs(opts LsOptions, gopts GlobalOptions, args []string) error {
 	for sn := range FindFilteredSnapshots(ctx, repo, opts.Host, opts.Tags, opts.Paths, args) {
 		Verbosef("snapshot %s of %v at %s):\n", sn.ID().Str(), sn.Paths, sn.Time)
 
-		if err = printTree(gopts.ctx, repo, sn.Tree, string(filepath.Separator)); err != nil {
+		err := walker.Walk(ctx, repo, *sn.Tree, nil, func(nodepath string, node *restic.Node, err error) (bool, error) {
+			if err != nil {
+				return false, err
+			}
+
+			if node == nil {
+				return false, nil
+			}
+			Printf("%s\n", formatNode(nodepath, node, lsOptions.ListLong))
+			return false, nil
+		})
+		if err != nil {
 			return err
 		}
 	}

cmd/restic/cmd_mount.go

@@ -1,4 +1,6 @@
+// +build !netbsd
 // +build !openbsd
+// +build !solaris
 // +build !windows
 
 package main

cmd/restic/cmd_restore.go

@@ -5,6 +5,7 @@ import (
 	"github.com/restic/restic/internal/errors"
 	"github.com/restic/restic/internal/filter"
 	"github.com/restic/restic/internal/restic"
+	"github.com/restic/restic/internal/restorer"
 
 	"github.com/spf13/cobra"
 )
@@ -33,6 +34,7 @@ type RestoreOptions struct {
 	Host    string
 	Paths   []string
 	Tags    restic.TagLists
+	Verify  bool
 }
 
 var restoreOptions RestoreOptions
@@ -48,6 +50,7 @@ func init() {
 	flags.StringVarP(&restoreOptions.Host, "host", "H", "", `only consider snapshots for this host when the snapshot ID is "latest"`)
 	flags.Var(&restoreOptions.Tags, "tag", "only consider snapshots which include this `taglist` for snapshot ID \"latest\"")
 	flags.StringArrayVar(&restoreOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path` for snapshot ID \"latest\"")
+	flags.BoolVar(&restoreOptions.Verify, "verify", false, "verify restored files content")
 }
 
 func runRestore(opts RestoreOptions, gopts GlobalOptions, args []string) error {
@@ -104,7 +107,7 @@ func runRestore(opts RestoreOptions, gopts GlobalOptions, args []string) error {
 		}
 	}
 
-	res, err := restic.NewRestorer(repo, id)
+	res, err := restorer.NewRestorer(repo, id)
 	if err != nil {
 		Exitf(2, "creating restorer failed: %v\n", err)
 	}
@@ -153,6 +156,12 @@ func runRestore(opts RestoreOptions, gopts GlobalOptions, args []string) error {
 	Verbosef("restoring %s to %s\n", res.Snapshot(), opts.Target)
 
 	err = res.RestoreTo(ctx, opts.Target)
+	if err == nil && opts.Verify {
+		Verbosef("verifying files in %s\n", opts.Target)
+		var count int
+		count, err = res.VerifyFiles(ctx, opts.Target)
+		Verbosef("finished verifying %d files in %s\n", count, opts.Target)
+	}
 	if totalErrors > 0 {
 		Printf("There were %d errors\n", totalErrors)
 	}

cmd/restic/cmd_stats.go

@@ -0,0 +1,314 @@
+package main
+
+import (
+	"context"
+	"crypto/sha256"
+	"encoding/json"
+	"fmt"
+	"os"
+	"path/filepath"
+
+	"github.com/restic/restic/internal/restic"
+	"github.com/restic/restic/internal/walker"
+	"github.com/spf13/cobra"
+)
+
+var cmdStats = &cobra.Command{
+	Use:   "stats [flags] [snapshot-ID]",
+	Short: "Scan the repository and show basic statistics",
+	Long: `
+The "stats" command walks one or all snapshots in a repository and
+accumulates statistics about the data stored therein. It reports on
+the number of unique files and their sizes, according to one of
+the counting modes as given by the --mode flag.
+
+If no snapshot is specified, all snapshots will be considered. Some
+modes make more sense over just a single snapshot, while others
+are useful across all snapshots, depending on what you are trying
+to calculate.
+
+The modes are:
+
+  restore-size: (default) Counts the size of the restored files.
+
+  files-by-contents: Counts total size of files, where a file is
+                     considered unique if it has unique contents.
+
+  raw-data: Counts the size of blobs in the repository, regardless
+			of how many files reference them.
+
+  blobs-per-file: A combination of files-by-contents and raw-data.
+
+Refer to the online manual for more details about each mode.
+`,
+	DisableAutoGenTag: true,
+	RunE: func(cmd *cobra.Command, args []string) error {
+		return runStats(globalOptions, args)
+	},
+}
+
+func init() {
+	cmdRoot.AddCommand(cmdStats)
+	f := cmdStats.Flags()
+	f.StringVar(&countMode, "mode", countModeRestoreSize, "counting mode: restore-size (default), files-by-contents, blobs-per-file, or raw-data")
+	f.StringVar(&snapshotByHost, "host", "", "filter latest snapshot by this hostname")
+}
+
+func runStats(gopts GlobalOptions, args []string) error {
+	err := verifyStatsInput(gopts, args)
+	if err != nil {
+		return err
+	}
+
+	ctx, cancel := context.WithCancel(gopts.ctx)
+	defer cancel()
+
+	repo, err := OpenRepository(gopts)
+	if err != nil {
+		return err
+	}
+
+	if err = repo.LoadIndex(ctx); err != nil {
+		return err
+	}
+
+	if !gopts.NoLock {
+		lock, err := lockRepo(repo)
+		defer unlockRepo(lock)
+		if err != nil {
+			return err
+		}
+	}
+
+	// create a container for the stats (and other needed state)
+	stats := &statsContainer{
+		uniqueFiles: make(map[fileID]struct{}),
+		fileBlobs:   make(map[string]restic.IDSet),
+		blobs:       restic.NewBlobSet(),
+		blobsSeen:   restic.NewBlobSet(),
+	}
+
+	if snapshotIDString != "" {
+		// scan just a single snapshot
+
+		var sID restic.ID
+		if snapshotIDString == "latest" {
+			sID, err = restic.FindLatestSnapshot(ctx, repo, []string{}, []restic.TagList{}, snapshotByHost)
+			if err != nil {
+				Exitf(1, "latest snapshot for criteria not found: %v", err)
+			}
+		} else {
+			sID, err = restic.FindSnapshot(repo, snapshotIDString)
+			if err != nil {
+				return err
+			}
+		}
+
+		snapshot, err := restic.LoadSnapshot(ctx, repo, sID)
+		if err != nil {
+			return err
+		}
+
+		err = statsWalkSnapshot(ctx, snapshot, repo, stats)
+	} else {
+		// iterate every snapshot in the repo
+		err = repo.List(ctx, restic.SnapshotFile, func(snapshotID restic.ID, size int64) error {
+			snapshot, err := restic.LoadSnapshot(ctx, repo, snapshotID)
+			if err != nil {
+				return fmt.Errorf("Error loading snapshot %s: %v", snapshotID.Str(), err)
+			}
+			return statsWalkSnapshot(ctx, snapshot, repo, stats)
+		})
+	}
+	if err != nil {
+		return err
+	}
+
+	if countMode == countModeRawData {
+		// the blob handles have been collected, but not yet counted
+		for blobHandle := range stats.blobs {
+			blobSize, found := repo.LookupBlobSize(blobHandle.ID, blobHandle.Type)
+			if !found {
+				return fmt.Errorf("blob %v not found", blobHandle)
+			}
+			stats.TotalSize += uint64(blobSize)
+			stats.TotalBlobCount++
+		}
+	}
+
+	if gopts.JSON {
+		err = json.NewEncoder(os.Stdout).Encode(stats)
+		if err != nil {
+			return fmt.Errorf("encoding output: %v", err)
+		}
+		return nil
+	}
+
+	if stats.TotalBlobCount > 0 {
+		Printf("  Total Blob Count:   %d\n", stats.TotalBlobCount)
+	}
+	if stats.TotalFileCount > 0 {
+		Printf("  Total File Count:   %d\n", stats.TotalFileCount)
+	}
+	Printf("        Total Size:   %-5s\n", formatBytes(stats.TotalSize))
+
+	return nil
+}
+
+func statsWalkSnapshot(ctx context.Context, snapshot *restic.Snapshot, repo restic.Repository, stats *statsContainer) error {
+	if snapshot.Tree == nil {
+		return fmt.Errorf("snapshot %s has nil tree", snapshot.ID().Str())
+	}
+
+	if countMode == countModeRawData {
+		// count just the sizes of unique blobs; we don't need to walk the tree
+		// ourselves in this case, since a nifty function does it for us
+		return restic.FindUsedBlobs(ctx, repo, *snapshot.Tree, stats.blobs, stats.blobsSeen)
+	}
+
+	err := walker.Walk(ctx, repo, *snapshot.Tree, restic.NewIDSet(), statsWalkTree(repo, stats))
+	if err != nil {
+		return fmt.Errorf("walking tree %s: %v", *snapshot.Tree, err)
+	}
+	return nil
+}
+
+func statsWalkTree(repo restic.Repository, stats *statsContainer) walker.WalkFunc {
+	return func(npath string, node *restic.Node, nodeErr error) (bool, error) {
+		if nodeErr != nil {
+			return true, nodeErr
+		}
+		if node == nil {
+			return true, nil
+		}
+
+		if countMode == countModeUniqueFilesByContents || countMode == countModeBlobsPerFile {
+			// only count this file if we haven't visited it before
+			fid := makeFileIDByContents(node)
+			if _, ok := stats.uniqueFiles[fid]; !ok {
+				// mark the file as visited
+				stats.uniqueFiles[fid] = struct{}{}
+
+				if countMode == countModeUniqueFilesByContents {
+					// simply count the size of each unique file (unique by contents only)
+					stats.TotalSize += node.Size
+					stats.TotalFileCount++
+				}
+				if countMode == countModeBlobsPerFile {
+					// count the size of each unique blob reference, which is
+					// by unique file (unique by contents and file path)
+					for _, blobID := range node.Content {
+						// ensure we have this file (by path) in our map; in this
+						// mode, a file is unique by both contents and path
+						nodePath := filepath.Join(npath, node.Name)
+						if _, ok := stats.fileBlobs[nodePath]; !ok {
+							stats.fileBlobs[nodePath] = restic.NewIDSet()
+							stats.TotalFileCount++
+						}
+						if _, ok := stats.fileBlobs[nodePath][blobID]; !ok {
+							// is always a data blob since we're accessing it via a file's Content array
+							blobSize, found := repo.LookupBlobSize(blobID, restic.DataBlob)
+							if !found {
+								return true, fmt.Errorf("blob %s not found for tree %s", blobID, *node.Subtree)
+							}
+
+							// count the blob's size, then add this blob by this
+							// file (path) so we don't double-count it
+							stats.TotalSize += uint64(blobSize)
+							stats.fileBlobs[nodePath].Insert(blobID)
+							// this mode also counts total unique blob _references_ per file
+							stats.TotalBlobCount++
+						}
+					}
+				}
+			}
+		}
+
+		if countMode == countModeRestoreSize {
+			// as this is a file in the snapshot, we can simply count its
+			// size without worrying about uniqueness, since duplicate files
+			// will still be restored
+			stats.TotalSize += node.Size
+			stats.TotalFileCount++
+		}
+
+		return true, nil
+	}
+}
+
+// makeFileIDByContents returns a hash of the blob IDs of the
+// node's Content in sequence.
+func makeFileIDByContents(node *restic.Node) fileID {
+	var bb []byte
+	for _, c := range node.Content {
+		bb = append(bb, []byte(c[:])...)
+	}
+	return sha256.Sum256(bb)
+}
+
+func verifyStatsInput(gopts GlobalOptions, args []string) error {
+	// require a recognized counting mode
+	switch countMode {
+	case countModeRestoreSize:
+	case countModeUniqueFilesByContents:
+	case countModeBlobsPerFile:
+	case countModeRawData:
+	default:
+		return fmt.Errorf("unknown counting mode: %s (use the -h flag to get a list of supported modes)", countMode)
+	}
+
+	// ensure at most one snapshot was specified
+	if len(args) > 1 {
+		return fmt.Errorf("only one snapshot may be specified")
+	}
+
+	// if a snapshot was specified, mark it as the one to scan
+	if len(args) == 1 {
+		snapshotIDString = args[0]
+	}
+
+	return nil
+}
+
+// statsContainer holds information during a walk of a repository
+// to collect information about it, as well as state needed
+// for a successful and efficient walk.
+type statsContainer struct {
+	TotalSize      uint64 `json:"total_size"`
+	TotalFileCount uint64 `json:"total_file_count"`
+	TotalBlobCount uint64 `json:"total_blob_count,omitempty"`
+
+	// uniqueFiles marks visited files according to their
+	// contents (hashed sequence of content blob IDs)
+	uniqueFiles map[fileID]struct{}
+
+	// fileBlobs maps a file name (path) to the set of
+	// blobs that have been seen as a part of the file
+	fileBlobs map[string]restic.IDSet
+
+	// blobs and blobsSeen are used to count indiviudal
+	// unique blobs, independent of references to files
+	blobs, blobsSeen restic.BlobSet
+}
+
+// fileID is a 256-bit hash that distinguishes unique files.
+type fileID [32]byte
+
+var (
+	// the mode of counting to perform
+	countMode string
+
+	// the snapshot to scan, as given by the user
+	snapshotIDString string
+
+	// snapshotByHost is the host to filter latest
+	// snapshot by, if given by user
+	snapshotByHost string
+)
+
+const (
+	countModeRestoreSize           = "restore-size"
+	countModeUniqueFilesByContents = "files-by-contents"
+	countModeBlobsPerFile          = "blobs-per-file"
+	countModeRawData               = "raw-data"
+)

cmd/restic/cmd_version.go

@@ -16,7 +16,7 @@ and the version of this software.
 `,
 	DisableAutoGenTag: true,
 	Run: func(cmd *cobra.Command, args []string) {
-		fmt.Printf("restic %s\ncompiled with %v on %v/%v\n",
+		fmt.Printf("restic %s compiled with %v on %v/%v\n",
 			version, runtime.Version(), runtime.GOOS, runtime.GOARCH)
 	},
 }

cmd/restic/exclude.go

@@ -185,6 +185,11 @@ func isDirExcludedByFile(dir, tagFilename, header string) bool {
 func gatherDevices(items []string) (deviceMap map[string]uint64, err error) {
 	deviceMap = make(map[string]uint64)
 	for _, item := range items {
+		item, err = filepath.Abs(filepath.Clean(item))
+		if err != nil {
+			return nil, err
+		}
+
 		fi, err := fs.Lstat(item)
 		if err != nil {
 			return nil, err
@@ -215,6 +220,8 @@ func rejectByDevice(samples []string) (RejectFunc, error) {
 			return false
 		}
 
+		item = filepath.Clean(item)
+
 		id, err := fs.DeviceID(fi)
 		if err != nil {
 			// This should never happen because gatherDevices() would have
@@ -222,11 +229,14 @@ func rejectByDevice(samples []string) (RejectFunc, error) {
 			panic(err)
 		}
 
-		for dir := item; dir != ""; dir = filepath.Dir(dir) {
+		for dir := item; ; dir = filepath.Dir(dir) {
 			debug.Log("item %v, test dir %v", item, dir)
 
 			allowedID, ok := allowed[dir]
 			if !ok {
+				if dir == filepath.Dir(dir) {
+					break
+				}
 				continue
 			}
 

cmd/restic/excludes

@@ -1,31 +0,0 @@
-/boot
-/dev
-/etc
-/home
-/lost+found
-/mnt
-/proc
-/root
-/run
-/sys
-/tmp
-/usr
-/var
-/opt/android-sdk
-/opt/bullet
-/opt/dex2jar
-/opt/jameica
-/opt/google
-/opt/JDownloader
-/opt/JDownloaderScripts
-/opt/opencascade
-/opt/vagrant
-/opt/visual-studio-code
-/opt/vtk6
-/bin
-/fonts*
-/srv/ftp
-/srv/http
-/sbin
-/lib
-/lib64

cmd/restic/format.go

@@ -3,7 +3,6 @@ package main
 import (
 	"fmt"
 	"os"
-	"path/filepath"
 	"time"
 
 	"github.com/restic/restic/internal/restic"
@@ -63,9 +62,9 @@ func formatDuration(d time.Duration) string {
 	return formatSeconds(sec)
 }
 
-func formatNode(prefix string, n *restic.Node, long bool) string {
+func formatNode(path string, n *restic.Node, long bool) string {
 	if !long {
-		return filepath.Join(prefix, n.Name)
+		return path
 	}
 
 	var mode os.FileMode
@@ -91,6 +90,6 @@ func formatNode(prefix string, n *restic.Node, long bool) string {
 
 	return fmt.Sprintf("%s %5d %5d %6d %s %s%s",
 		mode|n.Mode, n.UID, n.GID, n.Size,
-		n.ModTime.Format(TimeFormat), filepath.Join(prefix, n.Name),
+		n.ModTime.Format(TimeFormat), path,
 		target)
 }

cmd/restic/global.go

@@ -4,7 +4,6 @@ import (
 	"context"
 	"fmt"
 	"io"
-	"io/ioutil"
 	"os"
 	"path/filepath"
 	"runtime"
@@ -18,6 +17,7 @@ import (
 	"github.com/restic/restic/internal/backend/gs"
 	"github.com/restic/restic/internal/backend/local"
 	"github.com/restic/restic/internal/backend/location"
+	"github.com/restic/restic/internal/backend/rclone"
 	"github.com/restic/restic/internal/backend/rest"
 	"github.com/restic/restic/internal/backend/s3"
 	"github.com/restic/restic/internal/backend/sftp"
@@ -29,6 +29,7 @@ import (
 	"github.com/restic/restic/internal/options"
 	"github.com/restic/restic/internal/repository"
 	"github.com/restic/restic/internal/restic"
+	"github.com/restic/restic/internal/textfile"
 
 	"github.com/restic/restic/internal/errors"
 
@@ -42,6 +43,7 @@ type GlobalOptions struct {
 	Repo          string
 	PasswordFile  string
 	Quiet         bool
+	Verbose       int
 	NoLock        bool
 	JSON          bool
 	CacheDir      string
@@ -58,6 +60,13 @@ type GlobalOptions struct {
 	stdout   io.Writer
 	stderr   io.Writer
 
+	// verbosity is set as follows:
+	//  0 means: don't print any messages except errors, this is used when --quiet is specified
+	//  1 is the default: print essential messages
+	//  2 means: print more messages, report minor things, this is used when --verbose is specified
+	//  3 means: print very detailed debug messages, this is used when --debug is specified
+	verbosity uint
+
 	Options []string
 
 	extended options.Options
@@ -80,11 +89,12 @@ func init() {
 	f.StringVarP(&globalOptions.Repo, "repo", "r", os.Getenv("RESTIC_REPOSITORY"), "repository to backup to or restore from (default: $RESTIC_REPOSITORY)")
 	f.StringVarP(&globalOptions.PasswordFile, "password-file", "p", os.Getenv("RESTIC_PASSWORD_FILE"), "read the repository password from a file (default: $RESTIC_PASSWORD_FILE)")
 	f.BoolVarP(&globalOptions.Quiet, "quiet", "q", false, "do not output comprehensive progress report")
+	f.CountVarP(&globalOptions.Verbose, "verbose", "v", "be verbose (specify --verbose multiple times or level `n`)")
 	f.BoolVar(&globalOptions.NoLock, "no-lock", false, "do not lock the repo, this allows some operations on read-only repos")
 	f.BoolVarP(&globalOptions.JSON, "json", "", false, "set output mode to JSON for commands that support it")
 	f.StringVar(&globalOptions.CacheDir, "cache-dir", "", "set the cache directory")
 	f.BoolVar(&globalOptions.NoCache, "no-cache", false, "do not use a local cache")
-	f.StringSliceVar(&globalOptions.CACerts, "cacert", nil, "path to load root certificates from (default: use system certificates)")
+	f.StringSliceVar(&globalOptions.CACerts, "cacert", nil, "`file` to load root certificates from (default: use system certificates)")
 	f.StringVar(&globalOptions.TLSClientCert, "tls-client-cert", "", "path to a file containing PEM encoded TLS client certificate and private key")
 	f.BoolVar(&globalOptions.CleanupCache, "cleanup-cache", false, "auto remove old cache directories")
 	f.IntVar(&globalOptions.LimitUploadKb, "limit-upload", 0, "limits uploads to a maximum rate in KiB/s. (default: unlimited)")
@@ -172,11 +182,9 @@ func Printf(format string, args ...interface{}) {
 
 // Verbosef calls Printf to write the message when the verbose flag is set.
 func Verbosef(format string, args ...interface{}) {
-	if globalOptions.Quiet {
-		return
+	if globalOptions.verbosity >= 1 {
+		Printf(format, args...)
 	}
-
-	Printf(format, args...)
 }
 
 // PrintProgress wraps fmt.Printf to handle the difference in writing progress
@@ -227,8 +235,8 @@ func Exitf(exitcode int, format string, args ...interface{}) {
 // resolvePassword determines the password to be used for opening the repository.
 func resolvePassword(opts GlobalOptions, env string) (string, error) {
 	if opts.PasswordFile != "" {
-		s, err := ioutil.ReadFile(opts.PasswordFile)
-		if os.IsNotExist(err) {
+		s, err := textfile.Read(opts.PasswordFile)
+		if os.IsNotExist(errors.Cause(err)) {
 			return "", errors.Fatalf("%s does not exist", opts.PasswordFile)
 		}
 		return strings.TrimSpace(string(s)), errors.Wrap(err, "Readfile")
@@ -285,6 +293,7 @@ func ReadPassword(opts GlobalOptions, prompt string) (string, error) {
 		password, err = readPasswordTerminal(os.Stdin, os.Stderr, prompt)
 	} else {
 		password, err = readPassword(os.Stdin)
+		Verbosef("read password from stdin\n")
 	}
 
 	if err != nil {
@@ -347,7 +356,11 @@ func OpenRepository(opts GlobalOptions) (*repository.Repository, error) {
 	}
 
 	if stdoutIsTerminal() {
-		Verbosef("password is correct\n")
+		id := s.Config().ID
+		if len(id) > 8 {
+			id = id[:8]
+		}
+		Verbosef("repository %v opened successfully, password is correct\n", id)
 	}
 
 	if opts.NoCache {
@@ -378,7 +391,7 @@ func OpenRepository(opts GlobalOptions) (*repository.Repository, error) {
 		Printf("removing %d old cache dirs from %v\n", len(oldCacheDirs), c.Base)
 
 		for _, item := range oldCacheDirs {
-			dir := filepath.Join(c.Base, item)
+			dir := filepath.Join(c.Base, item.Name())
 			err = fs.RemoveAll(dir)
 			if err != nil {
 				Warnf("unable to remove %v: %v\n", dir, err)
@@ -440,18 +453,6 @@ func parseConfig(loc location.Location, opts options.Options) (interface{}, erro
 			cfg.ProjectID = os.Getenv("GOOGLE_PROJECT_ID")
 		}
 
-		if cfg.JSONKeyPath == "" {
-			if path := os.Getenv("GOOGLE_APPLICATION_CREDENTIALS"); path != "" {
-				// Check read access
-				if _, err := ioutil.ReadFile(path); err != nil {
-					return nil, errors.Fatalf("Failed to read google credential from file %v: %v", path, err)
-				}
-				cfg.JSONKeyPath = path
-			} else {
-				return nil, errors.Fatal("No credential file path is set")
-			}
-		}
-
 		if err := opts.Apply(loc.Scheme, &cfg); err != nil {
 			return nil, err
 		}
@@ -521,6 +522,14 @@ func parseConfig(loc location.Location, opts options.Options) (interface{}, erro
 			return nil, err
 		}
 
+		debug.Log("opening rest repository at %#v", cfg)
+		return cfg, nil
+	case "rclone":
+		cfg := loc.Config.(rclone.Config)
+		if err := opts.Apply(loc.Scheme, &cfg); err != nil {
+			return nil, err
+		}
+
 		debug.Log("opening rest repository at %#v", cfg)
 		return cfg, nil
 	}
@@ -553,17 +562,18 @@ func open(s string, gopts GlobalOptions, opts options.Options) (restic.Backend,
 	}
 
 	// wrap the transport so that the throughput via HTTP is limited
-	rt = limiter.NewStaticLimiter(gopts.LimitUploadKb, gopts.LimitDownloadKb).Transport(rt)
+	lim := limiter.NewStaticLimiter(gopts.LimitUploadKb, gopts.LimitDownloadKb)
+	rt = lim.Transport(rt)
 
 	switch loc.Scheme {
 	case "local":
 		be, err = local.Open(cfg.(local.Config))
 		// wrap the backend in a LimitBackend so that the throughput is limited
-		be = limiter.LimitBackend(be, limiter.NewStaticLimiter(gopts.LimitUploadKb, gopts.LimitDownloadKb))
+		be = limiter.LimitBackend(be, lim)
 	case "sftp":
 		be, err = sftp.Open(cfg.(sftp.Config))
 		// wrap the backend in a LimitBackend so that the throughput is limited
-		be = limiter.LimitBackend(be, limiter.NewStaticLimiter(gopts.LimitUploadKb, gopts.LimitDownloadKb))
+		be = limiter.LimitBackend(be, lim)
 	case "s3":
 		be, err = s3.Open(cfg.(s3.Config), rt)
 	case "gs":
@@ -576,6 +586,8 @@ func open(s string, gopts GlobalOptions, opts options.Options) (restic.Backend,
 		be, err = b2.Open(globalOptions.ctx, cfg.(b2.Config), rt)
 	case "rest":
 		be, err = rest.Open(cfg.(rest.Config), rt)
+	case "rclone":
+		be, err = rclone.Open(cfg.(rclone.Config), lim)
 
 	default:
 		return nil, errors.Fatalf("invalid backend: %q", loc.Scheme)
@@ -637,6 +649,8 @@ func create(s string, opts options.Options) (restic.Backend, error) {
 		return b2.Create(globalOptions.ctx, cfg.(b2.Config), rt)
 	case "rest":
 		return rest.Create(cfg.(rest.Config), rt)
+	case "rclone":
+		return rclone.Open(cfg.(rclone.Config), nil)
 	}
 
 	debug.Log("invalid repository scheme: %v", s)

cmd/restic/global_debug.go

@@ -1,4 +1,4 @@
-// +build debug
+// +build debug profile
 
 package main
 
@@ -15,17 +15,21 @@ import (
 )
 
 var (
-	listenMemoryProfile string
-	memProfilePath      string
-	cpuProfilePath      string
-	insecure            bool
+	listenProfile    string
+	memProfilePath   string
+	cpuProfilePath   string
+	traceProfilePath string
+	blockProfilePath string
+	insecure         bool
 )
 
 func init() {
 	f := cmdRoot.PersistentFlags()
-	f.StringVar(&listenMemoryProfile, "listen-profile", "", "listen on this `address:port` for memory profiling")
+	f.StringVar(&listenProfile, "listen-profile", "", "listen on this `address:port` for memory profiling")
 	f.StringVar(&memProfilePath, "mem-profile", "", "write memory profile to `dir`")
 	f.StringVar(&cpuProfilePath, "cpu-profile", "", "write cpu profile to `dir`")
+	f.StringVar(&traceProfilePath, "trace-profile", "", "write trace to `dir`")
+	f.StringVar(&blockProfilePath, "block-profile", "", "write block profile to `dir`")
 	f.BoolVar(&insecure, "insecure-kdf", false, "use insecure KDF settings")
 }
 
@@ -36,18 +40,32 @@ func (fakeTestingTB) Logf(msg string, args ...interface{}) {
 }
 
 func runDebug() error {
-	if listenMemoryProfile != "" {
-		fmt.Fprintf(os.Stderr, "running memory profile HTTP server on %v\n", listenMemoryProfile)
+	if listenProfile != "" {
+		fmt.Fprintf(os.Stderr, "running profile HTTP server on %v\n", listenProfile)
 		go func() {
-			err := http.ListenAndServe(listenMemoryProfile, nil)
+			err := http.ListenAndServe(listenProfile, nil)
 			if err != nil {
-				fmt.Fprintf(os.Stderr, "memory profile listen failed: %v\n", err)
+				fmt.Fprintf(os.Stderr, "profile HTTP server listen failed: %v\n", err)
 			}
 		}()
 	}
 
-	if memProfilePath != "" && cpuProfilePath != "" {
-		return errors.Fatal("only one profile (memory or CPU) may be activated at the same time")
+	profilesEnabled := 0
+	if memProfilePath != "" {
+		profilesEnabled++
+	}
+	if cpuProfilePath != "" {
+		profilesEnabled++
+	}
+	if traceProfilePath != "" {
+		profilesEnabled++
+	}
+	if blockProfilePath != "" {
+		profilesEnabled++
+	}
+
+	if profilesEnabled > 1 {
+		return errors.Fatal("only one profile (memory, CPU, trace, or block) may be activated at the same time")
 	}
 
 	var prof interface {
@@ -58,6 +76,10 @@ func runDebug() error {
 		prof = profile.Start(profile.Quiet, profile.NoShutdownHook, profile.MemProfile, profile.ProfilePath(memProfilePath))
 	} else if cpuProfilePath != "" {
 		prof = profile.Start(profile.Quiet, profile.NoShutdownHook, profile.CPUProfile, profile.ProfilePath(cpuProfilePath))
+	} else if traceProfilePath != "" {
+		prof = profile.Start(profile.Quiet, profile.NoShutdownHook, profile.TraceProfile, profile.ProfilePath(traceProfilePath))
+	} else if blockProfilePath != "" {
+		prof = profile.Start(profile.Quiet, profile.NoShutdownHook, profile.BlockProfile, profile.ProfilePath(blockProfilePath))
 	}
 
 	if prof != nil {

cmd/restic/global_release.go

@@ -1,4 +1,4 @@
-// +build !debug
+// +build !debug,!profile
 
 package main
 

cmd/restic/integration_fuse_test.go

@@ -1,4 +1,6 @@
+// +build !netbsd
 // +build !openbsd
+// +build !solaris
 // +build !windows
 
 package main
@@ -170,7 +172,7 @@ func TestMount(t *testing.T) {
 	rtest.SetupTarTestFixture(t, env.testdata, filepath.Join("testdata", "backup-data.tar.gz"))
 
 	// first backup
-	testRunBackup(t, []string{env.testdata}, BackupOptions{}, env.gopts)
+	testRunBackup(t, "", []string{env.testdata}, BackupOptions{}, env.gopts)
 	snapshotIDs := testRunList(t, "snapshots", env.gopts)
 	rtest.Assert(t, len(snapshotIDs) == 1,
 		"expected one snapshot, got %v", snapshotIDs)
@@ -178,7 +180,7 @@ func TestMount(t *testing.T) {
 	checkSnapshots(t, env.gopts, repo, env.mountpoint, env.repo, snapshotIDs, 2)
 
 	// second backup, implicit incremental
-	testRunBackup(t, []string{env.testdata}, BackupOptions{}, env.gopts)
+	testRunBackup(t, "", []string{env.testdata}, BackupOptions{}, env.gopts)
 	snapshotIDs = testRunList(t, "snapshots", env.gopts)
 	rtest.Assert(t, len(snapshotIDs) == 2,
 		"expected two snapshots, got %v", snapshotIDs)
@@ -187,7 +189,7 @@ func TestMount(t *testing.T) {
 
 	// third backup, explicit incremental
 	bopts := BackupOptions{Parent: snapshotIDs[0].String()}
-	testRunBackup(t, []string{env.testdata}, bopts, env.gopts)
+	testRunBackup(t, "", []string{env.testdata}, bopts, env.gopts)
 	snapshotIDs = testRunList(t, "snapshots", env.gopts)
 	rtest.Assert(t, len(snapshotIDs) == 3,
 		"expected three snapshots, got %v", snapshotIDs)

cmd/restic/integration_helpers_test.go

@@ -11,6 +11,7 @@ import (
 
 	"github.com/restic/restic/internal/options"
 	"github.com/restic/restic/internal/repository"
+	"github.com/restic/restic/internal/restic"
 	rtest "github.com/restic/restic/internal/test"
 )
 
@@ -65,7 +66,7 @@ func isSymlink(fi os.FileInfo) bool {
 
 func sameModTime(fi1, fi2 os.FileInfo) bool {
 	switch runtime.GOOS {
-	case "darwin", "freebsd", "openbsd":
+	case "darwin", "freebsd", "openbsd", "netbsd":
 		if isSymlink(fi1) && isSymlink(fi2) {
 			return true
 		}
@@ -189,6 +190,7 @@ func withTestEnvironment(t testing.TB) (env *testEnvironment, cleanup func()) {
 	}
 
 	repository.TestUseLowSecurityKDFParameters(t)
+	restic.TestDisableCheckPolynomial(t)
 
 	tempdir, err := ioutil.TempDir(rtest.TestTempDir, "restic-test-")
 	rtest.OK(t, err)

cmd/restic/integration_test.go

@@ -3,6 +3,7 @@ package main
 import (
 	"bufio"
 	"bytes"
+	"context"
 	"crypto/rand"
 	"encoding/json"
 	"fmt"
@@ -17,12 +18,14 @@ import (
 	"testing"
 	"time"
 
-	"github.com/restic/restic/internal/debug"
 	"github.com/restic/restic/internal/errors"
 	"github.com/restic/restic/internal/filter"
+	"github.com/restic/restic/internal/fs"
 	"github.com/restic/restic/internal/repository"
 	"github.com/restic/restic/internal/restic"
 	rtest "github.com/restic/restic/internal/test"
+	"github.com/restic/restic/internal/ui/termstatus"
+	"golang.org/x/sync/errgroup"
 )
 
 func parseIDsFromReader(t testing.TB, rd io.Reader) restic.IDs {
@@ -44,15 +47,36 @@ func parseIDsFromReader(t testing.TB, rd io.Reader) restic.IDs {
 
 func testRunInit(t testing.TB, opts GlobalOptions) {
 	repository.TestUseLowSecurityKDFParameters(t)
+	restic.TestDisableCheckPolynomial(t)
 	restic.TestSetLockTimeout(t, 0)
 
 	rtest.OK(t, runInit(opts, nil))
 	t.Logf("repository initialized at %v", opts.Repo)
 }
 
-func testRunBackup(t testing.TB, target []string, opts BackupOptions, gopts GlobalOptions) {
-	t.Logf("backing up %v", target)
-	rtest.OK(t, runBackup(opts, gopts, target))
+func testRunBackup(t testing.TB, dir string, target []string, opts BackupOptions, gopts GlobalOptions) {
+	ctx, cancel := context.WithCancel(gopts.ctx)
+	defer cancel()
+
+	var wg errgroup.Group
+	term := termstatus.New(gopts.stdout, gopts.stderr, gopts.Quiet)
+	wg.Go(func() error { term.Run(ctx); return nil })
+
+	gopts.stdout = ioutil.Discard
+	t.Logf("backing up %v in %v", target, dir)
+	if dir != "" {
+		cleanup := fs.TestChdir(t, dir)
+		defer cleanup()
+	}
+
+	rtest.OK(t, runBackup(opts, gopts, term, target))
+
+	cancel()
+
+	err := wg.Wait()
+	if err != nil {
+		t.Fatal(err)
+	}
 }
 
 func testRunList(t testing.TB, tpe string, opts GlobalOptions) restic.IDs {
@@ -62,7 +86,7 @@ func testRunList(t testing.TB, tpe string, opts GlobalOptions) restic.IDs {
 		globalOptions.stdout = os.Stdout
 	}()
 
-	rtest.OK(t, runList(opts, []string{tpe}))
+	rtest.OK(t, runList(cmdList, opts, []string{tpe}))
 	return parseIDsFromReader(t, buf)
 }
 
@@ -218,7 +242,7 @@ func TestBackup(t *testing.T) {
 	opts := BackupOptions{}
 
 	// first backup
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	testRunBackup(t, filepath.Dir(env.testdata), []string{"testdata"}, opts, env.gopts)
 	snapshotIDs := testRunList(t, "snapshots", env.gopts)
 	rtest.Assert(t, len(snapshotIDs) == 1,
 		"expected one snapshot, got %v", snapshotIDs)
@@ -227,7 +251,7 @@ func TestBackup(t *testing.T) {
 	stat1 := dirStats(env.repo)
 
 	// second backup, implicit incremental
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	testRunBackup(t, filepath.Dir(env.testdata), []string{"testdata"}, opts, env.gopts)
 	snapshotIDs = testRunList(t, "snapshots", env.gopts)
 	rtest.Assert(t, len(snapshotIDs) == 2,
 		"expected two snapshots, got %v", snapshotIDs)
@@ -241,7 +265,7 @@ func TestBackup(t *testing.T) {
 	testRunCheck(t, env.gopts)
 	// third backup, explicit incremental
 	opts.Parent = snapshotIDs[0].String()
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	testRunBackup(t, filepath.Dir(env.testdata), []string{"testdata"}, opts, env.gopts)
 	snapshotIDs = testRunList(t, "snapshots", env.gopts)
 	rtest.Assert(t, len(snapshotIDs) == 3,
 		"expected three snapshots, got %v", snapshotIDs)
@@ -285,7 +309,7 @@ func TestBackupNonExistingFile(t *testing.T) {
 		globalOptions.stderr = os.Stderr
 	}()
 
-	p := filepath.Join(env.testdata, "0", "0")
+	p := filepath.Join(env.testdata, "0", "0", "9")
 	dirs := []string{
 		filepath.Join(p, "0"),
 		filepath.Join(p, "1"),
@@ -295,198 +319,7 @@ func TestBackupNonExistingFile(t *testing.T) {
 
 	opts := BackupOptions{}
 
-	testRunBackup(t, dirs, opts, env.gopts)
-}
-
-func TestBackupMissingFile1(t *testing.T) {
-	env, cleanup := withTestEnvironment(t)
-	defer cleanup()
-
-	datafile := filepath.Join("testdata", "backup-data.tar.gz")
-	fd, err := os.Open(datafile)
-	if os.IsNotExist(errors.Cause(err)) {
-		t.Skipf("unable to find data file %q, skipping", datafile)
-		return
-	}
-	rtest.OK(t, err)
-	rtest.OK(t, fd.Close())
-
-	rtest.SetupTarTestFixture(t, env.testdata, datafile)
-
-	testRunInit(t, env.gopts)
-	globalOptions.stderr = ioutil.Discard
-	defer func() {
-		globalOptions.stderr = os.Stderr
-	}()
-
-	ranHook := false
-	debug.Hook("pipe.walk1", func(context interface{}) {
-		pathname := context.(string)
-
-		if pathname != filepath.Join("testdata", "0", "0", "9") {
-			return
-		}
-
-		t.Logf("in hook, removing test file testdata/0/0/9/37")
-		ranHook = true
-
-		rtest.OK(t, os.Remove(filepath.Join(env.testdata, "0", "0", "9", "37")))
-	})
-
-	opts := BackupOptions{}
-
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
-	testRunCheck(t, env.gopts)
-
-	rtest.Assert(t, ranHook, "hook did not run")
-	debug.RemoveHook("pipe.walk1")
-}
-
-func TestBackupMissingFile2(t *testing.T) {
-	env, cleanup := withTestEnvironment(t)
-	defer cleanup()
-
-	datafile := filepath.Join("testdata", "backup-data.tar.gz")
-	fd, err := os.Open(datafile)
-	if os.IsNotExist(errors.Cause(err)) {
-		t.Skipf("unable to find data file %q, skipping", datafile)
-		return
-	}
-	rtest.OK(t, err)
-	rtest.OK(t, fd.Close())
-
-	rtest.SetupTarTestFixture(t, env.testdata, datafile)
-
-	testRunInit(t, env.gopts)
-
-	globalOptions.stderr = ioutil.Discard
-	defer func() {
-		globalOptions.stderr = os.Stderr
-	}()
-
-	ranHook := false
-	debug.Hook("pipe.walk2", func(context interface{}) {
-		pathname := context.(string)
-
-		if pathname != filepath.Join("testdata", "0", "0", "9", "37") {
-			return
-		}
-
-		t.Logf("in hook, removing test file testdata/0/0/9/37")
-		ranHook = true
-
-		rtest.OK(t, os.Remove(filepath.Join(env.testdata, "0", "0", "9", "37")))
-	})
-
-	opts := BackupOptions{}
-
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
-	testRunCheck(t, env.gopts)
-
-	rtest.Assert(t, ranHook, "hook did not run")
-	debug.RemoveHook("pipe.walk2")
-}
-
-func TestBackupChangedFile(t *testing.T) {
-	env, cleanup := withTestEnvironment(t)
-	defer cleanup()
-
-	datafile := filepath.Join("testdata", "backup-data.tar.gz")
-	fd, err := os.Open(datafile)
-	if os.IsNotExist(errors.Cause(err)) {
-		t.Skipf("unable to find data file %q, skipping", datafile)
-		return
-	}
-	rtest.OK(t, err)
-	rtest.OK(t, fd.Close())
-
-	rtest.SetupTarTestFixture(t, env.testdata, datafile)
-
-	testRunInit(t, env.gopts)
-
-	globalOptions.stderr = ioutil.Discard
-	defer func() {
-		globalOptions.stderr = os.Stderr
-	}()
-
-	modFile := filepath.Join(env.testdata, "0", "0", "6", "18")
-
-	ranHook := false
-	debug.Hook("archiver.SaveFile", func(context interface{}) {
-		pathname := context.(string)
-
-		if pathname != modFile {
-			return
-		}
-
-		t.Logf("in hook, modifying test file %v", modFile)
-		ranHook = true
-
-		rtest.OK(t, ioutil.WriteFile(modFile, []byte("modified"), 0600))
-	})
-
-	opts := BackupOptions{}
-
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
-	testRunCheck(t, env.gopts)
-
-	rtest.Assert(t, ranHook, "hook did not run")
-	debug.RemoveHook("archiver.SaveFile")
-}
-
-func TestBackupDirectoryError(t *testing.T) {
-	env, cleanup := withTestEnvironment(t)
-	defer cleanup()
-
-	datafile := filepath.Join("testdata", "backup-data.tar.gz")
-	fd, err := os.Open(datafile)
-	if os.IsNotExist(errors.Cause(err)) {
-		t.Skipf("unable to find data file %q, skipping", datafile)
-		return
-	}
-	rtest.OK(t, err)
-	rtest.OK(t, fd.Close())
-
-	rtest.SetupTarTestFixture(t, env.testdata, datafile)
-
-	testRunInit(t, env.gopts)
-
-	globalOptions.stderr = ioutil.Discard
-	defer func() {
-		globalOptions.stderr = os.Stderr
-	}()
-
-	ranHook := false
-
-	testdir := filepath.Join(env.testdata, "0", "0", "9")
-
-	// install hook that removes the dir right before readdirnames()
-	debug.Hook("pipe.readdirnames", func(context interface{}) {
-		path := context.(string)
-
-		if path != testdir {
-			return
-		}
-
-		t.Logf("in hook, removing test file %v", testdir)
-		ranHook = true
-
-		rtest.OK(t, os.RemoveAll(testdir))
-	})
-
-	testRunBackup(t, []string{filepath.Join(env.testdata, "0", "0")}, BackupOptions{}, env.gopts)
-	testRunCheck(t, env.gopts)
-
-	rtest.Assert(t, ranHook, "hook did not run")
-	debug.RemoveHook("pipe.walk2")
-
-	snapshots := testRunList(t, "snapshots", env.gopts)
-	rtest.Assert(t, len(snapshots) > 0,
-		"no snapshots found in repo (%v)", datafile)
-
-	files := testRunLs(t, env.gopts, snapshots[0].String())
-
-	rtest.Assert(t, len(files) > 1, "snapshot is empty")
+	testRunBackup(t, "", dirs, opts, env.gopts)
 }
 
 func includes(haystack []string, needle string) bool {
@@ -551,33 +384,33 @@ func TestBackupExclude(t *testing.T) {
 
 	opts := BackupOptions{}
 
-	testRunBackup(t, []string{datadir}, opts, env.gopts)
+	testRunBackup(t, filepath.Dir(env.testdata), []string{"testdata"}, opts, env.gopts)
 	snapshots, snapshotID := lastSnapshot(snapshots, loadSnapshotMap(t, env.gopts))
 	files := testRunLs(t, env.gopts, snapshotID)
-	rtest.Assert(t, includes(files, filepath.Join(string(filepath.Separator), "testdata", "foo.tar.gz")),
+	rtest.Assert(t, includes(files, "/testdata/foo.tar.gz"),
 		"expected file %q in first snapshot, but it's not included", "foo.tar.gz")
 
 	opts.Excludes = []string{"*.tar.gz"}
-	testRunBackup(t, []string{datadir}, opts, env.gopts)
+	testRunBackup(t, filepath.Dir(env.testdata), []string{"testdata"}, opts, env.gopts)
 	snapshots, snapshotID = lastSnapshot(snapshots, loadSnapshotMap(t, env.gopts))
 	files = testRunLs(t, env.gopts, snapshotID)
-	rtest.Assert(t, !includes(files, filepath.Join(string(filepath.Separator), "testdata", "foo.tar.gz")),
+	rtest.Assert(t, !includes(files, "/testdata/foo.tar.gz"),
 		"expected file %q not in first snapshot, but it's included", "foo.tar.gz")
 
 	opts.Excludes = []string{"*.tar.gz", "private/secret"}
-	testRunBackup(t, []string{datadir}, opts, env.gopts)
+	testRunBackup(t, filepath.Dir(env.testdata), []string{"testdata"}, opts, env.gopts)
 	_, snapshotID = lastSnapshot(snapshots, loadSnapshotMap(t, env.gopts))
 	files = testRunLs(t, env.gopts, snapshotID)
-	rtest.Assert(t, !includes(files, filepath.Join(string(filepath.Separator), "testdata", "foo.tar.gz")),
+	rtest.Assert(t, !includes(files, "/testdata/foo.tar.gz"),
 		"expected file %q not in first snapshot, but it's included", "foo.tar.gz")
-	rtest.Assert(t, !includes(files, filepath.Join(string(filepath.Separator), "testdata", "private", "secret", "passwords.txt")),
+	rtest.Assert(t, !includes(files, "/testdata/private/secret/passwords.txt"),
 		"expected file %q not in first snapshot, but it's included", "passwords.txt")
 }
 
 const (
-	incrementalFirstWrite  = 20 * 1042 * 1024
-	incrementalSecondWrite = 12 * 1042 * 1024
-	incrementalThirdWrite  = 4 * 1042 * 1024
+	incrementalFirstWrite  = 10 * 1042 * 1024
+	incrementalSecondWrite = 1 * 1042 * 1024
+	incrementalThirdWrite  = 1 * 1042 * 1024
 )
 
 func appendRandomData(filename string, bytes uint) error {
@@ -615,13 +448,13 @@ func TestIncrementalBackup(t *testing.T) {
 
 	opts := BackupOptions{}
 
-	testRunBackup(t, []string{datadir}, opts, env.gopts)
+	testRunBackup(t, "", []string{datadir}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 	stat1 := dirStats(env.repo)
 
 	rtest.OK(t, appendRandomData(testfile, incrementalSecondWrite))
 
-	testRunBackup(t, []string{datadir}, opts, env.gopts)
+	testRunBackup(t, "", []string{datadir}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 	stat2 := dirStats(env.repo)
 	if stat2.size-stat1.size > incrementalFirstWrite {
@@ -631,7 +464,7 @@ func TestIncrementalBackup(t *testing.T) {
 
 	rtest.OK(t, appendRandomData(testfile, incrementalThirdWrite))
 
-	testRunBackup(t, []string{datadir}, opts, env.gopts)
+	testRunBackup(t, "", []string{datadir}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 	stat3 := dirStats(env.repo)
 	if stat3.size-stat2.size > incrementalFirstWrite {
@@ -650,7 +483,7 @@ func TestBackupTags(t *testing.T) {
 
 	opts := BackupOptions{}
 
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	testRunBackup(t, "", []string{env.testdata}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 	newest, _ := testRunSnapshots(t, env.gopts)
 	rtest.Assert(t, newest != nil, "expected a new backup, got nil")
@@ -659,7 +492,7 @@ func TestBackupTags(t *testing.T) {
 	parent := newest
 
 	opts.Tags = []string{"NL"}
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	testRunBackup(t, "", []string{env.testdata}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 	newest, _ = testRunSnapshots(t, env.gopts)
 	rtest.Assert(t, newest != nil, "expected a new backup, got nil")
@@ -682,7 +515,7 @@ func TestTag(t *testing.T) {
 	testRunInit(t, env.gopts)
 	rtest.SetupTarTestFixture(t, env.testdata, datafile)
 
-	testRunBackup(t, []string{env.testdata}, BackupOptions{}, env.gopts)
+	testRunBackup(t, "", []string{env.testdata}, BackupOptions{}, env.gopts)
 	testRunCheck(t, env.gopts)
 	newest, _ := testRunSnapshots(t, env.gopts)
 	rtest.Assert(t, newest != nil, "expected a new backup, got nil")
@@ -858,7 +691,7 @@ func TestRestoreFilter(t *testing.T) {
 
 	opts := BackupOptions{}
 
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	testRunBackup(t, filepath.Dir(env.testdata), []string{filepath.Base(env.testdata)}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 
 	snapshotID := testRunList(t, "snapshots", env.gopts)[0]
@@ -893,12 +726,12 @@ func TestRestore(t *testing.T) {
 	for i := 0; i < 10; i++ {
 		p := filepath.Join(env.testdata, fmt.Sprintf("foo/bar/testfile%v", i))
 		rtest.OK(t, os.MkdirAll(filepath.Dir(p), 0755))
-		rtest.OK(t, appendRandomData(p, uint(mrand.Intn(5<<21))))
+		rtest.OK(t, appendRandomData(p, uint(mrand.Intn(2<<21))))
 	}
 
 	opts := BackupOptions{}
 
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	testRunBackup(t, filepath.Dir(env.testdata), []string{filepath.Base(env.testdata)}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 
 	// Restore latest without any filters
@@ -921,12 +754,22 @@ func TestRestoreLatest(t *testing.T) {
 
 	opts := BackupOptions{}
 
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	// chdir manually here so we can get the current directory. This is not the
+	// same as the temp dir returned by ioutil.TempDir() on darwin.
+	back := fs.TestChdir(t, filepath.Dir(env.testdata))
+	defer back()
+
+	curdir, err := os.Getwd()
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	testRunBackup(t, "", []string{filepath.Base(env.testdata)}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 
 	os.Remove(p)
 	rtest.OK(t, appendRandomData(p, 101))
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	testRunBackup(t, "", []string{filepath.Base(env.testdata)}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 
 	// Restore latest without any filters
@@ -934,16 +777,18 @@ func TestRestoreLatest(t *testing.T) {
 	rtest.OK(t, testFileSize(filepath.Join(env.base, "restore0", "testdata", "testfile.c"), int64(101)))
 
 	// Setup test files in different directories backed up in different snapshots
-	p1 := filepath.Join(env.testdata, "p1/testfile.c")
+	p1 := filepath.Join(curdir, filepath.FromSlash("p1/testfile.c"))
+
 	rtest.OK(t, os.MkdirAll(filepath.Dir(p1), 0755))
 	rtest.OK(t, appendRandomData(p1, 102))
-	testRunBackup(t, []string{filepath.Dir(p1)}, opts, env.gopts)
+	testRunBackup(t, "", []string{"p1"}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 
-	p2 := filepath.Join(env.testdata, "p2/testfile.c")
+	p2 := filepath.Join(curdir, filepath.FromSlash("p2/testfile.c"))
+
 	rtest.OK(t, os.MkdirAll(filepath.Dir(p2), 0755))
 	rtest.OK(t, appendRandomData(p2, 103))
-	testRunBackup(t, []string{filepath.Dir(p2)}, opts, env.gopts)
+	testRunBackup(t, "", []string{"p2"}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 
 	p1rAbs := filepath.Join(env.base, "restore1", "p1/testfile.c")
@@ -1016,7 +861,7 @@ func TestRestoreNoMetadataOnIgnoredIntermediateDirs(t *testing.T) {
 
 	opts := BackupOptions{}
 
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	testRunBackup(t, filepath.Dir(env.testdata), []string{filepath.Base(env.testdata)}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 
 	snapshotID := testRunList(t, "snapshots", env.gopts)[0]
@@ -1030,9 +875,6 @@ func TestRestoreNoMetadataOnIgnoredIntermediateDirs(t *testing.T) {
 	fi, err := os.Stat(f1)
 	rtest.OK(t, err)
 
-	rtest.Assert(t, fi.ModTime() != time.Unix(0, 0),
-		"meta data of intermediate directory has been restore although it was ignored")
-
 	// restore with filter "*", this should restore meta data on everything.
 	testRunRestoreIncludes(t, env.gopts, filepath.Join(env.base, "restore1"), snapshotID, []string{"*"})
 
@@ -1054,7 +896,7 @@ func TestFind(t *testing.T) {
 
 	opts := BackupOptions{}
 
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	testRunBackup(t, "", []string{env.testdata}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 
 	results := testRunFind(t, false, env.gopts, "unexistingfile")
@@ -1094,7 +936,7 @@ func TestFindJSON(t *testing.T) {
 
 	opts := BackupOptions{}
 
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	testRunBackup(t, "", []string{env.testdata}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 
 	results := testRunFind(t, true, env.gopts, "unexistingfile")
@@ -1197,13 +1039,13 @@ func TestPrune(t *testing.T) {
 	rtest.SetupTarTestFixture(t, env.testdata, datafile)
 	opts := BackupOptions{}
 
-	testRunBackup(t, []string{filepath.Join(env.testdata, "0", "0")}, opts, env.gopts)
+	testRunBackup(t, "", []string{filepath.Join(env.testdata, "0", "0", "9")}, opts, env.gopts)
 	firstSnapshot := testRunList(t, "snapshots", env.gopts)
 	rtest.Assert(t, len(firstSnapshot) == 1,
 		"expected one snapshot, got %v", firstSnapshot)
 
-	testRunBackup(t, []string{filepath.Join(env.testdata, "0", "0", "2")}, opts, env.gopts)
-	testRunBackup(t, []string{filepath.Join(env.testdata, "0", "0", "3")}, opts, env.gopts)
+	testRunBackup(t, "", []string{filepath.Join(env.testdata, "0", "0", "9", "2")}, opts, env.gopts)
+	testRunBackup(t, "", []string{filepath.Join(env.testdata, "0", "0", "9", "3")}, opts, env.gopts)
 
 	snapshotIDs := testRunList(t, "snapshots", env.gopts)
 	rtest.Assert(t, len(snapshotIDs) == 3,
@@ -1237,7 +1079,7 @@ func TestHardLink(t *testing.T) {
 	opts := BackupOptions{}
 
 	// first backup
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	testRunBackup(t, filepath.Dir(env.testdata), []string{filepath.Base(env.testdata)}, opts, env.gopts)
 	snapshotIDs := testRunList(t, "snapshots", env.gopts)
 	rtest.Assert(t, len(snapshotIDs) == 1,
 		"expected one snapshot, got %v", snapshotIDs)
@@ -1311,3 +1153,38 @@ func linkEqual(source, dest []string) bool {
 
 	return true
 }
+
+func TestQuietBackup(t *testing.T) {
+	env, cleanup := withTestEnvironment(t)
+	defer cleanup()
+
+	datafile := filepath.Join("testdata", "backup-data.tar.gz")
+	fd, err := os.Open(datafile)
+	if os.IsNotExist(errors.Cause(err)) {
+		t.Skipf("unable to find data file %q, skipping", datafile)
+		return
+	}
+	rtest.OK(t, err)
+	rtest.OK(t, fd.Close())
+
+	testRunInit(t, env.gopts)
+
+	rtest.SetupTarTestFixture(t, env.testdata, datafile)
+	opts := BackupOptions{}
+
+	env.gopts.Quiet = false
+	testRunBackup(t, "", []string{env.testdata}, opts, env.gopts)
+	snapshotIDs := testRunList(t, "snapshots", env.gopts)
+	rtest.Assert(t, len(snapshotIDs) == 1,
+		"expected one snapshot, got %v", snapshotIDs)
+
+	testRunCheck(t, env.gopts)
+
+	env.gopts.Quiet = true
+	testRunBackup(t, "", []string{env.testdata}, opts, env.gopts)
+	snapshotIDs = testRunList(t, "snapshots", env.gopts)
+	rtest.Assert(t, len(snapshotIDs) == 2,
+		"expected two snapshots, got %v", snapshotIDs)
+
+	testRunCheck(t, env.gopts)
+}

cmd/restic/lock.go

@@ -91,19 +91,23 @@ func unlockRepo(lock *restic.Lock) error {
 	globalLocks.Lock()
 	defer globalLocks.Unlock()
 
-	debug.Log("unlocking repository with lock %p", lock)
-	if err := lock.Unlock(); err != nil {
-		debug.Log("error while unlocking: %v", err)
-		return err
-	}
-
 	for i := 0; i < len(globalLocks.locks); i++ {
 		if lock == globalLocks.locks[i] {
+			// remove the lock from the repo
+			debug.Log("unlocking repository with lock %v", lock)
+			if err := lock.Unlock(); err != nil {
+				debug.Log("error while unlocking: %v", err)
+				return err
+			}
+
+			// remove the lock from the list of locks
 			globalLocks.locks = append(globalLocks.locks[:i], globalLocks.locks[i+1:]...)
 			return nil
 		}
 	}
 
+	debug.Log("unable to find lock %v in the global list of locks, ignoring", lock)
+
 	return nil
 }
 
@@ -119,6 +123,7 @@ func unlockAll() error {
 		}
 		debug.Log("successfully removed lock")
 	}
+	globalLocks.locks = globalLocks.locks[:0]
 
 	return nil
 }

cmd/restic/main.go

@@ -29,14 +29,31 @@ directories in an encrypted repository stored on different backends.
 	SilenceUsage:      true,
 	DisableAutoGenTag: true,
 
-	PersistentPreRunE: func(*cobra.Command, []string) error {
+	PersistentPreRunE: func(c *cobra.Command, args []string) error {
+		// set verbosity, default is one
+		globalOptions.verbosity = 1
+		if globalOptions.Quiet && (globalOptions.Verbose > 1) {
+			return errors.Fatal("--quiet and --verbose cannot be specified at the same time")
+		}
+
+		switch {
+		case globalOptions.Verbose >= 2:
+			globalOptions.verbosity = 3
+		case globalOptions.Verbose > 0:
+			globalOptions.verbosity = 2
+		case globalOptions.Quiet:
+			globalOptions.verbosity = 0
+		}
+
 		// parse extended options
 		opts, err := options.Parse(globalOptions.Options)
 		if err != nil {
 			return err
 		}
 		globalOptions.extended = opts
-
+		if c.Name() == "version" {
+			return nil
+		}
 		pwd, err := resolvePassword(globalOptions, "RESTIC_PASSWORD")
 		if err != nil {
 			fmt.Fprintf(os.Stderr, "Resolving password failed: %v\n", err)
@@ -64,7 +81,7 @@ func init() {
 
 func main() {
 	debug.Log("main %#v", os.Args)
-	debug.Log("restic %s, compiled with %v on %v/%v",
+	debug.Log("restic %s compiled with %v on %v/%v",
 		version, runtime.Version(), runtime.GOOS, runtime.GOARCH)
 	err := cmdRoot.Execute()
 

doc/010_introduction.rst

@@ -14,3 +14,6 @@
 Introduction
 ############
 
+Restic is a fast and secure backup program. In the following sections, we will
+present typical workflows, starting with installing, preparing a new
+repository, and making the first backup.

doc/020_installation.rst

@@ -17,6 +17,14 @@ Installation
 Packages
 ********
 
+Note that if at any point the package you’re trying to use is outdated, you
+always have the option to use an official binary from the restic project.
+
+These are up to date binaries, built in a reproducible and verifiable way, that
+you can download and run without having to do additional installation work.
+
+Please see the :ref:`official_binaries` section below for various downloads.
+
 Mac OS X
 ========
 
@@ -62,32 +70,110 @@ installed from the official repos, e.g. with ``apt-get``:
 .. warning:: Please be aware that, at the time of writing, Debian *stable*
    has ``restic`` version 0.3.3 which is very old. The *testing* and *unstable*
    branches have recent versions of ``restic``.
+   
+RHEL & CentOS
+=============
 
-Pre-compiled Binary
-*******************
+restic can be installed via copr repository, for RHEL7/CentOS you can try the following:
 
-You can download the latest pre-compiled binary from the `restic release
-page <https://github.com/restic/restic/releases/latest>`__.
+.. code-block:: console
+
+    $ yum install yum-plugin-copr
+    $ yum copr enable copart/restic
+    $ yum install restic
+
+If that doesn't work, you can try adding the repository directly, for CentOS6 use:
+
+.. code-block:: console
+
+    $ yum-config-manager --add-repo https://copr.fedorainfracloud.org/coprs/copart/restic/repo/epel-6/copart-restic-epel-6.repo
+
+For CentOS7 use:
+
+.. code-block:: console
+
+    $ yum-config-manager --add-repo https://copr.fedorainfracloud.org/coprs/copart/restic/repo/epel-7/copart-restic-epel-7.repo
+
+Fedora
+======
+
+restic can be installed via copr repository.
+
+.. code-block:: console
+
+    $ dnf install dnf-plugin-core
+    $ dnf copr enable copart/restic
+    $ dnf install restic
+
+Solus
+=====
+
+restic can be installed from the official repo of Solus via the ``eopkg`` package manager:
+
+.. code-block:: console
+
+    $ eopkg install restic
+    
+OpenBSD
+=======
+
+On OpenBSD 6.3 and greater, you can install restic using ``pkg_add``:
+
+.. code-block:: console
+
+    # pkg_add restic
+
+.. _official_binaries:
+
+Official Binaries
+*****************
+
+Stable Releases
+===============
+
+You can download the latest stable release versions of restic from the `restic
+release page <https://github.com/restic/restic/releases/latest>`__. These builds
+are considered stable and releases are made regularly in a controlled manner.
+
+There's both pre-compiled binaries for different platforms as well as the source
+code available for download. Just download and run the one matching your system.
+
+Unstable Builds
+===============
+
+Another option is to use the latest builds for the master branch, available on
+the `restic beta download site
+<https://beta.restic.net/?sort=time&order=desc>`__. These too are pre-compiled
+and ready to run, and a new version is built every time a push is made to the
+master branch.
 
 Windows
 =======
 
-On Windows, put the `restic.exe` into `%SystemRoot%\System32` to use restic
+On Windows, put the `restic.exe` binary into `%SystemRoot%\\System32` to use restic
 in scripts without the need for absolute paths to the binary. This requires
-Admin rights.
+administrator rights.
 
 Docker Container
 ****************
 
+We're maintaining a bare docker container with just a few files and the restic
+binary, you can get it with `docker pull` like this:
+
+.. code-block:: console
+
+    $ docker pull restic/restic
+
 .. note::
-   | A docker container is available as a contribution (Thank you!).
-   | You can find it at https://github.com/Lobaro/restic-backup-docker
+   | Another docker container which offers more configuration options is
+   | available as a contribution (Thank you!). You can find it at
+   | https://github.com/Lobaro/restic-backup-docker
 
 From Source
 ***********
 
 restic is written in the Go programming language and you need at least
-Go version 1.8. Building restic may also work with older versions of Go,
+Go version 1.9. Building restic may also work with older versions of Go,
 but that's not supported. See the `Getting
 started <https://golang.org/doc/install>`__ guide of the Go project for
 instructions how to install Go.
@@ -107,7 +193,7 @@ You can easily cross-compile restic for all supported platforms, just
 supply the target OS and platform via the command-line options like this
 (for Windows and FreeBSD respectively):
 
-::
+.. code-block:: console
 
     $ go run build.go --goos windows --goarch amd64
 
@@ -124,35 +210,27 @@ compiler. Building restic with gccgo may work, but is not supported.
 Autocompletion
 **************
 
-Restic can write out a bash compatible autocompletion script:
+Restic can write out man pages and bash/zsh compatible autocompletion scripts:
 
 .. code-block:: console
 
-    $ ./restic autocomplete --help
-    The "autocomplete" command generates a shell autocompletion script.
+    $ ./restic generate --help
 
-    NOTE: The current version supports Bash only.
-          This should work for *nix systems with Bash installed.
-
-By default, the file is written directly to ``/etc/bash_completion.d/``
-for convenience, and the command may need superuser rights, e.g.
-
-.. code-block:: console
-
-    $ sudo restic autocomplete
+    The "generate" command writes automatically generated files like the man pages
+    and the auto-completion files for bash and zsh).
 
     Usage:
-      restic autocomplete [flags]
+      restic generate [command] [flags]
 
     Flags:
-          --completionfile string   autocompletion file (default "/etc/bash_completion.d/restic.sh")
+          --bash-completion file   write bash completion file
+      -h, --help                   help for generate
+          --man directory          write man pages to directory
+          --zsh-completion file    write zsh completion file
 
-    Global Flags:
-          --json                   set output mode to JSON for commands that support it
-          --no-lock                do not lock the repo, this allows some operations on read-only repos
-      -o, --option key=value       set extended option (key=value, can be specified multiple times)
-      -p, --password-file string   read the repository password from a file
-      -q, --quiet                  do not output comprehensive progress report
-      -r, --repo string            repository to backup to or restore from (default: $RESTIC_REPOSITORY)
+Example for using sudo to write a bash completion script directly to the system-wide location:
 
+.. code-block:: console
 
+    $ sudo ./restic generate --bash-completion /etc/bash_completion.d/restic
+    writing bash completion file to /etc/bash_completion.d/restic

doc/030_preparing_a_new_repo.rst

@@ -14,21 +14,25 @@
 Preparing a new repository
 ##########################
 
-The place where your backups will be saved at is called a "repository".
-This chapter explains how to create ("init") such a repository.
+The place where your backups will be saved is called a "repository".
+This chapter explains how to create ("init") such a repository. The repository
+can be stored locally, or on some remote server or service. We'll first cover
+using a local repository; the remaining sections of this chapter cover all the
+other options. You can skip to the next chapter once you've read the relevant
+section here.
 
 Local
 *****
 
-In order to create a repository at ``/tmp/backup``, run the following
+In order to create a repository at ``/srv/restic-repo``, run the following
 command and enter the same password twice:
 
 .. code-block:: console
 
-    $ restic init --repo /tmp/backup
+    $ restic init --repo /srv/restic-repo
     enter password for new backend:
     enter password again:
-    created restic backend 085b3c76b9 at /tmp/backup
+    created restic backend 085b3c76b9 at /srv/restic-repo
     Please note that knowledge of your password is required to access the repository.
     Losing your password means that your data is irrecoverably lost.
 
@@ -55,10 +59,10 @@ simply be achieved by changing the URL scheme in the ``init`` command:
 
 .. code-block:: console
 
-    $ restic -r sftp:user@host:/tmp/backup init
+    $ restic -r sftp:user@host:/srv/restic-repo init
     enter password for new backend:
     enter password again:
-    created restic backend f1c6108821 at sftp:user@host:/tmp/backup
+    created restic backend f1c6108821 at sftp:user@host:/srv/restic-repo
     Please note that knowledge of your password is required to access the repository.
     Losing your password means that your data is irrecoverably lost.
 
@@ -87,7 +91,7 @@ specify the user name in this case):
 
 ::
 
-    $ restic -r sftp:foo:/tmp/backup init
+    $ restic -r sftp:foo:/srv/restic-repo init
 
 You can also add an entry with a special host name which does not exist,
 just for use with restic, and use the ``Hostname`` option to set the
@@ -104,7 +108,7 @@ Then use it in the backend specification:
 
 ::
 
-    $ restic -r sftp:restic-backup-host:/tmp/backup init
+    $ restic -r sftp:restic-backup-host:/srv/restic-repo init
 
 Last, if you'd like to use an entirely different program to create the
 SFTP connection, you can specify the command to be run with the option
@@ -125,8 +129,8 @@ scheme like this:
     $ restic -r rest:http://host:8000/
 
 Depending on your REST server setup, you can use HTTPS protocol,
-password protection, or multiple repositories. Or any combination of
-those features, as you see fit. TCP/IP port is also configurable. Here
+password protection, multiple repositories or any combination of
+those features. The TCP/IP port is also configurable. Here
 are some more examples:
 
 .. code-block:: console
@@ -139,7 +143,10 @@ If you use TLS, restic will use the system's CA certificates to verify the
 server certificate. When the verification fails, restic refuses to proceed and
 exits with an error. If you have your own self-signed certificate, or a custom
 CA certificate should be used for verification, you can pass restic the
-certificate filename via the `--cacert` option.
+certificate filename via the ``--cacert`` option. It will then verify that the
+server's certificate is contained in the file passed to this option, or signed
+by a CA certificate in the file. In this case, the system CA certificates are
+not considered at all.
 
 REST server uses exactly the same directory structure as local backend,
 so you should be able to access it both locally and via HTTP, even
@@ -160,7 +167,7 @@ while creating the bucket.
     $ export AWS_SECRET_ACCESS_KEY=<MY_SECRET_ACCESS_KEY>
 
 You can then easily initialize a repository that uses your Amazon S3 as
-a backend, if the bucket does not exist yet it will be created in the
+a backend. If the bucket does not exist it will be created in the
 default location:
 
 .. code-block:: console
@@ -202,7 +209,7 @@ written in Go and compatible with AWS S3 API.
    on installation and getting started on Minio Client and Minio Server.
 
 You must first setup the following environment variables with the
-credentials of your running Minio Server.
+credentials of your Minio Server.
 
 .. code-block:: console
 
@@ -227,7 +234,7 @@ OpenStack Swift
 Restic can backup data to an OpenStack Swift container. Because Swift supports
 various authentication methods, credentials are passed through environment
 variables. In order to help integration with existing OpenStack installations,
-the naming convention of those variables follows official python swift client:
+the naming convention of those variables follows the official Python Swift client:
 
 .. code-block:: console
 
@@ -258,12 +265,12 @@ the naming convention of those variables follows official python swift client:
    $ export OS_AUTH_TOKEN=<MY_AUTH_TOKEN>
 
 
-Restic should be compatible with `OpenStack RC file
+Restic should be compatible with an `OpenStack RC file
 <https://docs.openstack.org/user-guide/common/cli-set-environment-variables-using-openstack-rc.html>`__
 in most cases.
 
 Once environment variables are set up, a new repository can be created. The
-name of swift container and optional path can be specified. If
+name of the Swift container and optional path can be specified. If
 the container does not exist, it will be created automatically:
 
 .. code-block:: console
@@ -275,7 +282,7 @@ the container does not exist, it will be created automatically:
    Please note that knowledge of your password is required to access the repository.
    Losing your password means that your data is irrecoverably lost.
 
-The policy of new container created by restic can be changed using environment variable:
+The policy of the new container created by restic can be changed using environment variable:
 
 .. code-block:: console
 
@@ -286,16 +293,19 @@ Backblaze B2
 ************
 
 Restic can backup data to any Backblaze B2 bucket. You need to first setup the
-following environment variables with the credentials you obtained when signed
-into your B2 account:
+following environment variables with the credentials you can find in the
+dashboard in on the "Buckets" page when signed into your B2 account:
 
 .. code-block:: console
 
     $ export B2_ACCOUNT_ID=<MY_ACCOUNT_ID>
     $ export B2_ACCOUNT_KEY=<MY_SECRET_ACCOUNT_KEY>
 
-You can then easily initialize a repository stored at Backblaze B2. If the
-bucket does not exist yet, it will be created:
+.. note:: In case you want to use Backblaze Application Keys replace <MY_ACCOUNT_ID> and <MY_SECRET_ACCOUNT_KEY> with <applicationKeyId> and <applicationKey> respectively.
+
+You can then initialize a repository stored at Backblaze B2. If the
+bucket does not exist yet and the credentials you passed to restic have the
+privilege to create buckets, it will be created automatically:
 
 .. code-block:: console
 
@@ -306,8 +316,10 @@ bucket does not exist yet, it will be created:
     Please note that knowledge of your password is required to access the repository.
     Losing your password means that your data is irrecoverably lost.
 
-The number of concurrent connections to the B2 service can be set with the `-o
-b2.connections=10`. By default, at most five parallel connections are
+Note that the bucket name must be unique across all of B2.
+
+The number of concurrent connections to the B2 service can be set with the ``-o
+b2.connections=10`` switch. By default, at most five parallel connections are
 established.
 
 Microsoft Azure Blob Storage
@@ -321,7 +333,7 @@ account name and key as follows:
     $ export AZURE_ACCOUNT_NAME=<ACCOUNT_NAME>
     $ export AZURE_ACCOUNT_KEY=<SECRET_KEY>
 
-Afterwards you can initialize a repository in a container called `foo` in the
+Afterwards you can initialize a repository in a container called ``foo`` in the
 root path like this:
 
 .. code-block:: console
@@ -334,15 +346,13 @@ root path like this:
     [...]
 
 The number of concurrent connections to the Azure Blob Storage service can be set with the
-`-o azure.connections=10`. By default, at most five parallel connections are
+``-o azure.connections=10`` switch. By default, at most five parallel connections are
 established.
 
 Google Cloud Storage
 ********************
 
-Restic supports Google Cloud Storage as a backend.
-
-Restic connects to Google Cloud Storage via a `service account`_.
+Restic supports Google Cloud Storage as a backend and connects via a `service account`_.
 
 For normal restic operation, the service account must have the
 ``storage.objects.{create,delete,get,list}`` permissions for the bucket. These
@@ -362,8 +372,13 @@ key file and the project ID as follows:
     $ export GOOGLE_PROJECT_ID=123123123123
     $ export GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gs-secret-restic-key.json
 
-Then you can use the ``gs:`` backend type to create a new repository in the
-bucket `foo` at the root path:
+Restic uses  Google's client library to generate `default authentication material`_,
+which means if you're running in Google Container Engine or are otherwise
+located on an instance with default service accounts then these should work out of 
+the box.
+
+Once authenticated, you can use the ``gs:`` backend type to create a new
+repository in the bucket ``foo`` at the root path:
 
 .. code-block:: console
 
@@ -375,11 +390,117 @@ bucket `foo` at the root path:
     [...]
 
 The number of concurrent connections to the GCS service can be set with the
-`-o gs.connections=10`. By default, at most five parallel connections are
+``-o gs.connections=10`` switch. By default, at most five parallel connections are
 established.
 
 .. _service account: https://cloud.google.com/storage/docs/authentication#service_accounts
 .. _create a service account key: https://cloud.google.com/storage/docs/authentication#generating-a-private-key
+.. _default authentication material: https://developers.google.com/identity/protocols/application-default-credentials
+
+Other Services via rclone
+*************************
+
+The program `rclone`_ can be used to access many other different services and
+store data there. First, you need to install and `configure`_ rclone.  The
+general backend specification format is ``rclone:<remote>:<path>``, the
+``<remote>:<path>`` component will be directly passed to rclone. When you
+configure a remote named ``foo``, you can then call restic as follows to
+initiate a new repository in the path ``bar`` in the repo:
+
+.. code-block:: console
+
+    $ restic -r rclone:foo:bar init
+
+Restic takes care of starting and stopping rclone.
+
+As a more concrete example, suppose you have configured a remote named
+``b2prod`` for Backblaze B2 with rclone, with a bucket called ``yggdrasil``.
+You can then use rclone to list files in the bucket like this:
+
+.. code-block:: console
+
+    $ rclone ls b2prod:yggdrasil
+
+In order to create a new repository in the root directory of the bucket, call
+restic like this:
+
+.. code-block:: console
+
+    $ restic -r rclone:b2prod:yggdrasil init
+
+If you want to use the path ``foo/bar/baz`` in the bucket instead, pass this to
+restic:
+
+.. code-block:: console
+
+    $ restic -r rclone:b2prod:yggdrasil/foo/bar/baz init
+
+Listing the files of an empty repository directly with rclone should return a
+listing similar to the following:
+
+.. code-block:: console
+
+    $ rclone ls b2prod:yggdrasil/foo/bar/baz
+        155 bar/baz/config
+        448 bar/baz/keys/4bf9c78049de689d73a56ed0546f83b8416795295cda12ec7fb9465af3900b44
+
+Rclone can be `configured with environment variables`_, so for instance
+configuring a bandwidth limit for rclone can be achieved by setting the
+``RCLONE_BWLIMIT`` environment variable:
+
+.. code-block:: console
+
+    $ export RCLONE_BWLIMIT=1M
+
+For debugging rclone, you can set the environment variable ``RCLONE_VERBOSE=2``.
+
+The rclone backend has two additional options:
+
+ * ``-o rclone.program`` specifies the path to rclone, the default value is just ``rclone``
+ * ``-o rclone.args`` allows setting the arguments passed to rclone, by default this is ``serve restic --stdio --b2-hard-delete --drive-use-trash=false``
+
+The reason for the two last parameters (``--b2-hard-delete`` and
+``--drive-use-trash=false``) can be found in the corresponding GitHub `issue #1657`_.
+
+In order to start rclone, restic will build a list of arguments by joining the
+following lists (in this order): ``rclone.program``, ``rclone.args`` and as the
+last parameter the value that follows the ``rclone:`` prefix of the repository
+specification.
+
+So, calling restic like this
+
+.. code-block:: console
+
+    $ restic -o rclone.program="/path/to/rclone" \
+      -o rclone.args="serve restic --stdio --bwlimit 1M --b2-hard-delete --verbose" \
+      -r rclone:b2:foo/bar
+
+runs rclone as follows:
+
+.. code-block:: console
+
+    $ /path/to/rclone serve restic --stdio --bwlimit 1M --b2-hard-delete --verbose b2:foo/bar
+
+Manually setting ``rclone.program`` also allows running a remote instance of
+rclone e.g. via SSH on a server, for example:
+
+.. code-block:: console
+
+    $ restic -o rclone.program="ssh user@host rclone" -r rclone:b2:foo/bar
+
+The rclone command may also be hard-coded in the SSH configuration or the
+user's public key, in this case it may be sufficient to just start the SSH
+connection (and it's irrelevant what's passed after ``rclone:`` in the
+repository specification):
+
+.. code-block:: console
+
+    $ restic -o rclone.program="ssh user@host" -r rclone:x
+
+.. _rclone: https://rclone.org/
+.. _configure: https://rclone.org/docs/
+.. _configured with environment variables: https://rclone.org/docs/#environment-variables
+.. _issue #1657: https://github.com/restic/restic/pull/1657#issuecomment-377707486
 
 Password prompt on Windows
 **************************
@@ -388,7 +509,7 @@ At the moment, restic only supports the default Windows console
 interaction. If you use emulation environments like
 `MSYS2 <https://msys2.github.io/>`__ or
 `Cygwin <https://www.cygwin.com/>`__, which use terminals like
-``Mintty`` or ``rxvt``, you may get a password error:
+``Mintty`` or ``rxvt``, you may get a password error.
 
 You can workaround this by using a special tool called ``winpty`` (look
 `here <https://sourceforge.net/p/msys2/wiki/Porting/>`__ and
@@ -398,5 +519,5 @@ On MSYS2, you can install ``winpty`` as follows:
 .. code-block:: console
 
     $ pacman -S winpty
-    $ winpty restic -r /tmp/backup init
+    $ winpty restic -r /srv/restic-repo init
 

doc/040_backup.rst

@@ -21,43 +21,87 @@ again:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup backup ~/work
+    $ restic -r /srv/restic-repo --verbose backup ~/work
+    open repository
     enter password for repository:
-    scan [/home/user/work]
-    scanned 764 directories, 1816 files in 0:00
-    [0:29] 100.00%  54.732 MiB/s  1.582 GiB / 1.582 GiB  2580 / 2580 items  0 errors  ETA 0:00
-    duration: 0:29, 54.47MiB/s
+    password is correct
+    lock repository
+    load index files
+    start scan
+    start backup
+    scan finished in 1.837s
+    processed 1.720 GiB in 0:12
+    Files:        5307 new,     0 changed,     0 unmodified
+    Dirs:         1867 new,     0 changed,     0 unmodified
+    Added:      1.700 GiB
     snapshot 40dc1520 saved
 
 As you can see, restic created a backup of the directory and was pretty
 fast! The specific snapshot just created is identified by a sequence of
 hexadecimal characters, ``40dc1520`` in this case.
 
+If you don't pass the ``--verbose`` option, restic will print less data. You'll still get a nice live status display. Be aware that the live status shows the processed files and not the transferred data. Transferred volume might be lower (due to deduplication) or higher.
+
 If you run the command again, restic will create another snapshot of
 your data, but this time it's even faster. This is de-duplication at
 work!
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup backup ~/work
+    $ restic -r /srv/restic-repo backup --verbose ~/work
+    open repository
     enter password for repository:
-    using parent snapshot 40dc1520aa6a07b7b3ae561786770a01951245d2367241e71e9485f18ae8228c
-    scan [/home/user/work]
-    scanned 764 directories, 1816 files in 0:00
-    [0:00] 100.00%  0B/s  1.582 GiB / 1.582 GiB  2580 / 2580 items  0 errors  ETA 0:00
-    duration: 0:00, 6572.38MiB/s
+    password is correct
+    lock repository
+    load index files
+    using parent snapshot d875ae93
+    start scan
+    start backup
+    scan finished in 1.881s
+    processed 1.720 GiB in 0:03
+    Files:           0 new,     0 changed,  5307 unmodified
+    Dirs:            0 new,     0 changed,  1867 unmodified
+    Added:      0 B
     snapshot 79766175 saved
 
-You can even backup individual files in the same repository.
+You can even backup individual files in the same repository (not passing
+``--verbose`` means less output):
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup backup ~/work.txt
-    scan [/home/user/work.txt]
-    scanned 0 directories, 1 files in 0:00
-    [0:00] 100.00%  0B/s  220B / 220B  1 / 1 items  0 errors  ETA 0:00
-    duration: 0:00, 0.03MiB/s
-    snapshot 31f7bd63 saved
+    $ restic -r /srv/restic-repo backup ~/work.txt
+    enter password for repository:
+    password is correct
+    snapshot 249d0210 saved
+
+If you're interested in what restic does, pass ``--verbose`` twice (or
+``--verbose 2``) to display detailed information about each file and directory
+restic encounters:
+
+.. code-block:: console
+
+    $ echo 'more data foo bar' >> ~/work.txt
+
+    $ restic -r /srv/restic-repo backup --verbose --verbose ~/work.txt
+    open repository
+    enter password for repository:
+    password is correct
+    lock repository
+    load index files
+    using parent snapshot f3f8d56b
+    start scan
+    start backup
+    scan finished in 2.115s
+    modified  /home/user/work.txt, saved in 0.007s (22 B added)
+    modified  /home/user/, saved in 0.008s (0 B added, 378 B metadata)
+    modified  /home/, saved in 0.009s (0 B added, 375 B metadata)
+    processed 22 B in 0:02
+    Files:           0 new,     1 changed,     0 unmodified
+    Dirs:            0 new,     2 changed,     0 unmodified
+    Data Blobs:      1 new
+    Tree Blobs:      3 new
+    Added:      1.116 KiB
+    snapshot 8dc503fc saved
 
 In fact several hosts may use the same repository to backup directories
 and files leading to a greater de-duplication.
@@ -75,6 +119,9 @@ Now is a good time to run ``restic check`` to verify that all data
 is properly stored in the repository. You should run this command regularly
 to make sure the internal structure of the repository is free of errors.
 
+Including and Excluding Files 
+*****************************
+
 You can exclude folders and files by specifying exclude patterns, currently
 the exclude options are:
 
@@ -84,33 +131,56 @@ the exclude options are:
 -  ``--exclude-if-present`` Specified one or more times to exclude a folders content
    if it contains a given file (optionally having a given header)
 
-Basic example:
+ Let's say we have a file called ``excludes.txt`` with the following content:
 
-.. code-block:: console
+::
 
-    $ cat exclude
     # exclude go-files
     *.go
     # exclude foo/x/y/z/bar foo/x/bar foo/bar
     foo/**/bar
-    $ restic -r /tmp/backup backup ~/work --exclude=*.c --exclude-file=exclude
+
+It can be used like this:
+
+.. code-block:: console
+
+    $ restic -r /srv/restic-repo backup ~/work --exclude="*.c" --exclude-file=excludes.txt
+
+This instruct restic to exclude files matching the following criteria:
+
+ * All files matching ``*.go`` (second line in ``excludes.txt``)
+ * All files and sub-directories named ``bar`` which reside somewhere below a directory called ``foo`` (fourth line in ``excludes.txt``)
+ * All files matching ``*.c`` (parameter ``--exclude``)
 
 Please see ``restic help backup`` for more specific information about each exclude option.
 
 Patterns use `filepath.Glob <https://golang.org/pkg/path/filepath/#Glob>`__ internally,
-see `filepath.Match <https://golang.org/pkg/path/filepath/#Match>`__ for syntax.
-Patterns are tested against the full path of a file/dir to be saved, not only
-against the relative path below the argument given to restic backup.
-Patterns need to match on complete path components. (``foo`` matches
-``/dir1/foo/dir2/file`` and ``/dir/foo`` but does not match ``/dir/foobar`` or
-``barfoo``.) A trailing ``/`` is ignored. A leading ``/`` anchors the
-pattern at the root directory. (``/bin`` matches ``/bin/bash`` but does not
-match ``/usr/bin/restic``.) Regular wildcards cannot be used to match over the
-directory separator ``/``. (``b*ash`` matches ``/bin/bash`` but does not match
-``/bin/ash``.) However ``**`` matches arbitrary subdirectories. (``foo/**/bar``
-matches ``/dir1/foo/dir2/bar/file``, ``/foo/bar/file`` and ``/tmp/foo/bar``.)
-Environment-variables in exclude-files are expanded with
-`os.ExpandEnv <https://golang.org/pkg/os/#ExpandEnv>`__.
+see `filepath.Match <https://golang.org/pkg/path/filepath/#Match>`__ for
+syntax. Patterns are tested against the full path of a file/dir to be saved,
+even if restic is passed a relative path to save. Environment-variables in
+exclude-files are expanded with `os.ExpandEnv <https://golang.org/pkg/os/#ExpandEnv>`__,
+so `/home/$USER/foo` will be expanded to `/home/bob/foo` for the user `bob`. To
+get a literal dollar sign, write `$$` to the file.
+
+Patterns need to match on complete path components. For example, the pattern ``foo``:
+
+ * matches ``/dir1/foo/dir2/file`` and ``/dir/foo``
+ * does not match ``/dir/foobar`` or ``barfoo``
+
+A trailing ``/`` is ignored, a leading ``/`` anchors the
+pattern at the root directory. This means, ``/bin`` matches ``/bin/bash`` but
+does not match ``/usr/bin/restic``.
+
+Regular wildcards cannot be used to match over the
+directory separator ``/``. For example: ``b*ash`` matches ``/bin/bash`` but does not match
+``/bin/ash``.
+
+For this, the special wildcard ``**`` can be used to match arbitrary
+sub-directories: The pattern ``foo/**/bar`` matches:
+
+ * ``/dir1/foo/dir2/bar/file``
+ * ``/foo/bar/file`` 
+ * ``/tmp/foo/bar``
 
 By specifying the option ``--one-file-system`` you can instruct restic
 to only backup files from the file systems the initially specified files
@@ -119,15 +189,15 @@ backup ``/sys`` or ``/dev`` on a Linux system:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup backup --one-file-system /
+    $ restic -r /srv/restic-repo backup --one-file-system /
 
 By using the ``--files-from`` option you can read the files you want to
 backup from a file. This is especially useful if a lot of files have to
 be backed up that are not in the same folder or are maybe pre-filtered
 by other software.
 
-or example maybe you want to backup files that have a certain filename
-in them:
+For example maybe you want to backup files which have a name that matches a
+certain pattern:
 
 .. code-block:: console
 
@@ -137,14 +207,16 @@ You can then use restic to backup the filtered files:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup backup --files-from /tmp/files_to_backup
+    $ restic -r /srv/restic-repo backup --files-from /tmp/files_to_backup
 
 Incidentally you can also combine ``--files-from`` with the normal files
 args:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup backup --files-from /tmp/files_to_backup /tmp/some_additional_file
+    $ restic -r /srv/restic-repo backup --files-from /tmp/files_to_backup /tmp/some_additional_file
+
+Paths in the listing file can be absolute or relative.
 
 Comparing Snapshots
 *******************
@@ -154,7 +226,7 @@ and displays a small statistic, just pass the command two snapshot IDs:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup diff 5845b002 2ab627a6
+    $ restic -r /srv/restic-repo diff 5845b002 2ab627a6
     password is correct
     comparing snapshot ea657ce5 to 2ab627a6:
 
@@ -201,7 +273,7 @@ this mode of operation, just supply the option ``--stdin`` to the
 
 .. code-block:: console
 
-    $ mysqldump [...] | restic -r /tmp/backup backup --stdin
+    $ mysqldump [...] | restic -r /srv/restic-repo backup --stdin
 
 This creates a new snapshot of the output of ``mysqldump``. You can then
 use e.g. the fuse mounting option (see below) to mount the repository
@@ -212,7 +284,7 @@ specified with ``--stdin-filename``, e.g. like this:
 
 .. code-block:: console
 
-    $ mysqldump [...] | restic -r /tmp/backup backup --stdin --stdin-filename production.sql
+    $ mysqldump [...] | restic -r /srv/restic-repo backup --stdin --stdin-filename production.sql
 
 Tags for backup
 ***************
@@ -222,9 +294,23 @@ information. Just specify the tags for a snapshot one by one with ``--tag``:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup backup --tag projectX --tag foo --tag bar ~/work
+    $ restic -r /srv/restic-repo backup --tag projectX --tag foo --tag bar ~/work
     [...]
 
 The tags can later be used to keep (or forget) snapshots with the ``forget``
 command. The command ``tag`` can be used to modify tags on an existing
 snapshot.
+
+Space requirements
+******************
+
+Restic currently assumes that your backup repository has sufficient space
+for the backup operation you are about to perform. This is a realistic
+assumption for many cloud providers, but may not be true when backing up
+to local disks.
+
+Should you run out of space during the middle of a backup, there will be
+some additional data in the repository, but the snapshot will never be
+created as it would only be written at the very (successful) end of
+the backup operation.  Previous snapshots will still be there and will still
+work.

doc/045_working_with_repos.rst

@@ -22,7 +22,7 @@ Now, you can list all the snapshots stored in the repository:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup snapshots
+    $ restic -r /srv/restic-repo snapshots
     enter password for repository:
     ID        Date                 Host    Tags   Directory
     ----------------------------------------------------------------------
@@ -36,7 +36,7 @@ You can filter the listing by directory path:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup snapshots --path="/srv"
+    $ restic -r /srv/restic-repo snapshots --path="/srv"
     enter password for repository:
     ID        Date                 Host    Tags   Directory
     ----------------------------------------------------------------------
@@ -47,7 +47,7 @@ Or filter by host:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup snapshots --host luigi
+    $ restic -r /srv/restic-repo snapshots --host luigi
     enter password for repository:
     ID        Date                 Host    Tags   Directory
     ----------------------------------------------------------------------
@@ -74,7 +74,7 @@ backup data is consistent and the integrity is unharmed:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup check
+    $ restic -r /srv/restic-repo check
     Load indexes
     ciphertext verification failed
 
@@ -83,7 +83,33 @@ yield the same error:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup restore 79766175 --target /tmp/restore-work
+    $ restic -r /srv/restic-repo restore 79766175 --target /tmp/restore-work
     Load indexes
     ciphertext verification failed
 
+By default, ``check`` command does not check that repository data files 
+are unmodified. Use ``--read-data`` parameter to check all repository
+data files:
+
+.. code-block:: console
+
+    $ restic -r /srv/restic-repo check --read-data
+    load indexes
+    check all packs
+    check snapshots, trees and blobs
+    read all data
+
+Use ``--read-data-subset=n/t`` parameter to check subset of repository data
+files. The parameter takes two values, ``n`` and ``t``. All repository data 
+files are logically devided in ``t`` roughly equal groups and only files that
+belong to the group number ``n`` are checked. For example, the following 
+commands check all repository data files over 5 separate invocations:
+
+.. code-block:: console
+
+    $ restic -r /srv/restic-repo check --read-data-subset=1/5
+    $ restic -r /srv/restic-repo check --read-data-subset=2/5
+    $ restic -r /srv/restic-repo check --read-data-subset=3/5
+    $ restic -r /srv/restic-repo check --read-data-subset=4/5
+    $ restic -r /srv/restic-repo check --read-data-subset=5/5
+

doc/050_restore.rst

@@ -23,7 +23,7 @@ command to restore the contents of the latest snapshot to
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup restore 79766175 --target /tmp/restore-work
+    $ restic -r /srv/restic-repo restore 79766175 --target /tmp/restore-work
     enter password for repository:
     restoring <Snapshot of [/home/user/work] at 2015-05-08 21:40:19.884408621 +0200 CEST> to /tmp/restore-work
 
@@ -33,7 +33,7 @@ backup for a specific host, path or both.
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup restore latest --target /tmp/restore-art --path "/home/art" --host luigi
+    $ restic -r /srv/restic-repo restore latest --target /tmp/restore-art --path "/home/art" --host luigi
     enter password for repository:
     restoring <Snapshot of [/home/art] at 2015-05-08 21:45:17.884408621 +0200 CEST> to /tmp/restore-art
 
@@ -42,12 +42,16 @@ files in the snapshot. For example, to restore a single file:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup restore 79766175 --target /tmp/restore-work --include /work/foo
+    $ restic -r /srv/restic-repo restore 79766175 --target /tmp/restore-work --include /work/foo
     enter password for repository:
     restoring <Snapshot of [/home/user/work] at 2015-05-08 21:40:19.884408621 +0200 CEST> to /tmp/restore-work
 
 This will restore the file ``foo`` to ``/tmp/restore-work/work/foo``.
 
+You can use the command ``restic ls latest`` or ``restic find foo`` to find the
+path to the file within the snapshot. This path you can then pass to
+`--include` in verbatim to only restore the single file or directory.
+
 Restore using mount
 ===================
 
@@ -58,12 +62,13 @@ command to serve the repository with FUSE:
 .. code-block:: console
 
     $ mkdir /mnt/restic
-    $ restic -r /tmp/backup mount /mnt/restic
+    $ restic -r /srv/restic-repo mount /mnt/restic
     enter password for repository:
-    Now serving /tmp/backup at /mnt/restic
+    Now serving /srv/restic-repo at /mnt/restic
     Don't forget to umount after quitting!
 
-Mounting repositories via FUSE is not possible on Windows and OpenBSD.
+Mounting repositories via FUSE is not possible on OpenBSD, Solaris/illumos
+and Windows.
 
 Restic supports storage and preservation of hard links. However, since
 hard links exist in the scope of a filesystem by definition, restoring
@@ -79,4 +84,37 @@ the data directly. This can be achieved by using the `dump` command, like this:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup dump latest production.sql | mysql
+    $ restic -r /srv/restic-repo dump latest production.sql | mysql
+
+If you have saved multiple different things into the same repo, the ``latest``
+snapshot may not be the right one. For example, consider the following
+snapshots in a repo:
+
+.. code-block:: console
+
+    $ restic -r /srv/restic-repo snapshots
+    ID        Date                 Host        Tags        Directory
+    ----------------------------------------------------------------------
+    562bfc5e  2018-07-14 20:18:01  mopped                  /home/user/file1
+    bbacb625  2018-07-14 20:18:07  mopped                  /home/other/work
+    e922c858  2018-07-14 20:18:10  mopped                  /home/other/work
+    098db9d5  2018-07-14 20:18:13  mopped                  /production.sql
+    b62f46ec  2018-07-14 20:18:16  mopped                  /home/user/file1
+    1541acae  2018-07-14 20:18:18  mopped                  /home/other/work
+    ----------------------------------------------------------------------
+
+Here, restic would resolve ``latest`` to the snapshot ``1541acae``, which does
+not contain the file we'd like to print at all (``production.sql``).  In this
+case, you can pass restic the snapshot ID of the snapshot you like to restore:
+
+.. code-block:: console
+
+    $ restic -r /srv/restic-repo dump 098db9d5 production.sql | mysql
+
+Or you can pass restic a path that should be used for selecting the latest
+snapshot. The path must match the patch printed in the "Directory" column,
+e.g.:
+
+.. code-block:: console
+
+    $ restic -r /srv/restic-repo dump --path /production.sql latest production.sql | mysql

doc/060_forget.rst

@@ -23,6 +23,13 @@ data that was referenced by the snapshot from the repository. This can
 be automated with the ``--prune`` option of the ``forget`` command,
 which runs ``prune`` automatically if snapshots have been removed.
 
+.. Warning::
+
+   Pruning snapshots can be a very time-consuming process, taking nearly
+   as long as backups themselves. During a prune operation, the index is
+   locked and backups cannot be completed. Performance improvements are 
+   planned for this feature.
+
 It is advisable to run ``restic check`` after pruning, to make sure
 you are alerted, should the internal data structures of the repository
 be damaged.
@@ -35,7 +42,7 @@ repository like this:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup snapshots
+    $ restic -r /srv/restic-repo snapshots
     enter password for repository:
     ID        Date                 Host      Tags  Directory
     ----------------------------------------------------------------------
@@ -50,7 +57,7 @@ command and specify the snapshot ID on the command line:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup forget bdbd3439
+    $ restic -r /srv/restic-repo forget bdbd3439
     enter password for repository:
     removed snapshot d3f01f63
 
@@ -58,7 +65,7 @@ Afterwards this snapshot is removed:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup snapshots
+    $ restic -r /srv/restic-repo snapshots
     enter password for repository:
     ID        Date                 Host     Tags  Directory
     ----------------------------------------------------------------------
@@ -73,7 +80,7 @@ command must be run:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup prune
+    $ restic -r /srv/restic-repo prune
     enter password for repository:
 
     counting files in repo
@@ -159,6 +166,13 @@ The ``forget`` command accepts the following parameters:
    snapshots, only keep the last one for that year.
 -  ``--keep-tag`` keep all snapshots which have all tags specified by
    this option (can be specified multiple times).
+-  ``--keep-within duration`` keep all snapshots which have been made within
+   the duration of the latest snapshot. ``duration`` needs to be a number of
+   years, months, and days, e.g. ``2y5m7d`` will keep all snapshots made in the
+   two years, five months, and seven days before the latest snapshot.
+
+Multiple policies will be ORed together so as to be as inclusive as possible
+for keeping snapshots.
 
 Additionally, you can restrict removing snapshots to those which have a
 particular hostname with the ``--hostname`` parameter, or tags with the

doc/070_encryption.rst

@@ -16,8 +16,8 @@ Encryption
 
 
 *"The design might not be perfect, but it’s good. Encryption is a first-class feature,
-the implementation looks sane and I guess the deduplication trade-off is worth it. So… I’m going to use restic for 
-my personal backups.*" `Filippo Valsorda`_
+the implementation looks sane and I guess the deduplication trade-off is worth
+it. So… I’m going to use restic for my personal backups.*" `Filippo Valsorda`_
 
 .. _Filippo Valsorda: https://blog.filippo.io/restic-cryptography/
 
@@ -31,19 +31,19 @@ per repository. In fact, you can use the ``list``, ``add``, ``remove``, and
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup key list
+    $ restic -r /srv/restic-repo key list
     enter password for repository:
      ID          User        Host        Created
     ----------------------------------------------------------------------
     *eb78040b    username    kasimir   2015-08-12 13:29:57
 
-    $ restic -r /tmp/backup key add
+    $ restic -r /srv/restic-repo key add
     enter password for repository:
     enter password for new key:
     enter password again:
     saved new key as <Key of username@kasimir, created on 2015-08-12 13:35:05.316831933 +0200 CEST>
 
-    $ restic -r backup key list
+    $ restic -r /srv/restic-repo key list
     enter password for repository:
      ID          User        Host        Created
     ----------------------------------------------------------------------

doc/075_scripting.rst

@@ -0,0 +1,39 @@
+..
+  Normally, there are no heading levels assigned to certain characters as the structure is
+  determined from the succession of headings. However, this convention is used in Python’s
+  Style Guide for documenting which you may follow:
+
+  # with overline, for parts
+  * for chapters
+  = for sections
+  - for subsections
+  ^ for subsubsections
+  " for paragraphs
+
+#########################
+Scripting
+#########################
+
+This is a list of how certain tasks may be accomplished when you use
+restic via scripts.
+
+Check if a repository is already initialized
+********************************************
+
+You may find a need to check if a repository is already initialized,
+perhaps to prevent your script from initializing a repository multiple
+times. The command ``snapshots`` may be used for this purpose:
+
+.. code-block:: console
+
+    $ restic -r /srv/restic-repo snapshots
+    Fatal: unable to open config file: Stat: stat /srv/restic-repo/config: no such file or directory
+    Is there a repository at the following location?
+    /srv/restic-repo
+
+If a repository does not exist, restic will return a non-zero exit code
+and print an error message. Note that restic will also return a non-zero
+exit code if a different error is encountered (e.g.: incorrect password
+to ``snapshots``) and it may print a different error message. If there
+are no errors, restic will return a zero exit code and print all the
+snapshots.

doc/080_examples.rst

@@ -312,7 +312,7 @@ directory.
 .. code-block:: console
 
    root@a3e580b6369d:/# mkdir ~restic/bin
-   root@a3e580b6369d:/# curl -L https://github.com/restic/restic/releases/download/v0.8.0/restic_0.8.0_linux_amd64.bz2 | bunzip2 > ~restic/bin/restic
+   root@a3e580b6369d:/# curl -L https://github.com/restic/restic/releases/download/v0.9.1/restic_0.9.1_linux_amd64.bz2 | bunzip2 > ~restic/bin/restic
 
 Before we assign any special capability to the restic binary we
 restrict its permissions so that only root and the newly created
@@ -338,5 +338,5 @@ system.
 
 .. code-block:: console
 
-   root@a3e580b6369d:/# sudo -u restic /opt/restic/bin/restic --exclude={/dev,/media,/mnt,/proc,/run,/sys,/tmp,/var/tmp} -r /tmp backup /
+   root@a3e580b6369d:/# sudo -u restic /home/restic/bin/restic --exclude={/dev,/media,/mnt,/proc,/run,/sys,/tmp,/var/tmp} -r /tmp backup /
 

doc/100_references.rst

@@ -18,804 +18,6 @@ References
 Design
 ******
 
-Terminology
-===========
-
-This section introduces terminology used in this document.
-
-*Repository*: All data produced during a backup is sent to and stored in
-a repository in a structured form, for example in a file system
-hierarchy with several subdirectories. A repository implementation must
-be able to fulfill a number of operations, e.g. list the contents.
-
-*Blob*: A Blob combines a number of data bytes with identifying
-information like the SHA-256 hash of the data and its length.
-
-*Pack*: A Pack combines one or more Blobs, e.g. in a single file.
-
-*Snapshot*: A Snapshot stands for the state of a file or directory that
-has been backed up at some point in time. The state here means the
-content and meta data like the name and modification time for the file
-or the directory and its contents.
-
-*Storage ID*: A storage ID is the SHA-256 hash of the content stored in
-the repository. This ID is required in order to load the file from the
-repository.
-
-Repository Format
-=================
-
-All data is stored in a restic repository. A repository is able to store
-data of several different types, which can later be requested based on
-an ID. This so-called "storage ID" is the SHA-256 hash of the content of
-a file. All files in a repository are only written once and never
-modified afterwards. This allows accessing and even writing to the
-repository with multiple clients in parallel. Only the ``prune`` operation
-removes data from the repository.
-
-Repositories consist of several directories and a top-level file called
-``config``. For all other files stored in the repository, the name for
-the file is the lower case hexadecimal representation of the storage ID,
-which is the SHA-256 hash of the file's contents. This allows for easy
-verification of files for accidental modifications, like disk read
-errors, by simply running the program ``sha256sum`` on the file and
-comparing its output to the file name. If the prefix of a filename is
-unique amongst all the other files in the same directory, the prefix may
-be used instead of the complete filename.
-
-Apart from the files stored within the ``keys`` directory, all files are
-encrypted with AES-256 in counter mode (CTR). The integrity of the
-encrypted data is secured by a Poly1305-AES message authentication code
-(sometimes also referred to as a "signature").
-
-In the first 16 bytes of each encrypted file the initialisation vector
-(IV) is stored. It is followed by the encrypted data and completed by
-the 16 byte MAC. The format is: ``IV || CIPHERTEXT || MAC``. The
-complete encryption overhead is 32 bytes. For each file, a new random IV
-is selected.
-
-The file ``config`` is encrypted this way and contains a JSON document
-like the following:
-
-.. code:: json
-
-    {
-      "version": 1,
-      "id": "5956a3f67a6230d4a92cefb29529f10196c7d92582ec305fd71ff6d331d6271b",
-      "chunker_polynomial": "25b468838dcb75"
-    }
-
-After decryption, restic first checks that the version field contains a
-version number that it understands, otherwise it aborts. At the moment,
-the version is expected to be 1. The field ``id`` holds a unique ID
-which consists of 32 random bytes, encoded in hexadecimal. This uniquely
-identifies the repository, regardless if it is accessed via SFTP or
-locally. The field ``chunker_polynomial`` contains a parameter that is
-used for splitting large files into smaller chunks (see below).
-
-Repository Layout
------------------
-
-The ``local`` and ``sftp`` backends are implemented using files and
-directories stored in a file system. The directory layout is the same
-for both backend types.
-
-The basic layout of a repository stored in a ``local`` or ``sftp``
-backend is shown here:
-
-::
-
-    /tmp/restic-repo
-    ├── config
-    ├── data
-    │   ├── 21
-    │   │   └── 2159dd48f8a24f33c307b750592773f8b71ff8d11452132a7b2e2a6a01611be1
-    │   ├── 32
-    │   │   └── 32ea976bc30771cebad8285cd99120ac8786f9ffd42141d452458089985043a5
-    │   ├── 59
-    │   │   └── 59fe4bcde59bd6222eba87795e35a90d82cd2f138a27b6835032b7b58173a426
-    │   ├── 73
-    │   │   └── 73d04e6125cf3c28a299cc2f3cca3b78ceac396e4fcf9575e34536b26782413c
-    │   [...]
-    ├── index
-    │   ├── c38f5fb68307c6a3e3aa945d556e325dc38f5fb68307c6a3e3aa945d556e325d
-    │   └── ca171b1b7394d90d330b265d90f506f9984043b342525f019788f97e745c71fd
-    ├── keys
-    │   └── b02de829beeb3c01a63e6b25cbd421a98fef144f03b9a02e46eff9e2ca3f0bd7
-    ├── locks
-    ├── snapshots
-    │   └── 22a5af1bdc6e616f8a29579458c49627e01b32210d09adb288d1ecda7c5711ec
-    └── tmp
-
-A local repository can be initialized with the ``restic init`` command,
-e.g.:
-
-.. code-block:: console
-
-    $ restic -r /tmp/restic-repo init
-
-The local and sftp backends will auto-detect and accept all layouts described
-in the following sections, so that remote repositories mounted locally e.g. via
-fuse can be accessed. The layout auto-detection can be overridden by specifying
-the option ``-o local.layout=default``, valid values are ``default`` and
-``s3legacy``. The option for the sftp backend is named ``sftp.layout``, for the
-s3 backend ``s3.layout``.
-
-S3 Legacy Layout
-----------------
-
-Unfortunately during development the AWS S3 backend uses slightly different
-paths (directory names use singular instead of plural for ``key``,
-``lock``, and ``snapshot`` files), and the data files are stored directly below
-the ``data`` directory. The S3 Legacy repository layout looks like this:
-
-::
-
-    /config
-    /data
-     ├── 2159dd48f8a24f33c307b750592773f8b71ff8d11452132a7b2e2a6a01611be1
-     ├── 32ea976bc30771cebad8285cd99120ac8786f9ffd42141d452458089985043a5
-     ├── 59fe4bcde59bd6222eba87795e35a90d82cd2f138a27b6835032b7b58173a426
-     ├── 73d04e6125cf3c28a299cc2f3cca3b78ceac396e4fcf9575e34536b26782413c
-    [...]
-    /index
-     ├── c38f5fb68307c6a3e3aa945d556e325dc38f5fb68307c6a3e3aa945d556e325d
-     └── ca171b1b7394d90d330b265d90f506f9984043b342525f019788f97e745c71fd
-    /key
-     └── b02de829beeb3c01a63e6b25cbd421a98fef144f03b9a02e46eff9e2ca3f0bd7
-    /lock
-    /snapshot
-     └── 22a5af1bdc6e616f8a29579458c49627e01b32210d09adb288d1ecda7c5711ec
-
-The S3 backend understands and accepts both forms, new backends are
-always created with the default layout for compatibility reasons.
-
-Pack Format
-===========
-
-All files in the repository except Key and Pack files just contain raw
-data, stored as ``IV || Ciphertext || MAC``. Pack files may contain one
-or more Blobs of data.
-
-A Pack's structure is as follows:
-
-::
-
-    EncryptedBlob1 || ... || EncryptedBlobN || EncryptedHeader || Header_Length
-
-At the end of the Pack file is a header, which describes the content.
-The header is encrypted and authenticated. ``Header_Length`` is the
-length of the encrypted header encoded as a four byte integer in
-little-endian encoding. Placing the header at the end of a file allows
-writing the blobs in a continuous stream as soon as they are read during
-the backup phase. This reduces code complexity and avoids having to
-re-write a file once the pack is complete and the content and length of
-the header is known.
-
-All the blobs (``EncryptedBlob1``, ``EncryptedBlobN`` etc.) are
-authenticated and encrypted independently. This enables repository
-reorganisation without having to touch the encrypted Blobs. In addition
-it also allows efficient indexing, for only the header needs to be read
-in order to find out which Blobs are contained in the Pack. Since the
-header is authenticated, authenticity of the header can be checked
-without having to read the complete Pack.
-
-After decryption, a Pack's header consists of the following elements:
-
-::
-
-    Type_Blob1 || Length(EncryptedBlob1) || Hash(Plaintext_Blob1) ||
-    [...]
-    Type_BlobN || Length(EncryptedBlobN) || Hash(Plaintext_Blobn) ||
-
-This is enough to calculate the offsets for all the Blobs in the Pack.
-Length is the length of a Blob as a four byte integer in little-endian
-format. The type field is a one byte field and labels the content of a
-blob according to the following table:
-
-+--------+-----------+
-| Type   | Meaning   |
-+========+===========+
-| 0      | data      |
-+--------+-----------+
-| 1      | tree      |
-+--------+-----------+
-
-All other types are invalid, more types may be added in the future.
-
-For reconstructing the index or parsing a pack without an index, first
-the last four bytes must be read in order to find the length of the
-header. Afterwards, the header can be read and parsed, which yields all
-plaintext hashes, types, offsets and lengths of all included blobs.
-
-Indexing
-========
-
-Index files contain information about Data and Tree Blobs and the Packs
-they are contained in and store this information in the repository. When
-the local cached index is not accessible any more, the index files can
-be downloaded and used to reconstruct the index. The files are encrypted
-and authenticated like Data and Tree Blobs, so the outer structure is
-``IV || Ciphertext || MAC`` again. The plaintext consists of a JSON
-document like the following:
-
-.. code:: json
-
-    {
-      "supersedes": [
-        "ed54ae36197f4745ebc4b54d10e0f623eaaaedd03013eb7ae90df881b7781452"
-      ],
-      "packs": [
-        {
-          "id": "73d04e6125cf3c28a299cc2f3cca3b78ceac396e4fcf9575e34536b26782413c",
-          "blobs": [
-            {
-              "id": "3ec79977ef0cf5de7b08cd12b874cd0f62bbaf7f07f3497a5b1bbcc8cb39b1ce",
-              "type": "data",
-              "offset": 0,
-              "length": 25
-            },{
-              "id": "9ccb846e60d90d4eb915848add7aa7ea1e4bbabfc60e573db9f7bfb2789afbae",
-              "type": "tree",
-              "offset": 38,
-              "length": 100
-            },
-            {
-              "id": "d3dc577b4ffd38cc4b32122cabf8655a0223ed22edfd93b353dc0c3f2b0fdf66",
-              "type": "data",
-              "offset": 150,
-              "length": 123
-            }
-          ]
-        }, [...]
-      ]
-    }
-
-This JSON document lists Packs and the blobs contained therein. In this
-example, the Pack ``73d04e61`` contains two data Blobs and one Tree
-blob, the plaintext hashes are listed afterwards.
-
-The field ``supersedes`` lists the storage IDs of index files that have
-been replaced with the current index file. This happens when index files
-are repacked, for example when old snapshots are removed and Packs are
-recombined.
-
-There may be an arbitrary number of index files, containing information
-on non-disjoint sets of Packs. The number of packs described in a single
-file is chosen so that the file size is kept below 8 MiB.
-
-Keys, Encryption and MAC
-========================
-
-All data stored by restic in the repository is encrypted with AES-256 in
-counter mode and authenticated using Poly1305-AES. For encrypting new
-data first 16 bytes are read from a cryptographically secure
-pseudorandom number generator as a random nonce. This is used both as
-the IV for counter mode and the nonce for Poly1305. This operation needs
-three keys: A 32 byte for AES-256 for encryption, a 16 byte AES key and
-a 16 byte key for Poly1305. For details see the original paper `The
-Poly1305-AES message-authentication
-code <http://cr.yp.to/mac/poly1305-20050329.pdf>`__ by Dan Bernstein.
-The data is then encrypted with AES-256 and afterwards a message
-authentication code (MAC) is computed over the ciphertext, everything is
-then stored as IV \|\| CIPHERTEXT \|\| MAC.
-
-The directory ``keys`` contains key files. These are simple JSON
-documents which contain all data that is needed to derive the
-repository's master encryption and message authentication keys from a
-user's password. The JSON document from the repository can be
-pretty-printed for example by using the Python module ``json``
-(shortened to increase readability):
-
-::
-
-    $ python -mjson.tool /tmp/restic-repo/keys/b02de82*
-    {
-        "hostname": "kasimir",
-        "username": "fd0"
-        "kdf": "scrypt",
-        "N": 65536,
-        "r": 8,
-        "p": 1,
-        "created": "2015-01-02T18:10:13.48307196+01:00",
-        "data": "tGwYeKoM0C4j4/9DFrVEmMGAldvEn/+iKC3te/QE/6ox/V4qz58FUOgMa0Bb1cIJ6asrypCx/Ti/pRXCPHLDkIJbNYd2ybC+fLhFIJVLCvkMS+trdywsUkglUbTbi+7+Ldsul5jpAj9vTZ25ajDc+4FKtWEcCWL5ICAOoTAxnPgT+Lh8ByGQBH6KbdWabqamLzTRWxePFoYuxa7yXgmj9A==",
-        "salt": "uW4fEI1+IOzj7ED9mVor+yTSJFd68DGlGOeLgJELYsTU5ikhG/83/+jGd4KKAaQdSrsfzrdOhAMftTSih5Ux6w==",
-    }
-
-When the repository is opened by restic, the user is prompted for the
-repository password. This is then used with ``scrypt``, a key derivation
-function (KDF), and the supplied parameters (``N``, ``r``, ``p`` and
-``salt``) to derive 64 key bytes. The first 32 bytes are used as the
-encryption key (for AES-256) and the last 32 bytes are used as the
-message authentication key (for Poly1305-AES). These last 32 bytes are
-divided into a 16 byte AES key ``k`` followed by 16 bytes of secret key
-``r``. The key ``r`` is then masked for use with Poly1305 (see the paper
-for details).
-
-Those keys are used to authenticate and decrypt the bytes contained in
-the JSON field ``data`` with AES-256 and Poly1305-AES as if they were
-any other blob (after removing the Base64 encoding). If the
-password is incorrect or the key file has been tampered with, the
-computed MAC will not match the last 16 bytes of the data, and restic
-exits with an error. Otherwise, the data yields a JSON document
-which contains the master encryption and message authentication keys for
-this repository (encoded in Base64). The command
-``restic cat masterkey`` can be used as follows to decrypt and
-pretty-print the master key:
-
-.. code-block:: console
-
-    $ restic -r /tmp/restic-repo cat masterkey
-    {
-        "mac": {
-          "k": "evFWd9wWlndL9jc501268g==",
-          "r": "E9eEDnSJZgqwTOkDtOp+Dw=="
-        },
-        "encrypt": "UQCqa0lKZ94PygPxMRqkePTZnHRYh1k1pX2k2lM2v3Q=",
-    }
-
-All data in the repository is encrypted and authenticated with these
-master keys. For encryption, the AES-256 algorithm in Counter mode is
-used. For message authentication, Poly1305-AES is used as described
-above.
-
-A repository can have several different passwords, with a key file for
-each. This way, the password can be changed without having to re-encrypt
-all data.
-
-Snapshots
-=========
-
-A snapshot represents a directory with all files and sub-directories at
-a given point in time. For each backup that is made, a new snapshot is
-created. A snapshot is a JSON document that is stored in an encrypted
-file below the directory ``snapshots`` in the repository. The filename
-is the storage ID. This string is unique and used within restic to
-uniquely identify a snapshot.
-
-The command ``restic cat snapshot`` can be used as follows to decrypt
-and pretty-print the contents of a snapshot file:
-
-.. code-block:: console
-
-    $ restic -r /tmp/restic-repo cat snapshot 251c2e58
-    enter password for repository:
-    {
-      "time": "2015-01-02T18:10:50.895208559+01:00",
-      "tree": "2da81727b6585232894cfbb8f8bdab8d1eccd3d8f7c92bc934d62e62e618ffdf",
-      "dir": "/tmp/testdata",
-      "hostname": "kasimir",
-      "username": "fd0",
-      "uid": 1000,
-      "gid": 100,
-      "tags": [
-        "NL"
-      ]
-    }
-
-Here it can be seen that this snapshot represents the contents of the
-directory ``/tmp/testdata``. The most important field is ``tree``. When
-the meta data (e.g. the tags) of a snapshot change, the snapshot needs
-to be re-encrypted and saved. This will change the storage ID, so in
-order to relate these seemingly different snapshots, a field
-``original`` is introduced which contains the ID of the original
-snapshot, e.g. after adding the tag ``DE`` to the snapshot above it
-becomes:
-
-.. code-block:: console
-
-    $ restic -r /tmp/restic-repo cat snapshot 22a5af1b
-    enter password for repository:
-    {
-      "time": "2015-01-02T18:10:50.895208559+01:00",
-      "tree": "2da81727b6585232894cfbb8f8bdab8d1eccd3d8f7c92bc934d62e62e618ffdf",
-      "dir": "/tmp/testdata",
-      "hostname": "kasimir",
-      "username": "fd0",
-      "uid": 1000,
-      "gid": 100,
-      "tags": [
-        "NL",
-        "DE"
-      ],
-      "original": "251c2e5841355f743f9d4ffd3260bee765acee40a6229857e32b60446991b837"
-    }
-
-Once introduced, the ``original`` field is not modified when the
-snapshot's meta data is changed again.
-
-All content within a restic repository is referenced according to its
-SHA-256 hash. Before saving, each file is split into variable sized
-Blobs of data. The SHA-256 hashes of all Blobs are saved in an ordered
-list which then represents the content of the file.
-
-In order to relate these plaintext hashes to the actual location within
-a Pack file , an index is used. If the index is not available, the
-header of all data Blobs can be read.
-
-Trees and Data
-==============
-
-A snapshot references a tree by the SHA-256 hash of the JSON string
-representation of its contents. Trees and data are saved in pack files
-in a subdirectory of the directory ``data``.
-
-The command ``restic cat blob`` can be used to inspect the tree
-referenced above (piping the output of the command to ``jq .`` so that
-the JSON is indented):
-
-.. code-block:: console
-
-    $ restic -r /tmp/restic-repo cat blob 2da81727b6585232894cfbb8f8bdab8d1eccd3d8f7c92bc934d62e62e618ffdf | jq .
-    enter password for repository:
-    {
-      "nodes": [
-        {
-          "name": "testdata",
-          "type": "dir",
-          "mode": 493,
-          "mtime": "2014-12-22T14:47:59.912418701+01:00",
-          "atime": "2014-12-06T17:49:21.748468803+01:00",
-          "ctime": "2014-12-22T14:47:59.912418701+01:00",
-          "uid": 1000,
-          "gid": 100,
-          "user": "fd0",
-          "inode": 409704562,
-          "content": null,
-          "subtree": "b26e315b0988ddcd1cee64c351d13a100fedbc9fdbb144a67d1b765ab280b4dc"
-        }
-      ]
-    }
-
-A tree contains a list of entries (in the field ``nodes``) which contain
-meta data like a name and timestamps. When the entry references a
-directory, the field ``subtree`` contains the plain text ID of another
-tree object.
-
-When the command ``restic cat blob`` is used, the plaintext ID is needed
-to print a tree. The tree referenced above can be dumped as follows:
-
-.. code-block:: console
-
-    $ restic -r /tmp/restic-repo cat blob b26e315b0988ddcd1cee64c351d13a100fedbc9fdbb144a67d1b765ab280b4dc
-    enter password for repository:
-    {
-      "nodes": [
-        {
-          "name": "testfile",
-          "type": "file",
-          "mode": 420,
-          "mtime": "2014-12-06T17:50:23.34513538+01:00",
-          "atime": "2014-12-06T17:50:23.338468713+01:00",
-          "ctime": "2014-12-06T17:50:23.34513538+01:00",
-          "uid": 1000,
-          "gid": 100,
-          "user": "fd0",
-          "inode": 416863351,
-          "size": 1234,
-          "links": 1,
-          "content": [
-            "50f77b3b4291e8411a027b9f9b9e64658181cc676ce6ba9958b95f268cb1109d"
-          ]
-        },
-        [...]
-      ]
-    }
-
-This tree contains a file entry. This time, the ``subtree`` field is not
-present and the ``content`` field contains a list with one plain text
-SHA-256 hash.
-
-The command ``restic cat blob`` can also be used to extract and decrypt
-data given a plaintext ID, e.g. for the data mentioned above:
-
-.. code-block:: console
-
-    $ restic -r /tmp/restic-repo cat blob 50f77b3b4291e8411a027b9f9b9e64658181cc676ce6ba9958b95f268cb1109d | sha256sum
-    enter password for repository:
-    50f77b3b4291e8411a027b9f9b9e64658181cc676ce6ba9958b95f268cb1109d  -
-
-As can be seen from the output of the program ``sha256sum``, the hash
-matches the plaintext hash from the map included in the tree above, so
-the correct data has been returned.
-
-Locks
-=====
-
-The restic repository structure is designed in a way that allows
-parallel access of multiple instance of restic and even parallel writes.
-However, there are some functions that work more efficient or even
-require exclusive access of the repository. In order to implement these
-functions, restic processes are required to create a lock on the
-repository before doing anything.
-
-Locks come in two types: Exclusive and non-exclusive locks. At most one
-process can have an exclusive lock on the repository, and during that
-time there must not be any other locks (exclusive and non-exclusive).
-There may be multiple non-exclusive locks in parallel.
-
-A lock is a file in the subdir ``locks`` whose filename is the storage
-ID of the contents. It is encrypted and authenticated the same way as
-other files in the repository and contains the following JSON structure:
-
-.. code:: json
-
-    {
-      "time": "2015-06-27T12:18:51.759239612+02:00",
-      "exclusive": false,
-      "hostname": "kasimir",
-      "username": "fd0",
-      "pid": 13607,
-      "uid": 1000,
-      "gid": 100
-    }
-
-The field ``exclusive`` defines the type of lock. When a new lock is to
-be created, restic checks all locks in the repository. When a lock is
-found, it is tested if the lock is stale, which is the case for locks
-with timestamps older than 30 minutes. If the lock was created on the
-same machine, even for younger locks it is tested whether the process is
-still alive by sending a signal to it. If that fails, restic assumes
-that the process is dead and considers the lock to be stale.
-
-When a new lock is to be created and no other conflicting locks are
-detected, restic creates a new lock, waits, and checks if other locks
-appeared in the repository. Depending on the type of the other locks and
-the lock to be created, restic either continues or fails.
-
-Backups and Deduplication
-=========================
-
-For creating a backup, restic scans the source directory for all files,
-sub-directories and other entries. The data from each file is split into
-variable length Blobs cut at offsets defined by a sliding window of 64
-byte. The implementation uses Rabin Fingerprints for implementing this
-Content Defined Chunking (CDC). An irreducible polynomial is selected at
-random and saved in the file ``config`` when a repository is
-initialized, so that watermark attacks are much harder.
-
-Files smaller than 512 KiB are not split, Blobs are of 512 KiB to 8 MiB
-in size. The implementation aims for 1 MiB Blob size on average.
-
-For modified files, only modified Blobs have to be saved in a subsequent
-backup. This even works if bytes are inserted or removed at arbitrary
-positions within the file.
-
-Threat Model
-============
-
-The design goals for restic include being able to securely store backups
-in a location that is not completely trusted, e.g. a shared system where
-others can potentially access the files or (in the case of the system
-administrator) even modify or delete them.
-
-General assumptions:
-
--  The host system a backup is created on is trusted. This is the most
-   basic requirement, and essential for creating trustworthy backups.
-
-The restic backup program guarantees the following:
-
--  Accessing the unencrypted content of stored files and metadata should
-   not be possible without a password for the repository. Everything
-   except the metadata included for informational purposes in the key
-   files is encrypted and authenticated.
-
--  Modifications (intentional or unintentional) can be detected
-   automatically on several layers:
-
-   1. For all accesses of data stored in the repository it is checked
-      whether the cryptographic hash of the contents matches the storage
-      ID (the file's name). This way, modifications (bad RAM, broken
-      harddisk) can be detected easily.
-
-   2. Before decrypting any data, the MAC on the encrypted data is
-      checked. If there has been a modification, the MAC check will
-      fail. This step happens even before the data is decrypted, so data
-      that has been tampered with is not decrypted at all.
-
-However, the restic backup program is not designed to protect against
-attackers deleting files at the storage location. There is nothing that
-can be done about this. If this needs to be guaranteed, get a secure
-location without any access from third parties. If you assume that
-attackers have write access to your files at the storage location,
-attackers are able to figure out (e.g. based on the timestamps of the
-stored files) which files belong to what snapshot. When only these files
-are deleted, the particular snapshot vanished and all snapshots
-depending on data that has been added in the snapshot cannot be restored
-completely. Restic is not designed to detect this attack.
-
-Local Cache
-===========
-
-In order to speed up certain operations, restic manages a local cache of data.
-This document describes the data structures for the local cache with version 1.
-
-Versions
---------
-
-The cache directory is selected according to the `XDG base dir specification
-<http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html>`__.
-Each repository has its own cache sub-directory, consting of the repository ID
-which is chosen at ``init``. All cache directories for different repos are
-independent of each other.
-
-The cache dir for a repo contains a file named ``version``, which contains a
-single ASCII integer line that stands for the current version of the cache. If
-a lower version number is found the cache is recreated with the current
-version. If a higher version number is found the cache is ignored and left as
-is.
-
-Snapshots and Indexes
----------------------
-
-Snapshot, Data and Index files are cached in the sub-directories ``snapshots``,
-``data`` and  ``index``, as read from the repository.
-
-
-************
-REST Backend
-************
-
-Restic can interact with HTTP Backend that respects the following REST
-API.
-
-The following values are valid for ``{type}``:
-
- * ``data``
- * ``keys``
- * ``locks``
- * ``snapshots``
- * ``index``
- * ``config``
-
-The API version is selected via the ``Accept`` HTTP header in the request. The
-following values are defined:
-
- * ``application/vnd.x.restic.rest.v1+json`` or empty: Select API version 1
- * ``application/vnd.x.restic.rest.v2+json``: Select API version 2
-
-The server will respond with the value of the highest version it supports in
-the ``Content-Type`` HTTP response header for the HTTP requests which should
-return JSON. Any different value for this header means API version 1.
-
-The placeholder ``{path}`` in this document is a path to the repository, so
-that multiple different repositories can be accessed. The default path is
-``/``.
-
-POST {path}?create=true
-=======================
-
-This request is used to initially create a new repository. The server
-responds with "200 OK" if the repository structure was created
-successfully or already exists, otherwise an error is returned.
-
-DELETE {path}
-=============
-
-Deletes the repository on the server side. The server responds with "200
-OK" if the repository was successfully removed. If this function is not
-implemented the server returns "501 Not Implemented", if this it is
-denied by the server it returns "403 Forbidden".
-
-HEAD {path}/config
-==================
-
-Returns "200 OK" if the repository has a configuration, an HTTP error
-otherwise.
-
-GET {path}/config
-=================
-
-Returns the content of the configuration file if the repository has a
-configuration, an HTTP error otherwise.
-
-Response format: binary/octet-stream
-
-POST {path}/config
-==================
-
-Returns "200 OK" if the configuration of the request body has been
-saved, an HTTP error otherwise.
-
-GET {path}/{type}/
-==================
-
-API version 1
--------------
-
-Returns a JSON array containing the names of all the blobs stored for a given
-type, example:
-
-.. code:: json
-
-    [
-      "245bc4c430d393f74fbe7b13325e30dbde9fb0745e50caad57c446c93d20096b",
-      "85b420239efa1132c41cea0065452a40ebc20c6f8e0b132a5b2f5848360973ec",
-      "8e2006bb5931a520f3c7009fe278d1ebb87eb72c3ff92a50c30e90f1b8cf3e60",
-      "e75c8c407ea31ba399ab4109f28dd18c4c68303d8d86cc275432820c42ce3649"
-    ]
-
-API version 2
--------------
-
-Returns a JSON array containing an object for each file of the given type. The
-objects have two keys: ``name`` for the file name, and ``size`` for the size in
-bytes.
-
-.. code:: json
-
-    [
-      {
-        "name": "245bc4c430d393f74fbe7b13325e30dbde9fb0745e50caad57c446c93d20096b",
-        "size": 2341058
-      },
-      {
-        "name": "85b420239efa1132c41cea0065452a40ebc20c6f8e0b132a5b2f5848360973ec",
-        "size": 2908900
-      },
-      {
-        "name": "8e2006bb5931a520f3c7009fe278d1ebb87eb72c3ff92a50c30e90f1b8cf3e60",
-        "size": 3030712
-      },
-      {
-        "name": "e75c8c407ea31ba399ab4109f28dd18c4c68303d8d86cc275432820c42ce3649",
-        "size": 2804
-      }
-    ]
-
-HEAD {path}/{type}/{name}
-=========================
-
-Returns "200 OK" if the blob with the given name and type is stored in
-the repository, "404 not found" otherwise. If the blob exists, the HTTP
-header ``Content-Length`` is set to the file size.
-
-GET {path}/{type}/{name}
-========================
-
-Returns the content of the blob with the given name and type if it is
-stored in the repository, "404 not found" otherwise.
-
-If the request specifies a partial read with a Range header field, then
-the status code of the response is 206 instead of 200 and the response
-only contains the specified range.
-
-Response format: binary/octet-stream
-
-POST {path}/{type}/{name}
-=========================
-
-Saves the content of the request body as a blob with the given name and
-type, an HTTP error otherwise.
-
-Request format: binary/octet-stream
-
-DELETE {path}/{type}/{name}
-===========================
-
-Returns "200 OK" if the blob with the given name and type has been
-deleted from the repository, an HTTP error otherwise.
-
-
-*****
-Talks
-*****
-
-The following talks will be or have been given about restic:
-
--  2016-01-31: Lightning Talk at the Go Devroom at FOSDEM 2016,
-   Brussels, Belgium
--  2016-01-29: `restic - Backups mal
-   richtig <https://media.ccc.de/v/c4.openchaos.2016.01.restic>`__:
-   Public lecture in German at `CCC Cologne
-   e.V. <https://koeln.ccc.de>`__ in Cologne, Germany
--  2015-08-23: `A Solution to the Backup
-   Inconvenience <https://programm.froscon.de/2015/events/1515.html>`__:
-   Lecture at `FROSCON 2015 <https://www.froscon.de>`__ in Bonn, Germany
--  2015-02-01: `Lightning Talk at FOSDEM
-   2015 <https://www.youtube.com/watch?v=oM-MfeflUZ8&t=11m40s>`__: A
-   short introduction (with slightly outdated command line)
--  2015-01-27: `Talk about restic at CCC
-   Aachen <https://videoag.fsmpi.rwth-aachen.de/?view=player&lectureid=4442#content>`__
-   (in German)
+.. include:: design.rst
+.. include:: cache.rst
+.. include:: REST_backend.rst

doc/110_talks.rst

@@ -0,0 +1,34 @@
+..
+  Normally, there are no heading levels assigned to certain characters as the structure is
+  determined from the succession of headings. However, this convention is used in Python’s
+  Style Guide for documenting which you may follow:
+
+  # with overline, for parts
+  * for chapters
+  = for sections
+  - for subsections
+  ^ for subsubsections
+  " for paragraphs
+
+
+#####
+Talks
+#####
+
+The following talks will be or have been given about restic:
+
+-  2016-01-31: Lightning Talk at the Go Devroom at FOSDEM 2016,
+   Brussels, Belgium
+-  2016-01-29: `restic - Backups mal
+   richtig <https://media.ccc.de/v/c4.openchaos.2016.01.restic>`__:
+   Public lecture in German at `CCC Cologne
+   e.V. <https://koeln.ccc.de>`__ in Cologne, Germany
+-  2015-08-23: `A Solution to the Backup
+   Inconvenience <https://programm.froscon.de/2015/events/1515.html>`__:
+   Lecture at `FROSCON 2015 <https://www.froscon.de>`__ in Bonn, Germany
+-  2015-02-01: `Lightning Talk at FOSDEM
+   2015 <https://www.youtube.com/watch?v=oM-MfeflUZ8&t=11m40s>`__: A
+   short introduction (with slightly outdated command line)
+-  2015-01-27: `Talk about restic at CCC
+   Aachen <https://videoag.fsmpi.rwth-aachen.de/?view=player&lectureid=4442#content>`__
+   (in German)