wip

Fix find, do not skip some snapshots

.github/ISSUE_TEMPLATE.md

@@ -1,14 +1,28 @@
 <!--
-NOTE: Not filling out the issue template needs a good reason, otherwise it may
-take a lot longer to find the problem! Please take the time to help us
-debugging the problem by collecting information, even if it seems irrelevant to
-you. Thanks!
 
-If you have a question, the forum at https://forum.restic.net is a better place.
-Please do not create issues for usage or documentation questions! We're using
-the GitHub issue tracker mainly for tracking bugs and feature requests.
+Welcome! - We kindly ask that you:
+
+  1. Fill out the issue template below - not doing so needs a good reason.
+  2. Use the forum if you have a question rather than a bug or feature request.
+
+The forum is at: https://forum.restic.net
+
+NOTE: Not filling out the issue template needs a good reason, as otherwise it
+may take a lot longer to find the problem, not to mention it can take up a lot
+more time which can otherwise be spent on development. Please also take the
+time to help us debug the issue by collecting relevant information, even if
+it doesn't seem to be relevant to you. Thanks!
+
+The forum is a better place for questions about restic or general suggestions
+and topics, e.g. usage or documentation questions! This issue tracker is mainly
+for tracking bugs and feature requests directly relating to the development of
+the software itself, rather than the project.
+
+Thanks for understanding, and for contributing to the project!
+
 -->
 
+
 ## Output of `restic version`
 
 
@@ -24,10 +38,10 @@ This section should include at least:
    information to diagnose the problem!
 -->
 
-
 ## What backend/server/service did you use to store the repository?
 
 
+
 ## Expected behavior
 
 <!--
@@ -48,12 +62,14 @@ The more time you spend describing an easy way to reproduce the behavior (if
 this is possible), the easier it is for the project developers to fix it!
 -->
 
-
 ## Do you have any idea what may have caused this?
 
 
+
 ## Do you have an idea how to solve the issue?
 
+
+
 ## Did restic help you or made you happy in any way?
 
 <!--

.github/PULL_REQUEST_TEMPLATE.md

@@ -25,7 +25,7 @@ Link issues and relevant forum posts here.
 - [ ] I have read the [Contribution Guidelines](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#providing-patches)
 - [ ] I have added tests for all changes in this PR
 - [ ] I have added documentation for the changes (in the manual)
-- [ ] There's a new file in a subdir of `changelog/x.y.z` that describe the changes for our users (template [here](https://github.com/restic/restic/blob/master/changelog/changelog-entry.tmpl))
+- [ ] There's a new file in `changelog/unreleased/` that describes the changes for our users (template [here](https://github.com/restic/restic/blob/master/changelog/TEMPLATE))
 - [ ] I have run `gofmt` on the code in all commits
 - [ ] All commit messages are formatted in the same style as [the other commits in the repo](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#git-commits)
 - [ ] I'm done, this Pull Request is ready for review

.travis.yml

@@ -1,33 +1,20 @@
 language: go
 sudo: false
 
-go:
-  - "1.8.x"
-  - "1.9.x"
-  - "1.10"
-
-os:
-  - linux
-  - osx
-
-env:
-  matrix:
-    RESTIC_TEST_FUSE=0
-
 matrix:
-  exclude:
-    - os: osx
-      go: "1.8.x"
-    - os: osx
-      go: "1.9.x"
-    - os: linux
-      go: "1.10"
   include:
     - os: linux
-      go: "1.10"
+      go: "1.9.x"
+      env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0 RESTIC_BUILD_SOLARIS=0
+
+    # only run fuse and cloud backends tests on Travis for the latest Go on Linux
+    - os: linux
+      go: "1.10.x"
       sudo: true
-      env:
-        RESTIC_TEST_FUSE=1
+
+    - os: osx
+      go: "1.10.x"
+      env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0
 
 branches:
   only:

CHANGELOG.md

@@ -1,3 +1,410 @@
+Changelog for restic 0.9.1 (2018-06-10)
+=======================================
+
+The following sections list the changes in restic 0.9.1 relevant to
+restic users. The changes are ordered by importance.
+
+Summary
+-------
+
+ * Fix #1801: Add limiting bandwidth to the rclone backend
+ * Fix #1822: Allow uploading large files to MS Azure
+ * Fix #1825: Correct `find` to not skip snapshots
+ * Fix #1833: Fix caching files on error
+ * Fix #1834: Resolve deadlock
+
+Details
+-------
+
+ * Bugfix #1801: Add limiting bandwidth to the rclone backend
+
+   The rclone backend did not respect `--limit-upload` or `--limit-download`. Oftentimes it's
+   not necessary to use this, as the limiting in rclone itself should be used because it gives much
+   better results, but in case a remote instance of rclone is used (e.g. called via ssh), it is still
+   relevant to limit the bandwidth from restic to rclone.
+
+   https://github.com/restic/restic/issues/1801
+
+ * Bugfix #1822: Allow uploading large files to MS Azure
+
+   Sometimes, restic creates files to be uploaded to the repository which are quite large, e.g.
+   when saving directories with many entries or very large files. The MS Azure API does not allow
+   uploading files larger that 256MiB directly, rather restic needs to upload them in blocks of
+   100MiB. This is now implemented.
+
+   https://github.com/restic/restic/issues/1822
+
+ * Bugfix #1825: Correct `find` to not skip snapshots
+
+   Under certain circumstances, the `find` command was found to skip snapshots containing
+   directories with files to look for when the directories haven't been modified at all, and were
+   already printed as part of a different snapshot. This is now corrected.
+
+   In addition, we've switched to our own matching/pattern implementation, so now things like
+   `restic find "/home/user/foo/**/main.go"` are possible.
+
+   https://github.com/restic/restic/issues/1825
+   https://github.com/restic/restic/issues/1823
+
+ * Bugfix #1833: Fix caching files on error
+
+   During `check` it may happen that different threads access the same file in the backend, which
+   is then downloaded into the cache only once. When that fails, only the thread which is
+   responsible for downloading the file signals the correct error. The other threads just assume
+   that the file has been downloaded successfully and then get an error when they try to access the
+   cached file.
+
+   https://github.com/restic/restic/issues/1833
+
+ * Bugfix #1834: Resolve deadlock
+
+   When the "scanning" process restic runs to find out how much data there is does not finish before
+   the backup itself is done, restic stops doing anything. This is resolved now.
+
+   https://github.com/restic/restic/issues/1834
+   https://github.com/restic/restic/pull/1835
+
+
+Changelog for restic 0.9.0 (2018-05-21)
+=======================================
+
+The following sections list the changes in restic 0.9.0 relevant to
+restic users. The changes are ordered by importance.
+
+Summary
+-------
+
+ * Fix #1608: Respect time stamp for new backup when reading from stdin
+ * Fix #1652: Ignore/remove invalid lock files
+ * Fix #1730: Ignore sockets for restore
+ * Fix #1684: Fix backend tests for rest-server
+ * Fix #1745: Correctly parse the argument to --tls-client-cert
+ * Enh #1433: Support UTF-16 encoding and process Byte Order Mark
+ * Enh #1561: Allow using rclone to access other services
+ * Enh #1665: Improve cache handling for `restic check`
+ * Enh #1721: Add `cache` command to list cache dirs
+ * Enh #1758: Allow saving OneDrive folders in Windows
+ * Enh #549: Rework archiver code
+ * Enh #1552: Use Google Application Default credentials
+ * Enh #1477: Accept AWS_SESSION_TOKEN for the s3 backend
+ * Enh #1648: Ignore AWS permission denied error when creating a repository
+ * Enh #1649: Add illumos/Solaris support
+ * Enh #1709: Improve messages `restic check` prints
+ * Enh #827: Add --new-password-file flag for non-interactive password changes
+ * Enh #1735: Allow keeping a time range of snaphots
+ * Enh #1782: Use default AWS credentials chain for S3 backend
+
+Details
+-------
+
+ * Bugfix #1608: Respect time stamp for new backup when reading from stdin
+
+   When reading backups from stdin (via `restic backup --stdin`), restic now uses the time stamp
+   for the new backup passed in `--time`.
+
+   https://github.com/restic/restic/issues/1608
+   https://github.com/restic/restic/pull/1703
+
+ * Bugfix #1652: Ignore/remove invalid lock files
+
+   This corrects a bug introduced recently: When an invalid lock file in the repo is encountered
+   (e.g. if the file is empty), the code used to ignore that, but now returns the error. Now, invalid
+   files are ignored for the normal lock check, and removed when `restic unlock --remove-all` is
+   run.
+
+   https://github.com/restic/restic/issues/1652
+   https://github.com/restic/restic/pull/1653
+
+ * Bugfix #1730: Ignore sockets for restore
+
+   We've received a report and correct the behavior in which the restore code aborted restoring a
+   directory when a socket was encountered. Unix domain socket files cannot be restored (they are
+   created on the fly once a process starts listening). The error handling was corrected, and in
+   addition we're now ignoring sockets during restore.
+
+   https://github.com/restic/restic/issues/1730
+   https://github.com/restic/restic/pull/1731
+
+ * Bugfix #1684: Fix backend tests for rest-server
+
+   The REST server for restic now requires an explicit parameter (`--no-auth`) if no
+   authentication should be allowed. This is fixed in the tests.
+
+   https://github.com/restic/restic/pull/1684
+
+ * Bugfix #1745: Correctly parse the argument to --tls-client-cert
+
+   Previously, the --tls-client-cert method attempt to read ARGV[1] (hardcoded) instead of the
+   argument that was passed to it. This has been corrected.
+
+   https://github.com/restic/restic/issues/1745
+   https://github.com/restic/restic/pull/1746
+
+ * Enhancement #1433: Support UTF-16 encoding and process Byte Order Mark
+
+   On Windows, text editors commonly leave a Byte Order Mark at the beginning of the file to define
+   which encoding is used (oftentimes UTF-16). We've added code to support processing the BOMs in
+   text files, like the exclude files, the password file and the file passed via `--files-from`.
+   This does not apply to any file being saved in a backup, those are not touched and archived as they
+   are.
+
+   https://github.com/restic/restic/issues/1433
+   https://github.com/restic/restic/issues/1738
+   https://github.com/restic/restic/pull/1748
+
+ * Enhancement #1561: Allow using rclone to access other services
+
+   We've added the ability to use rclone to store backup data on all backends that it supports. This
+   was done in collaboration with Nick, the author of rclone. You can now use it to first configure a
+   service, then restic manages the rest (starting and stopping rclone). For details, please see
+   the manual.
+
+   https://github.com/restic/restic/issues/1561
+   https://github.com/restic/restic/pull/1657
+   https://rclone.org
+
+ * Enhancement #1665: Improve cache handling for `restic check`
+
+   For safety reasons, restic does not use a local metadata cache for the `restic check` command,
+   so that data is loaded from the repository and restic can check it's in good condition. When the
+   cache is disabled, restic will fetch each tiny blob needed for checking the integrity using a
+   separate backend request. For non-local backends, that will take a long time, and depending on
+   the backend (e.g. B2) may also be much more expensive.
+
+   This PR adds a few commits which will change the behavior as follows:
+
+   * When `restic check` is called without any additional parameters, it will build a new cache in a
+   temporary directory, which is removed at the end of the check. This way, we'll get readahead for
+   metadata files (so restic will fetch the whole file when the first blob from the file is
+   requested), but all data is freshly fetched from the storage backend. This is the default
+   behavior and will work for almost all users.
+
+   * When `restic check` is called with `--with-cache`, the default on-disc cache is used. This
+   behavior hasn't changed since the cache was introduced.
+
+   * When `--no-cache` is specified, restic falls back to the old behavior, and read all tiny blobs
+   in separate requests.
+
+   https://github.com/restic/restic/issues/1665
+   https://github.com/restic/restic/issues/1694
+   https://github.com/restic/restic/pull/1696
+
+ * Enhancement #1721: Add `cache` command to list cache dirs
+
+   The command `cache` was added, it allows listing restic's cache directoriers together with
+   the last usage. It also allows removing old cache dirs without having to access a repo, via
+   `restic cache --cleanup`
+
+   https://github.com/restic/restic/issues/1721
+   https://github.com/restic/restic/pull/1749
+
+ * Enhancement #1758: Allow saving OneDrive folders in Windows
+
+   Restic now contains a bugfix to two libraries, which allows saving OneDrive folders in
+   Windows. In order to use the newer versions of the libraries, the minimal version required to
+   compile restic is now Go 1.9.
+
+   https://github.com/restic/restic/issues/1758
+   https://github.com/restic/restic/pull/1765
+
+ * Enhancement #549: Rework archiver code
+
+   The core archiver code and the complementary code for the `backup` command was rewritten
+   completely. This resolves very annoying issues such as 549. The first backup with this release
+   of restic will likely result in all files being re-read locally, so it will take a lot longer. The
+   next backup after that will be fast again.
+
+   Basically, with the old code, restic took the last path component of each to-be-saved file or
+   directory as the top-level file/directory within the snapshot. This meant that when called as
+   `restic backup /home/user/foo`, the snapshot would contain the files in the directory
+   `/home/user/foo` as `/foo`.
+
+   This is not the case any more with the new archiver code. Now, restic works very similar to what
+   `tar` does: When restic is called with an absolute path to save, then it'll preserve the
+   directory structure within the snapshot. For the example above, the snapshot would contain
+   the files in the directory within `/home/user/foo` in the snapshot. For relative
+   directories, it only preserves the relative path components. So `restic backup user/foo`
+   will save the files as `/user/foo` in the snapshot.
+
+   While we were at it, the status display and notification system was completely rewritten. By
+   default, restic now shows which files are currently read (unless `--quiet` is specified) in a
+   multi-line status display.
+
+   The `backup` command also gained a new option: `--verbose`. It can be specified once (which
+   prints a bit more detail what restic is doing) or twice (which prints a line for each
+   file/directory restic encountered, together with some statistics).
+
+   Another issue that was resolved is the new code only reads two files at most. The old code would
+   read way too many files in parallel, thereby slowing down the backup process on spinning discs a
+   lot.
+
+   https://github.com/restic/restic/issues/549
+   https://github.com/restic/restic/issues/1286
+   https://github.com/restic/restic/issues/446
+   https://github.com/restic/restic/issues/1344
+   https://github.com/restic/restic/issues/1416
+   https://github.com/restic/restic/issues/1456
+   https://github.com/restic/restic/issues/1145
+   https://github.com/restic/restic/issues/1160
+   https://github.com/restic/restic/pull/1494
+
+ * Enhancement #1552: Use Google Application Default credentials
+
+   Google provide libraries to generate appropriate credentials with various fallback
+   sources. This change uses the library to generate our GCS client, which allows us to make use of
+   these extra methods.
+
+   This should be backward compatible with previous restic behaviour while adding the
+   additional capabilities to auth from Google's internal metadata endpoints. For users
+   running restic in GCP this can make authentication far easier than it was before.
+
+   https://github.com/restic/restic/pull/1552
+   https://developers.google.com/identity/protocols/application-default-credentials
+
+ * Enhancement #1477: Accept AWS_SESSION_TOKEN for the s3 backend
+
+   Before, it was not possible to use s3 backend with AWS temporary security credentials(with
+   AWS_SESSION_TOKEN). This change gives higher priority to credentials.EnvAWS credentials
+   provider.
+
+   https://github.com/restic/restic/issues/1477
+   https://github.com/restic/restic/pull/1479
+   https://github.com/restic/restic/pull/1647
+
+ * Enhancement #1648: Ignore AWS permission denied error when creating a repository
+
+   It's not possible to use s3 backend scoped to a subdirectory(with specific permissions).
+   Restic doesn't try to create repository in a subdirectory, when 'bucket exists' of parent
+   directory check fails due to permission issues.
+
+   https://github.com/restic/restic/pull/1648
+
+ * Enhancement #1649: Add illumos/Solaris support
+
+   https://github.com/restic/restic/pull/1649
+
+ * Enhancement #1709: Improve messages `restic check` prints
+
+   Some messages `restic check` prints are not really errors, so from now on restic does not treat
+   them as errors any more and exits cleanly.
+
+   https://github.com/restic/restic/pull/1709
+   https://forum.restic.net/t/what-is-the-standard-procedure-to-follow-if-a-backup-or-restore-is-interrupted/571/2
+
+ * Enhancement #827: Add --new-password-file flag for non-interactive password changes
+
+   This makes it possible to change a repository password without being prompted.
+
+   https://github.com/restic/restic/issues/827
+   https://github.com/restic/restic/pull/1720
+   https://forum.restic.net/t/changing-repo-password-without-prompt/591
+
+ * Enhancement #1735: Allow keeping a time range of snaphots
+
+   We've added the `--keep-within` option to the `forget` command. It instructs restic to keep
+   all snapshots within the given duration since the newest snapshot. For example, running
+   `restic forget --keep-within 5m7d` will keep all snapshots which have been made in the five
+   months and seven days since the latest snapshot.
+
+   https://github.com/restic/restic/pull/1735
+
+ * Enhancement #1782: Use default AWS credentials chain for S3 backend
+
+   Adds support for file credentials to the S3 backend (e.g. ~/.aws/credentials), and reorders
+   the credentials chain for the S3 backend to match AWS's standard, which is static credentials,
+   env vars, credentials file, and finally remote.
+
+   https://github.com/restic/restic/pull/1782
+
+
+Changelog for restic 0.8.3 (2018-02-26)
+=======================================
+
+The following sections list the changes in restic 0.8.3 relevant to
+restic users. The changes are ordered by importance.
+
+Summary
+-------
+
+ * Fix #1633: Fixed unexpected 'pack file cannot be listed' error
+ * Fix #1641: Ignore files with invalid names in the repo
+ * Fix #1638: Handle errors listing files in the backend
+ * Enh #1497: Add --read-data-subset flag to check command
+ * Enh #1560: Retry all repository file download errors
+ * Enh #1623: Don't check for presence of files in the backend before writing
+ * Enh #1634: Upgrade B2 client library, reduce HTTP requests
+
+Details
+-------
+
+ * Bugfix #1633: Fixed unexpected 'pack file cannot be listed' error
+
+   Due to a regression introduced in 0.8.2, the `rebuild-index` and `prune` commands failed to
+   read pack files with size of 587, 588, 589 or 590 bytes.
+
+   https://github.com/restic/restic/issues/1633
+   https://github.com/restic/restic/pull/1635
+
+ * Bugfix #1641: Ignore files with invalid names in the repo
+
+   The release 0.8.2 introduced a bug: when restic encounters files in the repo which do not have a
+   valid name, it tries to load a file with a name of lots of zeroes instead of ignoring it. This is now
+   resolved, invalid file names are just ignored.
+
+   https://github.com/restic/restic/issues/1641
+   https://github.com/restic/restic/pull/1643
+   https://forum.restic.net/t/help-fixing-repo-no-such-file/485/3
+
+ * Bugfix #1638: Handle errors listing files in the backend
+
+   A user reported in the forum that restic completes a backup although a concurrent `prune`
+   operation was running. A few error messages were printed, but the backup was attempted and
+   completed successfully. No error code was returned.
+
+   This should not happen: The repository is exclusively locked during `prune`, so when `restic
+   backup` is run in parallel, it should abort and return an error code instead.
+
+   It was found that the bug was in the code introduced only recently, which retries a List()
+   operation on the backend should that fail. It is now corrected.
+
+   https://github.com/restic/restic/pull/1638
+   https://forum.restic.net/t/restic-backup-returns-0-exit-code-when-already-locked/484
+
+ * Enhancement #1497: Add --read-data-subset flag to check command
+
+   This change introduces ability to check integrity of a subset of repository data packs. This
+   can be used to spread integrity check of larger repositories over a period of time.
+
+   https://github.com/restic/restic/issues/1497
+   https://github.com/restic/restic/pull/1556
+
+ * Enhancement #1560: Retry all repository file download errors
+
+   Restic will now retry failed downloads, similar to other operations.
+
+   https://github.com/restic/restic/pull/1560
+
+ * Enhancement #1623: Don't check for presence of files in the backend before writing
+
+   Before, all backend implementations were required to return an error if the file that is to be
+   written already exists in the backend. For most backends, that means making a request (e.g. via
+   HTTP) and returning an error when the file already exists.
+
+   This is not accurate, the file could have been created between the HTTP request testing for it,
+   and when writing starts, so we've relaxed this requeriment, which saves one additional HTTP
+   request per newly added file.
+
+   https://github.com/restic/restic/pull/1623
+
+ * Enhancement #1634: Upgrade B2 client library, reduce HTTP requests
+
+   We've upgraded the B2 client library restic uses to access BackBlaze B2. This reduces the
+   number of HTTP requests needed to upload a new file from two to one, which should improve
+   throughput to B2.
+
+   https://github.com/restic/restic/pull/1634
+
+
 Changelog for restic 0.8.2 (2018-02-17)
 =======================================
 
@@ -69,6 +476,7 @@ Details
    of data loss, just minor inconvenience for our users.
 
    https://github.com/restic/restic/pull/1589
+   https://forum.restic.net/t/error-loading-tree-check-prune-and-forget-gives-error-b2-backend/406
 
  * Bugfix #1594: Google Cloud Storage: Use generic HTTP transport
 
@@ -607,7 +1015,7 @@ Details
  * Enhancement #1203: Print stats on all BSD systems when SIGINFO (ctrl+t) is received
 
    https://github.com/restic/restic/pull/1203
-   https://github.com/restic/restic/pull/1082
+   https://github.com/restic/restic/pull/1082#issuecomment-326279920
 
  * Enhancement #1205: Allow specifying time/date for a backup with `--time`
 
@@ -647,12 +1055,12 @@ Details
  * Enhancement #1055: Create subdirs below `data/` for local/sftp backends
 
    The local and sftp backends now create the subdirs below `data/` on open/init. This way, restic
-   makes sure that they always exist. This is connected to an issue for the sftp server:
+   makes sure that they always exist. This is connected to an issue for the sftp server.
 
    https://github.com/restic/restic/issues/1055
+   https://github.com/restic/rest-server/pull/11#issuecomment-309879710
    https://github.com/restic/restic/pull/1077
    https://github.com/restic/restic/pull/1105
-   https://github.com/restic/rest-server/pull/11#issuecomment-309879710
 
  * Enhancement #1067: Allow loading credentials for s3 from IAM
 
@@ -664,7 +1072,7 @@ Details
 
  * Enhancement #1073: Add `migrate` cmd to migrate from `s3legacy` to `default` layout
 
-   The `migrate` command for chaning the `s3legacy` layout to the `default` layout for s3
+   The `migrate` command for changing the `s3legacy` layout to the `default` layout for s3
    backends has been improved: It can now be restarted with `restic migrate --force s3_layout`
    and automatically retries operations on error.
 

GOVERNANCE.md

@@ -0,0 +1,27 @@
+# restic project governance
+
+## Overview
+
+The restic project uses a governance model commonly described as Benevolent
+Dictator For Life (BDFL). This document outlines our understanding of what this
+means. It is derived from the [i3 window manager project
+governance](https://raw.githubusercontent.com/i3/i3/next/.github/GOVERNANCE.md).
+
+## Roles
+
+* user: anyone who interacts with the restic project
+* core contributor: a handful of people who have contributed significantly to
+  the project by any means (issue triage, support, documentation, code, etc.).
+  Core contributors are recognizable via GitHub’s "Member" badge.
+* Benevolent Dictator For Life (BDFL): a single individual who makes decisions
+  when consensus cannot be reached. restic's current BDFL is [@fd0](https://github.com/fd0).
+
+## Decision making process
+
+In general, we try to reach consensus in discussions. In case consensus cannot
+be reached, the BDFL makes a decision.
+
+## Contribution process
+
+The contribution process is described in a separate document called
+[CONTRIBUTING](CONTRIBUTING.md).

Gopkg.lock

@@ -10,38 +10,38 @@
 [[projects]]
   name = "cloud.google.com/go"
   packages = ["compute/metadata"]
-  revision = "767c40d6a2e058483c25fa193e963a22da17236d"
-  version = "v0.18.0"
+  revision = "4b98a6370e36d7a85192e7bad08a4ebd82eac2a8"
+  version = "v0.20.0"
 
 [[projects]]
   name = "github.com/Azure/azure-sdk-for-go"
-  packages = ["storage"]
-  revision = "eae258195456be76b2ec9ad2ee2ab63cdda365d9"
-  version = "v12.2.0-beta"
+  packages = ["storage","version"]
+  revision = "56332fec5b308fbb6615fa1af6117394cdba186d"
+  version = "v15.0.0"
 
 [[projects]]
   name = "github.com/Azure/go-autorest"
   packages = ["autorest","autorest/adal","autorest/azure","autorest/date"]
-  revision = "c2a68353555b68de3ee8455a4fd3e890a0ac6d99"
-  version = "v9.8.1"
+  revision = "ed4b7f5bf1ec0c9ede1fda2681d96771282f2862"
+  version = "v10.4.0"
 
 [[projects]]
   name = "github.com/cenkalti/backoff"
   packages = ["."]
-  revision = "61153c768f31ee5f130071d08fc82b85208528de"
-  version = "v1.1.0"
+  revision = "2ea60e5f094469f9e65adb9cd103795b73ae743e"
+  version = "v2.0.0"
 
 [[projects]]
   name = "github.com/cpuguy83/go-md2man"
   packages = ["md2man"]
-  revision = "1d903dcb749992f3741d744c0f8376b4bd7eb3e1"
-  version = "v1.0.7"
+  revision = "20f5889cbdc3c73dbd2862796665e7c465ade7d1"
+  version = "v1.0.8"
 
 [[projects]]
   name = "github.com/dgrijalva/jwt-go"
   packages = ["."]
-  revision = "dbeaa9332f19a944acb5736b4456cfcc02140e29"
-  version = "v3.1.0"
+  revision = "06ea1031745cb8b3dab3f6a236daf2b0aa468b7e"
+  version = "v3.2.0"
 
 [[projects]]
   branch = "master"
@@ -52,20 +52,26 @@
 [[projects]]
   name = "github.com/elithrar/simple-scrypt"
   packages = ["."]
-  revision = "2325946f714c95de4a6088202c402fbdfa64163b"
-  version = "v1.2.0"
+  revision = "d150773194090feb6c897805a7bcea8d49544e2c"
+  version = "v1.3.0"
 
 [[projects]]
   name = "github.com/go-ini/ini"
   packages = ["."]
-  revision = "32e4c1e6bc4e7d0d8451aa6b75200d19e37a536a"
-  version = "v1.32.0"
+  revision = "6333e38ac20b8949a8dd68baa3650f4dee8f39f0"
+  version = "v1.33.0"
 
 [[projects]]
-  branch = "master"
   name = "github.com/golang/protobuf"
   packages = ["proto"]
-  revision = "c65a0412e71e8b9b3bfd22925720d23c0f054237"
+  revision = "925541529c1fa6821df4e44ce2723319eb2be768"
+  version = "v1.0.0"
+
+[[projects]]
+  name = "github.com/google/go-cmp"
+  packages = ["cmp","cmp/internal/diff","cmp/internal/function","cmp/internal/value"]
+  revision = "8099a9787ce5dc5984ed879a3bda47dc730a8e97"
+  version = "v0.1.0"
 
 [[projects]]
   name = "github.com/inconshreveable/mousetrap"
@@ -87,20 +93,27 @@
 
 [[projects]]
   name = "github.com/kurin/blazer"
-  packages = ["b2","base","internal/b2types","internal/blog"]
-  revision = "e269a1a17bb6aec278c06a57cb7e8f8d0d333e04"
-  version = "v0.2.1"
+  packages = ["b2","base","internal/b2assets","internal/b2types","internal/blog","x/window"]
+  revision = "318e9768bf9a0fe52a64b9f8fe74f4f5caef6452"
+  version = "v0.4.4"
 
 [[projects]]
   name = "github.com/marstr/guid"
   packages = ["."]
-  revision = "8bdf7d1a087ccc975cf37dd6507da50698fd19ca"
+  revision = "8bd9a64bf37eb297b492a4101fb28e80ac0b290f"
+  version = "v1.1.0"
+
+[[projects]]
+  name = "github.com/mattn/go-isatty"
+  packages = ["."]
+  revision = "0360b2af4f38e8d38c7fce2a9f4e702702d73a39"
+  version = "v0.0.3"
 
 [[projects]]
   name = "github.com/minio/minio-go"
   packages = [".","pkg/credentials","pkg/encrypt","pkg/policy","pkg/s3signer","pkg/s3utils","pkg/set"]
-  revision = "14f1d472d115bac5ca4804094aa87484a72ced61"
-  version = "4.0.6"
+  revision = "66252c2a3c15f7b90cc8493d497a04ac3b6e3606"
+  version = "5.0.0"
 
 [[projects]]
   branch = "master"
@@ -112,7 +125,7 @@
   branch = "master"
   name = "github.com/ncw/swift"
   packages = ["."]
-  revision = "ae9f0ea1605b9aa6434ed5c731ca35d83ba67c55"
+  revision = "b2a7479cf26fa841ff90dd932d0221cb5c50782d"
 
 [[projects]]
   name = "github.com/pkg/errors"
@@ -129,14 +142,14 @@
 [[projects]]
   name = "github.com/pkg/sftp"
   packages = ["."]
-  revision = "f6a9258a0f570c3a76681b897b6ded57cb0dfa88"
-  version = "1.2.0"
+  revision = "49488377fa2f14143ba3067cf7555f60f6c7b550"
+  version = "1.5.0"
 
 [[projects]]
   name = "github.com/pkg/xattr"
   packages = ["."]
-  revision = "23c75e3f6c1d8b13b3dd905b011a7f38a06044b7"
-  version = "v0.2.1"
+  revision = "1d7b7ffe7c46974a836eb583b7452f22de1c18cf"
+  version = "v0.2.3"
 
 [[projects]]
   name = "github.com/restic/chunker"
@@ -147,8 +160,8 @@
 [[projects]]
   name = "github.com/russross/blackfriday"
   packages = ["."]
-  revision = "4048872b16cc0fc2c5fd9eacf0ed2c2fedaa0c8c"
-  version = "v1.5"
+  revision = "55d61fa8aa702f59229e6cff85793c22e580eaf5"
+  version = "v1.5.1"
 
 [[projects]]
   name = "github.com/satori/go.uuid"
@@ -159,14 +172,14 @@
 [[projects]]
   name = "github.com/sirupsen/logrus"
   packages = ["."]
-  revision = "d682213848ed68c0a260ca37d6dd5ace8423f5ba"
-  version = "v1.0.4"
+  revision = "c155da19408a8799da419ed3eeb0cb5db0ad5dbc"
+  version = "v1.0.5"
 
 [[projects]]
   name = "github.com/spf13/cobra"
   packages = [".","doc"]
-  revision = "7b2c5ac9fc04fc5efafb60700713d4fa609b777b"
-  version = "v0.0.1"
+  revision = "a1f051bc3eba734da4772d60e2d677f47cf93ef4"
+  version = "v0.0.2"
 
 [[projects]]
   name = "github.com/spf13/pflag"
@@ -177,44 +190,44 @@
 [[projects]]
   branch = "master"
   name = "golang.org/x/crypto"
-  packages = ["curve25519","ed25519","ed25519/internal/edwards25519","internal/chacha20","pbkdf2","poly1305","scrypt","ssh","ssh/terminal"]
-  revision = "3d37316aaa6bd9929127ac9a527abf408178ea7b"
+  packages = ["argon2","blake2b","curve25519","ed25519","ed25519/internal/edwards25519","internal/chacha20","pbkdf2","poly1305","scrypt","ssh","ssh/terminal"]
+  revision = "4ec37c66abab2c7e02ae775328b2ff001c3f025a"
 
 [[projects]]
   branch = "master"
   name = "golang.org/x/net"
-  packages = ["context","context/ctxhttp","idna","lex/httplex"]
-  revision = "5ccada7d0a7ba9aeb5d3aca8d3501b4c2a509fec"
+  packages = ["context","context/ctxhttp","http2","http2/hpack","idna","lex/httplex"]
+  revision = "6078986fec03a1dcc236c34816c71b0e05018fda"
 
 [[projects]]
   branch = "master"
   name = "golang.org/x/oauth2"
   packages = [".","google","internal","jws","jwt"]
-  revision = "b28fcf2b08a19742b43084fb40ab78ac6c3d8067"
+  revision = "fdc9e635145ae97e6c2cb777c48305600cf515cb"
 
 [[projects]]
   branch = "master"
   name = "golang.org/x/sync"
   packages = ["errgroup"]
-  revision = "fd80eb99c8f653c847d294a001bdf2a3a6f768f5"
+  revision = "1d60e4601c6fd243af51cc01ddf169918a5407ca"
 
 [[projects]]
   branch = "master"
   name = "golang.org/x/sys"
-  packages = ["unix","windows"]
-  revision = "af50095a40f9041b3b38960738837185c26e9419"
+  packages = ["cpu","unix","windows"]
+  revision = "7db1c3b1a98089d0071c84f646ff5c96aad43682"
 
 [[projects]]
-  branch = "master"
   name = "golang.org/x/text"
-  packages = ["collate","collate/build","internal/colltab","internal/gen","internal/tag","internal/triegen","internal/ucd","language","secure/bidirule","transform","unicode/bidi","unicode/cldr","unicode/norm","unicode/rangetable"]
-  revision = "e19ae1496984b1c655b8044a65c0300a3c878dd3"
+  packages = ["collate","collate/build","encoding","encoding/internal","encoding/internal/identifier","encoding/unicode","internal/colltab","internal/gen","internal/tag","internal/triegen","internal/ucd","internal/utf8internal","language","runes","secure/bidirule","transform","unicode/bidi","unicode/cldr","unicode/norm","unicode/rangetable"]
+  revision = "f21a4dfb5e38f5895301dc265a8def02365cc3d0"
+  version = "v0.3.0"
 
 [[projects]]
   branch = "master"
   name = "google.golang.org/api"
   packages = ["gensupport","googleapi","googleapi/internal/uritemplates","storage/v1"]
-  revision = "65b0d8655182691ad23b4fac11e6f7b897d9b634"
+  revision = "dbbc13f71100fa6ece308335445fca6bb0dd5c2f"
 
 [[projects]]
   name = "google.golang.org/appengine"
@@ -224,13 +237,19 @@
 
 [[projects]]
   branch = "v2"
+  name = "gopkg.in/tomb.v2"
+  packages = ["."]
+  revision = "d5d1b5820637886def9eef33e03a27a9f166942c"
+
+[[projects]]
   name = "gopkg.in/yaml.v2"
   packages = ["."]
-  revision = "d670f9405373e636a5a2765eea47fac0c9bc91a4"
+  revision = "5420a8b6744d3b0345ab293f6fcba19c978f1183"
+  version = "v2.2.1"
 
 [solve-meta]
   analyzer-name = "dep"
   analyzer-version = 1
-  inputs-digest = "336ac5c261c174cac89f9a7102b493f08edfbd51fd61d1673d1d2ec4132d80ab"
+  inputs-digest = "a5de339cba7570216b212439b90e1e6c384c94be8342fe7755b7cb66aa0a3440"
   solver-name = "gps-cdcl"
   solver-version = 1

README.rst

@@ -29,7 +29,7 @@ and add some data:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup backup ~/work
+    $ restic --repo /tmp/backup backup ~/work
     enter password for repository:
     scan [/home/user/work]
     scanned 764 directories, 1816 files in 0:00
@@ -57,6 +57,7 @@ Therefore, restic supports the following backends for storing backups natively:
 - `BackBlaze B2 <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#backblaze-b2>`__
 - `Microsoft Azure Blob Storage <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#microsoft-azure-blob-storage>`__
 - `Google Cloud Storage <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#google-cloud-storage>`__
+- And many other services via the `rclone <https://rclone.org>`__ `Backend <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#other-services-via-rclone>`__
 
 Design Principles
 -----------------
@@ -112,8 +113,8 @@ complete text in ``LICENSE``.
 
 .. |Documentation| image:: https://readthedocs.org/projects/restic/badge/?version=latest
    :target: https://restic.readthedocs.io/en/latest/?badge=latest
-.. |Build Status| image:: https://travis-ci.org/restic/restic.svg?branch=master
-   :target: https://travis-ci.org/restic/restic
+.. |Build Status| image:: https://travis-ci.com/restic/restic.svg?branch=master
+   :target: https://travis-ci.com/restic/restic
 .. |Build status| image:: https://ci.appveyor.com/api/projects/status/nuy4lfbgfbytw92q/branch/master?svg=true
    :target: https://ci.appveyor.com/project/fd0/restic/branch/master
 .. |Report Card| image:: https://goreportcard.com/badge/github.com/restic/restic

VERSION

@@ -1 +1 @@
-0.8.2
+0.9.1

build.go

@@ -53,7 +53,7 @@ var config = Config{
 		"github.com/restic/restic/internal/...",
 		"github.com/restic/restic/cmd/...",
 	},
-	MinVersion: GoVersion{Major: 1, Minor: 8, Patch: 0}, // minimum Go version supported
+	MinVersion: GoVersion{Major: 1, Minor: 9, Patch: 0}, // minimum Go version supported
 }
 
 // Config configures the build.
@@ -70,6 +70,7 @@ var (
 	keepGopath bool
 	runTests   bool
 	enableCGO  bool
+	enablePIE  bool
 )
 
 // specialDir returns true if the file begins with a special character ('.' or '_').
@@ -225,9 +226,11 @@ func showUsage(output io.Writer) {
 	fmt.Fprintf(output, "  -T     --test          run tests\n")
 	fmt.Fprintf(output, "  -o     --output        set output file name\n")
 	fmt.Fprintf(output, "         --enable-cgo    use CGO to link against libc\n")
+	fmt.Fprintf(output, "         --enable-pie    use PIE buildmode\n")
 	fmt.Fprintf(output, "         --goos value    set GOOS for cross-compilation\n")
 	fmt.Fprintf(output, "         --goarch value  set GOARCH for cross-compilation\n")
-	fmt.Fprintf(output, "         --goarm value  set GOARM for cross-compilation\n")
+	fmt.Fprintf(output, "         --goarm value   set GOARM for cross-compilation\n")
+	fmt.Fprintf(output, "         --tempdir dir   use a specific directory for compilation\n")
 }
 
 func verbosePrintf(message string, args ...interface{}) {
@@ -253,10 +256,22 @@ func cleanEnv() (env []string) {
 }
 
 // build runs "go build args..." with GOPATH set to gopath.
-func build(cwd, goos, goarch, goarm, gopath string, args ...string) error {
+func build(cwd string, ver GoVersion, goos, goarch, goarm, gopath string, args ...string) error {
 	a := []string{"build"}
-	a = append(a, "-asmflags", fmt.Sprintf("-trimpath=%s", gopath))
-	a = append(a, "-gcflags", fmt.Sprintf("-trimpath=%s", gopath))
+
+	if ver.AtLeast(GoVersion{1, 10, 0}) {
+		verbosePrintf("Go version is at least 1.10, using new syntax for -gcflags\n")
+		// use new prefix
+		a = append(a, "-asmflags", fmt.Sprintf("all=-trimpath=%s", gopath))
+		a = append(a, "-gcflags", fmt.Sprintf("all=-trimpath=%s", gopath))
+	} else {
+		a = append(a, "-asmflags", fmt.Sprintf("-trimpath=%s", gopath))
+		a = append(a, "-gcflags", fmt.Sprintf("-trimpath=%s", gopath))
+	}
+	if enablePIE {
+		a = append(a, "-buildmode=pie")
+	}
+
 	a = append(a, args...)
 	cmd := exec.Command("go", a...)
 	cmd.Env = append(cleanEnv(), "GOPATH="+gopath, "GOARCH="+goarch, "GOOS="+goos)
@@ -270,7 +285,7 @@ func build(cwd, goos, goarch, goarm, gopath string, args ...string) error {
 	cmd.Dir = cwd
 	cmd.Stdout = os.Stdout
 	cmd.Stderr = os.Stderr
-	verbosePrintf("go %s\n", args)
+	verbosePrintf("go %s\n", a)
 
 	return cmd.Run()
 }
@@ -366,30 +381,39 @@ func ParseGoVersion(s string) (v GoVersion) {
 
 	s = s[2:]
 	data := strings.Split(s, ".")
-	if len(data) != 3 {
-		return
+	if len(data) < 2 || len(data) > 3 {
+		// invalid version
+		return GoVersion{}
 	}
 
-	major, err := strconv.Atoi(data[0])
+	var err error
+
+	v.Major, err = strconv.Atoi(data[0])
 	if err != nil {
-		return
+		return GoVersion{}
 	}
 
-	minor, err := strconv.Atoi(data[1])
-	if err != nil {
-		return
+	// try to parse the minor version while removing an eventual suffix (like
+	// "rc2" or so)
+	for s := data[1]; s != ""; s = s[:len(s)-1] {
+		v.Minor, err = strconv.Atoi(s)
+		if err == nil {
+			break
+		}
 	}
 
-	patch, err := strconv.Atoi(data[2])
-	if err != nil {
-		return
+	if v.Minor == 0 {
+		// no minor version found
+		return GoVersion{}
 	}
 
-	v = GoVersion{
-		Major: major,
-		Minor: minor,
-		Patch: patch,
+	if len(data) >= 3 {
+		v.Patch, err = strconv.Atoi(data[2])
+		if err != nil {
+			return GoVersion{}
+		}
 	}
+
 	return
 }
 
@@ -437,6 +461,8 @@ func main() {
 	targetGOARCH := runtime.GOARCH
 	targetGOARM := ""
 
+	gopath := ""
+
 	var outputFilename string
 
 	for i, arg := range params {
@@ -459,10 +485,15 @@ func main() {
 		case "-o", "--output":
 			skipNext = true
 			outputFilename = params[i+1]
+		case "--tempdir":
+			skipNext = true
+			gopath = params[i+1]
 		case "-T", "--test":
 			runTests = true
 		case "--enable-cgo":
 			enableCGO = true
+		case "--enable-pie":
+			enablePIE = true
 		case "--goos":
 			skipNext = true
 			targetGOOS = params[i+1]
@@ -482,6 +513,8 @@ func main() {
 		}
 	}
 
+	verbosePrintf("detected Go version %v\n", ver)
+
 	if len(buildTags) == 0 {
 		verbosePrintf("adding build-tag release\n")
 		buildTags = []string{"release"}
@@ -498,9 +531,11 @@ func main() {
 		die("Getwd(): %v\n", err)
 	}
 
-	gopath, err := ioutil.TempDir("", fmt.Sprintf("%v-build-", config.Name))
-	if err != nil {
-		die("TempDir(): %v\n", err)
+	if gopath == "" {
+		gopath, err = ioutil.TempDir("", fmt.Sprintf("%v-build-", config.Name))
+		if err != nil {
+			die("TempDir(): %v\n", err)
+		}
 	}
 
 	verbosePrintf("create GOPATH at %v\n", gopath)
@@ -556,7 +591,7 @@ func main() {
 		"-o", output, config.Main,
 	}
 
-	err = build(filepath.Join(gopath, "src"), targetGOOS, targetGOARCH, targetGOARM, gopath, args...)
+	err = build(filepath.Join(gopath, "src"), ver, targetGOOS, targetGOARCH, targetGOARM, gopath, args...)
 	if err != nil {
 		die("build failed: %v\n", err)
 	}

changelog/0.7.1_2017-07-22/issue-1055

@@ -2,9 +2,9 @@ Enhancement: Create subdirs below `data/` for local/sftp backends
 
 The local and sftp backends now create the subdirs below `data/` on
 open/init. This way, restic makes sure that they always exist. This is
-connected to an issue for the sftp server:
+connected to an issue for the sftp server.
 
-https://github.com/restic/rest-server/pull/11#issuecomment-309879710
 https://github.com/restic/restic/issues/1055
+https://github.com/restic/rest-server/pull/11#issuecomment-309879710
 https://github.com/restic/restic/pull/1077
 https://github.com/restic/restic/pull/1105

changelog/0.7.1_2017-07-22/issue-1073

@@ -1,6 +1,6 @@
 Enhancement: Add `migrate` cmd to migrate from `s3legacy` to `default` layout
 
-The `migrate` command for chaning the `s3legacy` layout to the `default` layout
+The `migrate` command for changing the `s3legacy` layout to the `default` layout
 for s3 backends has been improved: It can now be restarted with `restic migrate
 --force s3_layout` and automatically retries operations on error.
 

changelog/0.8.3_2018-02-26/issue-1497

@@ -0,0 +1,8 @@
+Enhancement: Add --read-data-subset flag to check command
+
+This change introduces ability to check integrity of a subset of repository 
+data packs. This can be used to spread integrity check of larger repositories
+over a period of time.
+
+https://github.com/restic/restic/issues/1497
+https://github.com/restic/restic/pull/1556

changelog/0.8.3_2018-02-26/issue-1633

@@ -0,0 +1,7 @@
+Bugfix: Fixed unexpected 'pack file cannot be listed' error
+
+Due to a regression introduced in 0.8.2, the `rebuild-index` and `prune`
+commands failed to read pack files with size of 587, 588, 589 or 590 bytes.
+
+https://github.com/restic/restic/issues/1633
+https://github.com/restic/restic/pull/1635

changelog/0.8.3_2018-02-26/issue-1641

@@ -0,0 +1,10 @@
+Bugfix: Ignore files with invalid names in the repo
+
+The release 0.8.2 introduced a bug: when restic encounters files in the repo
+which do not have a valid name, it tries to load a file with a name of lots of
+zeroes instead of ignoring it. This is now resolved, invalid file names are
+just ignored.
+
+https://github.com/restic/restic/issues/1641
+https://github.com/restic/restic/pull/1643
+https://forum.restic.net/t/help-fixing-repo-no-such-file/485/3

changelog/0.8.3_2018-02-26/pull-1560

@@ -0,0 +1,5 @@
+Enhancement: Retry all repository file download errors
+
+Restic will now retry failed downloads, similar to other operations.
+
+https://github.com/restic/restic/pull/1560

changelog/0.8.3_2018-02-26/pull-1623

@@ -0,0 +1,12 @@
+Enhancement: Don't check for presence of files in the backend before writing
+
+Before, all backend implementations were required to return an error if the
+file that is to be written already exists in the backend. For most backends,
+that means making a request (e.g. via HTTP) and returning an error when the
+file already exists.
+
+This is not accurate, the file could have been created between the HTTP request
+testing for it, and when writing starts, so we've relaxed this requeriment,
+which saves one additional HTTP request per newly added file.
+
+https://github.com/restic/restic/pull/1623

changelog/0.8.3_2018-02-26/pull-1634

@@ -0,0 +1,7 @@
+Enhancement: Upgrade B2 client library, reduce HTTP requests
+
+We've upgraded the B2 client library restic uses to access BackBlaze B2. This
+reduces the number of HTTP requests needed to upload a new file from two to
+one, which should improve throughput to B2.
+
+https://github.com/restic/restic/pull/1634

changelog/0.8.3_2018-02-26/pull-1638

@@ -0,0 +1,16 @@
+Bugfix: Handle errors listing files in the backend
+
+A user reported in the forum that restic completes a backup although a
+concurrent `prune` operation was running. A few error messages were printed,
+but the backup was attempted and completed successfully. No error code was
+returned.
+
+This should not happen: The repository is exclusively locked during `prune`, so
+when `restic backup` is run in parallel, it should abort and return an error
+code instead.
+
+It was found that the bug was in the code introduced only recently, which
+retries a List() operation on the backend should that fail. It is now corrected.
+
+https://github.com/restic/restic/pull/1638
+https://forum.restic.net/t/restic-backup-returns-0-exit-code-when-already-locked/484

changelog/0.9.0_2018-05-21/issue-1433

@@ -0,0 +1,12 @@
+Enhancement: Support UTF-16 encoding and process Byte Order Mark
+
+On Windows, text editors commonly leave a Byte Order Mark at the beginning of
+the file to define which encoding is used (oftentimes UTF-16). We've added code
+to support processing the BOMs in text files, like the exclude files, the
+password file and the file passed via `--files-from`. This does not apply to
+any file being saved in a backup, those are not touched and archived as they
+are.
+
+https://github.com/restic/restic/issues/1433
+https://github.com/restic/restic/issues/1738
+https://github.com/restic/restic/pull/1748

changelog/0.9.0_2018-05-21/issue-1561

@@ -0,0 +1,10 @@
+Enhancement: Allow using rclone to access other services
+
+We've added the ability to use rclone to store backup data on all backends that
+it supports. This was done in collaboration with Nick, the author of rclone.
+You can now use it to first configure a service, then restic manages the rest
+(starting and stopping rclone). For details, please see the manual.
+
+https://github.com/restic/restic/issues/1561
+https://github.com/restic/restic/pull/1657
+https://rclone.org

changelog/0.9.0_2018-05-21/issue-1608

@@ -0,0 +1,7 @@
+Bugfix: Respect time stamp for new backup when reading from stdin
+
+When reading backups from stdin (via `restic backup --stdin`), restic now uses
+the time stamp for the new backup passed in `--time`.
+
+https://github.com/restic/restic/issues/1608
+https://github.com/restic/restic/pull/1703

changelog/0.9.0_2018-05-21/issue-1652

@@ -0,0 +1,9 @@
+Bugfix: Ignore/remove invalid lock files
+
+This corrects a bug introduced recently: When an invalid lock file in the repo
+is encountered (e.g. if the file is empty), the code used to ignore that, but
+now returns the error. Now, invalid files are ignored for the normal lock
+check, and removed when `restic unlock --remove-all` is run.
+
+https://github.com/restic/restic/issues/1652
+https://github.com/restic/restic/pull/1653

changelog/0.9.0_2018-05-21/issue-1665

@@ -0,0 +1,27 @@
+Enhancement: Improve cache handling for `restic check`
+
+For safety reasons, restic does not use a local metadata cache for the `restic
+check` command, so that data is loaded from the repository and restic can check
+it's in good condition. When the cache is disabled, restic will fetch each tiny
+blob needed for checking the integrity using a separate backend request. For
+non-local backends, that will take a long time, and depending on the backend
+(e.g. B2) may also be much more expensive.
+
+This PR adds a few commits which will change the behavior as follows:
+
+ * When `restic check` is called without any additional parameters, it will
+   build a new cache in a temporary directory, which is removed at the end of
+   the check. This way, we'll get readahead for metadata files (so restic will
+   fetch the whole file when the first blob from the file is requested), but
+   all data is freshly fetched from the storage backend. This is the default
+   behavior and will work for almost all users.
+
+ * When `restic check` is called with `--with-cache`, the default on-disc cache
+   is used. This behavior hasn't changed since the cache was introduced.
+
+ * When `--no-cache` is specified, restic falls back to the old behavior, and
+   read all tiny blobs in separate requests.
+
+https://github.com/restic/restic/issues/1665
+https://github.com/restic/restic/issues/1694
+https://github.com/restic/restic/pull/1696

changelog/0.9.0_2018-05-21/issue-1721

@@ -0,0 +1,8 @@
+Enhancement: Add `cache` command to list cache dirs
+
+The command `cache` was added, it allows listing restic's cache directoriers
+together with the last usage. It also allows removing old cache dirs without
+having to access a repo, via `restic cache --cleanup`
+
+https://github.com/restic/restic/issues/1721
+https://github.com/restic/restic/pull/1749

changelog/0.9.0_2018-05-21/issue-1730

@@ -0,0 +1,11 @@
+Bugfix: Ignore sockets for restore
+
+We've received a report and correct the behavior in which the restore code
+aborted restoring a directory when a socket was encountered. Unix domain socket
+files cannot be restored (they are created on the fly once a process starts
+listening). The error handling was corrected, and in addition we're now
+ignoring sockets during restore.
+
+https://github.com/restic/restic/issues/1730
+https://github.com/restic/restic/pull/1731
+

changelog/0.9.0_2018-05-21/issue-1758

@@ -0,0 +1,8 @@
+Enhancement: Allow saving OneDrive folders in Windows
+
+Restic now contains a bugfix to two libraries, which allows saving OneDrive
+folders in Windows. In order to use the newer versions of the libraries, the
+minimal version required to compile restic is now Go 1.9.
+
+https://github.com/restic/restic/issues/1758
+https://github.com/restic/restic/pull/1765

changelog/0.9.0_2018-05-21/issue-549

@@ -0,0 +1,43 @@
+Enhancement: Rework archiver code
+
+The core archiver code and the complementary code for the `backup` command was
+rewritten completely. This resolves very annoying issues such as 549. The first
+backup with this release of restic will likely result in all files being
+re-read locally, so it will take a lot longer. The next backup after that will
+be fast again.
+
+Basically, with the old code, restic took the last path component of each
+to-be-saved file or directory as the top-level file/directory within the
+snapshot. This meant that when called as `restic backup /home/user/foo`, the
+snapshot would contain the files in the directory `/home/user/foo` as `/foo`.
+
+This is not the case any more with the new archiver code. Now, restic works
+very similar to what `tar` does: When restic is called with an absolute path to
+save, then it'll preserve the directory structure within the snapshot. For the
+example above, the snapshot would contain the files in the directory within
+`/home/user/foo` in the snapshot. For relative directories, it only preserves
+the relative path components. So `restic backup user/foo` will save the files
+as `/user/foo` in the snapshot.
+
+While we were at it, the status display and notification system was completely
+rewritten. By default, restic now shows which files are currently read (unless
+`--quiet` is specified) in a multi-line status display.
+
+The `backup` command also gained a new option: `--verbose`. It can be specified
+once (which prints a bit more detail what restic is doing) or twice (which
+prints a line for each file/directory restic encountered, together with some
+statistics).
+
+Another issue that was resolved is the new code only reads two files at most.
+The old code would read way too many files in parallel, thereby slowing down
+the backup process on spinning discs a lot.
+
+https://github.com/restic/restic/issues/549
+https://github.com/restic/restic/issues/1286
+https://github.com/restic/restic/issues/446
+https://github.com/restic/restic/issues/1344
+https://github.com/restic/restic/issues/1416
+https://github.com/restic/restic/issues/1456
+https://github.com/restic/restic/issues/1145
+https://github.com/restic/restic/issues/1160
+https://github.com/restic/restic/pull/1494

changelog/0.9.0_2018-05-21/pull-1552

@@ -0,0 +1,13 @@
+Enhancement: Use Google Application Default credentials
+
+Google provide libraries to generate appropriate credentials with various
+fallback sources. This change uses the library to generate our GCS client, which
+allows us to make use of these extra methods.
+
+This should be backward compatible with previous restic behaviour while adding
+the additional capabilities to auth from Google's internal metadata endpoints.
+For users running restic in GCP this can make authentication far easier than it
+was before.
+
+https://github.com/restic/restic/pull/1552
+https://developers.google.com/identity/protocols/application-default-credentials

changelog/0.9.0_2018-05-21/pull-1647

@@ -0,0 +1,9 @@
+Enhancement: Accept AWS_SESSION_TOKEN for the s3 backend
+
+Before, it was not possible to use s3 backend with AWS temporary security
+credentials(with AWS_SESSION_TOKEN). This change gives higher priority to
+credentials.EnvAWS credentials provider.
+
+https://github.com/restic/restic/issues/1477
+https://github.com/restic/restic/pull/1479
+https://github.com/restic/restic/pull/1647

changelog/0.9.0_2018-05-21/pull-1648

@@ -0,0 +1,6 @@
+Enhancement: Ignore AWS permission denied error when creating a repository
+
+It's not possible to use s3 backend scoped to a subdirectory(with specific permissions).
+Restic doesn't try to create repository in a subdirectory, when 'bucket exists' of parent directory check fails due to permission issues.
+
+https://github.com/restic/restic/pull/1648

changelog/0.9.0_2018-05-21/pull-1649

@@ -0,0 +1,3 @@
+Enhancement: Add illumos/Solaris support
+
+https://github.com/restic/restic/pull/1649

changelog/0.9.0_2018-05-21/pull-1684

@@ -0,0 +1,6 @@
+Bugfix: Fix backend tests for rest-server
+
+The REST server for restic now requires an explicit parameter (`--no-auth`) if
+no authentication should be allowed. This is fixed in the tests.
+
+https://github.com/restic/restic/pull/1684

changelog/0.9.0_2018-05-21/pull-1709

@@ -0,0 +1,7 @@
+Enhancement: Improve messages `restic check` prints
+
+Some messages `restic check` prints are not really errors, so from now on
+restic does not treat them as errors any more and exits cleanly.
+
+https://github.com/restic/restic/pull/1709
+https://forum.restic.net/t/what-is-the-standard-procedure-to-follow-if-a-backup-or-restore-is-interrupted/571/2

changelog/0.9.0_2018-05-21/pull-1720

@@ -0,0 +1,7 @@
+Enhancement: Add --new-password-file flag for non-interactive password changes
+
+This makes it possible to change a repository password without being prompted.
+
+https://github.com/restic/restic/issues/827
+https://github.com/restic/restic/pull/1720
+https://forum.restic.net/t/changing-repo-password-without-prompt/591

changelog/0.9.0_2018-05-21/pull-1735

@@ -0,0 +1,9 @@
+Enhancement: Allow keeping a time range of snaphots
+
+We've added the `--keep-within` option to the `forget` command. It instructs
+restic to keep all snapshots within the given duration since the newest
+snapshot. For example, running `restic forget --keep-within 5m7d` will keep all
+snapshots which have been made in the five months and seven days since the
+latest snapshot.
+
+https://github.com/restic/restic/pull/1735

changelog/0.9.0_2018-05-21/pull-1746

@@ -0,0 +1,7 @@
+Bugfix: Correctly parse the argument to --tls-client-cert
+
+Previously, the --tls-client-cert method attempt to read ARGV[1] (hardcoded)
+instead of the argument that was passed to it. This has been corrected.
+
+https://github.com/restic/restic/issues/1745
+https://github.com/restic/restic/pull/1746

changelog/0.9.0_2018-05-21/pull-1782

@@ -0,0 +1,7 @@
+Enhancement: Use default AWS credentials chain for S3 backend
+
+Adds support for file credentials to the S3 backend (e.g. ~/.aws/credentials),
+and reorders the credentials chain for the S3 backend to match AWS's standard,
+which is static credentials, env vars, credentials file, and finally remote.
+
+https://github.com/restic/restic/pull/1782

changelog/0.9.1_2018-06-10/issue-1801

@@ -0,0 +1,9 @@
+Bugfix: Add limiting bandwidth to the rclone backend
+
+The rclone backend did not respect `--limit-upload` or `--limit-download`.
+Oftentimes it's not necessary to use this, as the limiting in rclone itself
+should be used because it gives much better results, but in case a remote
+instance of rclone is used (e.g. called via ssh), it is still relevant to limit
+the bandwidth from restic to rclone.
+
+https://github.com/restic/restic/issues/1801

changelog/0.9.1_2018-06-10/issue-1822

@@ -0,0 +1,9 @@
+Bugfix: Allow uploading large files to MS Azure
+
+Sometimes, restic creates files to be uploaded to the repository which are
+quite large, e.g. when saving directories with many entries or very large
+files. The MS Azure API does not allow uploading files larger that 256MiB
+directly, rather restic needs to upload them in blocks of 100MiB. This is now
+implemented.
+
+https://github.com/restic/restic/issues/1822

changelog/0.9.1_2018-06-10/issue-1825

@@ -0,0 +1,12 @@
+Bugfix: Correct `find` to not skip snapshots
+
+Under certain circumstances, the `find` command was found to skip snapshots
+containing directories with files to look for when the directories haven't been
+modified at all, and were already printed as part of a different snapshot. This
+is now corrected.
+
+In addition, we've switched to our own matching/pattern implementation, so now
+things like `restic find "/home/user/foo/**/main.go"` are possible.
+
+https://github.com/restic/restic/issues/1825
+https://github.com/restic/restic/issues/1823

changelog/0.9.1_2018-06-10/issue-1833

@@ -0,0 +1,9 @@
+Bugfix: Fix caching files on error
+
+During `check` it may happen that different threads access the same file in the
+backend, which is then downloaded into the cache only once. When that fails,
+only the thread which is responsible for downloading the file signals the
+correct error. The other threads just assume that the file has been downloaded
+successfully and then get an error when they try to access the cached file.
+
+https://github.com/restic/restic/issues/1833

changelog/0.9.1_2018-06-10/issue-1834

@@ -0,0 +1,8 @@
+Bugfix: Resolve deadlock
+
+When the "scanning" process restic runs to find out how much data there is does
+not finish before the backup itself is done, restic stops doing anything. This
+is resolved now.
+
+https://github.com/restic/restic/issues/1834
+https://github.com/restic/restic/pull/1835

changelog/CHANGELOG.tmpl

@@ -18,11 +18,11 @@ Details
 {{ range $par := .Paragraphs }}
    {{ wrap $par 80 3 }}
 {{ end -}}
-{{ range $id := .Issues }}
-   https://github.com/restic/restic/issues/{{ $id -}}
+{{ range $url := .IssueURLs }}
+   {{ $url -}}
 {{ end -}}
-{{ range $id := .PRs }}
-   https://github.com/restic/restic/pull/{{ $id -}}
+{{ range $url := .PRURLs }}
+   {{ $url -}}
 {{ end -}}
 {{ range $url := .OtherURLs }}
    {{ $url -}}

changelog/changelog-github.tmpl

@@ -2,8 +2,7 @@
 Changelog for restic {{ .Version }} ({{ .Date }})
 =======================================
 
-The following sections list the changes in restic {{ .Version }} relevant to
-restic users. The changes are ordered by importance.
+The following sections list the changes in restic {{ .Version }} relevant to restic users. The changes are ordered by importance.
 
 Summary
 -------

cmd/restic/background.go

@@ -1,9 +0,0 @@
-// +build !linux
-
-package main
-
-// IsProcessBackground should return true if it is running in the background or false if not
-func IsProcessBackground() bool {
-	//TODO: Check if the process are running in the background in other OS than linux
-	return false
-}

cmd/restic/cleanup.go

@@ -64,7 +64,10 @@ func CleanupHandler(c <-chan os.Signal) {
 		fmt.Fprintf(stderr, "%ssignal %v received, cleaning up\n", ClearLine(), s)
 
 		code := 0
-		if s != syscall.SIGINT {
+
+		if s == syscall.SIGINT {
+			code = 130
+		} else {
 			code = 1
 		}
 

cmd/restic/cmd_backup.go

@@ -2,21 +2,27 @@ package main
 
 import (
 	"bufio"
-	"fmt"
-	"io"
+	"bytes"
+	"context"
+	"io/ioutil"
 	"os"
-	"path"
-	"path/filepath"
+	"strconv"
 	"strings"
 	"time"
 
 	"github.com/spf13/cobra"
+	tomb "gopkg.in/tomb.v2"
 
 	"github.com/restic/restic/internal/archiver"
 	"github.com/restic/restic/internal/debug"
 	"github.com/restic/restic/internal/errors"
 	"github.com/restic/restic/internal/fs"
+	"github.com/restic/restic/internal/repository"
 	"github.com/restic/restic/internal/restic"
+	"github.com/restic/restic/internal/textfile"
+	"github.com/restic/restic/internal/ui"
+	"github.com/restic/restic/internal/ui/config"
+	"github.com/restic/restic/internal/ui/termstatus"
 )
 
 var cmdBackup = &cobra.Command{
@@ -38,23 +44,34 @@ given as the arguments.
 	},
 	DisableAutoGenTag: true,
 	RunE: func(cmd *cobra.Command, args []string) error {
+		err := config.ApplyFlags(&backupOptions.Config, cmd.Flags())
+		if err != nil {
+			return err
+		}
+
 		if backupOptions.Stdin && backupOptions.FilesFrom == "-" {
 			return errors.Fatal("cannot use both `--stdin` and `--files-from -`")
 		}
 
-		if backupOptions.Stdin {
-			return readBackupFromStdin(backupOptions, globalOptions, args)
-		}
+		var t tomb.Tomb
+		term := termstatus.New(globalOptions.stdout, globalOptions.stderr, globalOptions.Quiet)
+		t.Go(func() error { term.Run(t.Context(globalOptions.ctx)); return nil })
 
-		return runBackup(backupOptions, globalOptions, args)
+		err = runBackup(backupOptions, globalOptions, term, args)
+		if err != nil {
+			return err
+		}
+		t.Kill(nil)
+		return t.Wait()
 	},
 }
 
 // BackupOptions bundles all options for the backup command.
 type BackupOptions struct {
+	Config config.Backup
+
 	Parent           string
 	Force            bool
-	Excludes         []string
 	ExcludeFiles     []string
 	ExcludeOtherFS   bool
 	ExcludeIfPresent []string
@@ -76,7 +93,9 @@ func init() {
 	f := cmdBackup.Flags()
 	f.StringVar(&backupOptions.Parent, "parent", "", "use this parent snapshot (default: last snapshot in the repo that has the same target files/directories)")
 	f.BoolVarP(&backupOptions.Force, "force", "f", false, `force re-reading the target files/directories (overrides the "parent" flag)`)
-	f.StringArrayVarP(&backupOptions.Excludes, "exclude", "e", nil, "exclude a `pattern` (can be specified multiple times)")
+
+	f.StringArrayP("exclude", "e", nil, "exclude a `pattern` (can be specified multiple times)")
+
 	f.StringArrayVar(&backupOptions.ExcludeFiles, "exclude-file", nil, "read exclude patterns from a `file` (can be specified multiple times)")
 	f.BoolVarP(&backupOptions.ExcludeOtherFS, "one-file-system", "x", false, "exclude other file systems")
 	f.StringArrayVar(&backupOptions.ExcludeIfPresent, "exclude-if-present", nil, "takes filename[:header], exclude contents of directories containing filename (except filename itself) if header of that file is as provided (can be specified multiple times)")
@@ -90,127 +109,6 @@ func init() {
 	f.BoolVar(&backupOptions.WithAtime, "with-atime", false, "store the atime for all files and directories")
 }
 
-func newScanProgress(gopts GlobalOptions) *restic.Progress {
-	if gopts.Quiet {
-		return nil
-	}
-
-	p := restic.NewProgress()
-	p.OnUpdate = func(s restic.Stat, d time.Duration, ticker bool) {
-		if IsProcessBackground() {
-			return
-		}
-
-		PrintProgress("[%s] %d directories, %d files, %s", formatDuration(d), s.Dirs, s.Files, formatBytes(s.Bytes))
-	}
-
-	p.OnDone = func(s restic.Stat, d time.Duration, ticker bool) {
-		PrintProgress("scanned %d directories, %d files in %s\n", s.Dirs, s.Files, formatDuration(d))
-	}
-
-	return p
-}
-
-func newArchiveProgress(gopts GlobalOptions, todo restic.Stat) *restic.Progress {
-	if gopts.Quiet {
-		return nil
-	}
-
-	archiveProgress := restic.NewProgress()
-
-	var bps, eta uint64
-	itemsTodo := todo.Files + todo.Dirs
-
-	archiveProgress.OnUpdate = func(s restic.Stat, d time.Duration, ticker bool) {
-		if IsProcessBackground() {
-			return
-		}
-
-		sec := uint64(d / time.Second)
-		if todo.Bytes > 0 && sec > 0 && ticker {
-			bps = s.Bytes / sec
-			if s.Bytes >= todo.Bytes {
-				eta = 0
-			} else if bps > 0 {
-				eta = (todo.Bytes - s.Bytes) / bps
-			}
-		}
-
-		itemsDone := s.Files + s.Dirs
-
-		status1 := fmt.Sprintf("[%s] %s  %s / %s  %d / %d items  %d errors  ",
-			formatDuration(d),
-			formatPercent(s.Bytes, todo.Bytes),
-			formatBytes(s.Bytes), formatBytes(todo.Bytes),
-			itemsDone, itemsTodo,
-			s.Errors)
-		status2 := fmt.Sprintf("ETA %s ", formatSeconds(eta))
-
-		if w := stdoutTerminalWidth(); w > 0 {
-			maxlen := w - len(status2) - 1
-
-			if maxlen < 4 {
-				status1 = ""
-			} else if len(status1) > maxlen {
-				status1 = status1[:maxlen-4]
-				status1 += "... "
-			}
-		}
-
-		PrintProgress("%s%s", status1, status2)
-	}
-
-	archiveProgress.OnDone = func(s restic.Stat, d time.Duration, ticker bool) {
-		fmt.Printf("\nduration: %s\n", formatDuration(d))
-	}
-
-	return archiveProgress
-}
-
-func newArchiveStdinProgress(gopts GlobalOptions) *restic.Progress {
-	if gopts.Quiet {
-		return nil
-	}
-
-	archiveProgress := restic.NewProgress()
-
-	var bps uint64
-
-	archiveProgress.OnUpdate = func(s restic.Stat, d time.Duration, ticker bool) {
-		if IsProcessBackground() {
-			return
-		}
-
-		sec := uint64(d / time.Second)
-		if s.Bytes > 0 && sec > 0 && ticker {
-			bps = s.Bytes / sec
-		}
-
-		status1 := fmt.Sprintf("[%s] %s  %s/s", formatDuration(d),
-			formatBytes(s.Bytes),
-			formatBytes(bps))
-
-		if w := stdoutTerminalWidth(); w > 0 {
-			maxlen := w - len(status1)
-
-			if maxlen < 4 {
-				status1 = ""
-			} else if len(status1) > maxlen {
-				status1 = status1[:maxlen-4]
-				status1 += "... "
-			}
-		}
-
-		PrintProgress("%s", status1)
-	}
-
-	archiveProgress.OnDone = func(s restic.Stat, d time.Duration, ticker bool) {
-		fmt.Printf("\nduration: %s\n", formatDuration(d))
-	}
-
-	return archiveProgress
-}
-
 // filterExisting returns a slice of all existing items, or an error if no
 // items exist at all.
 func filterExisting(items []string) (result []string, err error) {
@@ -231,78 +129,33 @@ func filterExisting(items []string) (result []string, err error) {
 	return
 }
 
-func readBackupFromStdin(opts BackupOptions, gopts GlobalOptions, args []string) error {
-	if len(args) != 0 {
-		return errors.Fatal("when reading from stdin, no additional files can be specified")
-	}
-
-	fn := opts.StdinFilename
-
-	if fn == "" {
-		return errors.Fatal("filename for backup from stdin must not be empty")
-	}
-
-	if filepath.Base(fn) != fn || path.Base(fn) != fn {
-		return errors.Fatal("filename is invalid (may not contain a directory, slash or backslash)")
-	}
-
-	if gopts.password == "" {
-		return errors.Fatal("unable to read password from stdin when data is to be read from stdin, use --password-file or $RESTIC_PASSWORD")
-	}
-
-	repo, err := OpenRepository(gopts)
-	if err != nil {
-		return err
-	}
-
-	lock, err := lockRepo(repo)
-	defer unlockRepo(lock)
-	if err != nil {
-		return err
-	}
-
-	err = repo.LoadIndex(gopts.ctx)
-	if err != nil {
-		return err
-	}
-
-	r := &archiver.Reader{
-		Repository: repo,
-		Tags:       opts.Tags,
-		Hostname:   opts.Hostname,
-	}
-
-	_, id, err := r.Archive(gopts.ctx, fn, os.Stdin, newArchiveStdinProgress(gopts))
-	if err != nil {
-		return err
-	}
-
-	Verbosef("archived as %v\n", id.Str())
-	return nil
-}
-
-// readFromFile will read all lines from the given filename and write them to a
-// string array, if filename is empty readFromFile returns and empty string
-// array. If filename is a dash (-), readFromFile will read the lines from
-// the standard input.
+// readFromFile will read all lines from the given filename and return them as
+// a string array, if filename is empty readFromFile returns and empty string
+// array. If filename is a dash (-), readFromFile will read the lines from the
+// standard input.
 func readLinesFromFile(filename string) ([]string, error) {
 	if filename == "" {
 		return nil, nil
 	}
 
-	var r io.Reader = os.Stdin
-	if filename != "-" {
-		f, err := os.Open(filename)
-		if err != nil {
-			return nil, err
-		}
-		defer f.Close()
-		r = f
+	var (
+		data []byte
+		err  error
+	)
+
+	if filename == "-" {
+		data, err = ioutil.ReadAll(os.Stdin)
+	} else {
+		data, err = textfile.Read(filename)
+	}
+
+	if err != nil {
+		return nil, err
 	}
 
 	var lines []string
 
-	scanner := bufio.NewScanner(r)
+	scanner := bufio.NewScanner(bytes.NewReader(data))
 	for scanner.Scan() {
 		line := strings.TrimSpace(scanner.Text())
 		// ignore empty lines
@@ -323,56 +176,56 @@ func readLinesFromFile(filename string) ([]string, error) {
 	return lines, nil
 }
 
-func runBackup(opts BackupOptions, gopts GlobalOptions, args []string) error {
+// Check returns an error when an invalid combination of options was set.
+func (opts BackupOptions) Check(gopts GlobalOptions, args []string) error {
 	if opts.FilesFrom == "-" && gopts.password == "" {
 		return errors.Fatal("unable to read password from stdin when data is to be read from stdin, use --password-file or $RESTIC_PASSWORD")
 	}
 
-	fromfile, err := readLinesFromFile(opts.FilesFrom)
-	if err != nil {
-		return err
-	}
-
-	// merge files from files-from into normal args so we can reuse the normal
-	// args checks and have the ability to use both files-from and args at the
-	// same time
-	args = append(args, fromfile...)
-	if len(args) == 0 {
-		return errors.Fatal("nothing to backup, please specify target files/dirs")
-	}
-
-	target := make([]string, 0, len(args))
-	for _, d := range args {
-		if a, err := filepath.Abs(d); err == nil {
-			d = a
+	if opts.Stdin {
+		if opts.FilesFrom != "" {
+			return errors.Fatal("--stdin and --files-from cannot be used together")
+		}
+
+		if len(args) > 0 {
+			return errors.Fatal("--stdin was specified and files/dirs were listed as arguments")
 		}
-		target = append(target, d)
 	}
 
-	target, err = filterExisting(target)
-	if err != nil {
-		return err
-	}
-
-	// rejectFuncs collect functions that can reject items from the backup
-	var rejectFuncs []RejectFunc
+	return nil
+}
 
+// collectRejectFuncs returns a list of all functions which may reject data
+// from being saved in a snapshot
+func collectRejectFuncs(opts BackupOptions, repo *repository.Repository, targets []string) (fs []RejectFunc, excludes []string, err error) {
 	// allowed devices
 	if opts.ExcludeOtherFS {
-		f, err := rejectByDevice(target)
+		f, err := rejectByDevice(targets)
 		if err != nil {
-			return err
+			return nil, nil, err
 		}
-		rejectFuncs = append(rejectFuncs, f)
+		fs = append(fs, f)
 	}
 
+	// exclude restic cache
+	if repo.Cache != nil {
+		f, err := rejectResticCache(repo)
+		if err != nil {
+			return nil, nil, err
+		}
+
+		fs = append(fs, f)
+	}
+
+	excludes = append(excludes, opts.Config.Excludes...)
+
 	// add patterns from file
 	if len(opts.ExcludeFiles) > 0 {
-		opts.Excludes = append(opts.Excludes, readExcludePatternsFromFiles(opts.ExcludeFiles)...)
+		excludes = append(excludes, readExcludePatternsFromFiles(opts.ExcludeFiles)...)
 	}
 
-	if len(opts.Excludes) > 0 {
-		rejectFuncs = append(rejectFuncs, rejectByPattern(opts.Excludes))
+	if len(excludes) > 0 {
+		fs = append(fs, rejectByPattern(excludes))
 	}
 
 	if opts.ExcludeCaches {
@@ -382,124 +235,27 @@ func runBackup(opts BackupOptions, gopts GlobalOptions, args []string) error {
 	for _, spec := range opts.ExcludeIfPresent {
 		f, err := rejectIfPresent(spec)
 		if err != nil {
-			return err
+			return nil, nil, err
 		}
 
-		rejectFuncs = append(rejectFuncs, f)
+		fs = append(fs, f)
 	}
 
-	repo, err := OpenRepository(gopts)
-	if err != nil {
-		return err
-	}
-
-	lock, err := lockRepo(repo)
-	defer unlockRepo(lock)
-	if err != nil {
-		return err
-	}
-
-	// exclude restic cache
-	if repo.Cache != nil {
-		f, err := rejectResticCache(repo)
-		if err != nil {
-			return err
-		}
-
-		rejectFuncs = append(rejectFuncs, f)
-	}
-
-	err = repo.LoadIndex(gopts.ctx)
-	if err != nil {
-		return err
-	}
-
-	var parentSnapshotID *restic.ID
-
-	// Force using a parent
-	if !opts.Force && opts.Parent != "" {
-		id, err := restic.FindSnapshot(repo, opts.Parent)
-		if err != nil {
-			return errors.Fatalf("invalid id %q: %v", opts.Parent, err)
-		}
-
-		parentSnapshotID = &id
-	}
-
-	// Find last snapshot to set it as parent, if not already set
-	if !opts.Force && parentSnapshotID == nil {
-		id, err := restic.FindLatestSnapshot(gopts.ctx, repo, target, []restic.TagList{}, opts.Hostname)
-		if err == nil {
-			parentSnapshotID = &id
-		} else if err != restic.ErrNoSnapshotFound {
-			return err
-		}
-	}
-
-	if parentSnapshotID != nil {
-		Verbosef("using parent snapshot %v\n", parentSnapshotID.Str())
-	}
-
-	Verbosef("scan %v\n", target)
-
-	selectFilter := func(item string, fi os.FileInfo) bool {
-		for _, reject := range rejectFuncs {
-			if reject(item, fi) {
-				return false
-			}
-		}
-		return true
-	}
-
-	stat, err := archiver.Scan(target, selectFilter, newScanProgress(gopts))
-	if err != nil {
-		return err
-	}
-
-	arch := archiver.New(repo)
-	arch.Excludes = opts.Excludes
-	arch.SelectFilter = selectFilter
-	arch.WithAccessTime = opts.WithAtime
-
-	arch.Warn = func(dir string, fi os.FileInfo, err error) {
-		// TODO: make ignoring errors configurable
-		Warnf("%s\rwarning for %s: %v\n", ClearLine(), dir, err)
-	}
-
-	timeStamp := time.Now()
-	if opts.TimeStamp != "" {
-		timeStamp, err = time.Parse(TimeFormat, opts.TimeStamp)
-		if err != nil {
-			return errors.Fatalf("error in time option: %v\n", err)
-		}
-	}
-
-	_, id, err := arch.Snapshot(gopts.ctx, newArchiveProgress(gopts, stat), target, opts.Tags, opts.Hostname, parentSnapshotID, timeStamp)
-	if err != nil {
-		return err
-	}
-
-	Verbosef("snapshot %s saved\n", id.Str())
-
-	return nil
+	return fs, excludes, nil
 }
 
+// readExcludePatternsFromFiles reads all exclude files and returns the list of
+// exclude patterns.
 func readExcludePatternsFromFiles(excludeFiles []string) []string {
 	var excludes []string
 	for _, filename := range excludeFiles {
 		err := func() (err error) {
-			file, err := fs.Open(filename)
+			data, err := textfile.Read(filename)
 			if err != nil {
 				return err
 			}
-			defer func() {
-				// return pre-close error if there was one
-				if errClose := file.Close(); err == nil {
-					err = errClose
-				}
-			}()
 
-			scanner := bufio.NewScanner(file)
+			scanner := bufio.NewScanner(bytes.NewReader(data))
 			for scanner.Scan() {
 				line := strings.TrimSpace(scanner.Text())
 
@@ -525,3 +281,217 @@ func readExcludePatternsFromFiles(excludeFiles []string) []string {
 	}
 	return excludes
 }
+
+// collectTargets returns a list of target files/dirs from several sources.
+func collectTargets(opts BackupOptions, args []string) (targets []string, err error) {
+	if opts.Stdin {
+		return nil, nil
+	}
+
+	fromfile, err := readLinesFromFile(opts.FilesFrom)
+	if err != nil {
+		return nil, err
+	}
+
+	// merge files from files-from into normal args so we can reuse the normal
+	// args checks and have the ability to use both files-from and args at the
+	// same time
+	args = append(args, fromfile...)
+	if len(args) == 0 && !opts.Stdin {
+		return nil, errors.Fatal("nothing to backup, please specify target files/dirs")
+	}
+
+	targets = args
+	targets, err = filterExisting(targets)
+	if err != nil {
+		return nil, err
+	}
+
+	return targets, nil
+}
+
+// parent returns the ID of the parent snapshot. If there is none, nil is
+// returned.
+func findParentSnapshot(ctx context.Context, repo restic.Repository, opts BackupOptions, targets []string) (parentID *restic.ID, err error) {
+	// Force using a parent
+	if !opts.Force && opts.Parent != "" {
+		id, err := restic.FindSnapshot(repo, opts.Parent)
+		if err != nil {
+			return nil, errors.Fatalf("invalid id %q: %v", opts.Parent, err)
+		}
+
+		parentID = &id
+	}
+
+	// Find last snapshot to set it as parent, if not already set
+	if !opts.Force && parentID == nil {
+		id, err := restic.FindLatestSnapshot(ctx, repo, targets, []restic.TagList{}, opts.Hostname)
+		if err == nil {
+			parentID = &id
+		} else if err != restic.ErrNoSnapshotFound {
+			return nil, err
+		}
+	}
+
+	return parentID, nil
+}
+
+func runBackup(opts BackupOptions, gopts GlobalOptions, term *termstatus.Terminal, args []string) error {
+	err := opts.Check(gopts, args)
+	if err != nil {
+		return err
+	}
+
+	targets, err := collectTargets(opts, args)
+	if err != nil {
+		return err
+	}
+
+	timeStamp := time.Now()
+	if opts.TimeStamp != "" {
+		timeStamp, err = time.Parse(TimeFormat, opts.TimeStamp)
+		if err != nil {
+			return errors.Fatalf("error in time option: %v\n", err)
+		}
+	}
+
+	var t tomb.Tomb
+
+	p := ui.NewBackup(term, gopts.verbosity)
+
+	// use the terminal for stdout/stderr
+	prevStdout, prevStderr := gopts.stdout, gopts.stderr
+	defer func() {
+		gopts.stdout, gopts.stderr = prevStdout, prevStderr
+	}()
+	gopts.stdout, gopts.stderr = p.Stdout(), p.Stderr()
+
+	if s, ok := os.LookupEnv("RESTIC_PROGRESS_FPS"); ok {
+		fps, err := strconv.Atoi(s)
+		if err == nil && fps >= 1 {
+			if fps > 60 {
+				fps = 60
+			}
+			p.MinUpdatePause = time.Second / time.Duration(fps)
+		}
+	}
+
+	t.Go(func() error { return p.Run(t.Context(gopts.ctx)) })
+
+	p.V("open repository")
+	repo, err := OpenRepository(gopts)
+	if err != nil {
+		return err
+	}
+
+	p.V("lock repository")
+	lock, err := lockRepo(repo)
+	defer unlockRepo(lock)
+	if err != nil {
+		return err
+	}
+
+	// rejectFuncs collect functions that can reject items from the backup
+	rejectFuncs, excludes, err := collectRejectFuncs(opts, repo, targets)
+	if err != nil {
+		return err
+	}
+
+	p.V("load index files")
+	err = repo.LoadIndex(gopts.ctx)
+	if err != nil {
+		return err
+	}
+
+	parentSnapshotID, err := findParentSnapshot(gopts.ctx, repo, opts, targets)
+	if err != nil {
+		return err
+	}
+
+	if parentSnapshotID != nil {
+		p.V("using parent snapshot %v\n", parentSnapshotID.Str())
+	}
+
+	selectFilter := func(item string, fi os.FileInfo) bool {
+		for _, reject := range rejectFuncs {
+			if reject(item, fi) {
+				return false
+			}
+		}
+		return true
+	}
+
+	var targetFS fs.FS = fs.Local{}
+	if opts.Stdin {
+		p.V("read data from stdin")
+		targetFS = &fs.Reader{
+			ModTime:    timeStamp,
+			Name:       opts.StdinFilename,
+			Mode:       0644,
+			ReadCloser: os.Stdin,
+		}
+		targets = []string{opts.StdinFilename}
+	}
+
+	sc := archiver.NewScanner(targetFS)
+	sc.Select = selectFilter
+	sc.Error = p.ScannerError
+	sc.Result = p.ReportTotal
+
+	p.V("start scan on %v", targets)
+	t.Go(func() error { return sc.Scan(t.Context(gopts.ctx), targets) })
+
+	arch := archiver.New(repo, targetFS, archiver.Options{})
+	arch.Select = selectFilter
+	arch.WithAtime = opts.WithAtime
+	arch.Error = p.Error
+	arch.CompleteItem = p.CompleteItemFn
+	arch.StartFile = p.StartFile
+	arch.CompleteBlob = p.CompleteBlob
+
+	if parentSnapshotID == nil {
+		parentSnapshotID = &restic.ID{}
+	}
+
+	snapshotOpts := archiver.SnapshotOptions{
+		Excludes:       excludes,
+		Tags:           opts.Tags,
+		Time:           timeStamp,
+		Hostname:       opts.Hostname,
+		ParentSnapshot: *parentSnapshotID,
+	}
+
+	uploader := archiver.IndexUploader{
+		Repository: repo,
+		Start: func() {
+			p.VV("uploading intermediate index")
+		},
+		Complete: func(id restic.ID) {
+			p.V("uploaded intermediate index %v", id.Str())
+		},
+	}
+
+	t.Go(func() error {
+		return uploader.Upload(gopts.ctx, t.Context(gopts.ctx), 30*time.Second)
+	})
+
+	p.V("start backup on %v", targets)
+	_, id, err := arch.Snapshot(gopts.ctx, targets, snapshotOpts)
+	if err != nil {
+		return errors.Fatalf("unable to save snapshot: %v", err)
+	}
+
+	p.Finish()
+	p.P("snapshot %s saved\n", id.Str())
+
+	// cleanly shutdown all running goroutines
+	t.Kill(nil)
+
+	// let's see if one returned an error
+	err = t.Wait()
+	if err != nil {
+		return err
+	}
+
+	return nil
+}

cmd/restic/cmd_cache.go

@@ -0,0 +1,122 @@
+package main
+
+import (
+	"fmt"
+	"path/filepath"
+	"sort"
+	"time"
+
+	"github.com/restic/restic/internal/cache"
+	"github.com/restic/restic/internal/errors"
+	"github.com/restic/restic/internal/fs"
+	"github.com/spf13/cobra"
+)
+
+var cmdCache = &cobra.Command{
+	Use:   "cache",
+	Short: "Operate on local cache directories",
+	Long: `
+The "cache" command allows listing and cleaning local cache directories.
+`,
+	DisableAutoGenTag: true,
+	RunE: func(cmd *cobra.Command, args []string) error {
+		return runCache(cacheOptions, globalOptions, args)
+	},
+}
+
+// CacheOptions bundles all options for the snapshots command.
+type CacheOptions struct {
+	Cleanup bool
+	MaxAge  uint
+}
+
+var cacheOptions CacheOptions
+
+func init() {
+	cmdRoot.AddCommand(cmdCache)
+
+	f := cmdCache.Flags()
+	f.BoolVar(&cacheOptions.Cleanup, "cleanup", false, "remove old cache directories")
+	f.UintVar(&cacheOptions.MaxAge, "max-age", 30, "max age in `days` for cache directories to be considered old")
+}
+
+func runCache(opts CacheOptions, gopts GlobalOptions, args []string) error {
+	if len(args) > 0 {
+		return errors.Fatal("the cache command has no arguments")
+	}
+
+	if gopts.NoCache {
+		return errors.Fatal("Refusing to do anything, the cache is disabled")
+	}
+
+	var (
+		cachedir = gopts.CacheDir
+		err      error
+	)
+
+	if cachedir == "" {
+		cachedir, err = cache.DefaultDir()
+		if err != nil {
+			return err
+		}
+	}
+
+	if opts.Cleanup || gopts.CleanupCache {
+		oldDirs, err := cache.OlderThan(cachedir, time.Duration(opts.MaxAge)*24*time.Hour)
+		if err != nil {
+			return err
+		}
+
+		if len(oldDirs) == 0 {
+			Verbosef("no old cache dirs found\n")
+			return nil
+		}
+
+		Verbosef("remove %d old cache directories\n", len(oldDirs))
+
+		for _, item := range oldDirs {
+			dir := filepath.Join(cachedir, item.Name())
+			err = fs.RemoveAll(dir)
+			if err != nil {
+				Warnf("unable to remove %v: %v\n", dir, err)
+			}
+		}
+
+		return nil
+	}
+
+	tab := NewTable()
+	tab.Header = fmt.Sprintf("%-14s   %-16s %s", "Repository ID", "Last Used", "Old")
+	tab.RowFormat = "%-14s   %-16s %s"
+
+	dirs, err := cache.All(cachedir)
+	if err != nil {
+		return err
+	}
+
+	if len(dirs) == 0 {
+		Printf("no cache dirs found, basedir is %v\n", cachedir)
+		return nil
+	}
+
+	sort.Slice(dirs, func(i, j int) bool {
+		return dirs[i].ModTime().Before(dirs[j].ModTime())
+	})
+
+	for _, entry := range dirs {
+		var old string
+		if cache.IsOld(entry.ModTime(), time.Duration(opts.MaxAge)*24*time.Hour) {
+			old = "yes"
+		}
+
+		tab.Rows = append(tab.Rows, []interface{}{
+			entry.Name()[:10],
+			fmt.Sprintf("%d days ago", uint(time.Since(entry.ModTime()).Hours()/24)),
+			old,
+		})
+	}
+
+	tab.Write(gopts.stdout)
+
+	return nil
+}

cmd/restic/cmd_cat.go

@@ -58,7 +58,7 @@ func runCat(gopts GlobalOptions, args []string) error {
 			// find snapshot id with prefix
 			id, err = restic.FindSnapshot(repo, args[1])
 			if err != nil {
-				return err
+				return errors.Fatalf("could not find snapshot: %v\n", err)
 			}
 		}
 	}

cmd/restic/cmd_check.go

@@ -2,13 +2,17 @@ package main
 
 import (
 	"fmt"
+	"io/ioutil"
 	"os"
+	"strconv"
+	"strings"
 	"time"
 
 	"github.com/spf13/cobra"
 
 	"github.com/restic/restic/internal/checker"
 	"github.com/restic/restic/internal/errors"
+	"github.com/restic/restic/internal/fs"
 	"github.com/restic/restic/internal/restic"
 )
 
@@ -26,13 +30,17 @@ repository and not use a local cache.
 	RunE: func(cmd *cobra.Command, args []string) error {
 		return runCheck(checkOptions, globalOptions, args)
 	},
+	PreRunE: func(cmd *cobra.Command, args []string) error {
+		return checkFlags(checkOptions)
+	},
 }
 
 // CheckOptions bundles all options for the 'check' command.
 type CheckOptions struct {
-	ReadData    bool
-	CheckUnused bool
-	WithCache   bool
+	ReadData       bool
+	ReadDataSubset string
+	CheckUnused    bool
+	WithCache      bool
 }
 
 var checkOptions CheckOptions
@@ -42,10 +50,45 @@ func init() {
 
 	f := cmdCheck.Flags()
 	f.BoolVar(&checkOptions.ReadData, "read-data", false, "read all data blobs")
+	f.StringVar(&checkOptions.ReadDataSubset, "read-data-subset", "", "read subset of data packs")
 	f.BoolVar(&checkOptions.CheckUnused, "check-unused", false, "find unused blobs")
 	f.BoolVar(&checkOptions.WithCache, "with-cache", false, "use the cache")
 }
 
+func checkFlags(opts CheckOptions) error {
+	if opts.ReadData && opts.ReadDataSubset != "" {
+		return errors.Fatalf("check flags --read-data and --read-data-subset cannot be used together")
+	}
+	if opts.ReadDataSubset != "" {
+		dataSubset, err := stringToIntSlice(opts.ReadDataSubset)
+		if err != nil || len(dataSubset) != 2 {
+			return errors.Fatalf("check flag --read-data-subset must have two positive integer values, e.g. --read-data-subset=1/2")
+		}
+		if dataSubset[0] == 0 || dataSubset[1] == 0 || dataSubset[0] > dataSubset[1] {
+			return errors.Fatalf("check flag --read-data-subset=n/t values must be positive integers, and n <= t, e.g. --read-data-subset=1/2")
+		}
+	}
+
+	return nil
+}
+
+// stringToIntSlice converts string to []uint, using '/' as element separator
+func stringToIntSlice(param string) (split []uint, err error) {
+	if param == "" {
+		return nil, nil
+	}
+	parts := strings.Split(param, "/")
+	result := make([]uint, len(parts))
+	for idx, part := range parts {
+		uintval, err := strconv.ParseUint(part, 10, 0)
+		if err != nil {
+			return nil, err
+		}
+		result[idx] = uint(uintval)
+	}
+	return result, nil
+}
+
 func newReadProgress(gopts GlobalOptions, todo restic.Stat) *restic.Progress {
 	if gopts.Quiet {
 		return nil
@@ -76,15 +119,55 @@ func newReadProgress(gopts GlobalOptions, todo restic.Stat) *restic.Progress {
 	return readProgress
 }
 
+// prepareCheckCache configures a special cache directory for check.
+//
+//  * if --with-cache is specified, the default cache is used
+//  * if the user explicitly requested --no-cache, we don't use any cache
+//  * by default, we use a cache in a temporary directory that is deleted after the check
+func prepareCheckCache(opts CheckOptions, gopts *GlobalOptions) (cleanup func()) {
+	cleanup = func() {}
+	if opts.WithCache {
+		// use the default cache, no setup needed
+		return cleanup
+	}
+
+	if gopts.NoCache {
+		// don't use any cache, no setup needed
+		return cleanup
+	}
+
+	// use a cache in a temporary directory
+	tempdir, err := ioutil.TempDir("", "restic-check-cache-")
+	if err != nil {
+		// if an error occurs, don't use any cache
+		Warnf("unable to create temporary directory for cache during check, disabling cache: %v\n", err)
+		gopts.NoCache = true
+		return cleanup
+	}
+
+	gopts.CacheDir = tempdir
+	Verbosef("using temporary cache in %v\n", tempdir)
+
+	cleanup = func() {
+		err := fs.RemoveAll(tempdir)
+		if err != nil {
+			Warnf("error removing temporary cache directory: %v\n", err)
+		}
+	}
+
+	return cleanup
+}
+
 func runCheck(opts CheckOptions, gopts GlobalOptions, args []string) error {
 	if len(args) != 0 {
 		return errors.Fatal("check has no arguments")
 	}
 
-	if !opts.WithCache {
-		// do not use a cache for the checker
-		gopts.NoCache = true
-	}
+	cleanup := prepareCheckCache(opts, &gopts)
+	AddCleanupHandler(func() error {
+		cleanup()
+		return nil
+	})
 
 	repo, err := OpenRepository(gopts)
 	if err != nil {
@@ -114,7 +197,7 @@ func runCheck(opts CheckOptions, gopts GlobalOptions, args []string) error {
 	}
 
 	if dupFound {
-		Printf("\nrun `restic rebuild-index' to correct this\n")
+		Printf("This is non-critical, you can run `restic rebuild-index' to correct this\n")
 	}
 
 	if len(errs) > 0 {
@@ -125,16 +208,26 @@ func runCheck(opts CheckOptions, gopts GlobalOptions, args []string) error {
 	}
 
 	errorsFound := false
+	orphanedPacks := 0
 	errChan := make(chan error)
 
 	Verbosef("check all packs\n")
 	go chkr.Packs(gopts.ctx, errChan)
 
 	for err := range errChan {
+		if checker.IsOrphanedPack(err) {
+			orphanedPacks++
+			Verbosef("%v\n", err)
+			continue
+		}
 		errorsFound = true
 		fmt.Fprintf(os.Stderr, "%v\n", err)
 	}
 
+	if orphanedPacks > 0 {
+		Verbosef("%d additional files were found in the repo, which likely contain duplicate data.\nYou can run `restic prune` to correct this.\n", orphanedPacks)
+	}
+
 	Verbosef("check snapshots, trees and blobs\n")
 	errChan = make(chan error)
 	go chkr.Structure(gopts.ctx, errChan)
@@ -158,13 +251,25 @@ func runCheck(opts CheckOptions, gopts GlobalOptions, args []string) error {
 		}
 	}
 
-	if opts.ReadData {
-		Verbosef("read all data\n")
+	doReadData := func(bucket, totalBuckets uint) {
+		packs := restic.IDSet{}
+		for pack := range chkr.GetPacks() {
+			if (uint(pack[0]) % totalBuckets) == (bucket - 1) {
+				packs.Insert(pack)
+			}
+		}
+		packCount := uint64(len(packs))
 
-		p := newReadProgress(gopts, restic.Stat{Blobs: chkr.CountPacks()})
+		if packCount < chkr.CountPacks() {
+			Verbosef(fmt.Sprintf("read group #%d of %d data packs (out of total %d packs in %d groups)\n", bucket, packCount, chkr.CountPacks(), totalBuckets))
+		} else {
+			Verbosef("read all data\n")
+		}
+
+		p := newReadProgress(gopts, restic.Stat{Blobs: packCount})
 		errChan := make(chan error)
 
-		go chkr.ReadData(gopts.ctx, p, errChan)
+		go chkr.ReadPacks(gopts.ctx, packs, p, errChan)
 
 		for err := range errChan {
 			errorsFound = true
@@ -172,6 +277,14 @@ func runCheck(opts CheckOptions, gopts GlobalOptions, args []string) error {
 		}
 	}
 
+	switch {
+	case opts.ReadData:
+		doReadData(1, 1)
+	case opts.ReadDataSubset != "":
+		dataSubset, _ := stringToIntSlice(opts.ReadDataSubset)
+		doReadData(dataSubset[0], dataSubset[1])
+	}
+
 	if errorsFound {
 		return errors.Fatal("repository contains errors")
 	}

cmd/restic/cmd_find.go

@@ -3,7 +3,6 @@ package main
 import (
 	"context"
 	"encoding/json"
-	"path/filepath"
 	"strings"
 	"time"
 
@@ -11,7 +10,9 @@ import (
 
 	"github.com/restic/restic/internal/debug"
 	"github.com/restic/restic/internal/errors"
+	"github.com/restic/restic/internal/filter"
 	"github.com/restic/restic/internal/restic"
+	"github.com/restic/restic/internal/walker"
 )
 
 var cmdFind = &cobra.Command{
@@ -94,7 +95,7 @@ type statefulOutput struct {
 	hits     int
 }
 
-func (s *statefulOutput) PrintJSON(prefix string, node *restic.Node) {
+func (s *statefulOutput) PrintJSON(path string, node *restic.Node) {
 	type findNode restic.Node
 	b, err := json.Marshal(struct {
 		// Add these attributes
@@ -111,7 +112,7 @@ func (s *statefulOutput) PrintJSON(prefix string, node *restic.Node) {
 		Content            byte `json:"content,omitempty"`
 		Subtree            byte `json:"subtree,omitempty"`
 	}{
-		Path:        filepath.Join(prefix, node.Name),
+		Path:        path,
 		Permissions: node.Mode.String(),
 		findNode:    (*findNode)(node),
 	})
@@ -138,22 +139,22 @@ func (s *statefulOutput) PrintJSON(prefix string, node *restic.Node) {
 	s.hits++
 }
 
-func (s *statefulOutput) PrintNormal(prefix string, node *restic.Node) {
+func (s *statefulOutput) PrintNormal(path string, node *restic.Node) {
 	if s.newsn != s.oldsn {
 		if s.oldsn != nil {
 			Verbosef("\n")
 		}
 		s.oldsn = s.newsn
-		Verbosef("Found matching entries in snapshot %s\n", s.oldsn.ID())
+		Verbosef("Found matching entries in snapshot %s\n", s.oldsn.ID().Str())
 	}
-	Printf(formatNode(prefix, node, s.ListLong) + "\n")
+	Printf(formatNode(path, node, s.ListLong) + "\n")
 }
 
-func (s *statefulOutput) Print(prefix string, node *restic.Node) {
+func (s *statefulOutput) Print(path string, node *restic.Node) {
 	if s.JSON {
-		s.PrintJSON(prefix, node)
+		s.PrintJSON(path, node)
 	} else {
-		s.PrintNormal(prefix, node)
+		s.PrintNormal(path, node)
 	}
 }
 
@@ -174,74 +175,75 @@ func (s *statefulOutput) Finish() {
 
 // Finder bundles information needed to find a file or directory.
 type Finder struct {
-	repo     restic.Repository
-	pat      findPattern
-	out      statefulOutput
-	notfound restic.IDSet
+	repo        restic.Repository
+	pat         findPattern
+	out         statefulOutput
+	ignoreTrees restic.IDSet
 }
 
-func (f *Finder) findInTree(ctx context.Context, treeID restic.ID, prefix string) error {
-	if f.notfound.Has(treeID) {
-		debug.Log("%v skipping tree %v, has already been checked", prefix, treeID)
-		return nil
+func (f *Finder) findInSnapshot(ctx context.Context, sn *restic.Snapshot) error {
+	debug.Log("searching in snapshot %s\n  for entries within [%s %s]", sn.ID(), f.pat.oldest, f.pat.newest)
+
+	if sn.Tree == nil {
+		return errors.Errorf("snapshot %v has no tree", sn.ID().Str())
 	}
 
-	debug.Log("%v checking tree %v\n", prefix, treeID)
+	f.out.newsn = sn
+	return walker.Walk(ctx, f.repo, *sn.Tree, f.ignoreTrees, func(nodepath string, node *restic.Node, err error) (bool, error) {
+		if err != nil {
+			return false, err
+		}
 
-	tree, err := f.repo.LoadTree(ctx, treeID)
-	if err != nil {
-		return err
-	}
-
-	var found bool
-	for _, node := range tree.Nodes {
-		debug.Log("  testing entry %q\n", node.Name)
+		if node == nil {
+			return false, nil
+		}
 
 		name := node.Name
 		if f.pat.ignoreCase {
 			name = strings.ToLower(name)
 		}
 
-		m, err := filepath.Match(f.pat.pattern, name)
+		foundMatch, err := filter.Match(f.pat.pattern, nodepath)
 		if err != nil {
-			return err
-		}
-
-		if m {
-			if !f.pat.oldest.IsZero() && node.ModTime.Before(f.pat.oldest) {
-				debug.Log("    ModTime is older than %s\n", f.pat.oldest)
-				continue
-			}
-
-			if !f.pat.newest.IsZero() && node.ModTime.After(f.pat.newest) {
-				debug.Log("    ModTime is newer than %s\n", f.pat.newest)
-				continue
-			}
-
-			debug.Log("    found match\n")
-			found = true
-			f.out.Print(prefix, node)
+			return false, err
 		}
 
+		var (
+			ignoreIfNoMatch = true
+			errIfNoMatch    error
+		)
 		if node.Type == "dir" {
-			if err := f.findInTree(ctx, *node.Subtree, filepath.Join(prefix, node.Name)); err != nil {
-				return err
+			childMayMatch, err := filter.ChildMatch(f.pat.pattern, nodepath)
+			if err != nil {
+				return false, err
+			}
+
+			if !childMayMatch {
+				ignoreIfNoMatch = true
+				errIfNoMatch = walker.SkipNode
+			} else {
+				ignoreIfNoMatch = false
 			}
 		}
-	}
 
-	if !found {
-		f.notfound.Insert(treeID)
-	}
+		if !foundMatch {
+			return ignoreIfNoMatch, errIfNoMatch
+		}
 
-	return nil
-}
+		if !f.pat.oldest.IsZero() && node.ModTime.Before(f.pat.oldest) {
+			debug.Log("    ModTime is older than %s\n", f.pat.oldest)
+			return ignoreIfNoMatch, errIfNoMatch
+		}
 
-func (f *Finder) findInSnapshot(ctx context.Context, sn *restic.Snapshot) error {
-	debug.Log("searching in snapshot %s\n  for entries within [%s %s]", sn.ID(), f.pat.oldest, f.pat.newest)
+		if !f.pat.newest.IsZero() && node.ModTime.After(f.pat.newest) {
+			debug.Log("    ModTime is newer than %s\n", f.pat.newest)
+			return ignoreIfNoMatch, errIfNoMatch
+		}
 
-	f.out.newsn = sn
-	return f.findInTree(ctx, *sn.Tree, string(filepath.Separator))
+		debug.Log("    found match\n")
+		f.out.Print(nodepath, node)
+		return false, nil
+	})
 }
 
 func runFind(opts FindOptions, gopts GlobalOptions, args []string) error {
@@ -289,10 +291,10 @@ func runFind(opts FindOptions, gopts GlobalOptions, args []string) error {
 	defer cancel()
 
 	f := &Finder{
-		repo:     repo,
-		pat:      pat,
-		out:      statefulOutput{ListLong: opts.ListLong, JSON: globalOptions.JSON},
-		notfound: restic.NewIDSet(),
+		repo:        repo,
+		pat:         pat,
+		out:         statefulOutput{ListLong: opts.ListLong, JSON: globalOptions.JSON},
+		ignoreTrees: restic.NewIDSet(),
 	}
 	for sn := range FindFilteredSnapshots(ctx, repo, opts.Host, opts.Tags, opts.Paths, opts.Snapshots) {
 		if err = f.findInSnapshot(ctx, sn); err != nil {

cmd/restic/cmd_forget.go

@@ -33,6 +33,7 @@ type ForgetOptions struct {
 	Weekly   int
 	Monthly  int
 	Yearly   int
+	Within   restic.Duration
 	KeepTags restic.TagLists
 
 	Host    string
@@ -58,6 +59,7 @@ func init() {
 	f.IntVarP(&forgetOptions.Weekly, "keep-weekly", "w", 0, "keep the last `n` weekly snapshots")
 	f.IntVarP(&forgetOptions.Monthly, "keep-monthly", "m", 0, "keep the last `n` monthly snapshots")
 	f.IntVarP(&forgetOptions.Yearly, "keep-yearly", "y", 0, "keep the last `n` yearly snapshots")
+	f.VarP(&forgetOptions.Within, "keep-within", "", "keep snapshots that were created within `duration` before the newest (e.g. 1y5m7d)")
 
 	f.Var(&forgetOptions.KeepTags, "keep-tag", "keep snapshots with this `taglist` (can be specified multiple times)")
 	// Sadly the commonly used shortcut `H` is already used.
@@ -170,6 +172,7 @@ func runForget(opts ForgetOptions, gopts GlobalOptions, args []string) error {
 		Weekly:  opts.Weekly,
 		Monthly: opts.Monthly,
 		Yearly:  opts.Yearly,
+		Within:  opts.Within,
 		Tags:    opts.KeepTags,
 	}
 
@@ -178,6 +181,8 @@ func runForget(opts ForgetOptions, gopts GlobalOptions, args []string) error {
 	}
 
 	if !policy.Empty() {
+		Verbosef("Applying Policy: %v\n", policy)
+
 		for k, snapshotGroup := range snapshotGroups {
 			var key key
 			if json.Unmarshal([]byte(k), &key) != nil {

cmd/restic/cmd_key.go

@@ -3,6 +3,9 @@ package main
 import (
 	"context"
 	"fmt"
+	"io/ioutil"
+	"os"
+	"strings"
 
 	"github.com/restic/restic/internal/errors"
 	"github.com/restic/restic/internal/repository"
@@ -23,8 +26,13 @@ The "key" command manages keys (passwords) for accessing the repository.
 	},
 }
 
+var newPasswordFile string
+
 func init() {
 	cmdRoot.AddCommand(cmdKey)
+
+	flags := cmdKey.Flags()
+	flags.StringVarP(&newPasswordFile, "new-password-file", "", "", "the file from which to load a new password")
 }
 
 func listKeys(ctx context.Context, s *repository.Repository) error {
@@ -64,6 +72,10 @@ func getNewPassword(gopts GlobalOptions) (string, error) {
 		return testKeyNewPassword, nil
 	}
 
+	if newPasswordFile != "" {
+		return loadPasswordFromFile(newPasswordFile)
+	}
+
 	// Since we already have an open repository, temporary remove the password
 	// to prompt the user for the passwd.
 	newopts := gopts
@@ -182,3 +194,11 @@ func runKey(gopts GlobalOptions, args []string) error {
 
 	return nil
 }
+
+func loadPasswordFromFile(pwdFile string) (string, error) {
+	s, err := ioutil.ReadFile(pwdFile)
+	if os.IsNotExist(err) {
+		return "", errors.Fatalf("%s does not exist", pwdFile)
+	}
+	return strings.TrimSpace(string(s)), errors.Wrap(err, "Readfile")
+}

cmd/restic/cmd_list.go

@@ -18,7 +18,7 @@ The "list" command allows listing objects in the repository based on type.
 `,
 	DisableAutoGenTag: true,
 	RunE: func(cmd *cobra.Command, args []string) error {
-		return runList(globalOptions, args)
+		return runList(cmd, globalOptions, args)
 	},
 }
 
@@ -26,9 +26,9 @@ func init() {
 	cmdRoot.AddCommand(cmdList)
 }
 
-func runList(opts GlobalOptions, args []string) error {
+func runList(cmd *cobra.Command, opts GlobalOptions, args []string) error {
 	if len(args) != 1 {
-		return errors.Fatal("type not specified")
+		return errors.Fatal("type not specified, usage: " + cmd.Use)
 	}
 
 	repo, err := OpenRepository(opts)

cmd/restic/cmd_ls.go

@@ -2,13 +2,12 @@ package main
 
 import (
 	"context"
-	"path/filepath"
 
 	"github.com/spf13/cobra"
 
 	"github.com/restic/restic/internal/errors"
-	"github.com/restic/restic/internal/repository"
 	"github.com/restic/restic/internal/restic"
+	"github.com/restic/restic/internal/walker"
 )
 
 var cmdLs = &cobra.Command{
@@ -46,25 +45,6 @@ func init() {
 	flags.StringArrayVar(&lsOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path`, when no snapshot ID is given")
 }
 
-func printTree(ctx context.Context, repo *repository.Repository, id *restic.ID, prefix string) error {
-	tree, err := repo.LoadTree(ctx, *id)
-	if err != nil {
-		return err
-	}
-
-	for _, entry := range tree.Nodes {
-		Printf("%s\n", formatNode(prefix, entry, lsOptions.ListLong))
-
-		if entry.Type == "dir" && entry.Subtree != nil {
-			if err = printTree(ctx, repo, entry.Subtree, filepath.Join(prefix, entry.Name)); err != nil {
-				return err
-			}
-		}
-	}
-
-	return nil
-}
-
 func runLs(opts LsOptions, gopts GlobalOptions, args []string) error {
 	if len(args) == 0 && opts.Host == "" && len(opts.Tags) == 0 && len(opts.Paths) == 0 {
 		return errors.Fatal("Invalid arguments, either give one or more snapshot IDs or set filters.")
@@ -84,7 +64,18 @@ func runLs(opts LsOptions, gopts GlobalOptions, args []string) error {
 	for sn := range FindFilteredSnapshots(ctx, repo, opts.Host, opts.Tags, opts.Paths, args) {
 		Verbosef("snapshot %s of %v at %s):\n", sn.ID().Str(), sn.Paths, sn.Time)
 
-		if err = printTree(gopts.ctx, repo, sn.Tree, string(filepath.Separator)); err != nil {
+		err := walker.Walk(ctx, repo, *sn.Tree, nil, func(nodepath string, node *restic.Node, err error) (bool, error) {
+			if err != nil {
+				return false, err
+			}
+
+			if node == nil {
+				return false, nil
+			}
+			Printf("%s\n", formatNode(nodepath, node, lsOptions.ListLong))
+			return false, nil
+		})
+		if err != nil {
 			return err
 		}
 	}

cmd/restic/cmd_mount.go

@@ -1,4 +1,5 @@
 // +build !openbsd
+// +build !solaris
 // +build !windows
 
 package main

cmd/restic/cmd_options.go

@@ -3,7 +3,7 @@ package main
 import (
 	"fmt"
 
-	"github.com/restic/restic/internal/options"
+	"github.com/restic/restic/internal/ui/options"
 
 	"github.com/spf13/cobra"
 )

cmd/restic/cmd_version.go

@@ -16,7 +16,7 @@ and the version of this software.
 `,
 	DisableAutoGenTag: true,
 	Run: func(cmd *cobra.Command, args []string) {
-		fmt.Printf("restic %s\ncompiled with %v on %v/%v\n",
+		fmt.Printf("restic %s compiled with %v on %v/%v\n",
 			version, runtime.Version(), runtime.GOOS, runtime.GOARCH)
 	},
 }

cmd/restic/exclude.go

@@ -185,6 +185,11 @@ func isDirExcludedByFile(dir, tagFilename, header string) bool {
 func gatherDevices(items []string) (deviceMap map[string]uint64, err error) {
 	deviceMap = make(map[string]uint64)
 	for _, item := range items {
+		item, err = filepath.Abs(filepath.Clean(item))
+		if err != nil {
+			return nil, err
+		}
+
 		fi, err := fs.Lstat(item)
 		if err != nil {
 			return nil, err
@@ -215,6 +220,8 @@ func rejectByDevice(samples []string) (RejectFunc, error) {
 			return false
 		}
 
+		item = filepath.Clean(item)
+
 		id, err := fs.DeviceID(fi)
 		if err != nil {
 			// This should never happen because gatherDevices() would have
@@ -222,11 +229,14 @@ func rejectByDevice(samples []string) (RejectFunc, error) {
 			panic(err)
 		}
 
-		for dir := item; dir != ""; dir = filepath.Dir(dir) {
+		for dir := item; ; dir = filepath.Dir(dir) {
 			debug.Log("item %v, test dir %v", item, dir)
 
 			allowedID, ok := allowed[dir]
 			if !ok {
+				if dir == filepath.Dir(dir) {
+					break
+				}
 				continue
 			}
 

cmd/restic/excludes

@@ -1,31 +0,0 @@
-/boot
-/dev
-/etc
-/home
-/lost+found
-/mnt
-/proc
-/root
-/run
-/sys
-/tmp
-/usr
-/var
-/opt/android-sdk
-/opt/bullet
-/opt/dex2jar
-/opt/jameica
-/opt/google
-/opt/JDownloader
-/opt/JDownloaderScripts
-/opt/opencascade
-/opt/vagrant
-/opt/visual-studio-code
-/opt/vtk6
-/bin
-/fonts*
-/srv/ftp
-/srv/http
-/sbin
-/lib
-/lib64

cmd/restic/format.go

@@ -3,7 +3,6 @@ package main
 import (
 	"fmt"
 	"os"
-	"path/filepath"
 	"time"
 
 	"github.com/restic/restic/internal/restic"
@@ -63,9 +62,9 @@ func formatDuration(d time.Duration) string {
 	return formatSeconds(sec)
 }
 
-func formatNode(prefix string, n *restic.Node, long bool) string {
+func formatNode(path string, n *restic.Node, long bool) string {
 	if !long {
-		return filepath.Join(prefix, n.Name)
+		return path
 	}
 
 	var mode os.FileMode
@@ -91,6 +90,6 @@ func formatNode(prefix string, n *restic.Node, long bool) string {
 
 	return fmt.Sprintf("%s %5d %5d %6d %s %s%s",
 		mode|n.Mode, n.UID, n.GID, n.Size,
-		n.ModTime.Format(TimeFormat), filepath.Join(prefix, n.Name),
+		n.ModTime.Format(TimeFormat), path,
 		target)
 }

cmd/restic/global.go

@@ -4,7 +4,6 @@ import (
 	"context"
 	"fmt"
 	"io"
-	"io/ioutil"
 	"os"
 	"path/filepath"
 	"runtime"
@@ -18,6 +17,7 @@ import (
 	"github.com/restic/restic/internal/backend/gs"
 	"github.com/restic/restic/internal/backend/local"
 	"github.com/restic/restic/internal/backend/location"
+	"github.com/restic/restic/internal/backend/rclone"
 	"github.com/restic/restic/internal/backend/rest"
 	"github.com/restic/restic/internal/backend/s3"
 	"github.com/restic/restic/internal/backend/sftp"
@@ -26,9 +26,11 @@ import (
 	"github.com/restic/restic/internal/debug"
 	"github.com/restic/restic/internal/fs"
 	"github.com/restic/restic/internal/limiter"
-	"github.com/restic/restic/internal/options"
 	"github.com/restic/restic/internal/repository"
 	"github.com/restic/restic/internal/restic"
+	"github.com/restic/restic/internal/textfile"
+	"github.com/restic/restic/internal/ui/config"
+	"github.com/restic/restic/internal/ui/options"
 
 	"github.com/restic/restic/internal/errors"
 
@@ -39,9 +41,10 @@ var version = "compiled manually"
 
 // GlobalOptions hold all global options for restic.
 type GlobalOptions struct {
-	Repo          string
-	PasswordFile  string
+	config.Config
+
 	Quiet         bool
+	Verbose       int
 	NoLock        bool
 	JSON          bool
 	CacheDir      string
@@ -58,6 +61,13 @@ type GlobalOptions struct {
 	stdout   io.Writer
 	stderr   io.Writer
 
+	// verbosity is set as follows:
+	//  0 means: don't print any messages except errors, this is used when --quiet is specified
+	//  1 is the default: print essential messages
+	//  2 means: print more messages, report minor things, this is used when --verbose is specified
+	//  3 means: print very detailed debug messages, this is used when --debug is specified
+	verbosity uint
+
 	Options []string
 
 	extended options.Options
@@ -77,14 +87,18 @@ func init() {
 	})
 
 	f := cmdRoot.PersistentFlags()
-	f.StringVarP(&globalOptions.Repo, "repo", "r", os.Getenv("RESTIC_REPOSITORY"), "repository to backup to or restore from (default: $RESTIC_REPOSITORY)")
-	f.StringVarP(&globalOptions.PasswordFile, "password-file", "p", os.Getenv("RESTIC_PASSWORD_FILE"), "read the repository password from a file (default: $RESTIC_PASSWORD_FILE)")
+
+	// these fields are embedded in config.Config and queried via f.Get[...]()
+	f.StringP("repo", "r", "", "repository to backup to or restore from (default: $RESTIC_REPOSITORY)")
+	f.StringP("password-file", "p", "", "read the repository password from a file (default: $RESTIC_PASSWORD_FILE)")
+
 	f.BoolVarP(&globalOptions.Quiet, "quiet", "q", false, "do not output comprehensive progress report")
+	f.CountVarP(&globalOptions.Verbose, "verbose", "v", "be verbose (specify --verbose multiple times or level `n`)")
 	f.BoolVar(&globalOptions.NoLock, "no-lock", false, "do not lock the repo, this allows some operations on read-only repos")
 	f.BoolVarP(&globalOptions.JSON, "json", "", false, "set output mode to JSON for commands that support it")
 	f.StringVar(&globalOptions.CacheDir, "cache-dir", "", "set the cache directory")
 	f.BoolVar(&globalOptions.NoCache, "no-cache", false, "do not use a local cache")
-	f.StringSliceVar(&globalOptions.CACerts, "cacert", nil, "path to load root certificates from (default: use system certificates)")
+	f.StringSliceVar(&globalOptions.CACerts, "cacert", nil, "`file` to load root certificates from (default: use system certificates)")
 	f.StringVar(&globalOptions.TLSClientCert, "tls-client-cert", "", "path to a file containing PEM encoded TLS client certificate and private key")
 	f.BoolVar(&globalOptions.CleanupCache, "cleanup-cache", false, "auto remove old cache directories")
 	f.IntVar(&globalOptions.LimitUploadKb, "limit-upload", 0, "limits uploads to a maximum rate in KiB/s. (default: unlimited)")
@@ -172,11 +186,9 @@ func Printf(format string, args ...interface{}) {
 
 // Verbosef calls Printf to write the message when the verbose flag is set.
 func Verbosef(format string, args ...interface{}) {
-	if globalOptions.Quiet {
-		return
+	if globalOptions.verbosity >= 1 {
+		Printf(format, args...)
 	}
-
-	Printf(format, args...)
 }
 
 // PrintProgress wraps fmt.Printf to handle the difference in writing progress
@@ -225,19 +237,19 @@ func Exitf(exitcode int, format string, args ...interface{}) {
 }
 
 // resolvePassword determines the password to be used for opening the repository.
-func resolvePassword(opts GlobalOptions, env string) (string, error) {
+func resolvePassword(opts GlobalOptions) (string, error) {
+	if opts.Password != "" {
+		return opts.Password, nil
+	}
+
 	if opts.PasswordFile != "" {
-		s, err := ioutil.ReadFile(opts.PasswordFile)
-		if os.IsNotExist(err) {
+		s, err := textfile.Read(opts.PasswordFile)
+		if os.IsNotExist(errors.Cause(err)) {
 			return "", errors.Fatalf("%s does not exist", opts.PasswordFile)
 		}
 		return strings.TrimSpace(string(s)), errors.Wrap(err, "Readfile")
 	}
 
-	if pwd := os.Getenv(env); pwd != "" {
-		return pwd, nil
-	}
-
 	return "", nil
 }
 
@@ -347,7 +359,11 @@ func OpenRepository(opts GlobalOptions) (*repository.Repository, error) {
 	}
 
 	if stdoutIsTerminal() {
-		Verbosef("password is correct\n")
+		id := s.Config().ID
+		if len(id) > 8 {
+			id = id[:8]
+		}
+		Verbosef("repository %v opened successfully, password is correct\n", id)
 	}
 
 	if opts.NoCache {
@@ -378,7 +394,7 @@ func OpenRepository(opts GlobalOptions) (*repository.Repository, error) {
 		Printf("removing %d old cache dirs from %v\n", len(oldCacheDirs), c.Base)
 
 		for _, item := range oldCacheDirs {
-			dir := filepath.Join(c.Base, item)
+			dir := filepath.Join(c.Base, item.Name())
 			err = fs.RemoveAll(dir)
 			if err != nil {
 				Warnf("unable to remove %v: %v\n", dir, err)
@@ -440,18 +456,6 @@ func parseConfig(loc location.Location, opts options.Options) (interface{}, erro
 			cfg.ProjectID = os.Getenv("GOOGLE_PROJECT_ID")
 		}
 
-		if cfg.JSONKeyPath == "" {
-			if path := os.Getenv("GOOGLE_APPLICATION_CREDENTIALS"); path != "" {
-				// Check read access
-				if _, err := ioutil.ReadFile(path); err != nil {
-					return nil, errors.Fatalf("Failed to read google credential from file %v: %v", path, err)
-				}
-				cfg.JSONKeyPath = path
-			} else {
-				return nil, errors.Fatal("No credential file path is set")
-			}
-		}
-
 		if err := opts.Apply(loc.Scheme, &cfg); err != nil {
 			return nil, err
 		}
@@ -521,6 +525,14 @@ func parseConfig(loc location.Location, opts options.Options) (interface{}, erro
 			return nil, err
 		}
 
+		debug.Log("opening rest repository at %#v", cfg)
+		return cfg, nil
+	case "rclone":
+		cfg := loc.Config.(rclone.Config)
+		if err := opts.Apply(loc.Scheme, &cfg); err != nil {
+			return nil, err
+		}
+
 		debug.Log("opening rest repository at %#v", cfg)
 		return cfg, nil
 	}
@@ -553,17 +565,18 @@ func open(s string, gopts GlobalOptions, opts options.Options) (restic.Backend,
 	}
 
 	// wrap the transport so that the throughput via HTTP is limited
-	rt = limiter.NewStaticLimiter(gopts.LimitUploadKb, gopts.LimitDownloadKb).Transport(rt)
+	lim := limiter.NewStaticLimiter(gopts.LimitUploadKb, gopts.LimitDownloadKb)
+	rt = lim.Transport(rt)
 
 	switch loc.Scheme {
 	case "local":
 		be, err = local.Open(cfg.(local.Config))
 		// wrap the backend in a LimitBackend so that the throughput is limited
-		be = limiter.LimitBackend(be, limiter.NewStaticLimiter(gopts.LimitUploadKb, gopts.LimitDownloadKb))
+		be = limiter.LimitBackend(be, lim)
 	case "sftp":
 		be, err = sftp.Open(cfg.(sftp.Config))
 		// wrap the backend in a LimitBackend so that the throughput is limited
-		be = limiter.LimitBackend(be, limiter.NewStaticLimiter(gopts.LimitUploadKb, gopts.LimitDownloadKb))
+		be = limiter.LimitBackend(be, lim)
 	case "s3":
 		be, err = s3.Open(cfg.(s3.Config), rt)
 	case "gs":
@@ -576,6 +589,8 @@ func open(s string, gopts GlobalOptions, opts options.Options) (restic.Backend,
 		be, err = b2.Open(globalOptions.ctx, cfg.(b2.Config), rt)
 	case "rest":
 		be, err = rest.Open(cfg.(rest.Config), rt)
+	case "rclone":
+		be, err = rclone.Open(cfg.(rclone.Config), lim)
 
 	default:
 		return nil, errors.Fatalf("invalid backend: %q", loc.Scheme)
@@ -637,6 +652,8 @@ func create(s string, opts options.Options) (restic.Backend, error) {
 		return b2.Create(globalOptions.ctx, cfg.(b2.Config), rt)
 	case "rest":
 		return rest.Create(cfg.(rest.Config), rt)
+	case "rclone":
+		return rclone.Open(cfg.(rclone.Config), nil)
 	}
 
 	debug.Log("invalid repository scheme: %v", s)

cmd/restic/global_debug.go

@@ -1,4 +1,4 @@
-// +build debug
+// +build debug profile
 
 package main
 
@@ -15,17 +15,21 @@ import (
 )
 
 var (
-	listenMemoryProfile string
-	memProfilePath      string
-	cpuProfilePath      string
-	insecure            bool
+	listenProfile    string
+	memProfilePath   string
+	cpuProfilePath   string
+	traceProfilePath string
+	blockProfilePath string
+	insecure         bool
 )
 
 func init() {
 	f := cmdRoot.PersistentFlags()
-	f.StringVar(&listenMemoryProfile, "listen-profile", "", "listen on this `address:port` for memory profiling")
+	f.StringVar(&listenProfile, "listen-profile", "", "listen on this `address:port` for memory profiling")
 	f.StringVar(&memProfilePath, "mem-profile", "", "write memory profile to `dir`")
 	f.StringVar(&cpuProfilePath, "cpu-profile", "", "write cpu profile to `dir`")
+	f.StringVar(&traceProfilePath, "trace-profile", "", "write trace to `dir`")
+	f.StringVar(&blockProfilePath, "block-profile", "", "write block profile to `dir`")
 	f.BoolVar(&insecure, "insecure-kdf", false, "use insecure KDF settings")
 }
 
@@ -36,18 +40,32 @@ func (fakeTestingTB) Logf(msg string, args ...interface{}) {
 }
 
 func runDebug() error {
-	if listenMemoryProfile != "" {
-		fmt.Fprintf(os.Stderr, "running memory profile HTTP server on %v\n", listenMemoryProfile)
+	if listenProfile != "" {
+		fmt.Fprintf(os.Stderr, "running profile HTTP server on %v\n", listenProfile)
 		go func() {
-			err := http.ListenAndServe(listenMemoryProfile, nil)
+			err := http.ListenAndServe(listenProfile, nil)
 			if err != nil {
-				fmt.Fprintf(os.Stderr, "memory profile listen failed: %v\n", err)
+				fmt.Fprintf(os.Stderr, "profile HTTP server listen failed: %v\n", err)
 			}
 		}()
 	}
 
-	if memProfilePath != "" && cpuProfilePath != "" {
-		return errors.Fatal("only one profile (memory or CPU) may be activated at the same time")
+	profilesEnabled := 0
+	if memProfilePath != "" {
+		profilesEnabled++
+	}
+	if cpuProfilePath != "" {
+		profilesEnabled++
+	}
+	if traceProfilePath != "" {
+		profilesEnabled++
+	}
+	if blockProfilePath != "" {
+		profilesEnabled++
+	}
+
+	if profilesEnabled > 1 {
+		return errors.Fatal("only one profile (memory, CPU, trace, or block) may be activated at the same time")
 	}
 
 	var prof interface {
@@ -58,6 +76,10 @@ func runDebug() error {
 		prof = profile.Start(profile.Quiet, profile.NoShutdownHook, profile.MemProfile, profile.ProfilePath(memProfilePath))
 	} else if cpuProfilePath != "" {
 		prof = profile.Start(profile.Quiet, profile.NoShutdownHook, profile.CPUProfile, profile.ProfilePath(cpuProfilePath))
+	} else if traceProfilePath != "" {
+		prof = profile.Start(profile.Quiet, profile.NoShutdownHook, profile.TraceProfile, profile.ProfilePath(traceProfilePath))
+	} else if blockProfilePath != "" {
+		prof = profile.Start(profile.Quiet, profile.NoShutdownHook, profile.BlockProfile, profile.ProfilePath(blockProfilePath))
 	}
 
 	if prof != nil {

cmd/restic/global_release.go

@@ -1,4 +1,4 @@
-// +build !debug
+// +build !debug,!profile
 
 package main
 

cmd/restic/integration_fuse_test.go

@@ -1,4 +1,5 @@
 // +build !openbsd
+// +build !solaris
 // +build !windows
 
 package main
@@ -170,7 +171,7 @@ func TestMount(t *testing.T) {
 	rtest.SetupTarTestFixture(t, env.testdata, filepath.Join("testdata", "backup-data.tar.gz"))
 
 	// first backup
-	testRunBackup(t, []string{env.testdata}, BackupOptions{}, env.gopts)
+	testRunBackup(t, "", []string{env.testdata}, BackupOptions{}, env.gopts)
 	snapshotIDs := testRunList(t, "snapshots", env.gopts)
 	rtest.Assert(t, len(snapshotIDs) == 1,
 		"expected one snapshot, got %v", snapshotIDs)
@@ -178,7 +179,7 @@ func TestMount(t *testing.T) {
 	checkSnapshots(t, env.gopts, repo, env.mountpoint, env.repo, snapshotIDs, 2)
 
 	// second backup, implicit incremental
-	testRunBackup(t, []string{env.testdata}, BackupOptions{}, env.gopts)
+	testRunBackup(t, "", []string{env.testdata}, BackupOptions{}, env.gopts)
 	snapshotIDs = testRunList(t, "snapshots", env.gopts)
 	rtest.Assert(t, len(snapshotIDs) == 2,
 		"expected two snapshots, got %v", snapshotIDs)
@@ -187,7 +188,7 @@ func TestMount(t *testing.T) {
 
 	// third backup, explicit incremental
 	bopts := BackupOptions{Parent: snapshotIDs[0].String()}
-	testRunBackup(t, []string{env.testdata}, bopts, env.gopts)
+	testRunBackup(t, "", []string{env.testdata}, bopts, env.gopts)
 	snapshotIDs = testRunList(t, "snapshots", env.gopts)
 	rtest.Assert(t, len(snapshotIDs) == 3,
 		"expected three snapshots, got %v", snapshotIDs)

cmd/restic/integration_helpers_test.go

@@ -9,9 +9,11 @@ import (
 	"runtime"
 	"testing"
 
-	"github.com/restic/restic/internal/options"
 	"github.com/restic/restic/internal/repository"
+	"github.com/restic/restic/internal/restic"
 	rtest "github.com/restic/restic/internal/test"
+	"github.com/restic/restic/internal/ui/config"
+	"github.com/restic/restic/internal/ui/options"
 )
 
 type dirEntry struct {
@@ -189,6 +191,7 @@ func withTestEnvironment(t testing.TB) (env *testEnvironment, cleanup func()) {
 	}
 
 	repository.TestUseLowSecurityKDFParameters(t)
+	restic.TestDisableCheckPolynomial(t)
 
 	tempdir, err := ioutil.TempDir(rtest.TestTempDir, "restic-test-")
 	rtest.OK(t, err)
@@ -207,7 +210,9 @@ func withTestEnvironment(t testing.TB) (env *testEnvironment, cleanup func()) {
 	rtest.OK(t, os.MkdirAll(env.repo, 0700))
 
 	env.gopts = GlobalOptions{
-		Repo:     env.repo,
+		Config: config.Config{
+			Repo: env.repo,
+		},
 		Quiet:    true,
 		CacheDir: env.cache,
 		ctx:      context.Background(),

cmd/restic/integration_test.go

@@ -3,6 +3,7 @@ package main
 import (
 	"bufio"
 	"bytes"
+	"context"
 	"crypto/rand"
 	"encoding/json"
 	"fmt"
@@ -17,12 +18,14 @@ import (
 	"testing"
 	"time"
 
-	"github.com/restic/restic/internal/debug"
 	"github.com/restic/restic/internal/errors"
 	"github.com/restic/restic/internal/filter"
+	"github.com/restic/restic/internal/fs"
 	"github.com/restic/restic/internal/repository"
 	"github.com/restic/restic/internal/restic"
 	rtest "github.com/restic/restic/internal/test"
+	"github.com/restic/restic/internal/ui/termstatus"
+	"golang.org/x/sync/errgroup"
 )
 
 func parseIDsFromReader(t testing.TB, rd io.Reader) restic.IDs {
@@ -44,15 +47,36 @@ func parseIDsFromReader(t testing.TB, rd io.Reader) restic.IDs {
 
 func testRunInit(t testing.TB, opts GlobalOptions) {
 	repository.TestUseLowSecurityKDFParameters(t)
+	restic.TestDisableCheckPolynomial(t)
 	restic.TestSetLockTimeout(t, 0)
 
 	rtest.OK(t, runInit(opts, nil))
 	t.Logf("repository initialized at %v", opts.Repo)
 }
 
-func testRunBackup(t testing.TB, target []string, opts BackupOptions, gopts GlobalOptions) {
-	t.Logf("backing up %v", target)
-	rtest.OK(t, runBackup(opts, gopts, target))
+func testRunBackup(t testing.TB, dir string, target []string, opts BackupOptions, gopts GlobalOptions) {
+	ctx, cancel := context.WithCancel(gopts.ctx)
+	defer cancel()
+
+	var wg errgroup.Group
+	term := termstatus.New(gopts.stdout, gopts.stderr, gopts.Quiet)
+	wg.Go(func() error { term.Run(ctx); return nil })
+
+	gopts.stdout = ioutil.Discard
+	t.Logf("backing up %v in %v", target, dir)
+	if dir != "" {
+		cleanup := fs.TestChdir(t, dir)
+		defer cleanup()
+	}
+
+	rtest.OK(t, runBackup(opts, gopts, term, target))
+
+	cancel()
+
+	err := wg.Wait()
+	if err != nil {
+		t.Fatal(err)
+	}
 }
 
 func testRunList(t testing.TB, tpe string, opts GlobalOptions) restic.IDs {
@@ -62,7 +86,7 @@ func testRunList(t testing.TB, tpe string, opts GlobalOptions) restic.IDs {
 		globalOptions.stdout = os.Stdout
 	}()
 
-	rtest.OK(t, runList(opts, []string{tpe}))
+	rtest.OK(t, runList(cmdList, opts, []string{tpe}))
 	return parseIDsFromReader(t, buf)
 }
 
@@ -218,7 +242,7 @@ func TestBackup(t *testing.T) {
 	opts := BackupOptions{}
 
 	// first backup
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	testRunBackup(t, filepath.Dir(env.testdata), []string{"testdata"}, opts, env.gopts)
 	snapshotIDs := testRunList(t, "snapshots", env.gopts)
 	rtest.Assert(t, len(snapshotIDs) == 1,
 		"expected one snapshot, got %v", snapshotIDs)
@@ -227,7 +251,7 @@ func TestBackup(t *testing.T) {
 	stat1 := dirStats(env.repo)
 
 	// second backup, implicit incremental
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	testRunBackup(t, filepath.Dir(env.testdata), []string{"testdata"}, opts, env.gopts)
 	snapshotIDs = testRunList(t, "snapshots", env.gopts)
 	rtest.Assert(t, len(snapshotIDs) == 2,
 		"expected two snapshots, got %v", snapshotIDs)
@@ -241,7 +265,7 @@ func TestBackup(t *testing.T) {
 	testRunCheck(t, env.gopts)
 	// third backup, explicit incremental
 	opts.Parent = snapshotIDs[0].String()
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	testRunBackup(t, filepath.Dir(env.testdata), []string{"testdata"}, opts, env.gopts)
 	snapshotIDs = testRunList(t, "snapshots", env.gopts)
 	rtest.Assert(t, len(snapshotIDs) == 3,
 		"expected three snapshots, got %v", snapshotIDs)
@@ -285,7 +309,7 @@ func TestBackupNonExistingFile(t *testing.T) {
 		globalOptions.stderr = os.Stderr
 	}()
 
-	p := filepath.Join(env.testdata, "0", "0")
+	p := filepath.Join(env.testdata, "0", "0", "9")
 	dirs := []string{
 		filepath.Join(p, "0"),
 		filepath.Join(p, "1"),
@@ -295,198 +319,7 @@ func TestBackupNonExistingFile(t *testing.T) {
 
 	opts := BackupOptions{}
 
-	testRunBackup(t, dirs, opts, env.gopts)
-}
-
-func TestBackupMissingFile1(t *testing.T) {
-	env, cleanup := withTestEnvironment(t)
-	defer cleanup()
-
-	datafile := filepath.Join("testdata", "backup-data.tar.gz")
-	fd, err := os.Open(datafile)
-	if os.IsNotExist(errors.Cause(err)) {
-		t.Skipf("unable to find data file %q, skipping", datafile)
-		return
-	}
-	rtest.OK(t, err)
-	rtest.OK(t, fd.Close())
-
-	rtest.SetupTarTestFixture(t, env.testdata, datafile)
-
-	testRunInit(t, env.gopts)
-	globalOptions.stderr = ioutil.Discard
-	defer func() {
-		globalOptions.stderr = os.Stderr
-	}()
-
-	ranHook := false
-	debug.Hook("pipe.walk1", func(context interface{}) {
-		pathname := context.(string)
-
-		if pathname != filepath.Join("testdata", "0", "0", "9") {
-			return
-		}
-
-		t.Logf("in hook, removing test file testdata/0/0/9/37")
-		ranHook = true
-
-		rtest.OK(t, os.Remove(filepath.Join(env.testdata, "0", "0", "9", "37")))
-	})
-
-	opts := BackupOptions{}
-
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
-	testRunCheck(t, env.gopts)
-
-	rtest.Assert(t, ranHook, "hook did not run")
-	debug.RemoveHook("pipe.walk1")
-}
-
-func TestBackupMissingFile2(t *testing.T) {
-	env, cleanup := withTestEnvironment(t)
-	defer cleanup()
-
-	datafile := filepath.Join("testdata", "backup-data.tar.gz")
-	fd, err := os.Open(datafile)
-	if os.IsNotExist(errors.Cause(err)) {
-		t.Skipf("unable to find data file %q, skipping", datafile)
-		return
-	}
-	rtest.OK(t, err)
-	rtest.OK(t, fd.Close())
-
-	rtest.SetupTarTestFixture(t, env.testdata, datafile)
-
-	testRunInit(t, env.gopts)
-
-	globalOptions.stderr = ioutil.Discard
-	defer func() {
-		globalOptions.stderr = os.Stderr
-	}()
-
-	ranHook := false
-	debug.Hook("pipe.walk2", func(context interface{}) {
-		pathname := context.(string)
-
-		if pathname != filepath.Join("testdata", "0", "0", "9", "37") {
-			return
-		}
-
-		t.Logf("in hook, removing test file testdata/0/0/9/37")
-		ranHook = true
-
-		rtest.OK(t, os.Remove(filepath.Join(env.testdata, "0", "0", "9", "37")))
-	})
-
-	opts := BackupOptions{}
-
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
-	testRunCheck(t, env.gopts)
-
-	rtest.Assert(t, ranHook, "hook did not run")
-	debug.RemoveHook("pipe.walk2")
-}
-
-func TestBackupChangedFile(t *testing.T) {
-	env, cleanup := withTestEnvironment(t)
-	defer cleanup()
-
-	datafile := filepath.Join("testdata", "backup-data.tar.gz")
-	fd, err := os.Open(datafile)
-	if os.IsNotExist(errors.Cause(err)) {
-		t.Skipf("unable to find data file %q, skipping", datafile)
-		return
-	}
-	rtest.OK(t, err)
-	rtest.OK(t, fd.Close())
-
-	rtest.SetupTarTestFixture(t, env.testdata, datafile)
-
-	testRunInit(t, env.gopts)
-
-	globalOptions.stderr = ioutil.Discard
-	defer func() {
-		globalOptions.stderr = os.Stderr
-	}()
-
-	modFile := filepath.Join(env.testdata, "0", "0", "6", "18")
-
-	ranHook := false
-	debug.Hook("archiver.SaveFile", func(context interface{}) {
-		pathname := context.(string)
-
-		if pathname != modFile {
-			return
-		}
-
-		t.Logf("in hook, modifying test file %v", modFile)
-		ranHook = true
-
-		rtest.OK(t, ioutil.WriteFile(modFile, []byte("modified"), 0600))
-	})
-
-	opts := BackupOptions{}
-
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
-	testRunCheck(t, env.gopts)
-
-	rtest.Assert(t, ranHook, "hook did not run")
-	debug.RemoveHook("archiver.SaveFile")
-}
-
-func TestBackupDirectoryError(t *testing.T) {
-	env, cleanup := withTestEnvironment(t)
-	defer cleanup()
-
-	datafile := filepath.Join("testdata", "backup-data.tar.gz")
-	fd, err := os.Open(datafile)
-	if os.IsNotExist(errors.Cause(err)) {
-		t.Skipf("unable to find data file %q, skipping", datafile)
-		return
-	}
-	rtest.OK(t, err)
-	rtest.OK(t, fd.Close())
-
-	rtest.SetupTarTestFixture(t, env.testdata, datafile)
-
-	testRunInit(t, env.gopts)
-
-	globalOptions.stderr = ioutil.Discard
-	defer func() {
-		globalOptions.stderr = os.Stderr
-	}()
-
-	ranHook := false
-
-	testdir := filepath.Join(env.testdata, "0", "0", "9")
-
-	// install hook that removes the dir right before readdirnames()
-	debug.Hook("pipe.readdirnames", func(context interface{}) {
-		path := context.(string)
-
-		if path != testdir {
-			return
-		}
-
-		t.Logf("in hook, removing test file %v", testdir)
-		ranHook = true
-
-		rtest.OK(t, os.RemoveAll(testdir))
-	})
-
-	testRunBackup(t, []string{filepath.Join(env.testdata, "0", "0")}, BackupOptions{}, env.gopts)
-	testRunCheck(t, env.gopts)
-
-	rtest.Assert(t, ranHook, "hook did not run")
-	debug.RemoveHook("pipe.walk2")
-
-	snapshots := testRunList(t, "snapshots", env.gopts)
-	rtest.Assert(t, len(snapshots) > 0,
-		"no snapshots found in repo (%v)", datafile)
-
-	files := testRunLs(t, env.gopts, snapshots[0].String())
-
-	rtest.Assert(t, len(files) > 1, "snapshot is empty")
+	testRunBackup(t, "", dirs, opts, env.gopts)
 }
 
 func includes(haystack []string, needle string) bool {
@@ -551,33 +384,33 @@ func TestBackupExclude(t *testing.T) {
 
 	opts := BackupOptions{}
 
-	testRunBackup(t, []string{datadir}, opts, env.gopts)
+	testRunBackup(t, filepath.Dir(env.testdata), []string{"testdata"}, opts, env.gopts)
 	snapshots, snapshotID := lastSnapshot(snapshots, loadSnapshotMap(t, env.gopts))
 	files := testRunLs(t, env.gopts, snapshotID)
-	rtest.Assert(t, includes(files, filepath.Join(string(filepath.Separator), "testdata", "foo.tar.gz")),
+	rtest.Assert(t, includes(files, "/testdata/foo.tar.gz"),
 		"expected file %q in first snapshot, but it's not included", "foo.tar.gz")
 
-	opts.Excludes = []string{"*.tar.gz"}
-	testRunBackup(t, []string{datadir}, opts, env.gopts)
+	opts.Config.Excludes = []string{"*.tar.gz"}
+	testRunBackup(t, filepath.Dir(env.testdata), []string{"testdata"}, opts, env.gopts)
 	snapshots, snapshotID = lastSnapshot(snapshots, loadSnapshotMap(t, env.gopts))
 	files = testRunLs(t, env.gopts, snapshotID)
-	rtest.Assert(t, !includes(files, filepath.Join(string(filepath.Separator), "testdata", "foo.tar.gz")),
+	rtest.Assert(t, !includes(files, "/testdata/foo.tar.gz"),
 		"expected file %q not in first snapshot, but it's included", "foo.tar.gz")
 
-	opts.Excludes = []string{"*.tar.gz", "private/secret"}
-	testRunBackup(t, []string{datadir}, opts, env.gopts)
+	opts.Config.Excludes = []string{"*.tar.gz", "private/secret"}
+	testRunBackup(t, filepath.Dir(env.testdata), []string{"testdata"}, opts, env.gopts)
 	_, snapshotID = lastSnapshot(snapshots, loadSnapshotMap(t, env.gopts))
 	files = testRunLs(t, env.gopts, snapshotID)
-	rtest.Assert(t, !includes(files, filepath.Join(string(filepath.Separator), "testdata", "foo.tar.gz")),
+	rtest.Assert(t, !includes(files, "/testdata/foo.tar.gz"),
 		"expected file %q not in first snapshot, but it's included", "foo.tar.gz")
-	rtest.Assert(t, !includes(files, filepath.Join(string(filepath.Separator), "testdata", "private", "secret", "passwords.txt")),
+	rtest.Assert(t, !includes(files, "/testdata/private/secret/passwords.txt"),
 		"expected file %q not in first snapshot, but it's included", "passwords.txt")
 }
 
 const (
-	incrementalFirstWrite  = 20 * 1042 * 1024
-	incrementalSecondWrite = 12 * 1042 * 1024
-	incrementalThirdWrite  = 4 * 1042 * 1024
+	incrementalFirstWrite  = 10 * 1042 * 1024
+	incrementalSecondWrite = 1 * 1042 * 1024
+	incrementalThirdWrite  = 1 * 1042 * 1024
 )
 
 func appendRandomData(filename string, bytes uint) error {
@@ -615,13 +448,13 @@ func TestIncrementalBackup(t *testing.T) {
 
 	opts := BackupOptions{}
 
-	testRunBackup(t, []string{datadir}, opts, env.gopts)
+	testRunBackup(t, "", []string{datadir}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 	stat1 := dirStats(env.repo)
 
 	rtest.OK(t, appendRandomData(testfile, incrementalSecondWrite))
 
-	testRunBackup(t, []string{datadir}, opts, env.gopts)
+	testRunBackup(t, "", []string{datadir}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 	stat2 := dirStats(env.repo)
 	if stat2.size-stat1.size > incrementalFirstWrite {
@@ -631,7 +464,7 @@ func TestIncrementalBackup(t *testing.T) {
 
 	rtest.OK(t, appendRandomData(testfile, incrementalThirdWrite))
 
-	testRunBackup(t, []string{datadir}, opts, env.gopts)
+	testRunBackup(t, "", []string{datadir}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 	stat3 := dirStats(env.repo)
 	if stat3.size-stat2.size > incrementalFirstWrite {
@@ -650,7 +483,7 @@ func TestBackupTags(t *testing.T) {
 
 	opts := BackupOptions{}
 
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	testRunBackup(t, "", []string{env.testdata}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 	newest, _ := testRunSnapshots(t, env.gopts)
 	rtest.Assert(t, newest != nil, "expected a new backup, got nil")
@@ -659,7 +492,7 @@ func TestBackupTags(t *testing.T) {
 	parent := newest
 
 	opts.Tags = []string{"NL"}
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	testRunBackup(t, "", []string{env.testdata}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 	newest, _ = testRunSnapshots(t, env.gopts)
 	rtest.Assert(t, newest != nil, "expected a new backup, got nil")
@@ -682,7 +515,7 @@ func TestTag(t *testing.T) {
 	testRunInit(t, env.gopts)
 	rtest.SetupTarTestFixture(t, env.testdata, datafile)
 
-	testRunBackup(t, []string{env.testdata}, BackupOptions{}, env.gopts)
+	testRunBackup(t, "", []string{env.testdata}, BackupOptions{}, env.gopts)
 	testRunCheck(t, env.gopts)
 	newest, _ := testRunSnapshots(t, env.gopts)
 	rtest.Assert(t, newest != nil, "expected a new backup, got nil")
@@ -858,7 +691,7 @@ func TestRestoreFilter(t *testing.T) {
 
 	opts := BackupOptions{}
 
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	testRunBackup(t, filepath.Dir(env.testdata), []string{filepath.Base(env.testdata)}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 
 	snapshotID := testRunList(t, "snapshots", env.gopts)[0]
@@ -893,12 +726,12 @@ func TestRestore(t *testing.T) {
 	for i := 0; i < 10; i++ {
 		p := filepath.Join(env.testdata, fmt.Sprintf("foo/bar/testfile%v", i))
 		rtest.OK(t, os.MkdirAll(filepath.Dir(p), 0755))
-		rtest.OK(t, appendRandomData(p, uint(mrand.Intn(5<<21))))
+		rtest.OK(t, appendRandomData(p, uint(mrand.Intn(2<<21))))
 	}
 
 	opts := BackupOptions{}
 
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	testRunBackup(t, filepath.Dir(env.testdata), []string{filepath.Base(env.testdata)}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 
 	// Restore latest without any filters
@@ -921,12 +754,22 @@ func TestRestoreLatest(t *testing.T) {
 
 	opts := BackupOptions{}
 
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	// chdir manually here so we can get the current directory. This is not the
+	// same as the temp dir returned by ioutil.TempDir() on darwin.
+	back := fs.TestChdir(t, filepath.Dir(env.testdata))
+	defer back()
+
+	curdir, err := os.Getwd()
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	testRunBackup(t, "", []string{filepath.Base(env.testdata)}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 
 	os.Remove(p)
 	rtest.OK(t, appendRandomData(p, 101))
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	testRunBackup(t, "", []string{filepath.Base(env.testdata)}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 
 	// Restore latest without any filters
@@ -934,16 +777,18 @@ func TestRestoreLatest(t *testing.T) {
 	rtest.OK(t, testFileSize(filepath.Join(env.base, "restore0", "testdata", "testfile.c"), int64(101)))
 
 	// Setup test files in different directories backed up in different snapshots
-	p1 := filepath.Join(env.testdata, "p1/testfile.c")
+	p1 := filepath.Join(curdir, filepath.FromSlash("p1/testfile.c"))
+
 	rtest.OK(t, os.MkdirAll(filepath.Dir(p1), 0755))
 	rtest.OK(t, appendRandomData(p1, 102))
-	testRunBackup(t, []string{filepath.Dir(p1)}, opts, env.gopts)
+	testRunBackup(t, "", []string{"p1"}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 
-	p2 := filepath.Join(env.testdata, "p2/testfile.c")
+	p2 := filepath.Join(curdir, filepath.FromSlash("p2/testfile.c"))
+
 	rtest.OK(t, os.MkdirAll(filepath.Dir(p2), 0755))
 	rtest.OK(t, appendRandomData(p2, 103))
-	testRunBackup(t, []string{filepath.Dir(p2)}, opts, env.gopts)
+	testRunBackup(t, "", []string{"p2"}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 
 	p1rAbs := filepath.Join(env.base, "restore1", "p1/testfile.c")
@@ -1016,7 +861,7 @@ func TestRestoreNoMetadataOnIgnoredIntermediateDirs(t *testing.T) {
 
 	opts := BackupOptions{}
 
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	testRunBackup(t, filepath.Dir(env.testdata), []string{filepath.Base(env.testdata)}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 
 	snapshotID := testRunList(t, "snapshots", env.gopts)[0]
@@ -1054,7 +899,7 @@ func TestFind(t *testing.T) {
 
 	opts := BackupOptions{}
 
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	testRunBackup(t, "", []string{env.testdata}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 
 	results := testRunFind(t, false, env.gopts, "unexistingfile")
@@ -1094,7 +939,7 @@ func TestFindJSON(t *testing.T) {
 
 	opts := BackupOptions{}
 
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	testRunBackup(t, "", []string{env.testdata}, opts, env.gopts)
 	testRunCheck(t, env.gopts)
 
 	results := testRunFind(t, true, env.gopts, "unexistingfile")
@@ -1197,13 +1042,13 @@ func TestPrune(t *testing.T) {
 	rtest.SetupTarTestFixture(t, env.testdata, datafile)
 	opts := BackupOptions{}
 
-	testRunBackup(t, []string{filepath.Join(env.testdata, "0", "0")}, opts, env.gopts)
+	testRunBackup(t, "", []string{filepath.Join(env.testdata, "0", "0", "9")}, opts, env.gopts)
 	firstSnapshot := testRunList(t, "snapshots", env.gopts)
 	rtest.Assert(t, len(firstSnapshot) == 1,
 		"expected one snapshot, got %v", firstSnapshot)
 
-	testRunBackup(t, []string{filepath.Join(env.testdata, "0", "0", "2")}, opts, env.gopts)
-	testRunBackup(t, []string{filepath.Join(env.testdata, "0", "0", "3")}, opts, env.gopts)
+	testRunBackup(t, "", []string{filepath.Join(env.testdata, "0", "0", "9", "2")}, opts, env.gopts)
+	testRunBackup(t, "", []string{filepath.Join(env.testdata, "0", "0", "9", "3")}, opts, env.gopts)
 
 	snapshotIDs := testRunList(t, "snapshots", env.gopts)
 	rtest.Assert(t, len(snapshotIDs) == 3,
@@ -1237,7 +1082,7 @@ func TestHardLink(t *testing.T) {
 	opts := BackupOptions{}
 
 	// first backup
-	testRunBackup(t, []string{env.testdata}, opts, env.gopts)
+	testRunBackup(t, filepath.Dir(env.testdata), []string{filepath.Base(env.testdata)}, opts, env.gopts)
 	snapshotIDs := testRunList(t, "snapshots", env.gopts)
 	rtest.Assert(t, len(snapshotIDs) == 1,
 		"expected one snapshot, got %v", snapshotIDs)
@@ -1311,3 +1156,38 @@ func linkEqual(source, dest []string) bool {
 
 	return true
 }
+
+func TestQuietBackup(t *testing.T) {
+	env, cleanup := withTestEnvironment(t)
+	defer cleanup()
+
+	datafile := filepath.Join("testdata", "backup-data.tar.gz")
+	fd, err := os.Open(datafile)
+	if os.IsNotExist(errors.Cause(err)) {
+		t.Skipf("unable to find data file %q, skipping", datafile)
+		return
+	}
+	rtest.OK(t, err)
+	rtest.OK(t, fd.Close())
+
+	testRunInit(t, env.gopts)
+
+	rtest.SetupTarTestFixture(t, env.testdata, datafile)
+	opts := BackupOptions{}
+
+	env.gopts.Quiet = false
+	testRunBackup(t, "", []string{env.testdata}, opts, env.gopts)
+	snapshotIDs := testRunList(t, "snapshots", env.gopts)
+	rtest.Assert(t, len(snapshotIDs) == 1,
+		"expected one snapshot, got %v", snapshotIDs)
+
+	testRunCheck(t, env.gopts)
+
+	env.gopts.Quiet = true
+	testRunBackup(t, "", []string{env.testdata}, opts, env.gopts)
+	snapshotIDs = testRunList(t, "snapshots", env.gopts)
+	rtest.Assert(t, len(snapshotIDs) == 2,
+		"expected two snapshots, got %v", snapshotIDs)
+
+	testRunCheck(t, env.gopts)
+}

cmd/restic/lock.go

@@ -91,19 +91,23 @@ func unlockRepo(lock *restic.Lock) error {
 	globalLocks.Lock()
 	defer globalLocks.Unlock()
 
-	debug.Log("unlocking repository with lock %p", lock)
-	if err := lock.Unlock(); err != nil {
-		debug.Log("error while unlocking: %v", err)
-		return err
-	}
-
 	for i := 0; i < len(globalLocks.locks); i++ {
 		if lock == globalLocks.locks[i] {
+			// remove the lock from the repo
+			debug.Log("unlocking repository with lock %v", lock)
+			if err := lock.Unlock(); err != nil {
+				debug.Log("error while unlocking: %v", err)
+				return err
+			}
+
+			// remove the lock from the list of locks
 			globalLocks.locks = append(globalLocks.locks[:i], globalLocks.locks[i+1:]...)
 			return nil
 		}
 	}
 
+	debug.Log("unable to find lock %v in the global list of locks, ignoring", lock)
+
 	return nil
 }
 
@@ -119,6 +123,7 @@ func unlockAll() error {
 		}
 		debug.Log("successfully removed lock")
 	}
+	globalLocks.locks = globalLocks.locks[:0]
 
 	return nil
 }

cmd/restic/main.go

@@ -8,9 +8,11 @@ import (
 	"os"
 	"runtime"
 
+	"github.com/davecgh/go-spew/spew"
 	"github.com/restic/restic/internal/debug"
-	"github.com/restic/restic/internal/options"
 	"github.com/restic/restic/internal/restic"
+	"github.com/restic/restic/internal/ui/config"
+	"github.com/restic/restic/internal/ui/options"
 
 	"github.com/spf13/cobra"
 
@@ -29,15 +31,49 @@ directories in an encrypted repository stored on different backends.
 	SilenceUsage:      true,
 	DisableAutoGenTag: true,
 
-	PersistentPreRunE: func(*cobra.Command, []string) error {
+	PersistentPreRunE: func(c *cobra.Command, args []string) (err error) {
+		globalOptions.Config, err = config.Load("restic.conf")
+		if err != nil {
+			return err
+		}
+
+		err = config.ApplyEnv(&globalOptions.Config, os.Environ())
+		if err != nil {
+			return err
+		}
+
+		err = config.ApplyFlags(&globalOptions.Config, c.Flags())
+		if err != nil {
+			return err
+		}
+
+		spew.Dump(globalOptions.Config)
+
+		// set verbosity, default is one
+		globalOptions.verbosity = 1
+		if globalOptions.Quiet && (globalOptions.Verbose > 1) {
+			return errors.Fatal("--quiet and --verbose cannot be specified at the same time")
+		}
+
+		switch {
+		case globalOptions.Verbose >= 2:
+			globalOptions.verbosity = 3
+		case globalOptions.Verbose > 0:
+			globalOptions.verbosity = 2
+		case globalOptions.Quiet:
+			globalOptions.verbosity = 0
+		}
+
 		// parse extended options
 		opts, err := options.Parse(globalOptions.Options)
 		if err != nil {
 			return err
 		}
 		globalOptions.extended = opts
-
-		pwd, err := resolvePassword(globalOptions, "RESTIC_PASSWORD")
+		if c.Name() == "version" {
+			return nil
+		}
+		pwd, err := resolvePassword(globalOptions)
 		if err != nil {
 			fmt.Fprintf(os.Stderr, "Resolving password failed: %v\n", err)
 			Exit(1)
@@ -64,7 +100,7 @@ func init() {
 
 func main() {
 	debug.Log("main %#v", os.Args)
-	debug.Log("restic %s, compiled with %v on %v/%v",
+	debug.Log("restic %s compiled with %v on %v/%v",
 		version, runtime.Version(), runtime.GOOS, runtime.GOARCH)
 	err := cmdRoot.Execute()
 

doc/010_introduction.rst

@@ -14,3 +14,6 @@
 Introduction
 ############
 
+Restic is a fast and secure backup program. In the following sections, we will
+present typical workflows, starting with installing, preparing a new
+repository, and making the first backup.

doc/020_installation.rst

@@ -17,6 +17,14 @@ Installation
 Packages
 ********
 
+Note that if at any point the package you’re trying to use is outdated, you
+always have the option to use an official binary from the restic project.
+
+These are up to date binaries, built in a reproducible and verifiable way, that
+you can download and run without having to do additional installation work.
+
+Please see the :ref:`official_binaries` section below for various downloads.
+
 Mac OS X
 ========
 
@@ -62,12 +70,82 @@ installed from the official repos, e.g. with ``apt-get``:
 .. warning:: Please be aware that, at the time of writing, Debian *stable*
    has ``restic`` version 0.3.3 which is very old. The *testing* and *unstable*
    branches have recent versions of ``restic``.
+   
+RHEL & CentOS
+=============
 
-Pre-compiled Binary
-*******************
+restic can be installed via copr repository, for RHEL7/CentOS you can try the following:
 
-You can download the latest pre-compiled binary from the `restic release
-page <https://github.com/restic/restic/releases/latest>`__.
+.. code-block:: console
+
+    $ yum install yum-plugin-copr
+    $ yum copr enable copart/restic
+    $ yum install restic
+
+If that doesn't work, you can try adding the repository directly, for CentOS6 use:
+
+.. code-block:: console
+
+    $ yum-config-manager --add-repo https://copr.fedorainfracloud.org/coprs/copart/restic/repo/epel-6/copart-restic-epel-6.repo
+
+For CentOS7 use:
+
+.. code-block:: console
+
+    $ yum-config-manager --add-repo https://copr.fedorainfracloud.org/coprs/copart/restic/repo/epel-7/copart-restic-epel-7.repo
+
+Fedora
+======
+
+restic can be installed via copr repository.
+
+.. code-block:: console
+
+    $ dnf install dnf-plugin-core
+    $ dnf copr enable copart/restic
+    $ dnf install restic
+
+Solus
+=====
+
+restic can be installed from the official repo of Solus via the ``eopkg`` package manager:
+
+.. code-block:: console
+
+    $ eopkg install restic
+    
+OpenBSD
+=======
+
+On OpenBSD 6.3 and greater, you can install restic using ``pkg_add``:
+
+.. code-block:: console
+
+    # pkg_add restic
+
+.. _official_binaries:
+
+Official Binaries
+*****************
+
+Stable Releases
+===============
+
+You can download the latest stable release versions of restic from the `restic
+release page <https://github.com/restic/restic/releases/latest>`__. These builds
+are considered stable and releases are made regularly in a controlled manner.
+
+There's both pre-compiled binaries for different platforms as well as the source
+code available for download. Just download and run the one matching your system.
+
+Unstable Builds
+===============
+
+Another option is to use the latest builds for the master branch, available on
+the `restic beta download site
+<https://beta.restic.net/?sort=time&order=desc>`__. These too are pre-compiled
+and ready to run, and a new version is built every time a push is made to the
+master branch.
 
 Windows
 =======
@@ -79,15 +157,23 @@ Admin rights.
 Docker Container
 ****************
 
+We're maintaining a bare docker container with just a few files and the restic
+binary, you can get it with `docker pull` like this:
+
+.. code-block:: console
+
+    $ docker pull restic/restic
+
 .. note::
-   | A docker container is available as a contribution (Thank you!).
-   | You can find it at https://github.com/Lobaro/restic-backup-docker
+   | Another docker container which offers more configuration options is
+   | available as a contribution (Thank you!). You can find it at
+   | https://github.com/Lobaro/restic-backup-docker
 
 From Source
 ***********
 
 restic is written in the Go programming language and you need at least
-Go version 1.8. Building restic may also work with older versions of Go,
+Go version 1.9. Building restic may also work with older versions of Go,
 but that's not supported. See the `Getting
 started <https://golang.org/doc/install>`__ guide of the Go project for
 instructions how to install Go.
@@ -107,7 +193,7 @@ You can easily cross-compile restic for all supported platforms, just
 supply the target OS and platform via the command-line options like this
 (for Windows and FreeBSD respectively):
 
-::
+.. code-block:: console
 
     $ go run build.go --goos windows --goarch amd64
 
@@ -124,35 +210,27 @@ compiler. Building restic with gccgo may work, but is not supported.
 Autocompletion
 **************
 
-Restic can write out a bash compatible autocompletion script:
+Restic can write out man pages and bash/zsh compatible autocompletion scripts:
 
 .. code-block:: console
 
-    $ ./restic autocomplete --help
-    The "autocomplete" command generates a shell autocompletion script.
+    $ ./restic generate --help
 
-    NOTE: The current version supports Bash only.
-          This should work for *nix systems with Bash installed.
-
-By default, the file is written directly to ``/etc/bash_completion.d/``
-for convenience, and the command may need superuser rights, e.g.
-
-.. code-block:: console
-
-    $ sudo restic autocomplete
+    The "generate" command writes automatically generated files like the man pages
+    and the auto-completion files for bash and zsh).
 
     Usage:
-      restic autocomplete [flags]
+      restic generate [command] [flags]
 
     Flags:
-          --completionfile string   autocompletion file (default "/etc/bash_completion.d/restic.sh")
+          --bash-completion file   write bash completion file
+      -h, --help                   help for generate
+          --man directory          write man pages to directory
+          --zsh-completion file    write zsh completion file
 
-    Global Flags:
-          --json                   set output mode to JSON for commands that support it
-          --no-lock                do not lock the repo, this allows some operations on read-only repos
-      -o, --option key=value       set extended option (key=value, can be specified multiple times)
-      -p, --password-file string   read the repository password from a file
-      -q, --quiet                  do not output comprehensive progress report
-      -r, --repo string            repository to backup to or restore from (default: $RESTIC_REPOSITORY)
+Example for using sudo to write a bash completion script directly to the system-wide location:
 
+.. code-block:: console
 
+    $ sudo ./restic generate --bash-completion /etc/bash_completion.d/restic
+    writing bash completion file to /etc/bash_completion.d/restic

doc/030_preparing_a_new_repo.rst

@@ -15,20 +15,24 @@ Preparing a new repository
 ##########################
 
 The place where your backups will be saved at is called a "repository".
-This chapter explains how to create ("init") such a repository.
+This chapter explains how to create ("init") such a repository. The repository
+can be stored locally, or on some remote server or service. We'll first cover
+using a local repository, the remaining sections of this chapter cover all the
+other options. You can skip to the next chapter once you've read the relevant
+section here.
 
 Local
 *****
 
-In order to create a repository at ``/tmp/backup``, run the following
+In order to create a repository at ``/srv/restic-repo``, run the following
 command and enter the same password twice:
 
 .. code-block:: console
 
-    $ restic init --repo /tmp/backup
+    $ restic init --repo /srv/restic-repo
     enter password for new backend:
     enter password again:
-    created restic backend 085b3c76b9 at /tmp/backup
+    created restic backend 085b3c76b9 at /srv/restic-repo
     Please note that knowledge of your password is required to access the repository.
     Losing your password means that your data is irrecoverably lost.
 
@@ -55,10 +59,10 @@ simply be achieved by changing the URL scheme in the ``init`` command:
 
 .. code-block:: console
 
-    $ restic -r sftp:user@host:/tmp/backup init
+    $ restic -r sftp:user@host:/srv/restic-repo init
     enter password for new backend:
     enter password again:
-    created restic backend f1c6108821 at sftp:user@host:/tmp/backup
+    created restic backend f1c6108821 at sftp:user@host:/srv/restic-repo
     Please note that knowledge of your password is required to access the repository.
     Losing your password means that your data is irrecoverably lost.
 
@@ -87,7 +91,7 @@ specify the user name in this case):
 
 ::
 
-    $ restic -r sftp:foo:/tmp/backup init
+    $ restic -r sftp:foo:/srv/restic-repo init
 
 You can also add an entry with a special host name which does not exist,
 just for use with restic, and use the ``Hostname`` option to set the
@@ -104,7 +108,7 @@ Then use it in the backend specification:
 
 ::
 
-    $ restic -r sftp:restic-backup-host:/tmp/backup init
+    $ restic -r sftp:restic-backup-host:/srv/restic-repo init
 
 Last, if you'd like to use an entirely different program to create the
 SFTP connection, you can specify the command to be run with the option
@@ -139,7 +143,10 @@ If you use TLS, restic will use the system's CA certificates to verify the
 server certificate. When the verification fails, restic refuses to proceed and
 exits with an error. If you have your own self-signed certificate, or a custom
 CA certificate should be used for verification, you can pass restic the
-certificate filename via the `--cacert` option.
+certificate filename via the ``--cacert`` option. It will then verify that the
+server's certificate is contained in the file passed to this option, or signed
+by a CA certificate in the file. In this case, the system CA certificates are
+not considered at all.
 
 REST server uses exactly the same directory structure as local backend,
 so you should be able to access it both locally and via HTTP, even
@@ -306,8 +313,8 @@ bucket does not exist yet, it will be created:
     Please note that knowledge of your password is required to access the repository.
     Losing your password means that your data is irrecoverably lost.
 
-The number of concurrent connections to the B2 service can be set with the `-o
-b2.connections=10`. By default, at most five parallel connections are
+The number of concurrent connections to the B2 service can be set with the ``-o
+b2.connections=10``. By default, at most five parallel connections are
 established.
 
 Microsoft Azure Blob Storage
@@ -321,7 +328,7 @@ account name and key as follows:
     $ export AZURE_ACCOUNT_NAME=<ACCOUNT_NAME>
     $ export AZURE_ACCOUNT_KEY=<SECRET_KEY>
 
-Afterwards you can initialize a repository in a container called `foo` in the
+Afterwards you can initialize a repository in a container called ``foo`` in the
 root path like this:
 
 .. code-block:: console
@@ -334,7 +341,7 @@ root path like this:
     [...]
 
 The number of concurrent connections to the Azure Blob Storage service can be set with the
-`-o azure.connections=10`. By default, at most five parallel connections are
+``-o azure.connections=10``. By default, at most five parallel connections are
 established.
 
 Google Cloud Storage
@@ -362,8 +369,14 @@ key file and the project ID as follows:
     $ export GOOGLE_PROJECT_ID=123123123123
     $ export GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gs-secret-restic-key.json
 
-Then you can use the ``gs:`` backend type to create a new repository in the
-bucket `foo` at the root path:
+Restic uses  Google's client library to generate [default authentication
+material](https://developers.google.com/identity/protocols/application-default-credentials),
+which means if you're running in Google Container Engine or are otherwise
+located on an instance with default service accounts then these should work out
+the box.
+
+Once authenticated, you can use the ``gs:`` backend type to create a new
+repository in the bucket ``foo`` at the root path:
 
 .. code-block:: console
 
@@ -375,12 +388,117 @@ bucket `foo` at the root path:
     [...]
 
 The number of concurrent connections to the GCS service can be set with the
-`-o gs.connections=10`. By default, at most five parallel connections are
+``-o gs.connections=10``. By default, at most five parallel connections are
 established.
 
 .. _service account: https://cloud.google.com/storage/docs/authentication#service_accounts
 .. _create a service account key: https://cloud.google.com/storage/docs/authentication#generating-a-private-key
 
+Other Services via rclone
+*************************
+
+The program `rclone`_ can be used to access many other different services and
+store data there. First, you need to install and `configure`_ rclone.  The
+general backend specification format is ``rclone:<remote>:<path>``, the
+``<remote>:<path>`` component will be directly passed to rclone. When you
+configure a remote named ``foo``, you can then call restic as follows to
+initiate a new repository in the path ``bar`` in the repo:
+
+.. code-block:: console
+
+    $ restic -r rclone:foo:bar init
+
+Restic takes care of starting and stopping rclone.
+
+As a more concrete example, suppose you have configured a remote named
+``b2prod`` for Backblaze B2 with rclone, with a bucket called ``yggdrasil``.
+You can then use rclone to list files in the bucket like this:
+
+.. code-block:: console
+
+    $ rclone ls b2prod:yggdrasil
+
+In order to create a new repository in the root directory of the bucket, call
+restic like this:
+
+.. code-block:: console
+
+    $ restic -r rclone:b2prod:yggdrasil init
+
+If you want to use the path ``foo/bar/baz`` in the bucket instead, pass this to
+restic:
+
+.. code-block:: console
+
+    $ restic -r rclone:b2prod:yggdrasil/foo/bar/baz init
+
+Listing the files of an empty repository directly with rclone should return a
+listing similar to the following:
+
+.. code-block:: console
+
+    $ rclone ls b2prod:yggdrasil/foo/bar/baz
+        155 bar/baz/config
+        448 bar/baz/keys/4bf9c78049de689d73a56ed0546f83b8416795295cda12ec7fb9465af3900b44
+
+Rclone can be `configured with environment variables`_, so for instance
+configuring a bandwidth limit for rclone can be achieved by setting the
+``RCLONE_BWLIMIT`` environment variable:
+
+.. code-block:: console
+
+    $ export RCLONE_BWLIMIT=1M
+
+For debugging rclone, you can set the environment variable ``RCLONE_VERBOSE=2``.
+
+The rclone backend has two additional options:
+
+ * ``-o rclone.program`` specifies the path to rclone, the default value is just ``rclone``
+ * ``-o rclone.args`` allows setting the arguments passed to rclone, by default this is ``serve restic --stdio --b2-hard-delete --drive-use-trash=false``
+
+The reason for the two last parameters (``--b2-hard-delete`` and
+``--drive-use-trash=false``) can be found in the corresponding GitHub `issue #1657`_.
+
+In order to start rclone, restic will build a list of arguments by joining the
+following lists (in this order): ``rclone.program``, ``rclone.args`` and as the
+last parameter the value that follows the ``rclone:`` prefix of the repository
+specification.
+
+So, calling restic like this
+
+.. code-block:: console
+
+    $ restic -o rclone.program="/path/to/rclone" \
+      -o rclone.args="serve restic --stdio --bwlimit 1M --b2-hard-delete --verbose" \
+      -r rclone:b2:foo/bar
+
+runs rclone as follows:
+
+.. code-block:: console
+
+    $ /path/to/rclone serve restic --stdio --bwlimit 1M --b2-hard-delete --verbose b2:foo/bar
+
+Manually setting ``rclone.program`` also allows running a remote instance of
+rclone e.g. via SSH on a server, for example:
+
+.. code-block:: console
+
+    $ restic -o rclone.program="ssh user@host rclone" -r rclone:b2:foo/bar
+
+The rclone command may also be hard-coded in the SSH configuration or the
+user's public key, in this case it may be sufficient to just start the SSH
+connection (and it's irrelevant what's passed after ``rclone:`` in the
+repository specification):
+
+.. code-block:: console
+
+    $ restic -o rclone.program="ssh user@host" -r rclone:x
+
+.. _rclone: https://rclone.org/
+.. _configure: https://rclone.org/docs/
+.. _configured with environment variables: https://rclone.org/docs/#environment-variables
+.. _issue #1657: https://github.com/restic/restic/pull/1657#issuecomment-377707486
+
 Password prompt on Windows
 **************************
 
@@ -398,5 +516,5 @@ On MSYS2, you can install ``winpty`` as follows:
 .. code-block:: console
 
     $ pacman -S winpty
-    $ winpty restic -r /tmp/backup init
+    $ winpty restic -r /srv/restic-repo init
 

doc/040_backup.rst

@@ -21,43 +21,88 @@ again:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup backup ~/work
+    $ restic -r /srv/restic-repo --verbose backup ~/work
+    open repository
     enter password for repository:
-    scan [/home/user/work]
-    scanned 764 directories, 1816 files in 0:00
-    [0:29] 100.00%  54.732 MiB/s  1.582 GiB / 1.582 GiB  2580 / 2580 items  0 errors  ETA 0:00
-    duration: 0:29, 54.47MiB/s
+    password is correct
+    lock repository
+    load index files
+    start scan
+    start backup
+    scan finished in 1.837s
+    processed 1.720 GiB in 0:12
+    Files:        5307 new,     0 changed,     0 unmodified
+    Dirs:         1867 new,     0 changed,     0 unmodified
+    Added:      1.700 GiB
     snapshot 40dc1520 saved
 
 As you can see, restic created a backup of the directory and was pretty
 fast! The specific snapshot just created is identified by a sequence of
 hexadecimal characters, ``40dc1520`` in this case.
 
+If you don't pass the ``--verbose`` option, restic will print less data (but
+you'll still get a nice live status display).
+
 If you run the command again, restic will create another snapshot of
 your data, but this time it's even faster. This is de-duplication at
 work!
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup backup ~/work
+    $ restic -r /srv/restic-repo backup --verbose ~/work
+    open repository
     enter password for repository:
-    using parent snapshot 40dc1520aa6a07b7b3ae561786770a01951245d2367241e71e9485f18ae8228c
-    scan [/home/user/work]
-    scanned 764 directories, 1816 files in 0:00
-    [0:00] 100.00%  0B/s  1.582 GiB / 1.582 GiB  2580 / 2580 items  0 errors  ETA 0:00
-    duration: 0:00, 6572.38MiB/s
+    password is correct
+    lock repository
+    load index files
+    using parent snapshot d875ae93
+    start scan
+    start backup
+    scan finished in 1.881s
+    processed 1.720 GiB in 0:03
+    Files:           0 new,     0 changed,  5307 unmodified
+    Dirs:            0 new,     0 changed,  1867 unmodified
+    Added:      0 B
     snapshot 79766175 saved
 
-You can even backup individual files in the same repository.
+You can even backup individual files in the same repository (not passing
+``--verbose`` means less output):
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup backup ~/work.txt
-    scan [/home/user/work.txt]
-    scanned 0 directories, 1 files in 0:00
-    [0:00] 100.00%  0B/s  220B / 220B  1 / 1 items  0 errors  ETA 0:00
-    duration: 0:00, 0.03MiB/s
-    snapshot 31f7bd63 saved
+    $ restic -r /srv/restic-repo backup ~/work.txt
+    enter password for repository:
+    password is correct
+    snapshot 249d0210 saved
+
+If you're interested in what restic does, pass ``--verbose`` twice (or
+``--verbose 2``) to display detailed information about each file and directory
+restic encounters:
+
+.. code-block:: console
+
+    $ echo 'more data foo bar' >> ~/work.txt
+
+    $ restic -r /srv/restic-repo backup --verbose --verbose ~/work.txt
+    open repository
+    enter password for repository:
+    password is correct
+    lock repository
+    load index files
+    using parent snapshot f3f8d56b
+    start scan
+    start backup
+    scan finished in 2.115s
+    modified  /home/user/work.txt, saved in 0.007s (22 B added)
+    modified  /home/user/, saved in 0.008s (0 B added, 378 B metadata)
+    modified  /home/, saved in 0.009s (0 B added, 375 B metadata)
+    processed 22 B in 0:02
+    Files:           0 new,     1 changed,     0 unmodified
+    Dirs:            0 new,     2 changed,     0 unmodified
+    Data Blobs:      1 new
+    Tree Blobs:      3 new
+    Added:      1.116 KiB
+    snapshot 8dc503fc saved
 
 In fact several hosts may use the same repository to backup directories
 and files leading to a greater de-duplication.
@@ -75,6 +120,9 @@ Now is a good time to run ``restic check`` to verify that all data
 is properly stored in the repository. You should run this command regularly
 to make sure the internal structure of the repository is free of errors.
 
+Including and Excluding Files 
+*****************************
+
 You can exclude folders and files by specifying exclude patterns, currently
 the exclude options are:
 
@@ -84,33 +132,54 @@ the exclude options are:
 -  ``--exclude-if-present`` Specified one or more times to exclude a folders content
    if it contains a given file (optionally having a given header)
 
-Basic example:
+ Let's say we have a file called ``excludes.txt`` with the following content:
 
-.. code-block:: console
+::
 
-    $ cat exclude
     # exclude go-files
     *.go
     # exclude foo/x/y/z/bar foo/x/bar foo/bar
     foo/**/bar
-    $ restic -r /tmp/backup backup ~/work --exclude=*.c --exclude-file=exclude
+
+It can be used like this:
+
+.. code-block:: console
+
+    $ restic -r /srv/restic-repo backup ~/work --exclude="*.c" --exclude-file=excludes.txt
+
+This instruct restic to exclude files matching the following criteria:
+
+ * All files matching ``*.go`` (second line in ``excludes.txt``)
+ * All files and sub-directories named ``bar`` which reside somewhere below a directory called ``foo`` (fourth line in ``excludes.txt``)
+ * All files matching ``*.c`` (parameter ``--exclude``)
 
 Please see ``restic help backup`` for more specific information about each exclude option.
 
 Patterns use `filepath.Glob <https://golang.org/pkg/path/filepath/#Glob>`__ internally,
-see `filepath.Match <https://golang.org/pkg/path/filepath/#Match>`__ for syntax.
-Patterns are tested against the full path of a file/dir to be saved, not only
-against the relative path below the argument given to restic backup.
-Patterns need to match on complete path components. (``foo`` matches
-``/dir1/foo/dir2/file`` and ``/dir/foo`` but does not match ``/dir/foobar`` or
-``barfoo``.) A trailing ``/`` is ignored. A leading ``/`` anchors the
-pattern at the root directory. (``/bin`` matches ``/bin/bash`` but does not
-match ``/usr/bin/restic``.) Regular wildcards cannot be used to match over the
-directory separator ``/``. (``b*ash`` matches ``/bin/bash`` but does not match
-``/bin/ash``.) However ``**`` matches arbitrary subdirectories. (``foo/**/bar``
-matches ``/dir1/foo/dir2/bar/file``, ``/foo/bar/file`` and ``/tmp/foo/bar``.)
-Environment-variables in exclude-files are expanded with
-`os.ExpandEnv <https://golang.org/pkg/os/#ExpandEnv>`__.
+see `filepath.Match <https://golang.org/pkg/path/filepath/#Match>`__ for
+syntax. Patterns are tested against the full path of a file/dir to be saved,
+even if restic is passed a relative path to save. Environment-variables in
+exclude-files are expanded with `os.ExpandEnv <https://golang.org/pkg/os/#ExpandEnv>`__.
+
+Patterns need to match on complete path components. For example, the pattern ``foo``:
+
+ * matches ``/dir1/foo/dir2/file`` and ``/dir/foo``
+ * does not match ``/dir/foobar`` or ``barfoo``
+
+A trailing ``/`` is ignored, a leading ``/`` anchors the
+pattern at the root directory. This means, ``/bin`` matches ``/bin/bash`` but
+does not match ``/usr/bin/restic``.
+
+Regular wildcards cannot be used to match over the
+directory separator ``/``. For example: ``b*ash`` matches ``/bin/bash`` but does not match
+``/bin/ash``.
+
+For this, the special wildcard ``**`` can be used to match arbitrary
+sub-directories: The pattern ``foo/**/bar`` matches:
+
+ * ``/dir1/foo/dir2/bar/file``
+ * ``/foo/bar/file`` 
+ * ``/tmp/foo/bar``
 
 By specifying the option ``--one-file-system`` you can instruct restic
 to only backup files from the file systems the initially specified files
@@ -119,15 +188,15 @@ backup ``/sys`` or ``/dev`` on a Linux system:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup backup --one-file-system /
+    $ restic -r /srv/restic-repo backup --one-file-system /
 
 By using the ``--files-from`` option you can read the files you want to
 backup from a file. This is especially useful if a lot of files have to
 be backed up that are not in the same folder or are maybe pre-filtered
 by other software.
 
-or example maybe you want to backup files that have a certain filename
-in them:
+For example maybe you want to backup files which have a name that matches a
+certain pattern:
 
 .. code-block:: console
 
@@ -137,14 +206,16 @@ You can then use restic to backup the filtered files:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup backup --files-from /tmp/files_to_backup
+    $ restic -r /srv/restic-repo backup --files-from /tmp/files_to_backup
 
 Incidentally you can also combine ``--files-from`` with the normal files
 args:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup backup --files-from /tmp/files_to_backup /tmp/some_additional_file
+    $ restic -r /srv/restic-repo backup --files-from /tmp/files_to_backup /tmp/some_additional_file
+
+Paths in the listing file can be absolute or relative.
 
 Comparing Snapshots
 *******************
@@ -154,7 +225,7 @@ and displays a small statistic, just pass the command two snapshot IDs:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup diff 5845b002 2ab627a6
+    $ restic -r /srv/restic-repo diff 5845b002 2ab627a6
     password is correct
     comparing snapshot ea657ce5 to 2ab627a6:
 
@@ -201,7 +272,7 @@ this mode of operation, just supply the option ``--stdin`` to the
 
 .. code-block:: console
 
-    $ mysqldump [...] | restic -r /tmp/backup backup --stdin
+    $ mysqldump [...] | restic -r /srv/restic-repo backup --stdin
 
 This creates a new snapshot of the output of ``mysqldump``. You can then
 use e.g. the fuse mounting option (see below) to mount the repository
@@ -212,7 +283,7 @@ specified with ``--stdin-filename``, e.g. like this:
 
 .. code-block:: console
 
-    $ mysqldump [...] | restic -r /tmp/backup backup --stdin --stdin-filename production.sql
+    $ mysqldump [...] | restic -r /srv/restic-repo backup --stdin --stdin-filename production.sql
 
 Tags for backup
 ***************
@@ -222,7 +293,7 @@ information. Just specify the tags for a snapshot one by one with ``--tag``:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup backup --tag projectX --tag foo --tag bar ~/work
+    $ restic -r /srv/restic-repo backup --tag projectX --tag foo --tag bar ~/work
     [...]
 
 The tags can later be used to keep (or forget) snapshots with the ``forget``

doc/045_working_with_repos.rst

@@ -22,7 +22,7 @@ Now, you can list all the snapshots stored in the repository:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup snapshots
+    $ restic -r /srv/restic-repo snapshots
     enter password for repository:
     ID        Date                 Host    Tags   Directory
     ----------------------------------------------------------------------
@@ -36,7 +36,7 @@ You can filter the listing by directory path:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup snapshots --path="/srv"
+    $ restic -r /srv/restic-repo snapshots --path="/srv"
     enter password for repository:
     ID        Date                 Host    Tags   Directory
     ----------------------------------------------------------------------
@@ -47,7 +47,7 @@ Or filter by host:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup snapshots --host luigi
+    $ restic -r /srv/restic-repo snapshots --host luigi
     enter password for repository:
     ID        Date                 Host    Tags   Directory
     ----------------------------------------------------------------------
@@ -74,7 +74,7 @@ backup data is consistent and the integrity is unharmed:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup check
+    $ restic -r /srv/restic-repo check
     Load indexes
     ciphertext verification failed
 
@@ -83,7 +83,33 @@ yield the same error:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup restore 79766175 --target /tmp/restore-work
+    $ restic -r /srv/restic-repo restore 79766175 --target /tmp/restore-work
     Load indexes
     ciphertext verification failed
 
+By default, ``check`` command does not check that repository data files 
+are unmodified. Use ``--read-data`` parameter to check all repository
+data files:
+
+.. code-block:: console
+
+    $ restic -r /srv/restic-repo check --read-data
+    load indexes
+    check all packs
+    check snapshots, trees and blobs
+    read all data
+
+Use ``--read-data-subset=n/t`` parameter to check subset of repository data
+files. The parameter takes two values, ``n`` and ``t``. All repository data 
+files are logically devided in ``t`` roughly equal groups and only files that
+belong to the group number ``n`` are checked. For example, the following 
+commands check all repository data files over 5 separate invocations:
+
+.. code-block:: console
+
+    $ restic -r /srv/restic-repo check --read-data-subset=1/5
+    $ restic -r /srv/restic-repo check --read-data-subset=2/5
+    $ restic -r /srv/restic-repo check --read-data-subset=3/5
+    $ restic -r /srv/restic-repo check --read-data-subset=4/5
+    $ restic -r /srv/restic-repo check --read-data-subset=5/5
+

doc/050_restore.rst

@@ -23,7 +23,7 @@ command to restore the contents of the latest snapshot to
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup restore 79766175 --target /tmp/restore-work
+    $ restic -r /srv/restic-repo restore 79766175 --target /tmp/restore-work
     enter password for repository:
     restoring <Snapshot of [/home/user/work] at 2015-05-08 21:40:19.884408621 +0200 CEST> to /tmp/restore-work
 
@@ -33,7 +33,7 @@ backup for a specific host, path or both.
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup restore latest --target /tmp/restore-art --path "/home/art" --host luigi
+    $ restic -r /srv/restic-repo restore latest --target /tmp/restore-art --path "/home/art" --host luigi
     enter password for repository:
     restoring <Snapshot of [/home/art] at 2015-05-08 21:45:17.884408621 +0200 CEST> to /tmp/restore-art
 
@@ -42,7 +42,7 @@ files in the snapshot. For example, to restore a single file:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup restore 79766175 --target /tmp/restore-work --include /work/foo
+    $ restic -r /srv/restic-repo restore 79766175 --target /tmp/restore-work --include /work/foo
     enter password for repository:
     restoring <Snapshot of [/home/user/work] at 2015-05-08 21:40:19.884408621 +0200 CEST> to /tmp/restore-work
 
@@ -58,12 +58,13 @@ command to serve the repository with FUSE:
 .. code-block:: console
 
     $ mkdir /mnt/restic
-    $ restic -r /tmp/backup mount /mnt/restic
+    $ restic -r /srv/restic-repo mount /mnt/restic
     enter password for repository:
-    Now serving /tmp/backup at /mnt/restic
+    Now serving /srv/restic-repo at /mnt/restic
     Don't forget to umount after quitting!
 
-Mounting repositories via FUSE is not possible on Windows and OpenBSD.
+Mounting repositories via FUSE is not possible on OpenBSD, Solaris/illumos
+and Windows.
 
 Restic supports storage and preservation of hard links. However, since
 hard links exist in the scope of a filesystem by definition, restoring
@@ -79,4 +80,4 @@ the data directly. This can be achieved by using the `dump` command, like this:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup dump latest production.sql | mysql
+    $ restic -r /srv/restic-repo dump latest production.sql | mysql

doc/060_forget.rst

@@ -35,7 +35,7 @@ repository like this:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup snapshots
+    $ restic -r /srv/restic-repo snapshots
     enter password for repository:
     ID        Date                 Host      Tags  Directory
     ----------------------------------------------------------------------
@@ -50,7 +50,7 @@ command and specify the snapshot ID on the command line:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup forget bdbd3439
+    $ restic -r /srv/restic-repo forget bdbd3439
     enter password for repository:
     removed snapshot d3f01f63
 
@@ -58,7 +58,7 @@ Afterwards this snapshot is removed:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup snapshots
+    $ restic -r /srv/restic-repo snapshots
     enter password for repository:
     ID        Date                 Host     Tags  Directory
     ----------------------------------------------------------------------
@@ -73,7 +73,7 @@ command must be run:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup prune
+    $ restic -r /srv/restic-repo prune
     enter password for repository:
 
     counting files in repo
@@ -159,6 +159,13 @@ The ``forget`` command accepts the following parameters:
    snapshots, only keep the last one for that year.
 -  ``--keep-tag`` keep all snapshots which have all tags specified by
    this option (can be specified multiple times).
+-  ``--keep-within duration`` keep all snapshots which have been made within
+   the duration of the latest snapshot. ``duration`` needs to be a number of
+   years, months, and days, e.g. ``2y5m7d`` will keep all snapshots made in the
+   two years, five months, and seven days before the latest snapshot.
+
+Multiple policies will be ORed together so as to be as inclusive as possible
+for keeping snapshots.
 
 Additionally, you can restrict removing snapshots to those which have a
 particular hostname with the ``--hostname`` parameter, or tags with the

doc/070_encryption.rst

@@ -16,8 +16,8 @@ Encryption
 
 
 *"The design might not be perfect, but it’s good. Encryption is a first-class feature,
-the implementation looks sane and I guess the deduplication trade-off is worth it. So… I’m going to use restic for 
-my personal backups.*" `Filippo Valsorda`_
+the implementation looks sane and I guess the deduplication trade-off is worth
+it. So… I’m going to use restic for my personal backups.*" `Filippo Valsorda`_
 
 .. _Filippo Valsorda: https://blog.filippo.io/restic-cryptography/
 
@@ -31,19 +31,19 @@ per repository. In fact, you can use the ``list``, ``add``, ``remove``, and
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup key list
+    $ restic -r /srv/restic-repo key list
     enter password for repository:
      ID          User        Host        Created
     ----------------------------------------------------------------------
     *eb78040b    username    kasimir   2015-08-12 13:29:57
 
-    $ restic -r /tmp/backup key add
+    $ restic -r /srv/restic-repo key add
     enter password for repository:
     enter password for new key:
     enter password again:
     saved new key as <Key of username@kasimir, created on 2015-08-12 13:35:05.316831933 +0200 CEST>
 
-    $ restic -r backup key list
+    $ restic -r /srv/restic-repo key list
     enter password for repository:
      ID          User        Host        Created
     ----------------------------------------------------------------------

doc/075_scripting.rst

@@ -0,0 +1,39 @@
+..
+  Normally, there are no heading levels assigned to certain characters as the structure is
+  determined from the succession of headings. However, this convention is used in Python’s
+  Style Guide for documenting which you may follow:
+
+  # with overline, for parts
+  * for chapters
+  = for sections
+  - for subsections
+  ^ for subsubsections
+  " for paragraphs
+
+#########################
+Scripting
+#########################
+
+This is a list of how certain tasks may be accomplished when you use
+restic via scripts.
+
+Check if a repository is already initialized
+********************************************
+
+You may find a need to check if a repository is already initialized,
+perhaps to prevent your script from initializing a repository multiple
+times. The command ``snapshots`` may be used for this purpose:
+
+.. code-block:: console
+
+    $ restic -r /srv/restic-repo snapshots
+    Fatal: unable to open config file: Stat: stat /srv/restic-repo/config: no such file or directory
+    Is there a repository at the following location?
+    /srv/restic-repo
+
+If a repository does not exist, restic will return a non-zero exit code
+and print an error message. Note that restic will also return a non-zero
+exit code if a different error is encountered (e.g.: incorrect password
+to ``snapshots``) and it may print a different error message. If there
+are no errors, restic will return a zero exit code and print all the
+snapshots.

doc/100_references.rst

@@ -625,14 +625,15 @@ are deleted, the particular snapshot vanished and all snapshots
 depending on data that has been added in the snapshot cannot be restored
 completely. Restic is not designed to detect this attack.
 
+******
 Local Cache
-===========
+******
 
 In order to speed up certain operations, restic manages a local cache of data.
 This document describes the data structures for the local cache with version 1.
 
 Versions
---------
+========
 
 The cache directory is selected according to the `XDG base dir specification
 <http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html>`__.
@@ -646,12 +647,21 @@ a lower version number is found the cache is recreated with the current
 version. If a higher version number is found the cache is ignored and left as
 is.
 
-Snapshots and Indexes
----------------------
+Snapshots, Data and Indexes
+===========================
 
 Snapshot, Data and Index files are cached in the sub-directories ``snapshots``,
 ``data`` and  ``index``, as read from the repository.
 
+Expiry
+======
+
+Whenever a cache directory for a repo is used, that directory's modification
+timestamp is updated to the current time. By looking at the modification
+timestamps of the repo cache directories it is easy to decide which directories
+are old and haven't been used in a long time. Those are probably stale and can
+be removed.
+
 
 ************
 REST Backend
@@ -672,8 +682,8 @@ The following values are valid for ``{type}``:
 The API version is selected via the ``Accept`` HTTP header in the request. The
 following values are defined:
 
- * ``application/vnd.x.restic.rest.v1+json`` or empty: Select API version 1
- * ``application/vnd.x.restic.rest.v2+json``: Select API version 2
+ * ``application/vnd.x.restic.rest.v1`` or empty: Select API version 1
+ * ``application/vnd.x.restic.rest.v2``: Select API version 2
 
 The server will respond with the value of the highest version it supports in
 the ``Content-Type`` HTTP response header for the HTTP requests which should
@@ -681,7 +691,7 @@ return JSON. Any different value for this header means API version 1.
 
 The placeholder ``{path}`` in this document is a path to the repository, so
 that multiple different repositories can be accessed. The default path is
-``/``.
+``/``. The path must end with a slash.
 
 POST {path}?create=true
 =======================
@@ -798,24 +808,3 @@ Returns "200 OK" if the blob with the given name and type has been
 deleted from the repository, an HTTP error otherwise.
 
 
-*****
-Talks
-*****
-
-The following talks will be or have been given about restic:
-
--  2016-01-31: Lightning Talk at the Go Devroom at FOSDEM 2016,
-   Brussels, Belgium
--  2016-01-29: `restic - Backups mal
-   richtig <https://media.ccc.de/v/c4.openchaos.2016.01.restic>`__:
-   Public lecture in German at `CCC Cologne
-   e.V. <https://koeln.ccc.de>`__ in Cologne, Germany
--  2015-08-23: `A Solution to the Backup
-   Inconvenience <https://programm.froscon.de/2015/events/1515.html>`__:
-   Lecture at `FROSCON 2015 <https://www.froscon.de>`__ in Bonn, Germany
--  2015-02-01: `Lightning Talk at FOSDEM
-   2015 <https://www.youtube.com/watch?v=oM-MfeflUZ8&t=11m40s>`__: A
-   short introduction (with slightly outdated command line)
--  2015-01-27: `Talk about restic at CCC
-   Aachen <https://videoag.fsmpi.rwth-aachen.de/?view=player&lectureid=4442#content>`__
-   (in German)

doc/110_talks.rst

@@ -0,0 +1,34 @@
+..
+  Normally, there are no heading levels assigned to certain characters as the structure is
+  determined from the succession of headings. However, this convention is used in Python’s
+  Style Guide for documenting which you may follow:
+
+  # with overline, for parts
+  * for chapters
+  = for sections
+  - for subsections
+  ^ for subsubsections
+  " for paragraphs
+
+
+#####
+Talks
+#####
+
+The following talks will be or have been given about restic:
+
+-  2016-01-31: Lightning Talk at the Go Devroom at FOSDEM 2016,
+   Brussels, Belgium
+-  2016-01-29: `restic - Backups mal
+   richtig <https://media.ccc.de/v/c4.openchaos.2016.01.restic>`__:
+   Public lecture in German at `CCC Cologne
+   e.V. <https://koeln.ccc.de>`__ in Cologne, Germany
+-  2015-08-23: `A Solution to the Backup
+   Inconvenience <https://programm.froscon.de/2015/events/1515.html>`__:
+   Lecture at `FROSCON 2015 <https://www.froscon.de>`__ in Bonn, Germany
+-  2015-02-01: `Lightning Talk at FOSDEM
+   2015 <https://www.youtube.com/watch?v=oM-MfeflUZ8&t=11m40s>`__: A
+   short introduction (with slightly outdated command line)
+-  2015-01-27: `Talk about restic at CCC
+   Aachen <https://videoag.fsmpi.rwth-aachen.de/?view=player&lectureid=4442#content>`__
+   (in German)

doc/bash-completion.sh

@@ -1,6 +1,6 @@
 # bash completion for restic                               -*- shell-script -*-
 
-__debug()
+__restic_debug()
 {
     if [[ -n ${BASH_COMP_DEBUG_FILE} ]]; then
         echo "$*" >> "${BASH_COMP_DEBUG_FILE}"
@@ -9,13 +9,13 @@ __debug()
 
 # Homebrew on Macs have version 1.3 of bash-completion which doesn't include
 # _init_completion. This is a very minimal version of that function.
-__my_init_completion()
+__restic_init_completion()
 {
     COMPREPLY=()
     _get_comp_words_by_ref "$@" cur prev words cword
 }
 
-__index_of_word()
+__restic_index_of_word()
 {
     local w word=$1
     shift
@@ -27,7 +27,7 @@ __index_of_word()
     index=-1
 }
 
-__contains_word()
+__restic_contains_word()
 {
     local w word=$1; shift
     for w in "$@"; do
@@ -36,9 +36,9 @@ __contains_word()
     return 1
 }
 
-__handle_reply()
+__restic_handle_reply()
 {
-    __debug "${FUNCNAME[0]}"
+    __restic_debug "${FUNCNAME[0]}"
     case $cur in
         -*)
             if [[ $(type -t compopt) = "builtin" ]]; then
@@ -62,8 +62,8 @@ __handle_reply()
                 fi
 
                 local index flag
-                flag="${cur%%=*}"
-                __index_of_word "${flag}" "${flags_with_completion[@]}"
+                flag="${cur%=*}"
+                __restic_index_of_word "${flag}" "${flags_with_completion[@]}"
                 COMPREPLY=()
                 if [[ ${index} -ge 0 ]]; then
                     PREFIX=""
@@ -81,7 +81,7 @@ __handle_reply()
 
     # check if we are handling a flag with special work handling
     local index
-    __index_of_word "${prev}" "${flags_with_completion[@]}"
+    __restic_index_of_word "${prev}" "${flags_with_completion[@]}"
     if [[ ${index} -ge 0 ]]; then
         ${flags_completion[${index}]}
         return
@@ -114,24 +114,30 @@ __handle_reply()
     if declare -F __ltrim_colon_completions >/dev/null; then
         __ltrim_colon_completions "$cur"
     fi
+
+    # If there is only 1 completion and it is a flag with an = it will be completed
+    # but we don't want a space after the =
+    if [[ "${#COMPREPLY[@]}" -eq "1" ]] && [[ $(type -t compopt) = "builtin" ]] && [[ "${COMPREPLY[0]}" == --*= ]]; then
+       compopt -o nospace
+    fi
 }
 
 # The arguments should be in the form "ext1|ext2|extn"
-__handle_filename_extension_flag()
+__restic_handle_filename_extension_flag()
 {
     local ext="$1"
     _filedir "@(${ext})"
 }
 
-__handle_subdirs_in_dir_flag()
+__restic_handle_subdirs_in_dir_flag()
 {
     local dir="$1"
     pushd "${dir}" >/dev/null 2>&1 && _filedir -d && popd >/dev/null 2>&1
 }
 
-__handle_flag()
+__restic_handle_flag()
 {
-    __debug "${FUNCNAME[0]}: c is $c words[c] is ${words[c]}"
+    __restic_debug "${FUNCNAME[0]}: c is $c words[c] is ${words[c]}"
 
     # if a command required a flag, and we found it, unset must_have_one_flag()
     local flagname=${words[c]}
@@ -139,30 +145,33 @@ __handle_flag()
     # if the word contained an =
     if [[ ${words[c]} == *"="* ]]; then
         flagvalue=${flagname#*=} # take in as flagvalue after the =
-        flagname=${flagname%%=*} # strip everything after the =
+        flagname=${flagname%=*} # strip everything after the =
         flagname="${flagname}=" # but put the = back
     fi
-    __debug "${FUNCNAME[0]}: looking for ${flagname}"
-    if __contains_word "${flagname}" "${must_have_one_flag[@]}"; then
+    __restic_debug "${FUNCNAME[0]}: looking for ${flagname}"
+    if __restic_contains_word "${flagname}" "${must_have_one_flag[@]}"; then
         must_have_one_flag=()
     fi
 
     # if you set a flag which only applies to this command, don't show subcommands
-    if __contains_word "${flagname}" "${local_nonpersistent_flags[@]}"; then
+    if __restic_contains_word "${flagname}" "${local_nonpersistent_flags[@]}"; then
       commands=()
     fi
 
     # keep flag value with flagname as flaghash
-    if [ -n "${flagvalue}" ] ; then
-        flaghash[${flagname}]=${flagvalue}
-    elif [ -n "${words[ $((c+1)) ]}" ] ; then
-        flaghash[${flagname}]=${words[ $((c+1)) ]}
-    else
-        flaghash[${flagname}]="true" # pad "true" for bool flag
+    # flaghash variable is an associative array which is only supported in bash > 3.
+    if [[ -z "${BASH_VERSION}" || "${BASH_VERSINFO[0]}" -gt 3 ]]; then
+        if [ -n "${flagvalue}" ] ; then
+            flaghash[${flagname}]=${flagvalue}
+        elif [ -n "${words[ $((c+1)) ]}" ] ; then
+            flaghash[${flagname}]=${words[ $((c+1)) ]}
+        else
+            flaghash[${flagname}]="true" # pad "true" for bool flag
+        fi
     fi
 
     # skip the argument to a two word flag
-    if __contains_word "${words[c]}" "${two_word_flags[@]}"; then
+    if __restic_contains_word "${words[c]}" "${two_word_flags[@]}"; then
         c=$((c+1))
         # if we are looking for a flags value, don't show commands
         if [[ $c -eq $cword ]]; then
@@ -174,13 +183,13 @@ __handle_flag()
 
 }
 
-__handle_noun()
+__restic_handle_noun()
 {
-    __debug "${FUNCNAME[0]}: c is $c words[c] is ${words[c]}"
+    __restic_debug "${FUNCNAME[0]}: c is $c words[c] is ${words[c]}"
 
-    if __contains_word "${words[c]}" "${must_have_one_noun[@]}"; then
+    if __restic_contains_word "${words[c]}" "${must_have_one_noun[@]}"; then
         must_have_one_noun=()
-    elif __contains_word "${words[c]}" "${noun_aliases[@]}"; then
+    elif __restic_contains_word "${words[c]}" "${noun_aliases[@]}"; then
         must_have_one_noun=()
     fi
 
@@ -188,42 +197,42 @@ __handle_noun()
     c=$((c+1))
 }
 
-__handle_command()
+__restic_handle_command()
 {
-    __debug "${FUNCNAME[0]}: c is $c words[c] is ${words[c]}"
+    __restic_debug "${FUNCNAME[0]}: c is $c words[c] is ${words[c]}"
 
     local next_command
     if [[ -n ${last_command} ]]; then
         next_command="_${last_command}_${words[c]//:/__}"
     else
         if [[ $c -eq 0 ]]; then
-            next_command="_$(basename "${words[c]//:/__}")"
+            next_command="_restic_root_command"
         else
             next_command="_${words[c]//:/__}"
         fi
     fi
     c=$((c+1))
-    __debug "${FUNCNAME[0]}: looking for ${next_command}"
+    __restic_debug "${FUNCNAME[0]}: looking for ${next_command}"
     declare -F "$next_command" >/dev/null && $next_command
 }
 
-__handle_word()
+__restic_handle_word()
 {
     if [[ $c -ge $cword ]]; then
-        __handle_reply
+        __restic_handle_reply
         return
     fi
-    __debug "${FUNCNAME[0]}: c is $c words[c] is ${words[c]}"
+    __restic_debug "${FUNCNAME[0]}: c is $c words[c] is ${words[c]}"
     if [[ "${words[c]}" == -* ]]; then
-        __handle_flag
-    elif __contains_word "${words[c]}" "${commands[@]}"; then
-        __handle_command
-    elif [[ $c -eq 0 ]] && __contains_word "$(basename "${words[c]}")" "${commands[@]}"; then
-        __handle_command
+        __restic_handle_flag
+    elif __restic_contains_word "${words[c]}" "${commands[@]}"; then
+        __restic_handle_command
+    elif [[ $c -eq 0 ]]; then
+        __restic_handle_command
     else
-        __handle_noun
+        __restic_handle_noun
     fi
-    __handle_word
+    __restic_handle_word
 }
 
 _restic_backup()
@@ -288,6 +297,51 @@ _restic_backup()
     flags+=("--repo=")
     two_word_flags+=("-r")
     flags+=("--tls-client-cert=")
+    flags+=("--verbose")
+    flags+=("-v")
+
+    must_have_one_flag=()
+    must_have_one_noun=()
+    noun_aliases=()
+}
+
+_restic_cache()
+{
+    last_command="restic_cache"
+    commands=()
+
+    flags=()
+    two_word_flags=()
+    local_nonpersistent_flags=()
+    flags_with_completion=()
+    flags_completion=()
+
+    flags+=("--cleanup")
+    local_nonpersistent_flags+=("--cleanup")
+    flags+=("--help")
+    flags+=("-h")
+    local_nonpersistent_flags+=("--help")
+    flags+=("--max-age=")
+    local_nonpersistent_flags+=("--max-age=")
+    flags+=("--cacert=")
+    flags+=("--cache-dir=")
+    flags+=("--cleanup-cache")
+    flags+=("--json")
+    flags+=("--limit-download=")
+    flags+=("--limit-upload=")
+    flags+=("--no-cache")
+    flags+=("--no-lock")
+    flags+=("--option=")
+    two_word_flags+=("-o")
+    flags+=("--password-file=")
+    two_word_flags+=("-p")
+    flags+=("--quiet")
+    flags+=("-q")
+    flags+=("--repo=")
+    two_word_flags+=("-r")
+    flags+=("--tls-client-cert=")
+    flags+=("--verbose")
+    flags+=("-v")
 
     must_have_one_flag=()
     must_have_one_noun=()
@@ -325,6 +379,8 @@ _restic_cat()
     flags+=("--repo=")
     two_word_flags+=("-r")
     flags+=("--tls-client-cert=")
+    flags+=("--verbose")
+    flags+=("-v")
 
     must_have_one_flag=()
     must_have_one_noun=()
@@ -349,6 +405,8 @@ _restic_check()
     local_nonpersistent_flags+=("--help")
     flags+=("--read-data")
     local_nonpersistent_flags+=("--read-data")
+    flags+=("--read-data-subset=")
+    local_nonpersistent_flags+=("--read-data-subset=")
     flags+=("--with-cache")
     local_nonpersistent_flags+=("--with-cache")
     flags+=("--cacert=")
@@ -368,6 +426,8 @@ _restic_check()
     flags+=("--repo=")
     two_word_flags+=("-r")
     flags+=("--tls-client-cert=")
+    flags+=("--verbose")
+    flags+=("-v")
 
     must_have_one_flag=()
     must_have_one_noun=()
@@ -407,6 +467,8 @@ _restic_diff()
     flags+=("--repo=")
     two_word_flags+=("-r")
     flags+=("--tls-client-cert=")
+    flags+=("--verbose")
+    flags+=("-v")
 
     must_have_one_flag=()
     must_have_one_noun=()
@@ -451,6 +513,8 @@ _restic_dump()
     flags+=("--repo=")
     two_word_flags+=("-r")
     flags+=("--tls-client-cert=")
+    flags+=("--verbose")
+    flags+=("-v")
 
     must_have_one_flag=()
     must_have_one_noun=()
@@ -510,6 +574,8 @@ _restic_find()
     flags+=("--repo=")
     two_word_flags+=("-r")
     flags+=("--tls-client-cert=")
+    flags+=("--verbose")
+    flags+=("-v")
 
     must_have_one_flag=()
     must_have_one_noun=()
@@ -545,6 +611,8 @@ _restic_forget()
     flags+=("--keep-yearly=")
     two_word_flags+=("-y")
     local_nonpersistent_flags+=("--keep-yearly=")
+    flags+=("--keep-within=")
+    local_nonpersistent_flags+=("--keep-within=")
     flags+=("--keep-tag=")
     local_nonpersistent_flags+=("--keep-tag=")
     flags+=("--host=")
@@ -586,6 +654,8 @@ _restic_forget()
     flags+=("--repo=")
     two_word_flags+=("-r")
     flags+=("--tls-client-cert=")
+    flags+=("--verbose")
+    flags+=("-v")
 
     must_have_one_flag=()
     must_have_one_noun=()
@@ -629,6 +699,8 @@ _restic_generate()
     flags+=("--repo=")
     two_word_flags+=("-r")
     flags+=("--tls-client-cert=")
+    flags+=("--verbose")
+    flags+=("-v")
 
     must_have_one_flag=()
     must_have_one_noun=()
@@ -666,6 +738,8 @@ _restic_init()
     flags+=("--repo=")
     two_word_flags+=("-r")
     flags+=("--tls-client-cert=")
+    flags+=("--verbose")
+    flags+=("-v")
 
     must_have_one_flag=()
     must_have_one_noun=()
@@ -686,6 +760,8 @@ _restic_key()
     flags+=("--help")
     flags+=("-h")
     local_nonpersistent_flags+=("--help")
+    flags+=("--new-password-file=")
+    local_nonpersistent_flags+=("--new-password-file=")
     flags+=("--cacert=")
     flags+=("--cache-dir=")
     flags+=("--cleanup-cache")
@@ -703,6 +779,8 @@ _restic_key()
     flags+=("--repo=")
     two_word_flags+=("-r")
     flags+=("--tls-client-cert=")
+    flags+=("--verbose")
+    flags+=("-v")
 
     must_have_one_flag=()
     must_have_one_noun=()
@@ -740,6 +818,8 @@ _restic_list()
     flags+=("--repo=")
     two_word_flags+=("-r")
     flags+=("--tls-client-cert=")
+    flags+=("--verbose")
+    flags+=("-v")
 
     must_have_one_flag=()
     must_have_one_noun=()
@@ -787,6 +867,8 @@ _restic_ls()
     flags+=("--repo=")
     two_word_flags+=("-r")
     flags+=("--tls-client-cert=")
+    flags+=("--verbose")
+    flags+=("-v")
 
     must_have_one_flag=()
     must_have_one_noun=()
@@ -827,6 +909,8 @@ _restic_migrate()
     flags+=("--repo=")
     two_word_flags+=("-r")
     flags+=("--tls-client-cert=")
+    flags+=("--verbose")
+    flags+=("-v")
 
     must_have_one_flag=()
     must_have_one_noun=()
@@ -879,6 +963,8 @@ _restic_mount()
     flags+=("--repo=")
     two_word_flags+=("-r")
     flags+=("--tls-client-cert=")
+    flags+=("--verbose")
+    flags+=("-v")
 
     must_have_one_flag=()
     must_have_one_noun=()
@@ -916,6 +1002,8 @@ _restic_prune()
     flags+=("--repo=")
     two_word_flags+=("-r")
     flags+=("--tls-client-cert=")
+    flags+=("--verbose")
+    flags+=("-v")
 
     must_have_one_flag=()
     must_have_one_noun=()
@@ -953,6 +1041,8 @@ _restic_rebuild-index()
     flags+=("--repo=")
     two_word_flags+=("-r")
     flags+=("--tls-client-cert=")
+    flags+=("--verbose")
+    flags+=("-v")
 
     must_have_one_flag=()
     must_have_one_noun=()
@@ -1006,6 +1096,8 @@ _restic_restore()
     flags+=("--repo=")
     two_word_flags+=("-r")
     flags+=("--tls-client-cert=")
+    flags+=("--verbose")
+    flags+=("-v")
 
     must_have_one_flag=()
     must_have_one_noun=()
@@ -1055,6 +1147,8 @@ _restic_snapshots()
     flags+=("--repo=")
     two_word_flags+=("-r")
     flags+=("--tls-client-cert=")
+    flags+=("--verbose")
+    flags+=("-v")
 
     must_have_one_flag=()
     must_have_one_noun=()
@@ -1105,6 +1199,8 @@ _restic_tag()
     flags+=("--repo=")
     two_word_flags+=("-r")
     flags+=("--tls-client-cert=")
+    flags+=("--verbose")
+    flags+=("-v")
 
     must_have_one_flag=()
     must_have_one_noun=()
@@ -1144,6 +1240,8 @@ _restic_unlock()
     flags+=("--repo=")
     two_word_flags+=("-r")
     flags+=("--tls-client-cert=")
+    flags+=("--verbose")
+    flags+=("-v")
 
     must_have_one_flag=()
     must_have_one_noun=()
@@ -1181,17 +1279,20 @@ _restic_version()
     flags+=("--repo=")
     two_word_flags+=("-r")
     flags+=("--tls-client-cert=")
+    flags+=("--verbose")
+    flags+=("-v")
 
     must_have_one_flag=()
     must_have_one_noun=()
     noun_aliases=()
 }
 
-_restic()
+_restic_root_command()
 {
     last_command="restic"
     commands=()
     commands+=("backup")
+    commands+=("cache")
     commands+=("cat")
     commands+=("check")
     commands+=("diff")
@@ -1239,6 +1340,8 @@ _restic()
     flags+=("--repo=")
     two_word_flags+=("-r")
     flags+=("--tls-client-cert=")
+    flags+=("--verbose")
+    flags+=("-v")
 
     must_have_one_flag=()
     must_have_one_noun=()
@@ -1252,7 +1355,7 @@ __start_restic()
     if declare -F _init_completion >/dev/null 2>&1; then
         _init_completion -s || return
     else
-        __my_init_completion -n "=" || return
+        __restic_init_completion -n "=" || return
     fi
 
     local c=0
@@ -1267,7 +1370,7 @@ __start_restic()
     local last_command
     local nouns=()
 
-    __handle_word
+    __restic_handle_word
 }
 
 if [[ $(type -t compopt) = "builtin" ]]; then

doc/cache.rst

@@ -1,36 +0,0 @@
-Local Cache
-===========
-
-In order to speed up certain operations, restic manages a local cache of data.
-This document describes the data structures for the local cache with version 1.
-
-Versions
---------
-
-The cache directory is selected according to the `XDG base dir specification
-<http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html>`__.
-Each repository has its own cache sub-directory, consting of the repository ID
-which is chosen at ``init``. All cache directories for different repos are
-independent of each other.
-
-The cache dir for a repo contains a file named ``version``, which contains a
-single ASCII integer line that stands for the current version of the cache. If
-a lower version number is found the cache is recreated with the current
-version. If a higher version number is found the cache is ignored and left as
-is.
-
-Snapshots, Data and Indexes
----------------------------
-
-Snapshot, Data and Index files are cached in the sub-directories ``snapshots``,
-``data`` and  ``index``, as read from the repository.
-
-Expiry
-------
-
-Whenever a cache directory for a repo is used, that directory's modification
-timestamp is updated to the current time. By looking at the modification
-timestamps of the repo cache directories it is easy to decide which directories
-are old and haven't been used in a long time. Those are probably stale and can
-be removed.
-

doc/conf.py

@@ -35,7 +35,7 @@ master_doc = 'index'
 
 # General information about the project.
 project = 'restic'
-copyright = '2017, restic authors'
+copyright = '2018, restic authors'
 author = 'fd0'
 
 # The version info for the project you're documenting, acts as replacement for

doc/faq.rst

@@ -111,6 +111,10 @@ directory structure via sftp, so the path that needs to be specified is
 different than the directory structure on the device and maybe even as exposed
 via other protocols.
 
+
+Try removing the /volume1 prefix in your paths. If this does not work, use sftp
+and ls to explore the SFTP file system hierarchy on your NAS.
+
 The following may work:
 
 ::

doc/index.rst

@@ -12,8 +12,10 @@ Restic Documentation
    050_restore
    060_forget
    070_encryption
+   075_scripting
    080_examples
    090_participating
    100_references
+   110_talks
    faq
    manual_rest

doc/man/restic-backup.1

@@ -84,7 +84,7 @@ given as the arguments.
 .SH OPTIONS INHERITED FROM PARENT COMMANDS
 .PP
 \fB\-\-cacert\fP=[]
-    path to load root certificates from (default: use system certificates)
+    \fB\fCfile\fR to load root certificates from (default: use system certificates)
 
 .PP
 \fB\-\-cache\-dir\fP=""
@@ -134,6 +134,10 @@ given as the arguments.
 \fB\-\-tls\-client\-cert\fP=""
     path to a file containing PEM encoded TLS client certificate and private key
 
+.PP
+\fB\-v\fP, \fB\-\-verbose\fP[=0]
+    be verbose (specify \-\-verbose multiple times or level \fB\fCn\fR)
+
 
 .SH SEE ALSO
 .PP

doc/man/restic-cache.1

@@ -0,0 +1,95 @@
+.TH "restic backup" "1" "Jan 2017" "generated by `restic generate`" "" 
+.nh
+.ad l
+
+
+.SH NAME
+.PP
+restic\-cache \- Operate on local cache directories
+
+
+.SH SYNOPSIS
+.PP
+\fBrestic cache [flags]\fP
+
+
+.SH DESCRIPTION
+.PP
+The "cache" command allows listing and cleaning local cache directories.
+
+
+.SH OPTIONS
+.PP
+\fB\-\-cleanup\fP[=false]
+    remove old cache directories
+
+.PP
+\fB\-h\fP, \fB\-\-help\fP[=false]
+    help for cache
+
+.PP
+\fB\-\-max\-age\fP=30
+    max age in \fB\fCdays\fR for cache directories to be considered old
+
+
+.SH OPTIONS INHERITED FROM PARENT COMMANDS
+.PP
+\fB\-\-cacert\fP=[]
+    \fB\fCfile\fR to load root certificates from (default: use system certificates)
+
+.PP
+\fB\-\-cache\-dir\fP=""
+    set the cache directory
+
+.PP
+\fB\-\-cleanup\-cache\fP[=false]
+    auto remove old cache directories
+
+.PP
+\fB\-\-json\fP[=false]
+    set output mode to JSON for commands that support it
+
+.PP
+\fB\-\-limit\-download\fP=0
+    limits downloads to a maximum rate in KiB/s. (default: unlimited)
+
+.PP
+\fB\-\-limit\-upload\fP=0
+    limits uploads to a maximum rate in KiB/s. (default: unlimited)
+
+.PP
+\fB\-\-no\-cache\fP[=false]
+    do not use a local cache
+
+.PP
+\fB\-\-no\-lock\fP[=false]
+    do not lock the repo, this allows some operations on read\-only repos
+
+.PP
+\fB\-o\fP, \fB\-\-option\fP=[]
+    set extended option (\fB\fCkey=value\fR, can be specified multiple times)
+
+.PP
+\fB\-p\fP, \fB\-\-password\-file\fP=""
+    read the repository password from a file (default: $RESTIC\_PASSWORD\_FILE)
+
+.PP
+\fB\-q\fP, \fB\-\-quiet\fP[=false]
+    do not output comprehensive progress report
+
+.PP
+\fB\-r\fP, \fB\-\-repo\fP=""
+    repository to backup to or restore from (default: $RESTIC\_REPOSITORY)
+
+.PP
+\fB\-\-tls\-client\-cert\fP=""
+    path to a file containing PEM encoded TLS client certificate and private key
+
+.PP
+\fB\-v\fP, \fB\-\-verbose\fP[=0]
+    be verbose (specify \-\-verbose multiple times or level \fB\fCn\fR)
+
+
+.SH SEE ALSO
+.PP
+\fBrestic(1)\fP

doc/man/restic-cat.1

@@ -27,7 +27,7 @@ The "cat" command is used to print internal objects to stdout.
 .SH OPTIONS INHERITED FROM PARENT COMMANDS
 .PP
 \fB\-\-cacert\fP=[]
-    path to load root certificates from (default: use system certificates)
+    \fB\fCfile\fR to load root certificates from (default: use system certificates)
 
 .PP
 \fB\-\-cache\-dir\fP=""
@@ -77,6 +77,10 @@ The "cat" command is used to print internal objects to stdout.
 \fB\-\-tls\-client\-cert\fP=""
     path to a file containing PEM encoded TLS client certificate and private key
 
+.PP
+\fB\-v\fP, \fB\-\-verbose\fP[=0]
+    be verbose (specify \-\-verbose multiple times or level \fB\fCn\fR)
+
 
 .SH SEE ALSO
 .PP

doc/man/restic-check.1

@@ -36,6 +36,10 @@ repository and not use a local cache.
 \fB\-\-read\-data\fP[=false]
     read all data blobs
 
+.PP
+\fB\-\-read\-data\-subset\fP=""
+    read subset of data packs
+
 .PP
 \fB\-\-with\-cache\fP[=false]
     use the cache
@@ -44,7 +48,7 @@ repository and not use a local cache.
 .SH OPTIONS INHERITED FROM PARENT COMMANDS
 .PP
 \fB\-\-cacert\fP=[]
-    path to load root certificates from (default: use system certificates)
+    \fB\fCfile\fR to load root certificates from (default: use system certificates)
 
 .PP
 \fB\-\-cache\-dir\fP=""
@@ -94,6 +98,10 @@ repository and not use a local cache.
 \fB\-\-tls\-client\-cert\fP=""
     path to a file containing PEM encoded TLS client certificate and private key
 
+.PP
+\fB\-v\fP, \fB\-\-verbose\fP[=0]
+    be verbose (specify \-\-verbose multiple times or level \fB\fCn\fR)
+
 
 .SH SEE ALSO
 .PP

doc/man/restic-diff.1

@@ -18,6 +18,8 @@ restic\-diff \- Show differences between two snapshots
 The "diff" command shows differences from the first to the second snapshot. The
 first characters in each line display what has happened to a particular file or
 directory:
+
+.RS
 .IP \(bu 2
 The item was added
 .IP \(bu 2
@@ -26,6 +28,8 @@ U  The metadata (access mode, timestamps, ...) for the item was updated
 M  The file's content was modified
 T  The type was changed, e.g. a file was made a symlink
 
+.RE
+
 
 .SH OPTIONS
 .PP
@@ -40,7 +44,7 @@ T  The type was changed, e.g. a file was made a symlink
 .SH OPTIONS INHERITED FROM PARENT COMMANDS
 .PP
 \fB\-\-cacert\fP=[]
-    path to load root certificates from (default: use system certificates)
+    \fB\fCfile\fR to load root certificates from (default: use system certificates)
 
 .PP
 \fB\-\-cache\-dir\fP=""
@@ -90,6 +94,10 @@ T  The type was changed, e.g. a file was made a symlink
 \fB\-\-tls\-client\-cert\fP=""
     path to a file containing PEM encoded TLS client certificate and private key
 
+.PP
+\fB\-v\fP, \fB\-\-verbose\fP[=0]
+    be verbose (specify \-\-verbose multiple times or level \fB\fCn\fR)
+
 
 .SH SEE ALSO
 .PP

doc/man/restic-dump.1

@@ -44,7 +44,7 @@ repository.
 .SH OPTIONS INHERITED FROM PARENT COMMANDS
 .PP
 \fB\-\-cacert\fP=[]
-    path to load root certificates from (default: use system certificates)
+    \fB\fCfile\fR to load root certificates from (default: use system certificates)
 
 .PP
 \fB\-\-cache\-dir\fP=""
@@ -94,6 +94,10 @@ repository.
 \fB\-\-tls\-client\-cert\fP=""
     path to a file containing PEM encoded TLS client certificate and private key
 
+.PP
+\fB\-v\fP, \fB\-\-verbose\fP[=0]
+    be verbose (specify \-\-verbose multiple times or level \fB\fCn\fR)
+
 
 .SH SEE ALSO
 .PP

doc/man/restic-find.1

@@ -60,7 +60,7 @@ repo.
 .SH OPTIONS INHERITED FROM PARENT COMMANDS
 .PP
 \fB\-\-cacert\fP=[]
-    path to load root certificates from (default: use system certificates)
+    \fB\fCfile\fR to load root certificates from (default: use system certificates)
 
 .PP
 \fB\-\-cache\-dir\fP=""
@@ -110,6 +110,10 @@ repo.
 \fB\-\-tls\-client\-cert\fP=""
     path to a file containing PEM encoded TLS client certificate and private key
 
+.PP
+\fB\-v\fP, \fB\-\-verbose\fP[=0]
+    be verbose (specify \-\-verbose multiple times or level \fB\fCn\fR)
+
 
 .SH SEE ALSO
 .PP

doc/man/restic-forget.1

@@ -46,6 +46,10 @@ data after 'forget' was run successfully, see the 'prune' command.
 \fB\-y\fP, \fB\-\-keep\-yearly\fP=0
     keep the last \fB\fCn\fR yearly snapshots
 
+.PP
+\fB\-\-keep\-within\fP=
+    keep snapshots that were created within \fB\fCduration\fR before the newest (e.g. 1y5m7d)
+
 .PP
 \fB\-\-keep\-tag\fP=[]
     keep snapshots with this \fB\fCtaglist\fR (can be specified multiple times)
@@ -90,7 +94,7 @@ data after 'forget' was run successfully, see the 'prune' command.
 .SH OPTIONS INHERITED FROM PARENT COMMANDS
 .PP
 \fB\-\-cacert\fP=[]
-    path to load root certificates from (default: use system certificates)
+    \fB\fCfile\fR to load root certificates from (default: use system certificates)
 
 .PP
 \fB\-\-cache\-dir\fP=""
@@ -140,6 +144,10 @@ data after 'forget' was run successfully, see the 'prune' command.
 \fB\-\-tls\-client\-cert\fP=""
     path to a file containing PEM encoded TLS client certificate and private key
 
+.PP
+\fB\-v\fP, \fB\-\-verbose\fP[=0]
+    be verbose (specify \-\-verbose multiple times or level \fB\fCn\fR)
+
 
 .SH SEE ALSO
 .PP

doc/man/restic-generate.1

@@ -40,7 +40,7 @@ and the auto\-completion files for bash and zsh).
 .SH OPTIONS INHERITED FROM PARENT COMMANDS
 .PP
 \fB\-\-cacert\fP=[]
-    path to load root certificates from (default: use system certificates)
+    \fB\fCfile\fR to load root certificates from (default: use system certificates)
 
 .PP
 \fB\-\-cache\-dir\fP=""
@@ -90,6 +90,10 @@ and the auto\-completion files for bash and zsh).
 \fB\-\-tls\-client\-cert\fP=""
     path to a file containing PEM encoded TLS client certificate and private key
 
+.PP
+\fB\-v\fP, \fB\-\-verbose\fP[=0]
+    be verbose (specify \-\-verbose multiple times or level \fB\fCn\fR)
+
 
 .SH SEE ALSO
 .PP

doc/man/restic-init.1

@@ -27,7 +27,7 @@ The "init" command initializes a new repository.
 .SH OPTIONS INHERITED FROM PARENT COMMANDS
 .PP
 \fB\-\-cacert\fP=[]
-    path to load root certificates from (default: use system certificates)
+    \fB\fCfile\fR to load root certificates from (default: use system certificates)
 
 .PP
 \fB\-\-cache\-dir\fP=""
@@ -77,6 +77,10 @@ The "init" command initializes a new repository.
 \fB\-\-tls\-client\-cert\fP=""
     path to a file containing PEM encoded TLS client certificate and private key
 
+.PP
+\fB\-v\fP, \fB\-\-verbose\fP[=0]
+    be verbose (specify \-\-verbose multiple times or level \fB\fCn\fR)
+
 
 .SH SEE ALSO
 .PP

doc/man/restic-key.1

@@ -23,11 +23,15 @@ The "key" command manages keys (passwords) for accessing the repository.
 \fB\-h\fP, \fB\-\-help\fP[=false]
     help for key
 
+.PP
+\fB\-\-new\-password\-file\fP=""
+    the file from which to load a new password
+
 
 .SH OPTIONS INHERITED FROM PARENT COMMANDS
 .PP
 \fB\-\-cacert\fP=[]
-    path to load root certificates from (default: use system certificates)
+    \fB\fCfile\fR to load root certificates from (default: use system certificates)
 
 .PP
 \fB\-\-cache\-dir\fP=""
@@ -77,6 +81,10 @@ The "key" command manages keys (passwords) for accessing the repository.
 \fB\-\-tls\-client\-cert\fP=""
     path to a file containing PEM encoded TLS client certificate and private key
 
+.PP
+\fB\-v\fP, \fB\-\-verbose\fP[=0]
+    be verbose (specify \-\-verbose multiple times or level \fB\fCn\fR)
+
 
 .SH SEE ALSO
 .PP

doc/man/restic-list.1

@@ -27,7 +27,7 @@ The "list" command allows listing objects in the repository based on type.
 .SH OPTIONS INHERITED FROM PARENT COMMANDS
 .PP
 \fB\-\-cacert\fP=[]
-    path to load root certificates from (default: use system certificates)
+    \fB\fCfile\fR to load root certificates from (default: use system certificates)
 
 .PP
 \fB\-\-cache\-dir\fP=""
@@ -77,6 +77,10 @@ The "list" command allows listing objects in the repository based on type.
 \fB\-\-tls\-client\-cert\fP=""
     path to a file containing PEM encoded TLS client certificate and private key
 
+.PP
+\fB\-v\fP, \fB\-\-verbose\fP[=0]
+    be verbose (specify \-\-verbose multiple times or level \fB\fCn\fR)
+
 
 .SH SEE ALSO
 .PP