Add VERSION for 0.8.1

Replace manual CHANGELOG with generated one

.github/ISSUE_TEMPLATE.md

@@ -53,3 +53,11 @@ this is possible), the easier it is for the project developers to fix it!
 
 
 ## Do you have an idea how to solve the issue?
+
+## Did restic help you or made you happy in any way?
+
+<!--
+Answering this question is not required, but if you have anything positive to share, please do so here!
+Sometimes we get tired of reading bug reports all day and a little positive end note does wonders.
+Idea by Joey Hess, https://joeyh.name/blog/entry/two_holiday_stories/
+-->

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,31 @@
+<!--
+Thank you very much for contributing code or documentation to restic! Please
+fill out the following questions to make it easier for us to review your
+changes.
+
+You do not need to check all the boxes below all at once, feel free to take
+your time and add more commits. If you're done and ready for review, please
+check the last box.
+-->
+
+### What is the purpose of this change? What does it change?
+
+<!--
+Describe the changes here, as detailed as needed.
+-->
+
+### Was the change discussed in an issue or in the forum before?
+
+<!--
+Link issues and relevant forum posts here.
+-->
+
+### Checklist
+
+- [ ] I have read the [Contribution Guidelines](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#providing-patches)
+- [ ] I have added tests for all changes in this PR
+- [ ] I have added documentation for the changes (in the manual)
+- [ ] There's an entry in the `CHANGELOG.md` file that describe the changes for our users
+- [ ] I have run `gofmt` on the code in all commits
+- [ ] All commit messages are formatted in the same style as [the other commits in the repo](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#git-commits)
+- [ ] I'm done, this Pull Request is ready for review

Gopkg.lock

@@ -10,20 +10,20 @@
 [[projects]]
   name = "cloud.google.com/go"
   packages = ["compute/metadata"]
-  revision = "eaddaf6dd7ee35fd3c2420c8d27478db176b0485"
-  version = "v0.15.0"
+  revision = "2d3a6656c17a60b0815b7e06ab0be04eacb6e613"
+  version = "v0.16.0"
 
 [[projects]]
   name = "github.com/Azure/azure-sdk-for-go"
   packages = ["storage"]
-  revision = "509eea43b93cec2f3f17acbe2578ef58703923f8"
-  version = "v11.1.1-beta"
+  revision = "7692b0cef22674113fcf71cc17ac3ccc1a7fef48"
+  version = "v11.2.2-beta"
 
 [[projects]]
   name = "github.com/Azure/go-autorest"
   packages = ["autorest","autorest/adal","autorest/azure","autorest/date"]
-  revision = "7aa5b8a6f18b5c15910c767ab005fc4585221177"
-  version = "v9.1.1"
+  revision = "c67b24a8e30d876542a85022ebbdecf0e5a935e8"
+  version = "v9.4.1"
 
 [[projects]]
   name = "github.com/cenkalti/backoff"
@@ -47,7 +47,7 @@
   branch = "master"
   name = "github.com/dustin/go-humanize"
   packages = ["."]
-  revision = "77ed807830b4df581417e7f89eb81d4872832b72"
+  revision = "bb3d318650d48840a39aa21a027c6630e198e626"
 
 [[projects]]
   name = "github.com/elithrar/simple-scrypt"
@@ -58,14 +58,14 @@
 [[projects]]
   name = "github.com/go-ini/ini"
   packages = ["."]
-  revision = "5b3e00af70a9484542169a976dcab8d03e601a17"
-  version = "v1.30.0"
+  revision = "32e4c1e6bc4e7d0d8451aa6b75200d19e37a536a"
+  version = "v1.32.0"
 
 [[projects]]
   branch = "master"
   name = "github.com/golang/protobuf"
   packages = ["proto"]
-  revision = "1643683e1b54a9e88ad26d98f81400c8c9d9f4f9"
+  revision = "1e59b77b52bf8e4b449a57e6f79f21226d571845"
 
 [[projects]]
   name = "github.com/inconshreveable/mousetrap"
@@ -77,7 +77,7 @@
   branch = "master"
   name = "github.com/juju/ratelimit"
   packages = ["."]
-  revision = "5b9ff866471762aa2ab2dced63c9fb6f53921342"
+  revision = "59fac5042749a5afb9af70e813da1dd5474f0167"
 
 [[projects]]
   branch = "master"
@@ -100,8 +100,8 @@
 [[projects]]
   name = "github.com/minio/minio-go"
   packages = [".","pkg/credentials","pkg/encrypt","pkg/policy","pkg/s3signer","pkg/s3utils","pkg/set"]
-  revision = "4e0f567303d4cc90ceb055a451959fb9fc391fb9"
-  version = "3.0.3"
+  revision = "57a8ae886b49af6eb0d2c27c2d007ed2f71e1da5"
+  version = "4.0.3"
 
 [[projects]]
   branch = "master"
@@ -158,10 +158,10 @@
   version = "v1.0.3"
 
 [[projects]]
-  branch = "master"
   name = "github.com/spf13/cobra"
   packages = [".","doc"]
   revision = "7b2c5ac9fc04fc5efafb60700713d4fa609b777b"
+  version = "v0.0.1"
 
 [[projects]]
   name = "github.com/spf13/pflag"
@@ -173,31 +173,31 @@
   branch = "master"
   name = "golang.org/x/crypto"
   packages = ["curve25519","ed25519","ed25519/internal/edwards25519","pbkdf2","poly1305","scrypt","ssh","ssh/terminal"]
-  revision = "edd5e9b0879d13ee6970a50153d85b8fec9f7686"
+  revision = "94eea52f7b742c7cbe0b03b22f0c4c8631ece122"
 
 [[projects]]
   branch = "master"
   name = "golang.org/x/net"
   packages = ["context","context/ctxhttp"]
-  revision = "cd69bc3fc700721b709c3a59e16e24c67b58f6ff"
+  revision = "a8b9294777976932365dabb6640cf1468d95c70f"
 
 [[projects]]
   branch = "master"
   name = "golang.org/x/oauth2"
   packages = [".","google","internal","jws","jwt"]
-  revision = "bb50c06baba3d0c76f9d125c0719093e315b5b44"
+  revision = "f95fa95eaa936d9d87489b15d1d18b97c1ba9c28"
 
 [[projects]]
   branch = "master"
   name = "golang.org/x/sys"
   packages = ["unix","windows"]
-  revision = "8dbc5d05d6edcc104950cc299a1ce6641235bc86"
+  revision = "8b4580aae2a0dd0c231a45d3ccb8434ff533b840"
 
 [[projects]]
   branch = "master"
   name = "google.golang.org/api"
   packages = ["gensupport","googleapi","googleapi/internal/uritemplates","storage/v1"]
-  revision = "7afc123cf726cd2f253faa3e144d2ab65477b18f"
+  revision = "3a1d936b7575b82197a1fea0632218dd07b1e65c"
 
 [[projects]]
   name = "google.golang.org/appengine"
@@ -209,7 +209,7 @@
   branch = "v2"
   name = "gopkg.in/yaml.v2"
   packages = ["."]
-  revision = "eb3733d160e74a9c7e442f435eb3bea458e1d19f"
+  revision = "287cf08546ab5e7e37d55a84f7ed3fd1db036de5"
 
 [solve-meta]
   analyzer-name = "dep"

README.rst

@@ -51,7 +51,7 @@ Therefore, restic supports the following backends for storing backups natively:
 
 - `Local directory <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#local>`__
 - `sftp server (via SSH) <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#sftp>`__
-- `HTTP REST server <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server>`__ (`protocol <doc/rest_backend.rst>`__ `rest-server <https://github.com/restic/rest-server>`__)
+- `HTTP REST server <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server>`__ (`protocol <doc/100_references.rst#rest-backend>`__ `rest-server <https://github.com/restic/rest-server>`__)
 - `AWS S3 <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#amazon-s3>`__ (either from Amazon or using the `Minio <https://minio.io>`__ server)
 - `OpenStack Swift <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#openstack-swift>`__
 - `BackBlaze B2 <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#backblaze-b2>`__

VERSION

@@ -1 +1 @@
-0.8.0
+0.8.1

changelog/0.6.0/issue-953

@@ -0,0 +1,8 @@
+Enhancement: Make `forget` consistent
+
+The `forget` command was corrected to be more consistent in which snapshots are
+to be forgotten. It is possible that the new code removes more snapshots than
+before, so please review what would be deleted by using the `--dry-run` option.
+
+https://github.com/restic/restic/pull/957
+https://github.com/restic/restic/issues/953

changelog/0.6.0/issue-965

@@ -0,0 +1,11 @@
+Enhancement: Unify repository layout for all backends
+
+Up to now the s3 backend used a special repository layout. We've decided to
+unify the repository layout and implemented the default layout also for the s3
+backend. For creating a new repository on s3 with the default layout, use
+`restic -o s3.layout=default init`. For further commands the option is not
+necessary any more, restic will automatically detect the correct layout to use.
+A future version will switch to the default layout for new repositories.
+
+https://github.com/restic/restic/pull/966
+https://github.com/restic/restic/issues/965

changelog/0.6.0/pull-962

@@ -0,0 +1,15 @@
+Enhancement: Improve memory and runtime for the s3 backend
+
+We've updated the library used for accessing s3, switched to using a lower
+level API and added caching for some requests. This lead to a decrease in
+memory usage and a great speedup. In addition, we added benchmark functions for
+all backends, so we can track improvements over time. The Continuous
+Integration test service we're using (Travis) now runs the s3 backend tests not
+only against a Minio server, but also against the Amazon s3 live service, so we
+should be notified of any regressions much sooner.
+
+https://github.com/restic/restic/pull/962
+https://github.com/restic/restic/pull/960
+https://github.com/restic/restic/pull/946
+https://github.com/restic/restic/pull/938
+https://github.com/restic/restic/pull/883

changelog/0.6.1/issue-985

@@ -0,0 +1,8 @@
+Enhancement: Allow multiple parallel idle HTTP connections
+
+Backends based on HTTP now allow several idle connections in parallel. This
+is especially important for the REST backend, which (when used with a local
+server) may create a lot connections and exhaust available ports quickly.
+
+https://github.com/restic/restic/issues/985
+https://github.com/restic/restic/pull/986

changelog/0.6.1/pull-891

@@ -0,0 +1,6 @@
+Enhancement: Remove temporary path from binary in `build.go`
+
+The `build.go` now strips the temporary directory used for compilation from
+the binary. This is the first step in enabling reproducible builds.
+
+https://github.com/restic/restic/pull/981

changelog/0.6.1/pull-974

@@ -0,0 +1,7 @@
+Enhancement: Remove regular status reports
+
+Regular status report: We've removed the status report that was printed
+every 10 seconds when restic is run non-interactively. You can still force
+reporting the current status by sending a `USR1` signal to the process.
+
+https://github.com/restic/restic/pull/974

changelog/0.7.0/issue-1013

@@ -0,0 +1,9 @@
+Bugfix: Switch back to using the high-level minio-go API for s3
+
+For the s3 backend we're back to using the high-level API the s3 client library
+for uploading data, a few users reported dropped connections (which the library
+will automatically retry now).
+
+https://github.com/restic/restic/issues/1013
+https://github.com/restic/restic/issues/1023
+https://github.com/restic/restic/pull/1025

changelog/0.7.0/issue-1021

@@ -0,0 +1,7 @@
+Enhancement: Detect invalid backend name and print error
+
+restic now tries to detect when an invalid/unknown backend is used and
+returns an error message.
+
+https://github.com/restic/restic/issues/1021
+https://github.com/restic/restic/pull/1070

changelog/0.7.0/issue-1029

@@ -0,0 +1,8 @@
+Enhancement: Remove invalid pack files when `prune` is run
+
+The `prune` command has been improved and will now remove invalid pack files,
+for example files that have not been uploaded completely because a backup was
+interrupted.
+
+https://github.com/restic/restic/issues/1029
+https://github.com/restic/restic/pull/1036

changelog/0.7.0/issue-512

@@ -0,0 +1,4 @@
+Enhancement: Add Backblaze B2 backend
+
+https://github.com/restic/restic/issues/512
+https://github.com/restic/restic/pull/978

changelog/0.7.0/issue-636

@@ -0,0 +1,9 @@
+Enhancement: Add dirs `tags` and `hosts` to fuse mount
+
+The fuse mount now has two more directories: `tags` contains a subdir for
+each tag, which in turn contains only the snapshots that have this tag. The
+subdir `hosts` contains a subdir for each host that has a snapshot, and the
+subdir contains the snapshots for that host.
+
+https://github.com/restic/restic/issues/636
+https://github.com/restic/restic/pull/1050

changelog/0.7.0/issue-965

@@ -0,0 +1,8 @@
+Bugfix: Switch to `default` repo layout for the s3 backend
+
+The default layout for the s3 backend is now `default` (instead of `s3legacy`).
+Also, there's a new `migrate` command to convert an existing repo, it can be
+run like this: `restic migrate s3_layout`
+
+https://github.com/restic/restic/issues/965
+https://github.com/restic/restic/pull/1004

changelog/0.7.0/issue-989

@@ -0,0 +1,8 @@
+Enhancement: Improve performance of the `find` command
+
+Improved performance for the `find` command: Restic recognizes paths it has
+already checked for the files in question, so the number of backend requests
+is reduced a lot.
+
+https://github.com/restic/restic/issues/989
+https://github.com/restic/restic/pull/993

changelog/0.7.0/pull-975

@@ -0,0 +1,4 @@
+Enhancement: Add new backend for OpenStack Swift
+
+https://github.com/restic/restic/pull/975
+https://github.com/restic/restic/pull/648

changelog/0.7.0/pull-998

@@ -0,0 +1,5 @@
+Enhancement: Improve performance of the fuse mount
+
+Listing directories which contain large files now is significantly faster.
+
+https://github.com/restic/restic/pull/998

changelog/0.7.1/issue-1055

@@ -0,0 +1,10 @@
+Enhancement: Create subdirs below `data/` for local/sftp backends
+
+The local and sftp backends now create the subdirs below `data/` on
+open/init. This way, restic makes sure that they always exist. This is
+connected to an issue for the sftp server:
+
+https://github.com/restic/rest-server/pull/11#issuecomment-309879710
+https://github.com/restic/restic/issues/1055
+https://github.com/restic/restic/pull/1077
+https://github.com/restic/restic/pull/1105

changelog/0.7.1/issue-1067

@@ -0,0 +1,8 @@
+Enhancement: Allow loading credentials for s3 from IAM
+
+When no S3 credentials are specified in the environment variables, restic
+now tries to load credentials from an IAM instance profile when the s3
+backend is used.
+
+https://github.com/restic/restic/issues/1067
+https://github.com/restic/restic/pull/1086

changelog/0.7.1/issue-1073

@@ -0,0 +1,8 @@
+Enhancement: Add `migrate` cmd to migrate from `s3legacy` to `default` layout
+
+The `migrate` command for chaning the `s3legacy` layout to the `default` layout
+for s3 backends has been improved: It can now be restarted with `restic migrate
+--force s3_layout` and automatically retries operations on error.
+
+https://github.com/restic/restic/issues/1073
+https://github.com/restic/restic/pull/1075

changelog/0.7.1/issue-1081

@@ -0,0 +1,4 @@
+Enhancement: Clarify semantic for `--tasg` for the `forget` command
+
+https://github.com/restic/restic/issues/1081
+https://github.com/restic/restic/pull/1090

changelog/0.7.1/plul-1080

@@ -0,0 +1,5 @@
+Enhancement: Ignore chmod() errors on filesystems which do not support it
+
+https://github.com/restic/restic/pull/1080
+https://github.com/restic/restic/pull/1112
+

changelog/0.7.1/pull-1082

@@ -0,0 +1,3 @@
+Enhancement: Print stats on SIGINFO on Darwin and FreeBSD (ctrl+t)
+
+https://github.com/restic/restic/pull/1082

changelog/0.7.1/pull-1115

@@ -0,0 +1,9 @@
+Bugfix: Fix `prune`, only include existing files in indexes
+
+A bug was found (and corrected) in the index rebuilding after prune, which led
+to indexes which include blobs that were not present in the repo any more.
+There were already checks in place which detected this situation and aborted
+with an error message. A new run of either `prune` or `rebuild-index` corrected
+the index files. This is now fixed and a test has been added to detect this.
+
+https://github.com/restic/restic/pull/1115

changelog/0.7.2/issue-1132

@@ -0,0 +1,8 @@
+Enhancement: Make `key` command always prompt for a password
+
+The `key` command now prompts for a password even if the original password
+to access a repo has been specified via the `RESTIC_PASSWORD` environment
+variable or a password file.
+
+https://github.com/restic/restic/issues/1132
+https://github.com/restic/restic/pull/1133

changelog/0.7.2/issue-1167

@@ -0,0 +1,8 @@
+Bugfix: Do not create a local repo unless `init` is used
+
+When a restic command other than `init` is used with a local repository and the
+repository directory does not exist, restic creates the directory structure.
+That's an error, only the `init` command should create the dir.
+
+https://github.com/restic/restic/issues/1167
+https://github.com/restic/restic/pull/1182

changelog/0.7.2/issue-1179

@@ -0,0 +1,4 @@
+Enhancement: Resolve name conflicts, append a counter
+
+https://github.com/restic/restic/issues/1179
+https://github.com/restic/restic/pull/1209

changelog/0.7.2/issue-1208

@@ -0,0 +1,7 @@
+Enhancement: Add `--compact` to `snapshots` command
+
+The option `--compact` was added to the `snapshots` command to get a better
+overview of the snapshots in a repo. It limits each snapshot to a single line.
+
+https://github.com/restic/restic/issues/1218
+https://github.com/restic/restic/pull/1223

changelog/0.7.2/issue-317

@@ -0,0 +1,10 @@
+Enhancement: Add `--exclude-caches` and `--exclude-if-present`
+
+A new option `--exclude-caches` was added that allows excluding cache
+directories (that are tagged as such). This is a special case of a more generic
+option `--exclude-if-present` which excludes a directory if a file with a
+specific name (and contents) is present.
+
+https://github.com/restic/restic/issues/317
+https://github.com/restic/restic/pull/1170
+https://github.com/restic/restic/pull/1224

changelog/0.7.2/issues-697

@@ -0,0 +1,4 @@
+Enhancement: Automatically generate man pages for all restic commands
+
+https://github.com/restic/restic/issues/697
+https://github.com/restic/restic/pull/1147

changelog/0.7.2/pull-1044

@@ -0,0 +1,3 @@
+Enhancement: Improve `restore`, do not traverse/load excluded directories
+
+https://github.com/restic/restic/pull/1044

changelog/0.7.2/pull-1061

@@ -0,0 +1,3 @@
+Enhancement: Add Dockerfile and official Docker image
+
+https://github.com/restic/restic/pull/1061

changelog/0.7.2/pull-1126

@@ -0,0 +1,7 @@
+Enhancement: Use the standard Go git repository layout, use `dep` for vendoring
+
+The git repository layout was changed to resemble the layout typically used in
+Go projects, we're not using `gb` for building restic any more and vendoring
+the dependencies is now taken care of by `dep`.
+
+https://github.com/restic/restic/pull/1126

changelog/0.7.2/pull-1134

@@ -0,0 +1,5 @@
+Enhancement: Add support for storing backups on Google Cloud Storage
+
+https://github.com/restic/restic/pull/1134
+https://github.com/restic/restic/pull/1052
+https://github.com/restic/restic/issues/211

changelog/0.7.2/pull-1144

@@ -0,0 +1,3 @@
+Enhancement: Properly report errors when reading files with exclude patterns.
+
+https://github.com/restic/restic/pull/1144

changelog/0.7.2/pull-1149

@@ -0,0 +1,8 @@
+Enhancement: Add support for storing backups on Microsoft Azure Blob Storage
+
+The library we're using to access the service requires Go 1.8, so restic now
+needs at least Go 1.8.
+
+https://github.com/restic/restic/pull/1149
+https://github.com/restic/restic/pull/1059
+https://github.com/restic/restic/issues/609

changelog/0.7.2/pull-1164

@@ -0,0 +1,3 @@
+Bugfix: Make the `key remove` command behave as documented
+
+https://github.com/restic/restic/pull/1164

changelog/0.7.2/pull-1191

@@ -0,0 +1,8 @@
+Bugfix: Make sure to write profiling files on interrupt
+
+Since a few releases restic had the ability to write profiling files for memory
+and CPU usage when `debug` is enabled. It was discovered that when restic is
+interrupted (ctrl+c is pressed), the proper shutdown hook is not run. This is
+now corrected.
+
+https://github.com/restic/restic/pull/1191

changelog/0.7.2/pull-1196

@@ -0,0 +1,3 @@
+Enhancement: Add `--group-by` to `forget` command for flexible grouping
+
+https://github.com/restic/restic/pull/1196

changelog/0.7.2/pull-1203

@@ -0,0 +1,5 @@
+Enhancement: Print stats on all BSD systems when SIGINFO (ctrl+t) is received
+
+https://github.com/restic/restic/pull/1203
+https://github.com/restic/restic/pull/1082#issuecomment-326279920
+

changelog/0.7.2/pull-1205

@@ -0,0 +1,3 @@
+Enhancement: Allow specifying time/date for a backup with `--time`
+
+https://github.com/restic/restic/pull/1205

changelog/0.7.3/issue-1246

@@ -0,0 +1,9 @@
+Bugfix: List all files stored in Google Cloud Storage
+
+For large backups stored in Google Cloud Storage, the `prune` command fails
+because listing only returns the first 1000 files. This has been corrected, no
+data is lost in the process. In addition, a plausibility check was added to
+`prune`.
+
+https://github.com/restic/restic/issues/1246
+https://github.com/restic/restic/pull/1247

changelog/0.8.0/issue-1102

@@ -0,0 +1,9 @@
+Enhancement: Add subdirectory `ids` to fuse mount
+
+The fuse mount now has an `ids` subdirectory which contains the snapshots below
+their (short) IDs.
+
+https://github.com/restic/restic/issues/1102
+https://github.com/restic/restic/pull/1299
+https://github.com/restic/restic/pull/1320
+

changelog/0.8.0/issue-1114

@@ -0,0 +1,10 @@
+Enhancement: Add `--cacert` to specify TLS certificates to check against
+
+We've added the `--cacert` option which can be used to pass one (or more) CA
+certificates to restic. These are used in addition to the system CA
+certificates to verify HTTPS certificates (e.g. for the REST backend).
+
+https://github.com/restic/restic/issues/1114
+https://github.com/restic/restic/pull/1276
+
+

changelog/0.8.0/issue-1216

@@ -0,0 +1,9 @@
+Enhancement: Add upload/download limiting
+
+We've added support for rate limiting through `--limit-upload` and
+`--limit-download` flags.
+
+https://github.com/restic/restic/issues/1216
+https://github.com/restic/restic/pull/1336
+https://github.com/restic/restic/pull/1358
+

changelog/0.8.0/issue-1256

@@ -0,0 +1,7 @@
+Bugfix: Re-enable workaround for S3 backend
+
+We've re-enabled a workaround for `minio-go` (the library we're using to
+access s3 backends), this reduces memory usage.
+
+https://github.com/restic/restic/issues/1256
+https://github.com/restic/restic/pull/1267

changelog/0.8.0/issue-1271

@@ -0,0 +1,6 @@
+Enhancement: Cache results for excludes for `backup`
+
+The `backup` command now caches the result of excludes for a directory.
+
+https://github.com/restic/restic/issues/1271
+https://github.com/restic/restic/pull/1326

changelog/0.8.0/issue-1274

@@ -0,0 +1,9 @@
+Enhancement: Add `generate` command, replaces `manpage` and `autocomplete`
+
+The `generate` command has been added, which replaces the now removed
+commands `manpage` and `autocomplete`. This release of restic contains the
+most recent manpages in `doc/man` and the auto-completion files for bash and
+zsh in `doc/bash-completion.sh` and `doc/zsh-completion.zsh`
+
+https://github.com/restic/restic/issues/1274
+https://github.com/restic/restic/pull/1282

changelog/0.8.0/issue-1291

@@ -0,0 +1,8 @@
+Bugfix: Reuse backend TCP connections to BackBlaze B2
+
+A bug was discovered in the library we're using to access Backblaze, it now
+reuses already established TCP connections which should be a lot faster and
+not cause network failures any more.
+
+https://github.com/restic/restic/issues/1291
+https://github.com/restic/restic/pull/1301

changelog/0.8.0/issue-1367

@@ -0,0 +1,7 @@
+Enhancement: Allow comments in files read from via `--file-from`
+
+When the list of files/dirs to be saved is read from a file with
+`--files-from`, comment lines (starting with `#`) are now ignored.
+
+https://github.com/restic/restic/issues/1367
+https://github.com/restic/restic/pull/1368

changelog/0.8.0/issue-1445

@@ -0,0 +1,18 @@
+Security: Prevent writing outside the target directory during restore
+
+A vulnerability was found in the restic restorer, which allowed attackers in
+special circumstances to restore files to a location outside of the target
+directory. Due to the circumstances we estimate this to be a low-risk
+vulnerability, but urge all users to upgrade to the latest version of restic.
+
+Exploiting the vulnerability requires a Linux/Unix system which saves backups
+via restic and a Windows systems which restores files from the repo. In
+addition, the attackers need to be able to create create files with arbitrary
+names which are then saved to the restic repo. For example, by creating a file
+named "..\test.txt" (which is a perfectly legal filename on Linux) and
+restoring a snapshot containing this file on Windows, it would be written to
+the parent of the target directory.
+
+We'd like to thank Tyler Spivey for reporting this responsibly!
+
+https://github.com/restic/restic/pull/1445

changelog/0.8.0/issue-448

@@ -0,0 +1,9 @@
+Enhancement: sftp backend prompts for password
+
+The sftp backend now prompts for the password if a password is necessary for
+login.
+
+https://github.com/restic/restic/issues/448
+https://github.com/restic/restic/pull/1270
+
+

changelog/0.8.0/issue-510

@@ -0,0 +1,7 @@
+Enhancement: Add `dump` command
+
+We've added the `dump` command which prints a file from a snapshot to
+stdout. This can e.g. be used to restore files read with `backup --stdin`.
+
+https://github.com/restic/restic/issues/510
+https://github.com/restic/restic/pull/1346

changelog/0.8.0/pull-1040

@@ -0,0 +1,23 @@
+Enhancement: Add local metadata cache
+
+We've added a local cache for metadata so that restic doesn't need to load
+all metadata (snapshots, indexes, ...) from the repo each time it starts. By
+default the cache is active, but there's a new global option `--no-cache`
+that can be used to disable the cache. By deafult, the cache a standard
+cache folder for the OS, which can be overridden with `--cache-dir`.  The
+cache will automatically populate, indexes and snapshots are saved as they
+are loaded. Cache directories for repos that haven't been used recently can
+automatically be removed by restic with the `--cleanup-cache` option.
+
+A related change was to by default create pack files in the repo that contain
+either data or metadata, not both mixed together. This allows easy caching of
+only the metadata files. The next run of `restic prune` will untangle mixed
+files automatically.
+
+https://github.com/restic/restic/pull/1040
+https://github.com/restic/restic/issues/29
+https://github.com/restic/restic/issues/738
+https://github.com/restic/restic/issues/282
+https://github.com/restic/restic/pull/1287
+https://github.com/restic/restic/pull/1436
+https://github.com/restic/restic/pull/1265

changelog/0.8.0/pull-1249

@@ -0,0 +1,6 @@
+Enhancement: Add `latest` symlink in fuse mount
+
+The directory structure in the fuse mount now exposes a symlink `latest`
+which points to the latest snapshot in that particular directory.
+
+https://github.com/restic/restic/pull/1249

changelog/0.8.0/pull-1269

@@ -0,0 +1,6 @@
+Enhancement: Add `--compact` to `forget` command
+
+The option `--compact` was added to the `forget` command to provide the same
+compact view as the `snapshots` command.
+
+https://github.com/restic/restic/pull/1269

changelog/0.8.0/pull-1281

@@ -0,0 +1,7 @@
+Enhancement: Google Cloud Storage backend needs less permissions
+
+The Google Cloud Storage backend no longer requires the service account to
+have the `storage.buckets.get` permission ("Storage Admin" role) in `restic
+init` if the bucket already exists.
+
+https://github.com/restic/restic/pull/1281

changelog/0.8.0/pull-1317

@@ -0,0 +1,7 @@
+Bugfix: Run prune when `forget --prune` is called with just snapshot IDs
+
+A bug in the `forget` command caused `prune` not to be run when `--prune` was
+specified without a policy, e.g. when only snapshot IDs that should be
+forgotten are listed manually.
+
+https://github.com/restic/restic/pull/1317

changelog/0.8.0/pull-1319

@@ -0,0 +1,8 @@
+Enhancement: Make `check` print `no errors found` explicitly
+
+The `check` command now explicetly prints `No errors were found` when no errors
+could be found.
+
+https://github.com/restic/restic/pull/1319
+https://github.com/restic/restic/issues/1303
+

changelog/0.8.0/pull-1353

@@ -0,0 +1,3 @@
+Enhancement: Retry failed backend requests
+
+https://github.com/restic/restic/pull/1353

changelog/0.8.0/pull-1437

@@ -0,0 +1,10 @@
+Bugfix: Remove implicit path `/restic` for the s3 backend
+
+The s3 backend used the subdir `restic` within a bucket if no explicit path
+after the bucket name was specified. Since this version, restic does not use
+this default path any more. If you created a repo on s3 in a bucket without
+specifying a path within the bucket, you need to add `/restic` at the end of
+the repository specification to access your repo: `s3:s3.amazonaws.com/bucket/restic`
+
+https://github.com/restic/restic/pull/1437
+https://github.com/restic/restic/issues/1292

changelog/0.8.1/issue-1457

@@ -0,0 +1,4 @@
+Bugfix: Improve s3 backend with DigitalOcean Spaces
+
+https://github.com/restic/restic/pull/1459
+https://github.com/restic/restic/issues/1457

changelog/0.8.1/pull-1436

@@ -0,0 +1,9 @@
+Enhancement: Add code to detect old cache directories
+
+We've added code to detect old cache directories of repositories that
+haven't been used in a long time, restic now prints a note when it detects
+that such dirs exist. Also, the option `--cleanup-cache` was added to
+automatically remove such directories. That's not a problem because the
+cache will be rebuild once a repo is accessed again.
+
+https://github.com/restic/restic/pull/1436

changelog/0.8.1/pull-1439

@@ -0,0 +1,6 @@
+Enhancement: Improve cancellation logic
+
+The cancellation logic was improved, restic can now shut down cleanly when
+requested to do so (e.g. via ctrl+c).
+
+https://github.com/restic/restic/pull/1439

changelog/0.8.1/pull-1452

@@ -0,0 +1,9 @@
+Change: Do not save atime by default
+
+By default, the access time for files and dirs is not saved any more. It is
+not possible to reliably disable updating the access time during a backup,
+so for the next backup the access time is different again. This means a lot
+of metadata is saved. If you want to save the access time anyway, pass
+`--with-atime` to the `backup` command.
+
+https://github.com/restic/restic/pull/1452

changelog/0.8.1/pull-1454

@@ -0,0 +1,6 @@
+Bugfix: Correct cache dir location for Windows and Darwin
+
+The cache directory on Windows and Darwin was not correct, instead the
+directory `.cache` was used.
+
+https://github.com/restic/restic/pull/1454

changelog/0.8.1/pull-1459

@@ -0,0 +1,9 @@
+Bugfix: Disable handling SIGPIPE
+
+We've disabled handling SIGPIPE again. Turns out, writing to broken TCP
+connections also raised SIGPIPE, so restic exits on the first write to a
+broken connection. Instead, restic should retry the request.
+
+https://github.com/restic/restic/pull/1459
+https://github.com/restic/restic/issues/1457
+https://github.com/restic/restic/issues/1466

changelog/0.8.1/pull-1462

@@ -0,0 +1,8 @@
+Enhancement: Add the `diff` command
+
+The command `diff` was added, it allows comparing two snapshots and listing
+all differences.
+
+https://github.com/restic/restic/issues/11
+https://github.com/restic/restic/issues/1460
+https://github.com/restic/restic/pull/1462

changelog/CHANGELOG.tmpl

@@ -0,0 +1,32 @@
+{{- range $changes := . }}{{ with $changes -}}
+Changelog for restic {{ .Version }} ({{ .Date }})
+=======================================
+
+The following sections list the changes in restic {{ .Version }} relevant to
+restic users. The changes are ordered by importance.
+
+Summary
+-------
+{{ range $entry := .Entries }}{{ with $entry }}
+ * {{ .TypeShort }} #{{ .PrimaryID }}: {{ .Title }}
+{{- end }}{{ end }}
+
+Details
+-------
+{{ range $entry := .Entries }}{{ with $entry }}
+ * {{ .Type }} #{{ .PrimaryID }}: {{ .Title }}
+{{ range $par := .Paragraphs }}
+   {{ wrap $par 80 3 }}
+{{ end -}}
+{{ range $id := .Issues }}
+   https://github.com/restic/restic/issues/{{ $id -}}
+{{ end -}}
+{{ range $id := .PRs }}
+   https://github.com/restic/restic/pull/{{ $id -}}
+{{ end -}}
+{{ range $url := .OtherURLs }}
+   {{ $url -}}
+{{ end }}
+{{ end }}{{ end }}
+
+{{ end }}{{ end -}}

changelog/changelog-entry.tmpl

@@ -0,0 +1,12 @@
+Bugfix: Fix behavior for foobar (in present tense)
+
+We've fixed the behavior for foobar, a long-standing annoyance for restic
+users.
+
+The text in the paragraphs is written in past tense. The last section is a list
+of issue URLs, PR URLs and other URLs. The first issue ID (or the first PR ID,
+in case there aren't any issue links) is used as the primary ID.
+
+https://github.com/restic/restic/issues/1234
+https://github.com/restic/restic/pull/55555
+https://forum.restic/.net/foo/bar/baz

changelog/changelog-github.tmpl

@@ -0,0 +1,32 @@
+{{- range $changes := . }}{{ with $changes -}}
+Changelog for restic {{ .Version }} ({{ .Date }})
+=======================================
+
+The following sections list the changes in restic {{ .Version }} relevant to
+restic users. The changes are ordered by importance.
+
+Summary
+-------
+{{ range $entry := .Entries }}{{ with $entry }}
+ * {{ .TypeShort }} [#{{ .PrimaryID }}]({{ .PrimaryURL }}): {{ .Title }}
+{{- end }}{{ end }}
+
+Details
+-------
+{{ range $entry := .Entries }}{{ with $entry }}
+ * {{ .Type }} #{{ .PrimaryID }}: {{ .Title }}
+{{ range $par := .Paragraphs }}
+   {{ $par }}
+{{ end }}
+   {{ range $id := .Issues -}}
+[{{ $id }}](https://github.com/restic/restic/issues/{{ $id -}})
+{{- end -}}
+{{ range $id := .PRs -}}
+{{ ` ` }}[#{{ $id }}](https://github.com/restic/restic/pull/{{ $id -}})
+{{- end -}}
+{{ ` ` }}{{ range $url := .OtherURLs -}}
+{{ $url -}}
+{{- end }}
+{{ end }}{{ end }}
+
+{{ end }}{{ end -}}

changelog/releases

@@ -0,0 +1,13 @@
+# This file lists all versions for the changelog. Each line consists of the
+# version string, followed by an optional release date.
+# 
+# The resulting changelog generated by `calens` will list all versions in
+# exactly this order.
+0.8.1
+0.8.0 2017-11-26
+0.7.3 2017-09-20
+0.7.2 2017-09-13
+0.7.1 2017-07-22
+0.7.0 2017-07-01
+0.6.1 2017-06-01
+0.6.0 2017-05-29

cmd/restic/cleanup.go

@@ -25,16 +25,14 @@ func init() {
 	InstallSignalHandler()
 }
 
-// InstallSignalHandler listens for SIGINT and SIGPIPE, and triggers the cleanup handlers.
+// InstallSignalHandler listens for SIGINT, and triggers the cleanup handlers.
 func InstallSignalHandler() {
 	signal.Notify(cleanupHandlers.ch, syscall.SIGINT)
-	signal.Notify(cleanupHandlers.ch, syscall.SIGPIPE)
 }
 
-// SuspendSignalHandler removes the signal handler for SIGINT and SIGPIPE.
+// SuspendSignalHandler removes the signal handler for SIGINT.
 func SuspendSignalHandler() {
 	signal.Reset(syscall.SIGINT)
-	signal.Reset(syscall.SIGPIPE)
 }
 
 // AddCleanupHandler adds the function f to the list of cleanup handlers so
@@ -69,7 +67,7 @@ func RunCleanupHandlers() {
 	cleanupHandlers.list = nil
 }
 
-// CleanupHandler handles the SIGINT and SIGPIPE signals.
+// CleanupHandler handles the SIGINT signals.
 func CleanupHandler(c <-chan os.Signal) {
 	for s := range c {
 		debug.Log("signal %v received, cleaning up", s)

cmd/restic/cmd_backup.go

@@ -2,10 +2,10 @@ package main
 
 import (
 	"bufio"
-	"context"
 	"fmt"
 	"io"
 	"os"
+	"path"
 	"path/filepath"
 	"strings"
 	"time"
@@ -65,6 +65,7 @@ type BackupOptions struct {
 	Hostname         string
 	FilesFrom        string
 	TimeStamp        string
+	WithAtime        bool
 }
 
 var backupOptions BackupOptions
@@ -86,6 +87,7 @@ func init() {
 	f.StringVar(&backupOptions.Hostname, "hostname", "", "set the `hostname` for the snapshot manually. To prevent an expensive rescan use the \"parent\" flag")
 	f.StringVar(&backupOptions.FilesFrom, "files-from", "", "read the files to backup from file (can be combined with file args)")
 	f.StringVar(&backupOptions.TimeStamp, "time", "", "time of the backup (ex. '2012-11-01 22:08:41') (default: now)")
+	f.BoolVar(&backupOptions.WithAtime, "with-atime", false, "store the atime for all files and directories")
 }
 
 func newScanProgress(gopts GlobalOptions) *restic.Progress {
@@ -235,10 +237,16 @@ func readBackupFromStdin(opts BackupOptions, gopts GlobalOptions, args []string)
 		return errors.Fatal("when reading from stdin, no additional files can be specified")
 	}
 
-	if opts.StdinFilename == "" {
+	fn := opts.StdinFilename
+
+	if fn == "" {
 		return errors.Fatal("filename for backup from stdin must not be empty")
 	}
 
+	if filepath.Base(fn) != fn || path.Base(fn) != fn {
+		return errors.Fatal("filename is invalid (may not contain a directory, slash or backslash)")
+	}
+
 	if gopts.password == "" {
 		return errors.Fatal("unable to read password from stdin when data is to be read from stdin, use --password-file or $RESTIC_PASSWORD")
 	}
@@ -254,7 +262,7 @@ func readBackupFromStdin(opts BackupOptions, gopts GlobalOptions, args []string)
 		return err
 	}
 
-	err = repo.LoadIndex(context.TODO())
+	err = repo.LoadIndex(gopts.ctx)
 	if err != nil {
 		return err
 	}
@@ -265,7 +273,7 @@ func readBackupFromStdin(opts BackupOptions, gopts GlobalOptions, args []string)
 		Hostname:   opts.Hostname,
 	}
 
-	_, id, err := r.Archive(context.TODO(), opts.StdinFilename, os.Stdin, newArchiveStdinProgress(gopts))
+	_, id, err := r.Archive(gopts.ctx, fn, os.Stdin, newArchiveStdinProgress(gopts))
 	if err != nil {
 		return err
 	}
@@ -372,9 +380,8 @@ func runBackup(opts BackupOptions, gopts GlobalOptions, args []string) error {
 		opts.ExcludeIfPresent = append(opts.ExcludeIfPresent, "CACHEDIR.TAG:Signature: 8a477f597d28d172789f06886806bc55")
 	}
 
-	rc := &rejectionCache{}
 	for _, spec := range opts.ExcludeIfPresent {
-		f, err := rejectIfPresent(spec, rc)
+		f, err := rejectIfPresent(spec)
 		if err != nil {
 			return err
 		}
@@ -403,7 +410,7 @@ func runBackup(opts BackupOptions, gopts GlobalOptions, args []string) error {
 		rejectFuncs = append(rejectFuncs, f)
 	}
 
-	err = repo.LoadIndex(context.TODO())
+	err = repo.LoadIndex(gopts.ctx)
 	if err != nil {
 		return err
 	}
@@ -422,7 +429,7 @@ func runBackup(opts BackupOptions, gopts GlobalOptions, args []string) error {
 
 	// Find last snapshot to set it as parent, if not already set
 	if !opts.Force && parentSnapshotID == nil {
-		id, err := restic.FindLatestSnapshot(context.TODO(), repo, target, []restic.TagList{}, opts.Hostname)
+		id, err := restic.FindLatestSnapshot(gopts.ctx, repo, target, []restic.TagList{}, opts.Hostname)
 		if err == nil {
 			parentSnapshotID = &id
 		} else if err != restic.ErrNoSnapshotFound {
@@ -453,6 +460,7 @@ func runBackup(opts BackupOptions, gopts GlobalOptions, args []string) error {
 	arch := archiver.New(repo)
 	arch.Excludes = opts.Excludes
 	arch.SelectFilter = selectFilter
+	arch.WithAccessTime = opts.WithAtime
 
 	arch.Warn = func(dir string, fi os.FileInfo, err error) {
 		// TODO: make ignoring errors configurable
@@ -467,7 +475,7 @@ func runBackup(opts BackupOptions, gopts GlobalOptions, args []string) error {
 		}
 	}
 
-	_, id, err := arch.Snapshot(context.TODO(), newArchiveProgress(gopts, stat), target, opts.Tags, opts.Hostname, parentSnapshotID, timeStamp)
+	_, id, err := arch.Snapshot(gopts.ctx, newArchiveProgress(gopts, stat), target, opts.Tags, opts.Hostname, parentSnapshotID, timeStamp)
 	if err != nil {
 		return err
 	}

cmd/restic/cmd_cat.go

@@ -1,7 +1,6 @@
 package main
 
 import (
-	"context"
 	"encoding/json"
 	"fmt"
 	"os"
@@ -75,7 +74,7 @@ func runCat(gopts GlobalOptions, args []string) error {
 		fmt.Println(string(buf))
 		return nil
 	case "index":
-		buf, err := repo.LoadAndDecrypt(context.TODO(), restic.IndexFile, id)
+		buf, err := repo.LoadAndDecrypt(gopts.ctx, restic.IndexFile, id)
 		if err != nil {
 			return err
 		}
@@ -85,7 +84,7 @@ func runCat(gopts GlobalOptions, args []string) error {
 
 	case "snapshot":
 		sn := &restic.Snapshot{}
-		err = repo.LoadJSONUnpacked(context.TODO(), restic.SnapshotFile, id, sn)
+		err = repo.LoadJSONUnpacked(gopts.ctx, restic.SnapshotFile, id, sn)
 		if err != nil {
 			return err
 		}
@@ -100,7 +99,7 @@ func runCat(gopts GlobalOptions, args []string) error {
 		return nil
 	case "key":
 		h := restic.Handle{Type: restic.KeyFile, Name: id.String()}
-		buf, err := backend.LoadAll(context.TODO(), repo.Backend(), h)
+		buf, err := backend.LoadAll(gopts.ctx, repo.Backend(), h)
 		if err != nil {
 			return err
 		}
@@ -127,7 +126,7 @@ func runCat(gopts GlobalOptions, args []string) error {
 		fmt.Println(string(buf))
 		return nil
 	case "lock":
-		lock, err := restic.LoadLock(context.TODO(), repo, id)
+		lock, err := restic.LoadLock(gopts.ctx, repo, id)
 		if err != nil {
 			return err
 		}
@@ -143,7 +142,7 @@ func runCat(gopts GlobalOptions, args []string) error {
 	}
 
 	// load index, handle all the other types
-	err = repo.LoadIndex(context.TODO())
+	err = repo.LoadIndex(gopts.ctx)
 	if err != nil {
 		return err
 	}
@@ -151,7 +150,7 @@ func runCat(gopts GlobalOptions, args []string) error {
 	switch tpe {
 	case "pack":
 		h := restic.Handle{Type: restic.DataFile, Name: id.String()}
-		buf, err := backend.LoadAll(context.TODO(), repo.Backend(), h)
+		buf, err := backend.LoadAll(gopts.ctx, repo.Backend(), h)
 		if err != nil {
 			return err
 		}
@@ -173,7 +172,7 @@ func runCat(gopts GlobalOptions, args []string) error {
 			blob := list[0]
 
 			buf := make([]byte, blob.Length)
-			n, err := repo.LoadBlob(context.TODO(), t, id, buf)
+			n, err := repo.LoadBlob(gopts.ctx, t, id, buf)
 			if err != nil {
 				return err
 			}

cmd/restic/cmd_check.go

@@ -1,7 +1,6 @@
 package main
 
 import (
-	"context"
 	"fmt"
 	"os"
 	"time"
@@ -104,7 +103,7 @@ func runCheck(opts CheckOptions, gopts GlobalOptions, args []string) error {
 	chkr := checker.New(repo)
 
 	Verbosef("load indexes\n")
-	hints, errs := chkr.LoadIndex(context.TODO())
+	hints, errs := chkr.LoadIndex(gopts.ctx)
 
 	dupFound := false
 	for _, hint := range hints {
@@ -129,7 +128,7 @@ func runCheck(opts CheckOptions, gopts GlobalOptions, args []string) error {
 	errChan := make(chan error)
 
 	Verbosef("check all packs\n")
-	go chkr.Packs(context.TODO(), errChan)
+	go chkr.Packs(gopts.ctx, errChan)
 
 	for err := range errChan {
 		errorsFound = true
@@ -138,7 +137,7 @@ func runCheck(opts CheckOptions, gopts GlobalOptions, args []string) error {
 
 	Verbosef("check snapshots, trees and blobs\n")
 	errChan = make(chan error)
-	go chkr.Structure(context.TODO(), errChan)
+	go chkr.Structure(gopts.ctx, errChan)
 
 	for err := range errChan {
 		errorsFound = true
@@ -165,7 +164,7 @@ func runCheck(opts CheckOptions, gopts GlobalOptions, args []string) error {
 		p := newReadProgress(gopts, restic.Stat{Blobs: chkr.CountPacks()})
 		errChan := make(chan error)
 
-		go chkr.ReadData(context.TODO(), p, errChan)
+		go chkr.ReadData(gopts.ctx, p, errChan)
 
 		for err := range errChan {
 			errorsFound = true

cmd/restic/cmd_debug.go

@@ -183,7 +183,7 @@ func runDebugDump(gopts GlobalOptions, args []string) error {
 		}
 	}
 
-	err = repo.LoadIndex(context.TODO())
+	err = repo.LoadIndex(gopts.ctx)
 	if err != nil {
 		return err
 	}

cmd/restic/cmd_diff.go

@@ -0,0 +1,356 @@
+package main
+
+import (
+	"context"
+	"path"
+	"reflect"
+	"sort"
+
+	"github.com/restic/restic/internal/debug"
+	"github.com/restic/restic/internal/errors"
+	"github.com/restic/restic/internal/repository"
+	"github.com/restic/restic/internal/restic"
+	"github.com/spf13/cobra"
+)
+
+var cmdDiff = &cobra.Command{
+	Use:   "diff snapshot-ID snapshot-ID",
+	Short: "Show differences between two snapshots",
+	Long: `
+The "diff" command shows differences from the first to the second snapshot. The
+first characters in each line display what has happened to a particular file or
+directory:
+
+ +  The item was added
+ -  The item was removed
+ U  The metadata (access mode, timestamps, ...) for the item was updated
+ M  The file's content was modified
+ T  The type was changed, e.g. a file was made a symlink
+`,
+	DisableAutoGenTag: true,
+	RunE: func(cmd *cobra.Command, args []string) error {
+		return runDiff(diffOptions, globalOptions, args)
+	},
+}
+
+// DiffOptions collects all options for the diff command.
+type DiffOptions struct {
+	ShowMetadata bool
+}
+
+var diffOptions DiffOptions
+
+func init() {
+	cmdRoot.AddCommand(cmdDiff)
+
+	f := cmdDiff.Flags()
+	f.BoolVar(&diffOptions.ShowMetadata, "metadata", false, "print changes in metadata")
+}
+
+func loadSnapshot(ctx context.Context, repo *repository.Repository, desc string) (*restic.Snapshot, error) {
+	id, err := restic.FindSnapshot(repo, desc)
+	if err != nil {
+		return nil, err
+	}
+
+	return restic.LoadSnapshot(ctx, repo, id)
+}
+
+// Comparer collects all things needed to compare two snapshots.
+type Comparer struct {
+	repo restic.Repository
+	opts DiffOptions
+}
+
+// DiffStat collects stats for all types of items.
+type DiffStat struct {
+	Files, Dirs, Others  int
+	DataBlobs, TreeBlobs int
+	Bytes                int
+}
+
+// Add adds stats information for node to s.
+func (s *DiffStat) Add(node *restic.Node) {
+	if node == nil {
+		return
+	}
+
+	switch node.Type {
+	case "file":
+		s.Files++
+	case "dir":
+		s.Dirs++
+	default:
+		s.Others++
+	}
+}
+
+// addBlobs adds the blobs of node to s.
+func addBlobs(bs restic.BlobSet, node *restic.Node) {
+	if node == nil {
+		return
+	}
+
+	switch node.Type {
+	case "file":
+		for _, blob := range node.Content {
+			h := restic.BlobHandle{
+				ID:   blob,
+				Type: restic.DataBlob,
+			}
+			bs.Insert(h)
+		}
+	case "dir":
+		h := restic.BlobHandle{
+			ID:   *node.Subtree,
+			Type: restic.TreeBlob,
+		}
+		bs.Insert(h)
+	}
+}
+
+// DiffStats collects the differences between two snapshots.
+type DiffStats struct {
+	ChangedFiles            int
+	Added                   DiffStat
+	Removed                 DiffStat
+	BlobsBefore, BlobsAfter restic.BlobSet
+}
+
+// NewDiffStats creates new stats for a diff run.
+func NewDiffStats() *DiffStats {
+	return &DiffStats{
+		BlobsBefore: restic.NewBlobSet(),
+		BlobsAfter:  restic.NewBlobSet(),
+	}
+}
+
+// updateBlobs updates the blob counters in the stats struct.
+func updateBlobs(repo restic.Repository, blobs restic.BlobSet, stats *DiffStat) {
+	for h := range blobs {
+		switch h.Type {
+		case restic.DataBlob:
+			stats.DataBlobs++
+		case restic.TreeBlob:
+			stats.TreeBlobs++
+		}
+
+		size, err := repo.LookupBlobSize(h.ID, h.Type)
+		if err != nil {
+			Warnf("unable to find blob size for %v: %v\n", h, err)
+			continue
+		}
+
+		stats.Bytes += int(size)
+	}
+}
+
+func (c *Comparer) printDir(ctx context.Context, mode string, stats *DiffStat, blobs restic.BlobSet, prefix string, id restic.ID) error {
+	debug.Log("print %v tree %v", mode, id)
+	tree, err := c.repo.LoadTree(ctx, id)
+	if err != nil {
+		return err
+	}
+
+	for _, node := range tree.Nodes {
+		name := path.Join(prefix, node.Name)
+		if node.Type == "dir" {
+			name += "/"
+		}
+		Printf("%-5s%v\n", mode, name)
+		stats.Add(node)
+		addBlobs(blobs, node)
+
+		if node.Type == "dir" {
+			err := c.printDir(ctx, mode, stats, blobs, name, *node.Subtree)
+			if err != nil {
+				Warnf("error: %v\n", err)
+			}
+		}
+	}
+
+	return nil
+}
+
+func uniqueNodeNames(tree1, tree2 *restic.Tree) (tree1Nodes, tree2Nodes map[string]*restic.Node, uniqueNames []string) {
+	names := make(map[string]struct{})
+	tree1Nodes = make(map[string]*restic.Node)
+	for _, node := range tree1.Nodes {
+		tree1Nodes[node.Name] = node
+		names[node.Name] = struct{}{}
+	}
+
+	tree2Nodes = make(map[string]*restic.Node)
+	for _, node := range tree2.Nodes {
+		tree2Nodes[node.Name] = node
+		names[node.Name] = struct{}{}
+	}
+
+	uniqueNames = make([]string, 0, len(names))
+	for name := range names {
+		uniqueNames = append(uniqueNames, name)
+	}
+
+	sort.Sort(sort.StringSlice(uniqueNames))
+	return tree1Nodes, tree2Nodes, uniqueNames
+}
+
+func (c *Comparer) diffTree(ctx context.Context, stats *DiffStats, prefix string, id1, id2 restic.ID) error {
+	debug.Log("diffing %v to %v", id1, id2)
+	tree1, err := c.repo.LoadTree(ctx, id1)
+	if err != nil {
+		return err
+	}
+
+	tree2, err := c.repo.LoadTree(ctx, id2)
+	if err != nil {
+		return err
+	}
+
+	tree1Nodes, tree2Nodes, names := uniqueNodeNames(tree1, tree2)
+
+	for _, name := range names {
+		node1, t1 := tree1Nodes[name]
+		node2, t2 := tree2Nodes[name]
+
+		addBlobs(stats.BlobsBefore, node1)
+		addBlobs(stats.BlobsAfter, node2)
+
+		switch {
+		case t1 && t2:
+			name := path.Join(prefix, name)
+			mod := ""
+
+			if node1.Type != node2.Type {
+				mod += "T"
+			}
+
+			if node2.Type == "dir" {
+				name += "/"
+			}
+
+			if node1.Type == "file" &&
+				node2.Type == "file" &&
+				!reflect.DeepEqual(node1.Content, node2.Content) {
+				mod += "M"
+				stats.ChangedFiles++
+			} else if c.opts.ShowMetadata && !node1.Equals(*node2) {
+				mod += "U"
+			}
+
+			if mod != "" {
+				Printf("%-5s%v\n", mod, name)
+			}
+
+			if node1.Type == "dir" && node2.Type == "dir" {
+				err := c.diffTree(ctx, stats, name, *node1.Subtree, *node2.Subtree)
+				if err != nil {
+					Warnf("error: %v\n", err)
+				}
+			}
+		case t1 && !t2:
+			prefix := path.Join(prefix, name)
+			if node1.Type == "dir" {
+				prefix += "/"
+			}
+			Printf("%-5s%v\n", "-", prefix)
+			stats.Removed.Add(node1)
+
+			if node1.Type == "dir" {
+				err := c.printDir(ctx, "-", &stats.Removed, stats.BlobsBefore, prefix, *node1.Subtree)
+				if err != nil {
+					Warnf("error: %v\n", err)
+				}
+			}
+		case !t1 && t2:
+			prefix := path.Join(prefix, name)
+			if node2.Type == "dir" {
+				prefix += "/"
+			}
+			Printf("%-5s%v\n", "+", prefix)
+			stats.Added.Add(node2)
+
+			if node2.Type == "dir" {
+				err := c.printDir(ctx, "+", &stats.Added, stats.BlobsAfter, prefix, *node2.Subtree)
+				if err != nil {
+					Warnf("error: %v\n", err)
+				}
+			}
+		}
+	}
+
+	return nil
+}
+
+func runDiff(opts DiffOptions, gopts GlobalOptions, args []string) error {
+	if len(args) != 2 {
+		return errors.Fatalf("specify two snapshot IDs")
+	}
+
+	ctx, cancel := context.WithCancel(gopts.ctx)
+	defer cancel()
+
+	repo, err := OpenRepository(gopts)
+	if err != nil {
+		return err
+	}
+
+	if err = repo.LoadIndex(ctx); err != nil {
+		return err
+	}
+
+	if !gopts.NoLock {
+		lock, err := lockRepo(repo)
+		defer unlockRepo(lock)
+		if err != nil {
+			return err
+		}
+	}
+
+	sn1, err := loadSnapshot(ctx, repo, args[0])
+	if err != nil {
+		return err
+	}
+
+	sn2, err := loadSnapshot(ctx, repo, args[1])
+	if err != nil {
+		return err
+	}
+
+	Verbosef("comparing snapshot %v to %v:\n\n", sn1.ID().Str(), sn2.ID().Str())
+
+	if sn1.Tree == nil {
+		return errors.Errorf("snapshot %v has nil tree", sn1.ID().Str())
+	}
+
+	if sn2.Tree == nil {
+		return errors.Errorf("snapshot %v has nil tree", sn2.ID().Str())
+	}
+
+	c := &Comparer{
+		repo: repo,
+		opts: diffOptions,
+	}
+
+	stats := NewDiffStats()
+
+	err = c.diffTree(ctx, stats, "/", *sn1.Tree, *sn2.Tree)
+	if err != nil {
+		return err
+	}
+
+	both := stats.BlobsBefore.Intersect(stats.BlobsAfter)
+	updateBlobs(repo, stats.BlobsBefore.Sub(both), &stats.Removed)
+	updateBlobs(repo, stats.BlobsAfter.Sub(both), &stats.Added)
+
+	Printf("\n")
+	Printf("Files:       %5d new, %5d removed, %5d changed\n", stats.Added.Files, stats.Removed.Files, stats.ChangedFiles)
+	Printf("Dirs:        %5d new, %5d removed\n", stats.Added.Dirs, stats.Removed.Dirs)
+	Printf("Others:      %5d new, %5d removed\n", stats.Added.Others, stats.Removed.Others)
+	Printf("Data Blobs:  %5d new, %5d removed\n", stats.Added.DataBlobs, stats.Removed.DataBlobs)
+	Printf("Tree Blobs:  %5d new, %5d removed\n", stats.Added.TreeBlobs, stats.Removed.TreeBlobs)
+	Printf("  Added:   %-5s\n", formatBytes(uint64(stats.Added.Bytes)))
+	Printf("  Removed: %-5s\n", formatBytes(uint64(stats.Removed.Bytes)))
+
+	return nil
+}

cmd/restic/cmd_dump.go

@@ -162,7 +162,7 @@ func runDump(opts DumpOptions, gopts GlobalOptions, args []string) error {
 		}
 	}
 
-	sn, err := restic.LoadSnapshot(context.TODO(), repo, id)
+	sn, err := restic.LoadSnapshot(gopts.ctx, repo, id)
 	if err != nil {
 		Exitf(2, "loading snapshot %q failed: %v", snapshotIDString, err)
 	}

cmd/restic/cmd_find.go

@@ -180,7 +180,7 @@ type Finder struct {
 	notfound restic.IDSet
 }
 
-func (f *Finder) findInTree(treeID restic.ID, prefix string) error {
+func (f *Finder) findInTree(ctx context.Context, treeID restic.ID, prefix string) error {
 	if f.notfound.Has(treeID) {
 		debug.Log("%v skipping tree %v, has already been checked", prefix, treeID.Str())
 		return nil
@@ -188,7 +188,7 @@ func (f *Finder) findInTree(treeID restic.ID, prefix string) error {
 
 	debug.Log("%v checking tree %v\n", prefix, treeID.Str())
 
-	tree, err := f.repo.LoadTree(context.TODO(), treeID)
+	tree, err := f.repo.LoadTree(ctx, treeID)
 	if err != nil {
 		return err
 	}
@@ -224,7 +224,7 @@ func (f *Finder) findInTree(treeID restic.ID, prefix string) error {
 		}
 
 		if node.Type == "dir" {
-			if err := f.findInTree(*node.Subtree, filepath.Join(prefix, node.Name)); err != nil {
+			if err := f.findInTree(ctx, *node.Subtree, filepath.Join(prefix, node.Name)); err != nil {
 				return err
 			}
 		}
@@ -237,14 +237,11 @@ func (f *Finder) findInTree(treeID restic.ID, prefix string) error {
 	return nil
 }
 
-func (f *Finder) findInSnapshot(sn *restic.Snapshot) error {
+func (f *Finder) findInSnapshot(ctx context.Context, sn *restic.Snapshot) error {
 	debug.Log("searching in snapshot %s\n  for entries within [%s %s]", sn.ID(), f.pat.oldest, f.pat.newest)
 
 	f.out.newsn = sn
-	if err := f.findInTree(*sn.Tree, string(filepath.Separator)); err != nil {
-		return err
-	}
-	return nil
+	return f.findInTree(ctx, *sn.Tree, string(filepath.Separator))
 }
 
 func runFind(opts FindOptions, gopts GlobalOptions, args []string) error {
@@ -284,7 +281,7 @@ func runFind(opts FindOptions, gopts GlobalOptions, args []string) error {
 		}
 	}
 
-	if err = repo.LoadIndex(context.TODO()); err != nil {
+	if err = repo.LoadIndex(gopts.ctx); err != nil {
 		return err
 	}
 
@@ -298,7 +295,7 @@ func runFind(opts FindOptions, gopts GlobalOptions, args []string) error {
 		notfound: restic.NewIDSet(),
 	}
 	for sn := range FindFilteredSnapshots(ctx, repo, opts.Host, opts.Tags, opts.Paths, opts.Snapshots) {
-		if err = f.findInSnapshot(sn); err != nil {
+		if err = f.findInSnapshot(ctx, sn); err != nil {
 			return err
 		}
 	}

cmd/restic/cmd_forget.go

@@ -125,7 +125,7 @@ func runForget(opts ForgetOptions, gopts GlobalOptions, args []string) error {
 			// When explicit snapshots args are given, remove them immediately.
 			if !opts.DryRun {
 				h := restic.Handle{Type: restic.SnapshotFile, Name: sn.ID().String()}
-				if err = repo.Backend().Remove(context.TODO(), h); err != nil {
+				if err = repo.Backend().Remove(gopts.ctx, h); err != nil {
 					return err
 				}
 				Verbosef("removed snapshot %v\n", sn.ID().Str())
@@ -223,7 +223,7 @@ func runForget(opts ForgetOptions, gopts GlobalOptions, args []string) error {
 		if !opts.DryRun {
 			for _, sn := range remove {
 				h := restic.Handle{Type: restic.SnapshotFile, Name: sn.ID().String()}
-				err = repo.Backend().Remove(context.TODO(), h)
+				err = repo.Backend().Remove(gopts.ctx, h)
 				if err != nil {
 					return err
 				}

cmd/restic/cmd_init.go

@@ -1,8 +1,6 @@
 package main
 
 import (
-	"context"
-
 	"github.com/restic/restic/internal/errors"
 	"github.com/restic/restic/internal/repository"
 
@@ -44,7 +42,7 @@ func runInit(gopts GlobalOptions, args []string) error {
 
 	s := repository.New(be)
 
-	err = s.Init(context.TODO(), gopts.password)
+	err = s.Init(gopts.ctx, gopts.password)
 	if err != nil {
 		return errors.Fatalf("create key in backend at %s failed: %v\n", gopts.Repo, err)
 	}

cmd/restic/cmd_key.go

@@ -76,7 +76,7 @@ func addKey(gopts GlobalOptions, repo *repository.Repository) error {
 		return err
 	}
 
-	id, err := repository.AddKey(context.TODO(), repo, pw, repo.Key())
+	id, err := repository.AddKey(gopts.ctx, repo, pw, repo.Key())
 	if err != nil {
 		return errors.Fatalf("creating new key failed: %v\n", err)
 	}
@@ -86,13 +86,13 @@ func addKey(gopts GlobalOptions, repo *repository.Repository) error {
 	return nil
 }
 
-func deleteKey(repo *repository.Repository, name string) error {
+func deleteKey(ctx context.Context, repo *repository.Repository, name string) error {
 	if name == repo.KeyName() {
 		return errors.Fatal("refusing to remove key currently used to access repository")
 	}
 
 	h := restic.Handle{Type: restic.KeyFile, Name: name}
-	err := repo.Backend().Remove(context.TODO(), h)
+	err := repo.Backend().Remove(ctx, h)
 	if err != nil {
 		return err
 	}
@@ -107,13 +107,13 @@ func changePassword(gopts GlobalOptions, repo *repository.Repository) error {
 		return err
 	}
 
-	id, err := repository.AddKey(context.TODO(), repo, pw, repo.Key())
+	id, err := repository.AddKey(gopts.ctx, repo, pw, repo.Key())
 	if err != nil {
 		return errors.Fatalf("creating new key failed: %v\n", err)
 	}
 
 	h := restic.Handle{Type: restic.KeyFile, Name: repo.KeyName()}
-	err = repo.Backend().Remove(context.TODO(), h)
+	err = repo.Backend().Remove(gopts.ctx, h)
 	if err != nil {
 		return err
 	}
@@ -165,7 +165,7 @@ func runKey(gopts GlobalOptions, args []string) error {
 			return err
 		}
 
-		return deleteKey(repo, id)
+		return deleteKey(gopts.ctx, repo, id)
 	case "passwd":
 		lock, err := lockRepoExclusive(repo)
 		defer unlockRepo(lock)

cmd/restic/cmd_list.go

@@ -1,7 +1,6 @@
 package main
 
 import (
-	"context"
 	"fmt"
 
 	"github.com/restic/restic/internal/errors"
@@ -58,7 +57,7 @@ func runList(opts GlobalOptions, args []string) error {
 	case "locks":
 		t = restic.LockFile
 	case "blobs":
-		idx, err := index.Load(context.TODO(), repo, nil)
+		idx, err := index.Load(opts.ctx, repo, nil)
 		if err != nil {
 			return err
 		}
@@ -74,7 +73,7 @@ func runList(opts GlobalOptions, args []string) error {
 		return errors.Fatal("invalid type")
 	}
 
-	for id := range repo.List(context.TODO(), t) {
+	for id := range repo.List(opts.ctx, t) {
 		Printf("%s\n", id)
 	}
 

cmd/restic/cmd_ls.go

@@ -46,8 +46,8 @@ func init() {
 	flags.StringArrayVar(&lsOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path`, when no snapshot ID is given")
 }
 
-func printTree(repo *repository.Repository, id *restic.ID, prefix string) error {
-	tree, err := repo.LoadTree(context.TODO(), *id)
+func printTree(ctx context.Context, repo *repository.Repository, id *restic.ID, prefix string) error {
+	tree, err := repo.LoadTree(ctx, *id)
 	if err != nil {
 		return err
 	}
@@ -56,7 +56,7 @@ func printTree(repo *repository.Repository, id *restic.ID, prefix string) error
 		Printf("%s\n", formatNode(prefix, entry, lsOptions.ListLong))
 
 		if entry.Type == "dir" && entry.Subtree != nil {
-			if err = printTree(repo, entry.Subtree, filepath.Join(prefix, entry.Name)); err != nil {
+			if err = printTree(ctx, repo, entry.Subtree, filepath.Join(prefix, entry.Name)); err != nil {
 				return err
 			}
 		}
@@ -75,7 +75,7 @@ func runLs(opts LsOptions, gopts GlobalOptions, args []string) error {
 		return err
 	}
 
-	if err = repo.LoadIndex(context.TODO()); err != nil {
+	if err = repo.LoadIndex(gopts.ctx); err != nil {
 		return err
 	}
 
@@ -84,7 +84,7 @@ func runLs(opts LsOptions, gopts GlobalOptions, args []string) error {
 	for sn := range FindFilteredSnapshots(ctx, repo, opts.Host, opts.Tags, opts.Paths, args) {
 		Verbosef("snapshot %s of %v at %s):\n", sn.ID().Str(), sn.Paths, sn.Time)
 
-		if err = printTree(repo, sn.Tree, string(filepath.Separator)); err != nil {
+		if err = printTree(gopts.ctx, repo, sn.Tree, string(filepath.Separator)); err != nil {
 			return err
 		}
 	}

cmd/restic/cmd_mount.go

@@ -4,7 +4,6 @@
 package main
 
 import (
-	"context"
 	"os"
 
 	"github.com/spf13/cobra"
@@ -73,7 +72,7 @@ func mount(opts MountOptions, gopts GlobalOptions, mountpoint string) error {
 		return err
 	}
 
-	err = repo.LoadIndex(context.TODO())
+	err = repo.LoadIndex(gopts.ctx)
 	if err != nil {
 		return err
 	}
@@ -114,7 +113,7 @@ func mount(opts MountOptions, gopts GlobalOptions, mountpoint string) error {
 		Tags:        opts.Tags,
 		Paths:       opts.Paths,
 	}
-	root, err := fuse.NewRoot(context.TODO(), repo, cfg)
+	root, err := fuse.NewRoot(gopts.ctx, repo, cfg)
 	if err != nil {
 		return err
 	}

cmd/restic/cmd_rebuild_index.go

@@ -3,6 +3,7 @@ package main
 import (
 	"context"
 
+	"github.com/restic/restic/internal/errors"
 	"github.com/restic/restic/internal/index"
 	"github.com/restic/restic/internal/restic"
 
@@ -66,7 +67,7 @@ func rebuildIndex(ctx context.Context, repo restic.Repository, ignorePacks resti
 
 	id, err := idx.Save(ctx, repo, supersedes)
 	if err != nil {
-		return err
+		return errors.Fatalf("unable to save index, last error was: %v", err)
 	}
 
 	Verbosef("saved new index as %v\n", id.Str())

cmd/restic/cmd_tag.go

@@ -53,7 +53,7 @@ func init() {
 	tagFlags.StringArrayVar(&tagOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path`, when no snapshot-ID is given")
 }
 
-func changeTags(repo *repository.Repository, sn *restic.Snapshot, setTags, addTags, removeTags []string) (bool, error) {
+func changeTags(ctx context.Context, repo *repository.Repository, sn *restic.Snapshot, setTags, addTags, removeTags []string) (bool, error) {
 	var changed bool
 
 	if len(setTags) != 0 {
@@ -77,20 +77,20 @@ func changeTags(repo *repository.Repository, sn *restic.Snapshot, setTags, addTa
 		}
 
 		// Save the new snapshot.
-		id, err := repo.SaveJSONUnpacked(context.TODO(), restic.SnapshotFile, sn)
+		id, err := repo.SaveJSONUnpacked(ctx, restic.SnapshotFile, sn)
 		if err != nil {
 			return false, err
 		}
 
 		debug.Log("new snapshot saved as %v", id.Str())
 
-		if err = repo.Flush(); err != nil {
+		if err = repo.Flush(ctx); err != nil {
 			return false, err
 		}
 
 		// Remove the old snapshot.
 		h := restic.Handle{Type: restic.SnapshotFile, Name: sn.ID().String()}
-		if err = repo.Backend().Remove(context.TODO(), h); err != nil {
+		if err = repo.Backend().Remove(ctx, h); err != nil {
 			return false, err
 		}
 
@@ -125,7 +125,7 @@ func runTag(opts TagOptions, gopts GlobalOptions, args []string) error {
 	ctx, cancel := context.WithCancel(gopts.ctx)
 	defer cancel()
 	for sn := range FindFilteredSnapshots(ctx, repo, opts.Host, opts.Tags, opts.Paths, args) {
-		changed, err := changeTags(repo, sn, opts.SetTags, opts.AddTags, opts.RemoveTags)
+		changed, err := changeTags(ctx, repo, sn, opts.SetTags, opts.AddTags, opts.RemoveTags)
 		if err != nil {
 			Warnf("unable to modify the tags for snapshot ID %q, ignoring: %v\n", sn.ID(), err)
 			continue

cmd/restic/cmd_unlock.go

@@ -1,8 +1,6 @@
 package main
 
 import (
-	"context"
-
 	"github.com/restic/restic/internal/restic"
 	"github.com/spf13/cobra"
 )
@@ -43,7 +41,7 @@ func runUnlock(opts UnlockOptions, gopts GlobalOptions) error {
 		fn = restic.RemoveAllLocks
 	}
 
-	err = fn(context.TODO(), repo)
+	err = fn(gopts.ctx, repo)
 	if err != nil {
 		return err
 	}

cmd/restic/exclude.go

@@ -90,7 +90,7 @@ func rejectByPattern(patterns []string) RejectFunc {
 // non-nil if the filename component of excludeFileSpec is empty. If rc is
 // non-nil, it is going to be used in the RejectFunc to expedite the evaluation
 // of a directory based on previous visits.
-func rejectIfPresent(excludeFileSpec string, rc *rejectionCache) (RejectFunc, error) {
+func rejectIfPresent(excludeFileSpec string) (RejectFunc, error) {
 	if excludeFileSpec == "" {
 		return nil, errors.New("name for exclusion tagfile is empty")
 	}
@@ -106,6 +106,7 @@ func rejectIfPresent(excludeFileSpec string, rc *rejectionCache) (RejectFunc, er
 		tf = excludeFileSpec
 	}
 	debug.Log("using %q as exclusion tagfile", tf)
+	rc := &rejectionCache{}
 	fn := func(filename string, _ os.FileInfo) bool {
 		return isExcludedByFile(filename, tf, tc, rc)
 	}

cmd/restic/exclude_test.go

@@ -2,6 +2,7 @@ package main
 
 import (
 	"io/ioutil"
+	"os"
 	"path/filepath"
 	"testing"
 
@@ -82,3 +83,82 @@ func TestIsExcludedByFile(t *testing.T) {
 		})
 	}
 }
+
+// TestMultipleIsExcludedByFile is for testing that multiple instances of
+// the --exclude-if-present parameter (or the shortcut --exclude-caches do not
+// cancel each other out. It was initially written to demonstrate a bug in
+// rejectIfPresent.
+func TestMultipleIsExcludedByFile(t *testing.T) {
+	tempDir, cleanup := test.TempDir(t)
+	defer cleanup()
+
+	// Create some files in a temporary directory.
+	// Files in UPPERCASE will be used as exclusion triggers later on.
+	// We will test the inclusion later, so we add the expected value as
+	// a bool.
+	files := []struct {
+		path string
+		incl bool
+	}{
+		{"42", true},
+
+		// everything in foodir except the NOFOO tagfile
+		// should not be included.
+		{"foodir/NOFOO", true},
+		{"foodir/foo", false},
+		{"foodir/foosub/underfoo", false},
+
+		// everything in bardir except the NOBAR tagfile
+		// should not be included.
+		{"bardir/NOBAR", true},
+		{"bardir/bar", false},
+		{"bardir/barsub/underbar", false},
+
+		// everything in bazdir should be included.
+		{"bazdir/baz", true},
+		{"bazdir/bazsub/underbaz", true},
+	}
+	var errs []error
+	for _, f := range files {
+		// create directories first, then the file
+		p := filepath.Join(tempDir, filepath.FromSlash(f.path))
+		errs = append(errs, os.MkdirAll(filepath.Dir(p), 0700))
+		errs = append(errs, ioutil.WriteFile(p, []byte(f.path), 0600))
+	}
+	test.OKs(t, errs) // see if anything went wrong during the creation
+
+	// create two rejection functions, one that tests for the NOFOO file
+	// and one for the NOBAR file
+	fooExclude, _ := rejectIfPresent("NOFOO")
+	barExclude, _ := rejectIfPresent("NOBAR")
+
+	// To mock the archiver scanning walk, we create filepath.WalkFn
+	// that tests against the two rejection functions and stores
+	// the result in a map against we can test later.
+	m := make(map[string]bool)
+	walk := func(p string, fi os.FileInfo, err error) error {
+		if err != nil {
+			return err
+		}
+		excludedByFoo := fooExclude(p, fi)
+		excludedByBar := barExclude(p, fi)
+		excluded := excludedByFoo || excludedByBar
+		// the log message helps debugging in case the test fails
+		t.Logf("%q: %v || %v = %v", p, excludedByFoo, excludedByBar, excluded)
+		m[p] = !excluded
+		if excluded {
+			return filepath.SkipDir
+		}
+		return nil
+	}
+	// walk through the temporary file and check the error
+	test.OK(t, filepath.Walk(tempDir, walk))
+
+	// compare whether the walk gave the expected values for the test cases
+	for _, f := range files {
+		p := filepath.Join(tempDir, filepath.FromSlash(f.path))
+		if m[p] != f.incl {
+			t.Errorf("inclusion status of %s is wrong: want %v, got %v", f.path, f.incl, m[p])
+		}
+	}
+}

cmd/restic/format.go

@@ -68,17 +68,29 @@ func formatNode(prefix string, n *restic.Node, long bool) string {
 		return filepath.Join(prefix, n.Name)
 	}
 
+	var mode os.FileMode
+	var target string
+
 	switch n.Type {
 	case "file":
-		return fmt.Sprintf("%s %5d %5d %6d %s %s",
-			n.Mode, n.UID, n.GID, n.Size, n.ModTime.Format(TimeFormat), filepath.Join(prefix, n.Name))
+		mode = 0
 	case "dir":
-		return fmt.Sprintf("%s %5d %5d %6d %s %s",
-			n.Mode|os.ModeDir, n.UID, n.GID, n.Size, n.ModTime.Format(TimeFormat), filepath.Join(prefix, n.Name))
+		mode = os.ModeDir
 	case "symlink":
-		return fmt.Sprintf("%s %5d %5d %6d %s %s -> %s",
-			n.Mode|os.ModeSymlink, n.UID, n.GID, n.Size, n.ModTime.Format(TimeFormat), filepath.Join(prefix, n.Name), n.LinkTarget)
-	default:
-		return fmt.Sprintf("<Node(%s) %s>", n.Type, n.Name)
+		mode = os.ModeSymlink
+		target = fmt.Sprintf(" -> %v", n.LinkTarget)
+	case "dev":
+		mode = os.ModeDevice
+	case "chardev":
+		mode = os.ModeDevice | os.ModeCharDevice
+	case "fifo":
+		mode = os.ModeNamedPipe
+	case "socket":
+		mode = os.ModeSocket
 	}
+
+	return fmt.Sprintf("%s %5d %5d %6d %s %s%s",
+		mode|n.Mode, n.UID, n.GID, n.Size,
+		n.ModTime.Format(TimeFormat), filepath.Join(prefix, n.Name),
+		target)
 }

cmd/restic/global.go

@@ -6,6 +6,7 @@ import (
 	"io"
 	"io/ioutil"
 	"os"
+	"path/filepath"
 	"runtime"
 	"strings"
 	"syscall"
@@ -23,6 +24,7 @@ import (
 	"github.com/restic/restic/internal/backend/swift"
 	"github.com/restic/restic/internal/cache"
 	"github.com/restic/restic/internal/debug"
+	"github.com/restic/restic/internal/fs"
 	"github.com/restic/restic/internal/limiter"
 	"github.com/restic/restic/internal/options"
 	"github.com/restic/restic/internal/repository"
@@ -45,6 +47,7 @@ type GlobalOptions struct {
 	CacheDir     string
 	NoCache      bool
 	CACerts      []string
+	CleanupCache bool
 
 	LimitUploadKb   int
 	LimitDownloadKb int
@@ -81,6 +84,7 @@ func init() {
 	f.StringVar(&globalOptions.CacheDir, "cache-dir", "", "set the cache directory")
 	f.BoolVar(&globalOptions.NoCache, "no-cache", false, "do not use a local cache")
 	f.StringSliceVar(&globalOptions.CACerts, "cacert", nil, "path to load root certificates from (default: use system certificates)")
+	f.BoolVar(&globalOptions.CleanupCache, "cleanup-cache", false, "auto remove old cache directories")
 	f.IntVar(&globalOptions.LimitUploadKb, "limit-upload", 0, "limits uploads to a maximum rate in KiB/s. (default: unlimited)")
 	f.IntVar(&globalOptions.LimitDownloadKb, "limit-download", 0, "limits downloads to a maximum rate in KiB/s. (default: unlimited)")
 	f.StringSliceVarP(&globalOptions.Options, "option", "o", []string{}, "set extended option (`key=value`, can be specified multiple times)")
@@ -340,7 +344,7 @@ func OpenRepository(opts GlobalOptions) (*repository.Repository, error) {
 		return nil, err
 	}
 
-	err = s.SearchKey(context.TODO(), opts.password, maxKeys)
+	err = s.SearchKey(opts.ctx, opts.password, maxKeys)
 	if err != nil {
 		return nil, err
 	}
@@ -353,11 +357,41 @@ func OpenRepository(opts GlobalOptions) (*repository.Repository, error) {
 		return s, nil
 	}
 
-	cache, err := cache.New(s.Config().ID, opts.CacheDir)
+	c, err := cache.New(s.Config().ID, opts.CacheDir)
 	if err != nil {
 		Warnf("unable to open cache: %v\n", err)
+		return s, nil
+	}
+
+	// start using the cache
+	s.UseCache(c)
+
+	oldCacheDirs, err := cache.Old(c.Base)
+	if err != nil {
+		Warnf("unable to find old cache directories: %v", err)
+	}
+
+	// nothing more to do if no old cache dirs could be found
+	if len(oldCacheDirs) == 0 {
+		return s, nil
+	}
+
+	// cleanup old cache dirs if instructed to do so
+	if opts.CleanupCache {
+		Printf("removing %d old cache dirs from %v\n", len(oldCacheDirs), c.Base)
+
+		for _, item := range oldCacheDirs {
+			dir := filepath.Join(c.Base, item)
+			err = fs.RemoveAll(dir)
+			if err != nil {
+				Warnf("unable to remove %v: %v\n", dir, err)
+			}
+		}
 	} else {
-		s.UseCache(cache)
+		if stdoutIsTerminal() {
+			Verbosef("found %d old cache directories in %v, pass --cleanup-cache to remove them\n",
+				len(oldCacheDirs), c.Base)
+		}
 	}
 
 	return s, nil
@@ -466,10 +500,18 @@ func parseConfig(loc location.Location, opts options.Options) (interface{}, erro
 			cfg.AccountID = os.Getenv("B2_ACCOUNT_ID")
 		}
 
+		if cfg.AccountID == "" {
+			return nil, errors.Fatalf("unable to open B2 backend: Account ID ($B2_ACCOUNT_ID) is empty")
+		}
+
 		if cfg.Key == "" {
 			cfg.Key = os.Getenv("B2_ACCOUNT_KEY")
 		}
 
+		if cfg.Key == "" {
+			return nil, errors.Fatalf("unable to open B2 backend: Key ($B2_ACCOUNT_KEY) is empty")
+		}
+
 		if err := opts.Apply(loc.Scheme, &cfg); err != nil {
 			return nil, err
 		}
@@ -523,7 +565,7 @@ func open(s string, opts options.Options) (restic.Backend, error) {
 	case "swift":
 		be, err = swift.Open(cfg.(swift.Config), rt)
 	case "b2":
-		be, err = b2.Open(cfg.(b2.Config), rt)
+		be, err = b2.Open(globalOptions.ctx, cfg.(b2.Config), rt)
 	case "rest":
 		be, err = rest.Open(cfg.(rest.Config), rt)
 
@@ -536,7 +578,7 @@ func open(s string, opts options.Options) (restic.Backend, error) {
 	}
 
 	// check if config is there
-	fi, err := be.Stat(context.TODO(), restic.Handle{Type: restic.ConfigFile})
+	fi, err := be.Stat(globalOptions.ctx, restic.Handle{Type: restic.ConfigFile})
 	if err != nil {
 		return nil, errors.Fatalf("unable to open config file: %v\nIs there a repository at the following location?\n%v", err, s)
 	}
@@ -580,7 +622,7 @@ func create(s string, opts options.Options) (restic.Backend, error) {
 	case "swift":
 		return swift.Open(cfg.(swift.Config), rt)
 	case "b2":
-		return b2.Create(cfg.(b2.Config), rt)
+		return b2.Create(globalOptions.ctx, cfg.(b2.Config), rt)
 	case "rest":
 		return rest.Create(cfg.(rest.Config), rt)
 	}

doc/030_preparing_a_new_repo.rst

@@ -333,7 +333,7 @@ root path like this:
     created restic backend a934bac191 at azure:foo:/
     [...]
 
-The number of concurrent connections to the B2 service can be set with the
+The number of concurrent connections to the Azure Blob Storage service can be set with the
 `-o azure.connections=10`. By default, at most five parallel connections are
 established.
 

doc/040_backup.rst

@@ -89,7 +89,16 @@ specify them with multiple ``--exclude``'s or one ``--exclude-file``
 
 Patterns use `filepath.Glob <https://golang.org/pkg/path/filepath/#Glob>`__ internally,
 see `filepath.Match <https://golang.org/pkg/path/filepath/#Match>`__ for syntax.
-Additionally ``**`` excludes arbitrary subdirectories.
+Patterns are tested against the full path of a file/dir to be saved, not only
+against the relative path below the argument given to restic backup.
+Patterns need to match on complete path components. (``foo`` matches
+``/dir1/foo/dir2/file`` and ``/dir/foo`` but does not match ``/dir/foobar`` or
+``barfoo``.) A trailing ``/`` is ignored. A leading ``/`` anchors the
+pattern at the root directory. (``/bin`` matches ``/bin/bash`` but does not
+match ``/usr/bin/restic``.) Regular wildcards cannot be used to match over the
+directory separator ``/``. (``b*ash`` matches ``/bin/bash`` but does not match
+``/bin/ash``.) However ``**`` matches arbitrary subdirectories. (``foo/**/bar``
+matches ``/dir1/foo/dir2/bar/file``, ``/foo/bar/file`` and ``/tmp/foo/bar``.)
 Environment-variables in exclude-files are expanded with
 `os.ExpandEnv <https://golang.org/pkg/os/#ExpandEnv>`__.
 
@@ -127,8 +136,33 @@ args:
 
     $ restic -r /tmp/backup backup --files-from /tmp/files_to_backup /tmp/some_additional_file
 
-Backing up special items
-************************
+Comparing Snapshots
+*******************
+
+Restic has a `diff` command which shows the difference between two snapshots
+and displays a small statistic, just pass the command two snapshot IDs:
+
+.. code-block:: console
+
+    $ restic -r /tmp/backup diff 5845b002 2ab627a6
+    password is correct
+    comparing snapshot ea657ce5 to 2ab627a6:
+
+     C   /restic/cmd_diff.go
+    +    /restic/foo
+     C   /restic/restic
+
+    Files:           0 new,     0 removed,     2 changed
+    Dirs:            1 new,     0 removed
+    Others:          0 new,     0 removed
+    Data Blobs:     14 new,    15 removed
+    Tree Blobs:      2 new,     1 removed
+      Added:   16.403 MiB
+      Removed: 16.402 MiB
+
+
+Backing up special items and metadata
+*************************************
 
 **Symlinks** are archived as symlinks, ``restic`` does not follow them.
 When you restore, you get the same symlink again, with the same link target
@@ -140,6 +174,12 @@ If there is a **bind-mount** below a directory that is to be saved, restic desce
 archived as a block device file and restored as such. This also means that the content of the
 corresponding disk is not read, at least not from the device file.
 
+By default, restic does not save the access time (atime) for any files or other
+items, since it is not possible to reliably disable updating the access time by
+restic itself. This means that for each new backup a lot of metadata is
+written, and the next backup needs to write new metadata again. If you really
+want to save the access time for files and directories, you can pass the
+``--with-atime`` option to the ``backup`` command.
 
 Reading data from stdin
 ***********************

doc/070_encryption.rst

@@ -26,8 +26,8 @@ Manage repository keys
 **********************
 
 The ``key`` command allows you to set multiple access keys or passwords
-per repository. In fact, you can use the ``list``, ``add``, ``remove``
-and ``passwd`` sub-commands to manage these keys very precisely:
+per repository. In fact, you can use the ``list``, ``add``, ``remove``, and
+``passwd`` (changes a password) sub-commands to manage these keys very precisely:
 
 .. code-block:: console