Add version for 0.9.5

backend find: Check prefix length < snapshot IDs when searching

.github/ISSUE_TEMPLATE.md

@@ -1,17 +1,9 @@
 <!--
 
-Welcome! - We kindly ask that you:
+Welcome! If you have a question or are unsure if you should open an issue,
+please use the forum instead!
 
-  1. Fill out the issue template below - not doing so needs a good reason.
-  2. Use the forum if you have a question rather than a bug or feature request.
-
-The forum is at: https://forum.restic.net
-
-NOTE: Not filling out the issue template needs a good reason, as otherwise it
-may take a lot longer to find the problem, not to mention it can take up a lot
-more time which can otherwise be spent on development. Please also take the
-time to help us debug the issue by collecting relevant information, even if
-it doesn't seem to be relevant to you. Thanks!
+   https://forum.restic.net
 
 The forum is a better place for questions about restic or general suggestions
 and topics, e.g. usage or documentation questions! This issue tracker is mainly
@@ -19,61 +11,17 @@ for tracking bugs and feature requests directly relating to the development of
 the software itself, rather than the project.
 
 Thanks for understanding, and for contributing to the project!
-
 -->
 
 
-## Output of `restic version`
-
-
-## How did you run restic exactly?
+Output of `restic version`
+--------------------------
 
 <!--
-This section should include at least:
-
- * The complete command line and any environment variables you used to
-   configure restic's backend access. Make sure to replace sensitive values!
-
- * The output of the commands, what restic prints gives may give us much
-   information to diagnose the problem!
+Please add the version of restic you're currently using here, this helps us
+later to see what has changed in restic when we revisit this issue after some
+time.
 -->
 
-## What backend/server/service did you use to store the repository?
-
-
-
-## Expected behavior
-
-<!--
-Describe what you'd like restic to do differently.
--->
-
-## Actual behavior
-
-<!--
-In this section, please try to concentrate on observations, so only describe
-what you observed directly.
--->
-
-## Steps to reproduce the behavior
-
-<!--
-The more time you spend describing an easy way to reproduce the behavior (if
-this is possible), the easier it is for the project developers to fix it!
--->
-
-## Do you have any idea what may have caused this?
-
-
-
-## Do you have an idea how to solve the issue?
-
-
-
-## Did restic help you or made you happy in any way?
-
-<!--
-Answering this question is not required, but if you have anything positive to share, please do so here!
-Sometimes we get tired of reading bug reports all day and a little positive end note does wonders.
-Idea by Joey Hess, https://joeyh.name/blog/entry/two_holiday_stories/
--->
+Describe the issue
+------------------

.github/ISSUE_TEMPLATE/Bug.md

@@ -0,0 +1,93 @@
+---
+name: Bug report
+about: Report a problem with restic to help us resolve it and improve
+---
+
+<!--
+
+Welcome! - We kindly ask that you:
+
+  1. Fill out the issue template below - not doing so needs a good reason.
+  2. Use the forum if you have a question rather than a bug or feature request.
+
+The forum is at: https://forum.restic.net
+
+NOTE: Not filling out the issue template needs a good reason, as otherwise it
+may take a lot longer to find the problem, not to mention it can take up a lot
+more time which can otherwise be spent on development. Please also take the
+time to help us debug the issue by collecting relevant information, even if
+it doesn't seem to be relevant to you. Thanks!
+
+The forum is a better place for questions about restic or general suggestions
+and topics, e.g. usage or documentation questions! This issue tracker is mainly
+for tracking bugs and feature requests directly relating to the development of
+the software itself, rather than the project.
+
+Thanks for understanding, and for contributing to the project!
+
+-->
+
+
+Output of `restic version`
+--------------------------
+
+
+How did you run restic exactly?
+-------------------------------
+
+<!--
+This section should include at least:
+
+ * The complete command line and any environment variables you used to
+   configure restic's backend access. Make sure to replace sensitive values!
+
+ * The output of the commands, what restic prints gives may give us much
+   information to diagnose the problem!
+-->
+
+What backend/server/service did you use to store the repository?
+----------------------------------------------------------------
+
+
+
+Expected behavior
+-----------------
+
+<!--
+Describe what you'd like restic to do differently.
+-->
+
+Actual behavior
+---------------
+
+<!--
+In this section, please try to concentrate on observations, so only describe
+what you observed directly.
+-->
+
+Steps to reproduce the behavior
+-------------------------------
+
+<!--
+The more time you spend describing an easy way to reproduce the behavior (if
+this is possible), the easier it is for the project developers to fix it!
+-->
+
+Do you have any idea what may have caused this?
+-----------------------------------------------
+
+
+
+Do you have an idea how to solve the issue?
+-------------------------------------------
+
+
+
+Did restic help you or made you happy in any way?
+-------------------------------------------------
+
+<!--
+Answering this question is not required, but if you have anything positive to share, please do so here!
+Sometimes we get tired of reading bug reports all day and a little positive end note does wonders.
+Idea by Joey Hess, https://joeyh.name/blog/entry/two_holiday_stories/
+-->

.github/ISSUE_TEMPLATE/Feature.md

@@ -0,0 +1,57 @@
+---
+name: Feature request
+about: Suggest a new feature or enhancement for restic
+---
+
+<!--
+
+Welcome! - We kindly ask that you:
+
+  1. Fill out the issue template below - not doing so needs a good reason.
+  2. Use the forum if you have a question rather than a bug or feature request.
+
+The forum is at: https://forum.restic.net
+
+The forum is a better place for questions about restic or general suggestions
+and topics, e.g. usage or documentation questions! This issue tracker is mainly
+for tracking bugs and feature requests directly relating to the development of
+the software itself, rather than the project.
+
+Thanks for understanding, and for contributing to the project!
+
+-->
+
+
+Output of `restic version`
+--------------------------
+
+<!--
+Please add the version of restic you're currently using here, this helps us
+later to see what has changed in restic when we revisit this issue after some
+time.
+-->
+
+What should restic do differently? Which functionality do you think we should add?
+----------------------------------------------------------------------------------
+
+<!--
+Please describe the feature you'd like us to add here.
+-->
+
+
+What are you trying to do?
+--------------------------
+
+<!--
+This section should contain a brief description what you're trying to do, which
+would be possible after implementing the new feature.
+-->
+
+Did restic help you or made you happy in any way?
+-------------------------------------------------
+
+<!--
+Answering this question is not required, but if you have anything positive to share, please do so here!
+Sometimes we get tired of reading bug reports all day and a little positive end note does wonders.
+Idea by Joey Hess, https://joeyh.name/blog/entry/two_holiday_stories/
+-->

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,3 +1,5 @@
+
+
 <!--
 Thank you very much for contributing code or documentation to restic! Please
 fill out the following questions to make it easier for us to review your
@@ -8,19 +10,22 @@ your time and add more commits. If you're done and ready for review, please
 check the last box.
 -->
 
-### What is the purpose of this change? What does it change?
+What is the purpose of this change? What does it change?
+--------------------------------------------------------
 
 <!--
 Describe the changes here, as detailed as needed.
 -->
 
-### Was the change discussed in an issue or in the forum before?
+Was the change discussed in an issue or in the forum before?
+------------------------------------------------------------
 
 <!--
 Link issues and relevant forum posts here.
 -->
 
-### Checklist
+Checklist
+---------
 
 - [ ] I have read the [Contribution Guidelines](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#providing-patches)
 - [ ] I have added tests for all changes in this PR

.gitignore

@@ -1,3 +1,2 @@
 /restic
 /.vagrant
-/doc/_build

.travis.yml

@@ -4,17 +4,37 @@ sudo: false
 matrix:
   include:
     - os: linux
-      go: "1.9.x"
-      env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0 RESTIC_BUILD_SOLARIS=0
+      go: "1.10.x"
+      env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0
+      cache:
+        directories:
+          - $HOME/.cache/go-build
+          - $HOME/gopath/pkg/mod
+
+    - os: linux
+      go: "1.11.x"
+      env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0
+      cache:
+        directories:
+          - $HOME/.cache/go-build
+          - $HOME/gopath/pkg/mod
 
     # only run fuse and cloud backends tests on Travis for the latest Go on Linux
     - os: linux
-      go: "1.10.x"
+      go: "1.12.x"
       sudo: true
+      cache:
+        directories:
+          - $HOME/.cache/go-build
+          - $HOME/gopath/pkg/mod
 
     - os: osx
-      go: "1.10.x"
+      go: "1.12.x"
       env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0
+      cache:
+        directories:
+          - $HOME/Library/Caches/go-build
+          - $HOME/gopath/pkg/mod
 
 branches:
   only:
@@ -38,4 +58,4 @@ script:
   - go run run_integration_tests.go
 
 after_success:
-  - bash <(curl -s https://codecov.io/bash) -f all.cov
+  - test -r all.cov && bash <(curl -s https://codecov.io/bash) -f all.cov

CHANGELOG.md

@@ -1,3 +1,566 @@
+Changelog for restic 0.9.5 (2019-04-23)
+=======================================
+
+The following sections list the changes in restic 0.9.5 relevant to
+restic users. The changes are ordered by importance.
+
+Summary
+-------
+
+ * Fix #2135: Return error when no bytes could be read from stdin
+ * Fix #2181: Don't cancel timeout after 30 seconds for self-update
+ * Fix #2203: Fix reading passwords from stdin
+ * Fix #2224: Don't abort the find command when a tree can't be loaded
+ * Enh #1895: Add case insensitive include & exclude options
+ * Enh #1937: Support streaming JSON output for backup
+ * Enh #2155: Add Openstack application credential auth for Swift
+ * Enh #2184: Add --json support to forget command
+ * Enh #2037: Add group-by option to snapshots command
+ * Enh #2124: Ability to dump folders to tar via stdout
+ * Enh #2139: Return error if no bytes could be read for `backup --stdin`
+ * Enh #2205: Add --ignore-inode option to backup cmd
+ * Enh #2220: Add config option to set S3 storage class
+
+Details
+-------
+
+ * Bugfix #2135: Return error when no bytes could be read from stdin
+
+   We assume that users reading backup data from stdin want to know when no data could be read, so now
+   restic returns an error when `backup --stdin` is called but no bytes could be read. Usually,
+   this means that an earlier command in a pipe has failed. The documentation was amended and now
+   recommends setting the `pipefail` option (`set -o pipefail`).
+
+   https://github.com/restic/restic/pull/2135
+   https://github.com/restic/restic/pull/2139
+
+ * Bugfix #2181: Don't cancel timeout after 30 seconds for self-update
+
+   https://github.com/restic/restic/issues/2181
+
+ * Bugfix #2203: Fix reading passwords from stdin
+
+   Passwords for the `init`, `key add`, and `key passwd` commands can now be read from
+   non-terminal stdin.
+
+   https://github.com/restic/restic/issues/2203
+
+ * Bugfix #2224: Don't abort the find command when a tree can't be loaded
+
+   Change the find command so that missing trees don't result in a crash. Instead, the error is
+   logged to the debug log, and the tree ID is displayed along with the snapshot it belongs to. This
+   makes it possible to recover repositories that are missing trees by forgetting the snapshots
+   they are used in.
+
+   https://github.com/restic/restic/issues/2224
+
+ * Enhancement #1895: Add case insensitive include & exclude options
+
+   The backup and restore commands now have --iexclude and --iinclude flags as case insensitive
+   variants of --exclude and --include.
+
+   https://github.com/restic/restic/issues/1895
+   https://github.com/restic/restic/pull/2032
+
+ * Enhancement #1937: Support streaming JSON output for backup
+
+   We've added support for getting machine-readable status output during backup, just pass the
+   flag `--json` for `restic backup` and restic will output a stream of JSON objects which contain
+   the current progress.
+
+   https://github.com/restic/restic/issues/1937
+   https://github.com/restic/restic/pull/1944
+
+ * Enhancement #2155: Add Openstack application credential auth for Swift
+
+   Since Openstack Queens Identity (auth V3) service supports an application credential auth
+   method. It allows to create a technical account with the limited roles. This commit adds an
+   application credential authentication method for the Swift backend.
+
+   https://github.com/restic/restic/issues/2155
+
+ * Enhancement #2184: Add --json support to forget command
+
+   The forget command now supports the --json argument, outputting the information about what is
+   (or would-be) kept and removed from the repository.
+
+   https://github.com/restic/restic/issues/2184
+   https://github.com/restic/restic/pull/2185
+
+ * Enhancement #2037: Add group-by option to snapshots command
+
+   We have added an option to group the output of the snapshots command, similar to the output of the
+   forget command. The option has been called "--group-by" and accepts any combination of the
+   values "host", "paths" and "tags", separated by commas. Default behavior (not specifying
+   --group-by) has not been changed. We have added support of the grouping to the JSON output.
+
+   https://github.com/restic/restic/issues/2037
+   https://github.com/restic/restic/pull/2087
+
+ * Enhancement #2124: Ability to dump folders to tar via stdout
+
+   We've added the ability to dump whole folders to stdout via the `dump` command. Restic now
+   requires at least Go 1.10 due to a limitation of the standard library for Go <= 1.9.
+
+   https://github.com/restic/restic/issues/2123
+   https://github.com/restic/restic/pull/2124
+
+ * Enhancement #2139: Return error if no bytes could be read for `backup --stdin`
+
+   When restic is used to backup the output of a program, like `mysqldump | restic backup --stdin`,
+   it now returns an error if no bytes could be read at all. This catches the failure case when
+   `mysqldump` failed for some reason and did not output any data to stdout.
+
+   https://github.com/restic/restic/pull/2139
+
+ * Enhancement #2205: Add --ignore-inode option to backup cmd
+
+   This option handles backup of virtual filesystems that do not keep fixed inodes for files, like
+   Fuse-based, pCloud, etc. Ignoring inode changes allows to consider the file as unchanged if
+   last modification date and size are unchanged.
+
+   https://github.com/restic/restic/issues/1631
+   https://github.com/restic/restic/pull/2205
+   https://github.com/restic/restic/pull/2047
+
+ * Enhancement #2220: Add config option to set S3 storage class
+
+   The `s3.storage-class` option can be passed to restic (using `-o`) to specify the storage
+   class to be used for S3 objects created by restic.
+
+   The storage class is passed as-is to S3, so it needs to be understood by the API. On AWS, it can be
+   one of `STANDARD`, `STANDARD_IA`, `ONEZONE_IA`, `INTELLIGENT_TIERING` and
+   `REDUCED_REDUNDANCY`. If unspecified, the default storage class is used (`STANDARD` on
+   AWS).
+
+   You can mix storage classes in the same bucket, and the setting isn't stored in the restic
+   repository, so be sure to specify it with each command that writes to S3.
+
+   https://github.com/restic/restic/issues/706
+   https://github.com/restic/restic/pull/2220
+
+
+Changelog for restic 0.9.4 (2019-01-06)
+=======================================
+
+The following sections list the changes in restic 0.9.4 relevant to
+restic users. The changes are ordered by importance.
+
+Summary
+-------
+
+ * Fix #1989: Google Cloud Storage: Respect bandwidth limit
+ * Fix #2040: Add host name filter shorthand flag for `stats` command
+ * Fix #2068: Correctly return error loading data
+ * Fix #2095: Consistently use local time for snapshots times
+ * Enh #1605: Concurrent restore
+ * Enh #2089: Increase granularity of the "keep within" retention policy
+ * Enh #2097: Add key hinting
+ * Enh #2017: Mount: Enforce FUSE Unix permissions with allow-other
+ * Enh #2070: Make all commands display timestamps in local time
+ * Enh #2085: Allow --files-from to be specified multiple times
+ * Enh #2094: Run command to get password
+
+Details
+-------
+
+ * Bugfix #1989: Google Cloud Storage: Respect bandwidth limit
+
+   The GCS backend did not respect the bandwidth limit configured, a previous commit
+   accidentally removed support for it.
+
+   https://github.com/restic/restic/issues/1989
+   https://github.com/restic/restic/pull/2100
+
+ * Bugfix #2040: Add host name filter shorthand flag for `stats` command
+
+   The default value for `--host` flag was set to 'H' (the shorthand version of the flag), this
+   caused the lookup for the latest snapshot to fail.
+
+   Add shorthand flag `-H` for `--host` (with empty default so if these flags are not specified the
+   latest snapshot will not filter by host name).
+
+   Also add shorthand `-H` for `backup` command.
+
+   https://github.com/restic/restic/issues/2040
+
+ * Bugfix #2068: Correctly return error loading data
+
+   In one case during `prune` and `check`, an error loading data from the backend is not returned
+   properly. This is now corrected.
+
+   https://github.com/restic/restic/issues/1999#issuecomment-433737921
+   https://github.com/restic/restic/pull/2068
+
+ * Bugfix #2095: Consistently use local time for snapshots times
+
+   By default snapshots created with restic backup were set to local time, but when the --time flag
+   was used the provided timestamp was parsed as UTC. With this change all snapshots times are set
+   to local time.
+
+   https://github.com/restic/restic/pull/2095
+
+ * Enhancement #1605: Concurrent restore
+
+   This change significantly improves restore performance, especially when using
+   high-latency remote repositories like B2.
+
+   The implementation now uses several concurrent threads to download and process multiple
+   remote files concurrently. To further reduce restore time, each remote file is downloaded
+   using a single repository request.
+
+   https://github.com/restic/restic/issues/1605
+   https://github.com/restic/restic/pull/1719
+
+ * Enhancement #2089: Increase granularity of the "keep within" retention policy
+
+   The `keep-within` option of the `forget` command now accepts time ranges with an hourly
+   granularity. For example, running `restic forget --keep-within 3d12h` will keep all the
+   snapshots made within three days and twelve hours from the time of the latest snapshot.
+
+   https://github.com/restic/restic/issues/2089
+   https://github.com/restic/restic/pull/2090
+
+ * Enhancement #2097: Add key hinting
+
+   Added a new option `--key-hint` and corresponding environment variable `RESTIC_KEY_HINT`.
+   The key hint is a key ID to try decrypting first, before other keys in the repository.
+
+   This change will benefit repositories with many keys; if the correct key hint is supplied then
+   restic only needs to check one key. If the key hint is incorrect (the key does not exist, or the
+   password is incorrect) then restic will check all keys, as usual.
+
+   https://github.com/restic/restic/issues/2097
+
+ * Enhancement #2017: Mount: Enforce FUSE Unix permissions with allow-other
+
+   The fuse mount (`restic mount`) now lets the kernel check the permissions of the files within
+   snapshots (this is done through the `DefaultPermissions` FUSE option) when the option
+   `--allow-other` is specified.
+
+   To restore the old behavior, we've added the `--no-default-permissions` option. This allows
+   all users that have access to the mount point to access all files within the snapshots.
+
+   https://github.com/restic/restic/pull/2017
+
+ * Enhancement #2070: Make all commands display timestamps in local time
+
+   Restic used to drop the timezone information from displayed timestamps, it now converts
+   timestamps to local time before printing them so the times can be easily compared to.
+
+   https://github.com/restic/restic/pull/2070
+
+ * Enhancement #2085: Allow --files-from to be specified multiple times
+
+   Before, restic took only the last file specified with `--files-from` into account, this is now
+   corrected.
+
+   https://github.com/restic/restic/issues/2085
+   https://github.com/restic/restic/pull/2086
+
+ * Enhancement #2094: Run command to get password
+
+   We've added the `--password-command` option which allows specifying a command that restic
+   runs every time the password for the repository is needed, so it can be integrated with a
+   password manager or keyring. The option can also be set via the environment variable
+   `$RESTIC_PASSWORD_COMMAND`.
+
+   https://github.com/restic/restic/pull/2094
+
+
+Changelog for restic 0.9.3 (2018-10-13)
+=======================================
+
+The following sections list the changes in restic 0.9.3 relevant to
+restic users. The changes are ordered by importance.
+
+Summary
+-------
+
+ * Fix #1935: Remove truncated files from cache
+ * Fix #1978: Do not return an error when the scanner is slower than backup
+ * Enh #1766: Restore: suppress lchown errors when not running as root
+ * Enh #1909: Reject files/dirs by name first
+ * Enh #1940: Add directory filter to ls command
+ * Enh #1967: Use `--host` everywhere
+ * Enh #2028: Display size of cache directories
+ * Enh #1777: Improve the `find` command
+ * Enh #1876: Display reason why forget keeps snapshots
+ * Enh #1891: Accept glob in paths loaded via --files-from
+ * Enh #1920: Vendor dependencies with Go 1.11 Modules
+ * Enh #1949: Add new command `self-update`
+ * Enh #1953: Ls: Add JSON output support for restic ls cmd
+ * Enh #1962: Stream JSON output for ls command
+
+Details
+-------
+
+ * Bugfix #1935: Remove truncated files from cache
+
+   When a file in the local cache is truncated, and restic tries to access data beyond the end of the
+   (cached) file, it used to return an error "EOF". This is now fixed, such truncated files are
+   removed and the data is fetched directly from the backend.
+
+   https://github.com/restic/restic/issues/1935
+
+ * Bugfix #1978: Do not return an error when the scanner is slower than backup
+
+   When restic makes a backup, there's a background task called "scanner" which collects
+   information on how many files and directories are to be saved, in order to display progress
+   information to the user. When the backup finishes faster than the scanner, it is aborted
+   because the result is not needed any more. This logic contained a bug, where quitting the
+   scanner process was treated as an error, and caused restic to print an unhelpful error message
+   ("context canceled").
+
+   https://github.com/restic/restic/issues/1978
+   https://github.com/restic/restic/pull/1991
+
+ * Enhancement #1766: Restore: suppress lchown errors when not running as root
+
+   Like "cp" and "rsync" do, restic now only reports errors for changing the ownership of files
+   during restore if it is run as root, on non-Windows operating systems. On Windows, the error
+   is reported as usual.
+
+   https://github.com/restic/restic/issues/1766
+
+ * Enhancement #1909: Reject files/dirs by name first
+
+   The current scanner/archiver code had an architectural limitation: it always ran the
+   `lstat()` system call on all files and directories before a decision to include/exclude the
+   file/dir was made. This lead to a lot of unnecessary system calls for items that could have been
+   rejected by their name or path only.
+
+   We've changed the archiver/scanner implementation so that it now first rejects by name/path,
+   and only runs the system call on the remaining items. This reduces the number of `lstat()`
+   system calls a lot (depending on the exclude settings).
+
+   https://github.com/restic/restic/issues/1909
+   https://github.com/restic/restic/pull/1912
+
+ * Enhancement #1940: Add directory filter to ls command
+
+   The ls command can now be filtered by directories, so that only files in the given directories
+   will be shown. If the --recursive flag is specified, then ls will traverse subfolders and list
+   their files as well.
+
+   It used to be possible to specify multiple snapshots, but that has been replaced by only one
+   snapshot and the possibility of specifying multiple directories.
+
+   Specifying directories constrains the walk, which can significantly speed up the listing.
+
+   https://github.com/restic/restic/issues/1940
+   https://github.com/restic/restic/pull/1941
+
+ * Enhancement #1967: Use `--host` everywhere
+
+   We now use the flag `--host` for all commands which need a host name, using `--hostname` (e.g.
+   for `restic backup`) still works, but will print a deprecation warning. Also, add the short
+   option `-H` where possible.
+
+   https://github.com/restic/restic/issues/1967
+
+ * Enhancement #2028: Display size of cache directories
+
+   The `cache` command now by default shows the size of the individual cache directories. It can be
+   disabled with `--no-size`.
+
+   https://github.com/restic/restic/issues/2028
+   https://github.com/restic/restic/pull/2033
+
+ * Enhancement #1777: Improve the `find` command
+
+   We've updated the `find` command to support multiple patterns.
+
+   `restic find` is now able to list the snapshots containing a specific tree or blob, or even the
+   snapshots that contain blobs belonging to a given pack. A list of IDs can be given, as long as they
+   all have the same type.
+
+   The command `find` can also display the pack IDs the blobs belong to, if the `--show-pack-id`
+   flag is provided.
+
+   https://github.com/restic/restic/issues/1777
+   https://github.com/restic/restic/pull/1780
+
+ * Enhancement #1876: Display reason why forget keeps snapshots
+
+   We've added a column to the list of snapshots `forget` keeps which details the reasons to keep a
+   particuliar snapshot. This makes debugging policies for forget much easier. Please remember
+   to always try things out with `--dry-run`!
+
+   https://github.com/restic/restic/pull/1876
+
+ * Enhancement #1891: Accept glob in paths loaded via --files-from
+
+   Before that, behaviour was different if paths were appended to command line or from a file,
+   because wild card characters were expanded by shell if appended to command line, but not
+   expanded if loaded from file.
+
+   https://github.com/restic/restic/issues/1891
+
+ * Enhancement #1920: Vendor dependencies with Go 1.11 Modules
+
+   Until now, we've used `dep` for managing dependencies, we've now switch to using Go modules.
+   For users this does not change much, only if you want to compile restic without downloading
+   anything with Go 1.11, then you need to run: `go build -mod=vendor build.go`
+
+   https://github.com/restic/restic/pull/1920
+
+ * Enhancement #1949: Add new command `self-update`
+
+   We have added a new command called `self-update` which downloads the latest released version
+   of restic from GitHub and replaces the current binary with it. It does not rely on any external
+   program (so it'll work everywhere), but still verifies the GPG signature using the embedded
+   GPG public key.
+
+   By default, the `self-update` command is hidden behind the `selfupdate` built tag, which is
+   only set when restic is built using `build.go` (including official releases). The reason for
+   this is that downstream distributions will then not include the command by default, so users
+   are encouraged to use the platform-specific distribution mechanism.
+
+   https://github.com/restic/restic/pull/1949
+
+ * Enhancement #1953: Ls: Add JSON output support for restic ls cmd
+
+   We've implemented listing files in the repository with JSON as output, just pass `--json` as an
+   option to `restic ls`. This makes the output of the command machine readable.
+
+   https://github.com/restic/restic/pull/1953
+
+ * Enhancement #1962: Stream JSON output for ls command
+
+   The `ls` command now supports JSON output with the global `--json` flag, and this change
+   streams out JSON messages one object at a time rather than en entire array buffered in memory
+   before encoding. The advantage is it allows large listings to be handled efficiently.
+
+   Two message types are printed: snapshots and nodes. A snapshot object will precede node
+   objects which belong to that snapshot. The `struct_type` field can be used to determine which
+   kind of message an object is.
+
+   https://github.com/restic/restic/pull/1962
+
+
+Changelog for restic 0.9.2 (2018-08-06)
+=======================================
+
+The following sections list the changes in restic 0.9.2 relevant to
+restic users. The changes are ordered by importance.
+
+Summary
+-------
+
+ * Fix #1854: Allow saving files/dirs on different fs with `--one-file-system`
+ * Fix #1870: Fix restore with --include
+ * Fix #1880: Use `--cache-dir` argument for `check` command
+ * Fix #1893: Return error when exclude file cannot be read
+ * Fix #1861: Fix case-insensitive search with restic find
+ * Enh #1906: Add support for B2 application keys
+ * Enh #874: Add stats command to get information about a repository
+ * Enh #1772: Add restore --verify to verify restored file content
+ * Enh #1853: Add JSON output support to `restic key list`
+ * Enh #1477: S3 backend: accept AWS_SESSION_TOKEN
+ * Enh #1901: Update the Backblaze B2 library
+
+Details
+-------
+
+ * Bugfix #1854: Allow saving files/dirs on different fs with `--one-file-system`
+
+   Restic now allows saving files/dirs on a different file system in a subdir correctly even when
+   `--one-file-system` is specified.
+
+   The first thing the restic archiver code does is to build a tree of the target
+   files/directories. If it detects that a parent directory is already included (e.g. `restic
+   backup /foo /foo/bar/baz`), it'll ignore the latter argument.
+
+   Without `--one-file-system`, that's perfectly valid: If `/foo` is to be archived, it will
+   include `/foo/bar/baz`. But with `--one-file-system`, `/foo/bar/baz` may reside on a
+   different file system, so it won't be included with `/foo`.
+
+   https://github.com/restic/restic/issues/1854
+   https://github.com/restic/restic/pull/1855
+
+ * Bugfix #1870: Fix restore with --include
+
+   We fixed a bug which prevented restic to restore files with an include filter.
+
+   https://github.com/restic/restic/issues/1870
+   https://github.com/restic/restic/pull/1900
+
+ * Bugfix #1880: Use `--cache-dir` argument for `check` command
+
+   `check` command now uses a temporary sub-directory of the specified directory if set using the
+   `--cache-dir` argument. If not set, the cache directory is created in the default temporary
+   directory as before. In either case a temporary cache is used to ensure the actual repository is
+   checked (rather than a local copy).
+
+   The `--cache-dir` argument was not used by the `check` command, instead a cache directory was
+   created in the temporary directory.
+
+   https://github.com/restic/restic/issues/1880
+
+ * Bugfix #1893: Return error when exclude file cannot be read
+
+   A bug was found: when multiple exclude files were passed to restic and one of them could not be
+   read, an error was printed and restic continued, ignoring even the existing exclude files.
+   Now, an error message is printed and restic aborts when an exclude file cannot be read.
+
+   https://github.com/restic/restic/issues/1893
+
+ * Bugfix #1861: Fix case-insensitive search with restic find
+
+   We've fixed the behavior for `restic find -i PATTERN`, which was broken in v0.9.1.
+
+   https://github.com/restic/restic/pull/1861
+
+ * Enhancement #1906: Add support for B2 application keys
+
+   Restic can now use so-called "application keys" which can be created in the B2 dashboard and
+   were only introduced recently. In contrast to the "master key", such keys can be restricted to a
+   specific bucket and/or path.
+
+   https://github.com/restic/restic/issues/1906
+   https://github.com/restic/restic/pull/1914
+
+ * Enhancement #874: Add stats command to get information about a repository
+
+   https://github.com/restic/restic/issues/874
+   https://github.com/restic/restic/pull/1729
+
+ * Enhancement #1772: Add restore --verify to verify restored file content
+
+   Restore will print error message if restored file content does not match expected SHA256
+   checksum
+
+   https://github.com/restic/restic/pull/1772
+
+ * Enhancement #1853: Add JSON output support to `restic key list`
+
+   This PR enables users to get the output of `restic key list` in JSON in addition to the existing
+   table format.
+
+   https://github.com/restic/restic/pull/1853
+
+ * Enhancement #1477: S3 backend: accept AWS_SESSION_TOKEN
+
+   Before, it was not possible to use s3 backend with AWS temporary security credentials(with
+   AWS_SESSION_TOKEN). This change gives higher priority to credentials.EnvAWS credentials
+   provider.
+
+   https://github.com/restic/restic/issues/1477
+   https://github.com/restic/restic/pull/1479
+   https://github.com/restic/restic/pull/1647
+
+ * Enhancement #1901: Update the Backblaze B2 library
+
+   We've updated the library we're using for accessing the Backblaze B2 service to 0.5.0 to
+   include support for upcoming so-called "application keys". With this feature, you can create
+   access credentials for B2 which are restricted to e.g. a single bucket or even a sub-directory
+   of a bucket.
+
+   https://github.com/restic/restic/pull/1901
+   https://github.com/kurin/blazer
+
+
 Changelog for restic 0.9.1 (2018-06-10)
 =======================================
 

CONTRIBUTING.md

@@ -46,12 +46,15 @@ Remember, the easier it is for us to reproduce the bug, the earlier it will be
 corrected!
 
 In addition, you can compile restic with debug support by running
-`go run build.go -tags debug` and instructing it to create a debug log by
-setting the environment variable `DEBUG_LOG` to a file, e.g. like this:
+`go run -mod=vendor build.go -tags debug` and instructing it to create a debug
+log by setting the environment variable `DEBUG_LOG` to a file, e.g. like this:
 
     $ export DEBUG_LOG=/tmp/restic-debug.log
     $ restic backup ~/work
 
+For Go < 1.11, you need to remove the `-mod=vendor` option from the build
+command.
+
 Please be aware that the debug log file will contain potentially sensitive
 things like file and directory names, so please either redact it before
 uploading it somewhere or post only the parts that are really relevant.
@@ -60,9 +63,37 @@ uploading it somewhere or post only the parts that are really relevant.
 Development Environment
 =======================
 
-In order to compile restic with the `go` tool directly, it needs to be checked
-out at the right path within a `GOPATH`. The concept of a `GOPATH` is explained
-in ["How to write Go code"](https://golang.org/doc/code.html).
+The repository contains several sets of directories with code: `cmd/` and
+`internal/` contain the code written for restic, whereas `vendor/` contains
+copies of libraries restic depends on. The libraries are managed with the
+command `go mod vendor`.
+
+Go >= 1.11
+----------
+
+For Go version 1.11 or later, you should clone the repo (without having
+`$GOPATH` set) and `cd` into the directory:
+
+    $ unset GOPATH
+    $ git clone https://github.com/restic/restic
+    $ cd restic
+
+Then use the `go` tool to build restic:
+
+    $ go build ./cmd/restic
+    $ ./restic version
+    restic 0.9.2-dev (compiled manually) compiled with go1.11 on linux/amd64
+
+You can run all tests with the following command:
+
+    $ go test ./...
+
+Go < 1.11
+---------
+
+In order to compile restic with Go before 1.11, it needs to be checked out at
+the right path within a `GOPATH`. The concept of a `GOPATH` is explained in
+["How to write Go code"](https://golang.org/doc/code.html).
 
 If you do not have a directory with Go code yet, executing the following
 instructions in your shell will create one for you and check out the restic
@@ -83,12 +114,7 @@ You can then build restic as follows:
 
 The following commands can be used to run all the tests:
 
-    $ go test ./cmd/... ./internal/...
-
-The repository contains two sets of directories with code: `cmd/` and
-`internal/` contain the code written for restic, whereas `vendor/` contains
-copies of libraries restic depends on. The libraries are managed with the
-[`dep`](https://github.com/golang/dep) tool.
+    $ go test ./...
 
 Providing Patches
 =================
@@ -141,13 +167,14 @@ run
 
     gofmt -w **/*.go
 
-in the project root directory before committing. Installing the script
-`fmt-check` from https://github.com/edsrzf/gofmt-git-hook locally as a
-pre-commit hook checks formatting before committing automatically, just copy
-this script to `.git/hooks/pre-commit`.
+in the project root directory before committing. For each Pull Request, the
+formatting is tested with `gofmt` for the latest stable version of Go.
+Installing the script `fmt-check` from https://github.com/edsrzf/gofmt-git-hook
+locally as a pre-commit hook checks formatting before committing automatically,
+just copy this script to `.git/hooks/pre-commit`.
 
 For each pull request, several different systems run the integration tests on
-Linux, OS X and Windows. We won't merge any code that does not pass all tests
+Linux, macOS and Windows. We won't merge any code that does not pass all tests
 for all systems, so when a tests fails, try to find out what's wrong and fix
 it. If you need help on this, please leave a comment in the pull request, and
 we'll be glad to assist. Having a PR with failing integration tests is nothing
@@ -164,7 +191,7 @@ history and triaging bugs much easier.
 Git commit messages have a very terse summary in the first line of the commit
 message, followed by an empty line, followed by a more verbose description or a
 List of changed things. For examples, please refer to the excellent [How to
-Write a Git Commit Message](http://chris.beams.io/posts/git-commit/).
+Write a Git Commit Message](https://chris.beams.io/posts/git-commit/).
 
 If you change/add multiple different things that aren't related at all, try to
 make several smaller commits. This is much easier to review. Using `git add -p`

Gopkg.lock

@@ -1,255 +0,0 @@
-# This file is autogenerated, do not edit; changes may be undone by the next 'dep ensure'.
-
-
-[[projects]]
-  branch = "master"
-  name = "bazil.org/fuse"
-  packages = [".","fs","fuseutil"]
-  revision = "371fbbdaa8987b715bdd21d6adc4c9b20155f748"
-
-[[projects]]
-  name = "cloud.google.com/go"
-  packages = ["compute/metadata"]
-  revision = "4b98a6370e36d7a85192e7bad08a4ebd82eac2a8"
-  version = "v0.20.0"
-
-[[projects]]
-  name = "github.com/Azure/azure-sdk-for-go"
-  packages = ["storage","version"]
-  revision = "56332fec5b308fbb6615fa1af6117394cdba186d"
-  version = "v15.0.0"
-
-[[projects]]
-  name = "github.com/Azure/go-autorest"
-  packages = ["autorest","autorest/adal","autorest/azure","autorest/date"]
-  revision = "ed4b7f5bf1ec0c9ede1fda2681d96771282f2862"
-  version = "v10.4.0"
-
-[[projects]]
-  name = "github.com/cenkalti/backoff"
-  packages = ["."]
-  revision = "2ea60e5f094469f9e65adb9cd103795b73ae743e"
-  version = "v2.0.0"
-
-[[projects]]
-  name = "github.com/cpuguy83/go-md2man"
-  packages = ["md2man"]
-  revision = "20f5889cbdc3c73dbd2862796665e7c465ade7d1"
-  version = "v1.0.8"
-
-[[projects]]
-  name = "github.com/dgrijalva/jwt-go"
-  packages = ["."]
-  revision = "06ea1031745cb8b3dab3f6a236daf2b0aa468b7e"
-  version = "v3.2.0"
-
-[[projects]]
-  branch = "master"
-  name = "github.com/dustin/go-humanize"
-  packages = ["."]
-  revision = "bb3d318650d48840a39aa21a027c6630e198e626"
-
-[[projects]]
-  name = "github.com/elithrar/simple-scrypt"
-  packages = ["."]
-  revision = "d150773194090feb6c897805a7bcea8d49544e2c"
-  version = "v1.3.0"
-
-[[projects]]
-  name = "github.com/go-ini/ini"
-  packages = ["."]
-  revision = "6333e38ac20b8949a8dd68baa3650f4dee8f39f0"
-  version = "v1.33.0"
-
-[[projects]]
-  name = "github.com/golang/protobuf"
-  packages = ["proto"]
-  revision = "925541529c1fa6821df4e44ce2723319eb2be768"
-  version = "v1.0.0"
-
-[[projects]]
-  name = "github.com/google/go-cmp"
-  packages = ["cmp","cmp/internal/diff","cmp/internal/function","cmp/internal/value"]
-  revision = "8099a9787ce5dc5984ed879a3bda47dc730a8e97"
-  version = "v0.1.0"
-
-[[projects]]
-  name = "github.com/inconshreveable/mousetrap"
-  packages = ["."]
-  revision = "76626ae9c91c4f2a10f34cad8ce83ea42c93bb75"
-  version = "v1.0"
-
-[[projects]]
-  name = "github.com/juju/ratelimit"
-  packages = ["."]
-  revision = "59fac5042749a5afb9af70e813da1dd5474f0167"
-  version = "1.0.1"
-
-[[projects]]
-  branch = "master"
-  name = "github.com/kr/fs"
-  packages = ["."]
-  revision = "2788f0dbd16903de03cb8186e5c7d97b69ad387b"
-
-[[projects]]
-  name = "github.com/kurin/blazer"
-  packages = ["b2","base","internal/b2assets","internal/b2types","internal/blog","x/window"]
-  revision = "318e9768bf9a0fe52a64b9f8fe74f4f5caef6452"
-  version = "v0.4.4"
-
-[[projects]]
-  name = "github.com/marstr/guid"
-  packages = ["."]
-  revision = "8bd9a64bf37eb297b492a4101fb28e80ac0b290f"
-  version = "v1.1.0"
-
-[[projects]]
-  name = "github.com/mattn/go-isatty"
-  packages = ["."]
-  revision = "0360b2af4f38e8d38c7fce2a9f4e702702d73a39"
-  version = "v0.0.3"
-
-[[projects]]
-  name = "github.com/minio/minio-go"
-  packages = [".","pkg/credentials","pkg/encrypt","pkg/policy","pkg/s3signer","pkg/s3utils","pkg/set"]
-  revision = "66252c2a3c15f7b90cc8493d497a04ac3b6e3606"
-  version = "5.0.0"
-
-[[projects]]
-  branch = "master"
-  name = "github.com/mitchellh/go-homedir"
-  packages = ["."]
-  revision = "b8bc1bf767474819792c23f32d8286a45736f1c6"
-
-[[projects]]
-  branch = "master"
-  name = "github.com/ncw/swift"
-  packages = ["."]
-  revision = "b2a7479cf26fa841ff90dd932d0221cb5c50782d"
-
-[[projects]]
-  name = "github.com/pkg/errors"
-  packages = ["."]
-  revision = "645ef00459ed84a119197bfb8d8205042c6df63d"
-  version = "v0.8.0"
-
-[[projects]]
-  name = "github.com/pkg/profile"
-  packages = ["."]
-  revision = "5b67d428864e92711fcbd2f8629456121a56d91f"
-  version = "v1.2.1"
-
-[[projects]]
-  name = "github.com/pkg/sftp"
-  packages = ["."]
-  revision = "49488377fa2f14143ba3067cf7555f60f6c7b550"
-  version = "1.5.0"
-
-[[projects]]
-  name = "github.com/pkg/xattr"
-  packages = ["."]
-  revision = "1d7b7ffe7c46974a836eb583b7452f22de1c18cf"
-  version = "v0.2.3"
-
-[[projects]]
-  name = "github.com/restic/chunker"
-  packages = ["."]
-  revision = "db83917be3b88cc307464b7d8a221c173e34a0db"
-  version = "v0.2.0"
-
-[[projects]]
-  name = "github.com/russross/blackfriday"
-  packages = ["."]
-  revision = "55d61fa8aa702f59229e6cff85793c22e580eaf5"
-  version = "v1.5.1"
-
-[[projects]]
-  name = "github.com/satori/go.uuid"
-  packages = ["."]
-  revision = "f58768cc1a7a7e77a3bd49e98cdd21419399b6a3"
-  version = "v1.2.0"
-
-[[projects]]
-  name = "github.com/sirupsen/logrus"
-  packages = ["."]
-  revision = "c155da19408a8799da419ed3eeb0cb5db0ad5dbc"
-  version = "v1.0.5"
-
-[[projects]]
-  name = "github.com/spf13/cobra"
-  packages = [".","doc"]
-  revision = "a1f051bc3eba734da4772d60e2d677f47cf93ef4"
-  version = "v0.0.2"
-
-[[projects]]
-  name = "github.com/spf13/pflag"
-  packages = ["."]
-  revision = "e57e3eeb33f795204c1ca35f56c44f83227c6e66"
-  version = "v1.0.0"
-
-[[projects]]
-  branch = "master"
-  name = "golang.org/x/crypto"
-  packages = ["argon2","blake2b","curve25519","ed25519","ed25519/internal/edwards25519","internal/chacha20","pbkdf2","poly1305","scrypt","ssh","ssh/terminal"]
-  revision = "4ec37c66abab2c7e02ae775328b2ff001c3f025a"
-
-[[projects]]
-  branch = "master"
-  name = "golang.org/x/net"
-  packages = ["context","context/ctxhttp","http2","http2/hpack","idna","lex/httplex"]
-  revision = "6078986fec03a1dcc236c34816c71b0e05018fda"
-
-[[projects]]
-  branch = "master"
-  name = "golang.org/x/oauth2"
-  packages = [".","google","internal","jws","jwt"]
-  revision = "fdc9e635145ae97e6c2cb777c48305600cf515cb"
-
-[[projects]]
-  branch = "master"
-  name = "golang.org/x/sync"
-  packages = ["errgroup"]
-  revision = "1d60e4601c6fd243af51cc01ddf169918a5407ca"
-
-[[projects]]
-  branch = "master"
-  name = "golang.org/x/sys"
-  packages = ["cpu","unix","windows"]
-  revision = "7db1c3b1a98089d0071c84f646ff5c96aad43682"
-
-[[projects]]
-  name = "golang.org/x/text"
-  packages = ["collate","collate/build","encoding","encoding/internal","encoding/internal/identifier","encoding/unicode","internal/colltab","internal/gen","internal/tag","internal/triegen","internal/ucd","internal/utf8internal","language","runes","secure/bidirule","transform","unicode/bidi","unicode/cldr","unicode/norm","unicode/rangetable"]
-  revision = "f21a4dfb5e38f5895301dc265a8def02365cc3d0"
-  version = "v0.3.0"
-
-[[projects]]
-  branch = "master"
-  name = "google.golang.org/api"
-  packages = ["gensupport","googleapi","googleapi/internal/uritemplates","storage/v1"]
-  revision = "dbbc13f71100fa6ece308335445fca6bb0dd5c2f"
-
-[[projects]]
-  name = "google.golang.org/appengine"
-  packages = [".","internal","internal/app_identity","internal/base","internal/datastore","internal/log","internal/modules","internal/remote_api","internal/urlfetch","urlfetch"]
-  revision = "150dc57a1b433e64154302bdc40b6bb8aefa313a"
-  version = "v1.0.0"
-
-[[projects]]
-  branch = "v2"
-  name = "gopkg.in/tomb.v2"
-  packages = ["."]
-  revision = "d5d1b5820637886def9eef33e03a27a9f166942c"
-
-[[projects]]
-  name = "gopkg.in/yaml.v2"
-  packages = ["."]
-  revision = "5420a8b6744d3b0345ab293f6fcba19c978f1183"
-  version = "v2.2.1"
-
-[solve-meta]
-  analyzer-name = "dep"
-  analyzer-version = 1
-  inputs-digest = "a5de339cba7570216b212439b90e1e6c384c94be8342fe7755b7cb66aa0a3440"
-  solver-name = "gps-cdcl"
-  solver-version = 1

Gopkg.toml

@@ -1,21 +0,0 @@
-
-# Gopkg.toml example
-#
-# Refer to https://github.com/golang/dep/blob/master/docs/Gopkg.toml.md
-# for detailed Gopkg.toml documentation.
-#
-# required = ["github.com/user/thing/cmd/thing"]
-# ignored = ["github.com/user/project/pkgX", "bitbucket.org/user/project/pkgA/pkgY"]
-#
-# [[constraint]]
-#   name = "github.com/user/project"
-#   version = "1.0.0"
-#
-# [[constraint]]
-#   name = "github.com/user/project2"
-#   branch = "dev"
-#   source = "github.com/myfork/project2"
-#
-# [[override]]
-#  name = "github.com/x/y"
-#  version = "2.4.0"

Makefile

@@ -3,7 +3,7 @@
 all: restic
 
 restic:
-	go run build.go
+	go run -mod=vendor build.go || go run build.go
 
 clean:
 	rm -f restic

README.rst

@@ -1,9 +1,9 @@
-|Documentation| |Build Status| |Build status| |Report Card| |Say Thanks| |TestCoverage|
+|Documentation| |Build Status| |Build status| |Report Card| |Say Thanks| |TestCoverage| |Reviewed by Hound|
 
 Introduction
 ------------
 
-restic is a backup program that is fast, efficient and secure.
+restic is a backup program that is fast, efficient and secure. It supports the three major operating systems (Linux, macOS, Windows) and a few smaller ones (FreeBSD, OpenBSD).
 
 For detailed usage and installation instructions check out the `documentation <https://restic.readthedocs.io/en/latest>`__.
 
@@ -111,6 +111,14 @@ License
 Restic is licensed under `BSD 2-Clause License <https://opensource.org/licenses/BSD-2-Clause>`__. You can find the
 complete text in ``LICENSE``.
 
+Sponsorship
+-----------
+
+Backend integration tests for Google Cloud Storage and Microsoft Azure Blob
+Storage are sponsored by `AppsCode <https://appscode.com>`__!
+
+|AppsCode|
+
 .. |Documentation| image:: https://readthedocs.org/projects/restic/badge/?version=latest
    :target: https://restic.readthedocs.io/en/latest/?badge=latest
 .. |Build Status| image:: https://travis-ci.com/restic/restic.svg?branch=master
@@ -123,3 +131,7 @@ complete text in ``LICENSE``.
    :target: https://saythanks.io/to/restic
 .. |TestCoverage| image:: https://codecov.io/gh/restic/restic/branch/master/graph/badge.svg
    :target: https://codecov.io/gh/restic/restic
+.. |AppsCode| image:: https://cdn.appscode.com/images/logo/appscode/ac-logo-color.png
+   :target: https://appscode.com
+.. |Reviewed by Hound| image:: https://img.shields.io/badge/Reviewed_by-Hound-8E64B0.svg
+   :target: https://houndci.com

VERSION

@@ -1 +1 @@
-0.9.1
+0.9.5

appveyor.yml

@@ -7,6 +7,9 @@ branches:
   only:
     - master
 
+cache:
+  - '%LocalAppData%\go-build'
+
 init:
   - ps: >-
       $app = Get-WmiObject -Class Win32_Product -Filter "Vendor = 'http://golang.org'"
@@ -17,8 +20,8 @@ init:
 
 install:
   - rmdir c:\go /s /q
-  - appveyor DownloadFile https://dl.google.com/go/go1.10.windows-amd64.msi
-  - msiexec /i go1.10.windows-amd64.msi /q
+  - appveyor DownloadFile https://dl.google.com/go/go1.12.1.windows-amd64.msi
+  - msiexec /i go1.12.1.windows-amd64.msi /q
   - go version
   - go env
   - appveyor DownloadFile http://sourceforge.netcologne.de/project/gnuwin32/tar/1.13-1/tar-1.13-1-bin.zip -FileName tar.zip
@@ -26,4 +29,4 @@ install:
   - set PATH=bin/;%PATH%
 
 build_script:
-  - go run run_integration_tests.go
+  - go run -mod=vendor run_integration_tests.go

build.go

@@ -1,3 +1,18 @@
+// Description
+//
+// This program aims to make building Go programs for end users easier by just
+// calling it with `go run`, without having to setup a GOPATH.
+//
+// For Go < 1.11, it'll create a new GOPATH in a temporary directory, then run
+// `go build` on the package configured as Main in the Config struct.
+//
+// For Go >= 1.11 if the file go.mod is present, it'll use Go modules and not
+// setup a GOPATH. It builds the package configured as Main in the Config
+// struct with `go build -mod=vendor` to use the vendored dependencies.
+// The variable GOPROXY is set to `off` so that no network calls are made. All
+// files are copied to a temporary directory before `go build` is called within
+// that directory.
+
 // BSD 2-Clause License
 //
 // Copyright (c) 2016-2018, Alexander Neumann <alexander@bumpern.de>
@@ -37,7 +52,6 @@ import (
 	"io/ioutil"
 	"os"
 	"os/exec"
-	"path"
 	"path/filepath"
 	"runtime"
 	"strconv"
@@ -46,23 +60,22 @@ import (
 
 // config contains the configuration for the program to build.
 var config = Config{
-	Name:      "restic",                              // name of the program executable and directory
-	Namespace: "github.com/restic/restic",            // subdir of GOPATH, e.g. "github.com/foo/bar"
-	Main:      "github.com/restic/restic/cmd/restic", // package name for the main package
-	Tests: []string{ // tests to run
-		"github.com/restic/restic/internal/...",
-		"github.com/restic/restic/cmd/...",
-	},
-	MinVersion: GoVersion{Major: 1, Minor: 9, Patch: 0}, // minimum Go version supported
+	Name:             "restic",                                 // name of the program executable and directory
+	Namespace:        "github.com/restic/restic",               // subdir of GOPATH, e.g. "github.com/foo/bar"
+	Main:             "./cmd/restic",                           // package name for the main package
+	DefaultBuildTags: []string{"selfupdate"},                   // specify build tags which are always used
+	Tests:            []string{"./..."},                        // tests to run
+	MinVersion:       GoVersion{Major: 1, Minor: 10, Patch: 0}, // minimum Go version supported
 }
 
 // Config configures the build.
 type Config struct {
-	Name       string
-	Namespace  string
-	Main       string
-	Tests      []string
-	MinVersion GoVersion
+	Name             string
+	Namespace        string
+	Main             string
+	DefaultBuildTags []string
+	Tests            []string
+	MinVersion       GoVersion
 }
 
 var (
@@ -71,41 +84,12 @@ var (
 	runTests   bool
 	enableCGO  bool
 	enablePIE  bool
+	goVersion  = ParseGoVersion(runtime.Version())
 )
 
-// specialDir returns true if the file begins with a special character ('.' or '_').
-func specialDir(name string) bool {
-	if name == "." {
-		return false
-	}
-
-	base := filepath.Base(name)
-	if base == "vendor" || base[0] == '_' || base[0] == '.' {
-		return true
-	}
-
-	return false
-}
-
-// excludePath returns true if the file should not be copied to the new GOPATH.
-func excludePath(name string) bool {
-	ext := path.Ext(name)
-	if ext == ".go" || ext == ".s" || ext == ".h" {
-		return false
-	}
-
-	parentDir := filepath.Base(filepath.Dir(name))
-	if parentDir == "testdata" {
-		return false
-	}
-
-	return true
-}
-
-// updateGopath builds a valid GOPATH at dst, with all Go files in src/ copied
-// to dst/prefix/, so calling
+// copy all Go files in src to dst, creating directories on the fly, so calling
 //
-//   updateGopath("/tmp/gopath", "/home/u/restic", "github.com/restic/restic")
+//   copy("/tmp/gopath/src/github.com/restic/restic", "/home/u/restic")
 //
 // with "/home/u/restic" containing the file "foo.go" yields the following tree
 // at "/tmp/gopath":
@@ -116,19 +100,15 @@ func excludePath(name string) bool {
 //           └── restic
 //               └── restic
 //                   └── foo.go
-func updateGopath(dst, src, prefix string) error {
-	verbosePrintf("copy contents of %v to %v\n", src, filepath.Join(dst, prefix))
+func copy(dst, src string) error {
+	verbosePrintf("copy contents of %v to %v\n", src, dst)
 	return filepath.Walk(src, func(name string, fi os.FileInfo, err error) error {
 		if name == src {
 			return err
 		}
 
-		if specialDir(name) {
-			if fi.IsDir() {
-				return filepath.SkipDir
-			}
-
-			return nil
+		if name == ".git" {
+			return filepath.SkipDir
 		}
 
 		if err != nil {
@@ -139,17 +119,13 @@ func updateGopath(dst, src, prefix string) error {
 			return nil
 		}
 
-		if excludePath(name) {
-			return nil
-		}
-
 		intermediatePath, err := filepath.Rel(src, name)
 		if err != nil {
 			return err
 		}
 
 		fileSrc := filepath.Join(src, intermediatePath)
-		fileDst := filepath.Join(dst, "src", prefix, intermediatePath)
+		fileDst := filepath.Join(dst, intermediatePath)
 
 		return copyFile(fileDst, fileSrc)
 	})
@@ -164,6 +140,15 @@ func directoryExists(dirname string) bool {
 	return stat.IsDir()
 }
 
+func fileExists(filename string) bool {
+	stat, err := os.Stat(filename)
+	if err != nil && os.IsNotExist(err) {
+		return false
+	}
+
+	return stat.Mode().IsRegular()
+}
+
 // copyFile creates dst from src, preserving file attributes and timestamps.
 func copyFile(dst, src string) error {
 	fi, err := os.Stat(src)
@@ -183,30 +168,34 @@ func copyFile(dst, src string) error {
 
 	fdst, err := os.Create(dst)
 	if err != nil {
+		_ = fsrc.Close()
 		return err
 	}
 
-	if _, err = io.Copy(fdst, fsrc); err != nil {
+	_, err = io.Copy(fdst, fsrc)
+	if err != nil {
+		_ = fsrc.Close()
+		_ = fdst.Close()
 		return err
 	}
 
-	if err == nil {
-		err = fsrc.Close()
+	err = fdst.Close()
+	if err != nil {
+		_ = fsrc.Close()
+		return err
 	}
 
-	if err == nil {
-		err = fdst.Close()
+	err = fsrc.Close()
+	if err != nil {
+		return err
 	}
 
-	if err == nil {
-		err = os.Chmod(dst, fi.Mode())
+	err = os.Chmod(dst, fi.Mode())
+	if err != nil {
+		return err
 	}
 
-	if err == nil {
-		err = os.Chtimes(dst, fi.ModTime(), fi.ModTime())
-	}
-
-	return nil
+	return os.Chtimes(dst, fi.ModTime(), fi.ModTime())
 }
 
 // die prints the message with fmt.Fprintf() to stderr and exits with an error
@@ -222,7 +211,7 @@ func showUsage(output io.Writer) {
 	fmt.Fprintf(output, "OPTIONS:\n")
 	fmt.Fprintf(output, "  -v     --verbose       output more messages\n")
 	fmt.Fprintf(output, "  -t     --tags          specify additional build tags\n")
-	fmt.Fprintf(output, "  -k     --keep-gopath   do not remove the GOPATH after build\n")
+	fmt.Fprintf(output, "  -k     --keep-tempdir  do not remove the temporary directory after build\n")
 	fmt.Fprintf(output, "  -T     --test          run tests\n")
 	fmt.Fprintf(output, "  -o     --output        set output file name\n")
 	fmt.Fprintf(output, "         --enable-cgo    use CGO to link against libc\n")
@@ -241,11 +230,20 @@ func verbosePrintf(message string, args ...interface{}) {
 	fmt.Printf("build: "+message, args...)
 }
 
-// cleanEnv returns a clean environment with GOPATH and GOBIN removed (if
-// present).
+// cleanEnv returns a clean environment with GOPATH, GOBIN and GO111MODULE
+// removed (if present).
 func cleanEnv() (env []string) {
+	removeKeys := map[string]struct{}{
+		"GOPATH":      struct{}{},
+		"GOBIN":       struct{}{},
+		"GO111MODULE": struct{}{},
+	}
+
 	for _, v := range os.Environ() {
-		if strings.HasPrefix(v, "GOPATH=") || strings.HasPrefix(v, "GOBIN=") {
+		data := strings.SplitN(v, "=", 2)
+		name := data[0]
+
+		if _, ok := removeKeys[name]; ok {
 			continue
 		}
 
@@ -256,17 +254,17 @@ func cleanEnv() (env []string) {
 }
 
 // build runs "go build args..." with GOPATH set to gopath.
-func build(cwd string, ver GoVersion, goos, goarch, goarm, gopath string, args ...string) error {
+func build(cwd string, env map[string]string, args ...string) error {
 	a := []string{"build"}
 
-	if ver.AtLeast(GoVersion{1, 10, 0}) {
+	if goVersion.AtLeast(GoVersion{1, 10, 0}) {
 		verbosePrintf("Go version is at least 1.10, using new syntax for -gcflags\n")
 		// use new prefix
-		a = append(a, "-asmflags", fmt.Sprintf("all=-trimpath=%s", gopath))
-		a = append(a, "-gcflags", fmt.Sprintf("all=-trimpath=%s", gopath))
+		a = append(a, "-asmflags", fmt.Sprintf("all=-trimpath=%s", cwd))
+		a = append(a, "-gcflags", fmt.Sprintf("all=-trimpath=%s", cwd))
 	} else {
-		a = append(a, "-asmflags", fmt.Sprintf("-trimpath=%s", gopath))
-		a = append(a, "-gcflags", fmt.Sprintf("-trimpath=%s", gopath))
+		a = append(a, "-asmflags", fmt.Sprintf("-trimpath=%s", cwd))
+		a = append(a, "-gcflags", fmt.Sprintf("-trimpath=%s", cwd))
 	}
 	if enablePIE {
 		a = append(a, "-buildmode=pie")
@@ -274,9 +272,9 @@ func build(cwd string, ver GoVersion, goos, goarch, goarm, gopath string, args .
 
 	a = append(a, args...)
 	cmd := exec.Command("go", a...)
-	cmd.Env = append(cleanEnv(), "GOPATH="+gopath, "GOARCH="+goarch, "GOOS="+goos)
-	if goarm != "" {
-		cmd.Env = append(cmd.Env, "GOARM="+goarm)
+	cmd.Env = append(cleanEnv(), "GOPROXY=off")
+	for k, v := range env {
+		cmd.Env = append(cmd.Env, k+"="+v)
 	}
 	if !enableCGO {
 		cmd.Env = append(cmd.Env, "CGO_ENABLED=0")
@@ -285,20 +283,30 @@ func build(cwd string, ver GoVersion, goos, goarch, goarm, gopath string, args .
 	cmd.Dir = cwd
 	cmd.Stdout = os.Stdout
 	cmd.Stderr = os.Stderr
-	verbosePrintf("go %s\n", a)
+
+	verbosePrintf("chdir %q\n", cwd)
+	verbosePrintf("go %q\n", a)
 
 	return cmd.Run()
 }
 
 // test runs "go test args..." with GOPATH set to gopath.
-func test(cwd, gopath string, args ...string) error {
-	args = append([]string{"test"}, args...)
+func test(cwd string, env map[string]string, args ...string) error {
+	args = append([]string{"test", "-count", "1"}, args...)
 	cmd := exec.Command("go", args...)
-	cmd.Env = append(cleanEnv(), "GOPATH="+gopath)
+	cmd.Env = append(cleanEnv(), "GOPROXY=off")
+	for k, v := range env {
+		cmd.Env = append(cmd.Env, k+"="+v)
+	}
+	if !enableCGO {
+		cmd.Env = append(cmd.Env, "CGO_ENABLED=0")
+	}
 	cmd.Dir = cwd
 	cmd.Stdout = os.Stdout
 	cmd.Stderr = os.Stderr
-	verbosePrintf("go %s\n", args)
+
+	verbosePrintf("chdir %q\n", cwd)
+	verbosePrintf("go %q\n", args)
 
 	return cmd.Run()
 }
@@ -446,22 +454,24 @@ func (v GoVersion) String() string {
 }
 
 func main() {
-	ver := ParseGoVersion(runtime.Version())
-	if !ver.AtLeast(config.MinVersion) {
-		fmt.Fprintf(os.Stderr, "%s detected, this program requires at least %s\n", ver, config.MinVersion)
+	if !goVersion.AtLeast(config.MinVersion) {
+		fmt.Fprintf(os.Stderr, "%s detected, this program requires at least %s\n", goVersion, config.MinVersion)
 		os.Exit(1)
 	}
 
-	buildTags := []string{}
+	buildTags := config.DefaultBuildTags
 
 	skipNext := false
 	params := os.Args[1:]
 
-	targetGOOS := runtime.GOOS
-	targetGOARCH := runtime.GOARCH
-	targetGOARM := ""
+	goEnv := map[string]string{}
+	buildEnv := map[string]string{
+		"GOOS":   runtime.GOOS,
+		"GOARCH": runtime.GOARCH,
+		"GOARM":  "",
+	}
 
-	gopath := ""
+	tempdir := ""
 
 	var outputFilename string
 
@@ -481,13 +491,13 @@ func main() {
 				die("-t given but no tag specified")
 			}
 			skipNext = true
-			buildTags = strings.Split(params[i+1], " ")
+			buildTags = append(buildTags, strings.Split(params[i+1], " ")...)
 		case "-o", "--output":
 			skipNext = true
 			outputFilename = params[i+1]
 		case "--tempdir":
 			skipNext = true
-			gopath = params[i+1]
+			tempdir = params[i+1]
 		case "-T", "--test":
 			runTests = true
 		case "--enable-cgo":
@@ -496,13 +506,13 @@ func main() {
 			enablePIE = true
 		case "--goos":
 			skipNext = true
-			targetGOOS = params[i+1]
+			buildEnv["GOOS"] = params[i+1]
 		case "--goarch":
 			skipNext = true
-			targetGOARCH = params[i+1]
+			buildEnv["GOARCH"] = params[i+1]
 		case "--goarm":
 			skipNext = true
-			targetGOARM = params[i+1]
+			buildEnv["GOARM"] = params[i+1]
 		case "-h":
 			showUsage(os.Stdout)
 			return
@@ -513,12 +523,7 @@ func main() {
 		}
 	}
 
-	verbosePrintf("detected Go version %v\n", ver)
-
-	if len(buildTags) == 0 {
-		verbosePrintf("adding build-tag release\n")
-		buildTags = []string{"release"}
-	}
+	verbosePrintf("detected Go version %v\n", goVersion)
 
 	for i := range buildTags {
 		buildTags[i] = strings.TrimSpace(buildTags[i])
@@ -531,50 +536,16 @@ func main() {
 		die("Getwd(): %v\n", err)
 	}
 
-	if gopath == "" {
-		gopath, err = ioutil.TempDir("", fmt.Sprintf("%v-build-", config.Name))
-		if err != nil {
-			die("TempDir(): %v\n", err)
-		}
-	}
-
-	verbosePrintf("create GOPATH at %v\n", gopath)
-	if err = updateGopath(gopath, root, config.Namespace); err != nil {
-		die("copying files from %v/src to %v/src failed: %v\n", root, gopath, err)
-	}
-
-	vendor := filepath.Join(root, "vendor")
-	if directoryExists(vendor) {
-		if err = updateGopath(gopath, vendor, filepath.Join(config.Namespace, "vendor")); err != nil {
-			die("copying files from %v to %v failed: %v\n", root, gopath, err)
-		}
-	}
-
-	defer func() {
-		if !keepGopath {
-			verbosePrintf("remove %v\n", gopath)
-			if err = os.RemoveAll(gopath); err != nil {
-				die("remove GOPATH at %s failed: %v\n", err)
-			}
-		} else {
-			verbosePrintf("leaving temporary GOPATH at %v\n", gopath)
-		}
-	}()
-
 	if outputFilename == "" {
 		outputFilename = config.Name
-		if targetGOOS == "windows" {
+		if buildEnv["GOOS"] == "windows" {
 			outputFilename += ".exe"
 		}
 	}
 
-	cwd, err := os.Getwd()
-	if err != nil {
-		die("Getwd() returned %v\n", err)
-	}
 	output := outputFilename
 	if !filepath.IsAbs(output) {
-		output = filepath.Join(cwd, output)
+		output = filepath.Join(root, output)
 	}
 
 	version := getVersion()
@@ -585,13 +556,65 @@ func main() {
 	ldflags := "-s -w " + constants.LDFlags()
 	verbosePrintf("ldflags: %s\n", ldflags)
 
-	args := []string{
-		"-tags", strings.Join(buildTags, " "),
-		"-ldflags", ldflags,
-		"-o", output, config.Main,
+	var (
+		buildArgs []string
+		testArgs  []string
+	)
+
+	mainPackage := config.Main
+	if strings.HasPrefix(mainPackage, config.Namespace) {
+		mainPackage = strings.Replace(mainPackage, config.Namespace, "./", 1)
 	}
 
-	err = build(filepath.Join(gopath, "src"), ver, targetGOOS, targetGOARCH, targetGOARM, gopath, args...)
+	buildTarget := filepath.FromSlash(mainPackage)
+	buildCWD := ""
+
+	if goVersion.AtLeast(GoVersion{1, 11, 0}) && fileExists("go.mod") {
+		verbosePrintf("Go >= 1.11 and 'go.mod' found, building with modules\n")
+		buildCWD = root
+
+		buildArgs = append(buildArgs, "-mod=vendor")
+		testArgs = append(testArgs, "-mod=vendor")
+	} else {
+		if tempdir == "" {
+			tempdir, err = ioutil.TempDir("", fmt.Sprintf("%v-build-", config.Name))
+			if err != nil {
+				die("TempDir(): %v\n", err)
+			}
+		}
+
+		verbosePrintf("Go < 1.11 or 'go.mod' not found, create GOPATH at %v\n", tempdir)
+		targetdir := filepath.Join(tempdir, "src", filepath.FromSlash(config.Namespace))
+		if err = copy(targetdir, root); err != nil {
+			die("copying files from %v to %v/src failed: %v\n", root, tempdir, err)
+		}
+
+		defer func() {
+			if !keepGopath {
+				verbosePrintf("remove %v\n", tempdir)
+				if err = os.RemoveAll(tempdir); err != nil {
+					die("remove GOPATH at %s failed: %v\n", tempdir, err)
+				}
+			} else {
+				verbosePrintf("leaving temporary GOPATH at %v\n", tempdir)
+			}
+		}()
+
+		buildCWD = targetdir
+
+		goEnv["GOPATH"] = tempdir
+		buildEnv["GOPATH"] = tempdir
+	}
+
+	verbosePrintf("environment:\n  go: %v\n  build: %v\n", goEnv, buildEnv)
+
+	buildArgs = append(buildArgs,
+		"-tags", strings.Join(buildTags, " "),
+		"-ldflags", ldflags,
+		"-o", output, buildTarget,
+	)
+
+	err = build(buildCWD, buildEnv, buildArgs...)
 	if err != nil {
 		die("build failed: %v\n", err)
 	}
@@ -599,7 +622,9 @@ func main() {
 	if runTests {
 		verbosePrintf("running tests\n")
 
-		err = test(cwd, gopath, config.Tests...)
+		testArgs = append(testArgs, config.Tests...)
+
+		err = test(buildCWD, goEnv, testArgs...)
 		if err != nil {
 			die("running tests failed: %v\n", err)
 		}

changelog/0.9.2_2018-08-06/issue-1854

@@ -0,0 +1,16 @@
+Bugfix: Allow saving files/dirs on different fs with `--one-file-system`
+
+restic now allows saving files/dirs on a different file system in a subdir
+correctly even when `--one-file-system` is specified. 
+
+The first thing the restic archiver code does is to build a tree of the target
+files/directories. If it detects that a parent directory is already included
+(e.g. `restic backup /foo /foo/bar/baz`), it'll ignore the latter argument.
+
+Without `--one-file-system`, that's perfectly valid: If `/foo` is to be
+archived, it will include `/foo/bar/baz`. But with `--one-file-system`,
+`/foo/bar/baz` may reside on a different file system, so it won't be included
+with `/foo`.
+
+https://github.com/restic/restic/issues/1854
+https://github.com/restic/restic/pull/1855

changelog/0.9.2_2018-08-06/issue-1870

@@ -0,0 +1,6 @@
+Bugfix: Fix restore with --include
+
+We fixed a bug which prevented restic to restore files with an include filter.
+
+https://github.com/restic/restic/issues/1870
+https://github.com/restic/restic/pull/1900

changelog/0.9.2_2018-08-06/issue-1880

@@ -0,0 +1,12 @@
+Bugfix: Use `--cache-dir` argument for `check` command
+
+`check` command now uses a temporary sub-directory of the specified directory
+if set using the `--cache-dir` argument. If not set, the cache directory is
+created in the default temporary directory as before.
+In either case a temporary cache is used to ensure the actual repository is
+checked (rather than a local copy).
+
+The `--cache-dir` argument was not used by the `check` command, instead a
+cache directory was created in the temporary directory.
+
+https://github.com/restic/restic/issues/1880

changelog/0.9.2_2018-08-06/issue-1893

@@ -0,0 +1,8 @@
+Bugfix: Return error when exclude file cannot be read
+
+A bug was found: when multiple exclude files were passed to restic and one of
+them could not be read, an error was printed and restic continued, ignoring
+even the existing exclude files. Now, an error message is printed and restic
+aborts when an exclude file cannot be read.
+
+https://github.com/restic/restic/issues/1893

changelog/0.9.2_2018-08-06/issue-1906

@@ -0,0 +1,8 @@
+Enhancement: Add support for B2 application keys
+
+Restic can now use so-called "application keys" which can be created in the B2
+dashboard and were only introduced recently. In contrast to the "master key",
+such keys can be restricted to a specific bucket and/or path.
+
+https://github.com/restic/restic/issues/1906
+https://github.com/restic/restic/pull/1914

changelog/0.9.2_2018-08-06/pull-1729

@@ -0,0 +1,4 @@
+Enhancement: Add stats command to get information about a repository
+
+https://github.com/restic/restic/issues/874
+https://github.com/restic/restic/pull/1729

changelog/0.9.2_2018-08-06/pull-1772

@@ -0,0 +1,6 @@
+Enhancement: Add restore --verify to verify restored file content
+
+Restore will print error message if restored file content does not match
+expected SHA256 checksum
+
+https://github.com/restic/restic/pull/1772

changelog/0.9.2_2018-08-06/pull-1853

@@ -0,0 +1,6 @@
+Enhancement: Add JSON output support to `restic key list`
+
+This PR enables users to get the output of `restic key list` in JSON in addition
+to the existing table format.
+
+https://github.com/restic/restic/pull/1853

changelog/0.9.2_2018-08-06/pull-1861

@@ -0,0 +1,6 @@
+Bugfix: Fix case-insensitive search with restic find
+
+We've fixed the behavior for `restic find -i PATTERN`, which was
+broken in v0.9.1.
+
+https://github.com/restic/restic/pull/1861

changelog/0.9.2_2018-08-06/pull-1882

@@ -0,0 +1,8 @@
+Enhancement: S3 backend: accept AWS_SESSION_TOKEN
+
+Before, it was not possible to use s3 backend with AWS temporary security credentials(with AWS_SESSION_TOKEN).
+This change gives higher priority to credentials.EnvAWS credentials provider.
+
+https://github.com/restic/restic/issues/1477
+https://github.com/restic/restic/pull/1479
+https://github.com/restic/restic/pull/1647

changelog/0.9.2_2018-08-06/pull-1901

@@ -0,0 +1,9 @@
+Enhancement: Update the Backblaze B2 library
+
+We've updated the library we're using for accessing the Backblaze B2 service to
+0.5.0 to include support for upcoming so-called "application keys". With this
+feature, you can create access credentials for B2 which are restricted to e.g.
+a single bucket or even a sub-directory of a bucket.
+
+https://github.com/restic/restic/pull/1901
+https://github.com/kurin/blazer

changelog/0.9.3_2018-10-13/issue-1766

@@ -0,0 +1,7 @@
+Enhancement: restore: suppress lchown errors when not running as root
+
+Like "cp" and "rsync" do, restic now only reports errors for changing
+the ownership of files during restore if it is run as root, on non-Windows
+operating systems. On Windows, the error is reported as usual.
+
+https://github.com/restic/restic/issues/1766

changelog/0.9.3_2018-10-13/issue-1909

@@ -0,0 +1,14 @@
+Enhancement: Reject files/dirs by name first
+
+The current scanner/archiver code had an architectural limitation: it always
+ran the `lstat()` system call on all files and directories before a decision to
+include/exclude the file/dir was made. This lead to a lot of unnecessary system
+calls for items that could have been rejected by their name or path only.
+
+We've changed the archiver/scanner implementation so that it now first rejects
+by name/path, and only runs the system call on the remaining items. This
+reduces the number of `lstat()` system calls a lot (depending on the exclude
+settings).
+
+https://github.com/restic/restic/issues/1909
+https://github.com/restic/restic/pull/1912

changelog/0.9.3_2018-10-13/issue-1935

@@ -0,0 +1,8 @@
+Bugfix: Remove truncated files from cache
+
+When a file in the local cache is truncated, and restic tries to access data
+beyond the end of the (cached) file, it used to return an error "EOF". This is
+now fixed, such truncated files are removed and the data is fetched directly
+from the backend.
+
+https://github.com/restic/restic/issues/1935

changelog/0.9.3_2018-10-13/issue-1941

@@ -0,0 +1,15 @@
+Enhancement: Add directory filter to ls command
+
+The ls command can now be filtered by directories, so that only files in the
+given directories will be shown. If the --recursive flag is specified, then
+ls will traverse subfolders and list their files as well.
+
+It used to be possible to specify multiple snapshots, but that has been
+replaced by only one snapshot and the possibility of specifying multiple
+directories.
+
+Specifying directories constrains the walk, which can significantly speed up
+the listing.
+
+https://github.com/restic/restic/issues/1940
+https://github.com/restic/restic/pull/1941

changelog/0.9.3_2018-10-13/issue-1967

@@ -0,0 +1,7 @@
+Enhancement: Use `--host` everywhere
+
+We now use the flag `--host` for all commands which need a host name, using
+`--hostname` (e.g. for `restic backup`) still works, but will print a
+deprecation warning. Also, add the short option `-H` where possible.
+
+https://github.com/restic/restic/issues/1967

changelog/0.9.3_2018-10-13/issue-1978

@@ -0,0 +1,12 @@
+Bugfix: Do not return an error when the scanner is slower than backup
+
+When restic makes a backup, there's a background task called "scanner" which
+collects information on how many files and directories are to be saved, in
+order to display progress information to the user. When the backup finishes
+faster than the scanner, it is aborted because the result is not needed any
+more. This logic contained a bug, where quitting the scanner process was
+treated as an error, and caused restic to print an unhelpful error message
+("context canceled").
+
+https://github.com/restic/restic/issues/1978
+https://github.com/restic/restic/pull/1991

changelog/0.9.3_2018-10-13/issue-2028

@@ -0,0 +1,7 @@
+Enhancement: Display size of cache directories
+
+The `cache` command now by default shows the size of the individual cache
+directories. It can be disabled with `--no-size`.
+
+https://github.com/restic/restic/issues/2028
+https://github.com/restic/restic/pull/2033

changelog/0.9.3_2018-10-13/pull-1780

@@ -0,0 +1,13 @@
+Enhancement: Improve the `find` command
+
+We've updated the `find` command to support multiple patterns.
+
+`restic find` is now able to list the snapshots containing a specific tree
+or blob, or even the snapshots that contain blobs belonging to a given pack.
+A list of IDs can be given, as long as they all have the same type.
+
+The command `find` can also display the pack IDs the blobs belong to, if
+the `--show-pack-id` flag is provided.
+
+https://github.com/restic/restic/issues/1777
+https://github.com/restic/restic/pull/1780

changelog/0.9.3_2018-10-13/pull-1876

@@ -0,0 +1,7 @@
+Enhancement: Display reason why forget keeps snapshots
+
+We've added a column to the list of snapshots `forget` keeps which details the
+reasons to keep a particuliar snapshot. This makes debugging policies for
+forget much easier. Please remember to always try things out with `--dry-run`!
+
+https://github.com/restic/restic/pull/1876

changelog/0.9.3_2018-10-13/pull-1891

@@ -0,0 +1,7 @@
+Enhancement: Accept glob in paths loaded via --files-from
+
+Before that, behaviour was different if paths were appended to command line or
+from a file, because wild card characters were expanded by shell if appended to
+command line, but not expanded if loaded from file.
+
+https://github.com/restic/restic/issues/1891

changelog/0.9.3_2018-10-13/pull-1920

@@ -0,0 +1,8 @@
+Enhancement: Vendor dependencies with Go 1.11 Modules
+
+Until now, we've used `dep` for managing dependencies, we've now switch to
+using Go modules. For users this does not change much, only if you want to
+compile restic without downloading anything with Go 1.11, then you need to run:
+`go build -mod=vendor build.go`
+
+https://github.com/restic/restic/pull/1920

changelog/0.9.3_2018-10-13/pull-1949

@@ -0,0 +1,15 @@
+Enhancement: Add new command `self-update`
+
+We have added a new command called `self-update` which downloads the
+latest released version of restic from GitHub and replaces the current
+binary with it. It does not rely on any external program (so it'll work
+everywhere), but still verifies the GPG signature using the embedded GPG
+public key.
+
+By default, the `self-update` command is hidden behind the `selfupdate`
+built tag, which is only set when restic is built using `build.go` (including
+official releases). The reason for this is that downstream distributions will
+then not include the command by default, so users are encouraged to use the
+platform-specific distribution mechanism.
+
+https://github.com/restic/restic/pull/1949

changelog/0.9.3_2018-10-13/pull-1953

@@ -0,0 +1,7 @@
+Enhancement: ls: Add JSON output support for restic ls cmd
+
+We've implemented listing files in the repository with JSON as output, just
+pass `--json` as an option to `restic ls`. This makes the output of the command
+machine readable.
+
+https://github.com/restic/restic/pull/1953

changelog/0.9.3_2018-10-13/pull-1962

@@ -0,0 +1,13 @@
+Enhancement: Stream JSON output for ls command
+
+The `ls` command now supports JSON output with the global `--json`
+flag, and this change streams out JSON messages one object at a time
+rather than en entire array buffered in memory before encoding. The
+advantage is it allows large listings to be handled efficiently.
+
+Two message types are printed: snapshots and nodes. A snapshot
+object will precede node objects which belong to that snapshot.
+The `struct_type` field can be used to determine which kind of
+message an object is.
+
+https://github.com/restic/restic/pull/1962

changelog/0.9.4_2019-01-06/issue-1605

@@ -0,0 +1,11 @@
+Enhancement: Concurrent restore
+
+This change significantly improves restore performance, especially
+when using high-latency remote repositories like B2.
+
+The implementation now uses several concurrent threads to download and process
+multiple remote files concurrently. To further reduce restore time, each remote
+file is downloaded using a single repository request.
+
+https://github.com/restic/restic/issues/1605
+https://github.com/restic/restic/pull/1719

changelog/0.9.4_2019-01-06/issue-1989

@@ -0,0 +1,7 @@
+Bugfix: Google Cloud Storage: Respect bandwidth limit
+
+The GCS backend did not respect the bandwidth limit configured, a previous
+commit accidentally removed support for it.
+
+https://github.com/restic/restic/issues/1989
+https://github.com/restic/restic/pull/2100

changelog/0.9.4_2019-01-06/issue-2040

@@ -0,0 +1,11 @@
+Bugfix: Add host name filter shorthand flag for `stats` command
+
+The default value for `--host` flag was set to 'H' (the shorthand version of
+the flag), this caused the lookup for the latest snapshot to fail.
+
+Add shorthand flag `-H` for `--host` (with empty default so if these flags
+are not specified the latest snapshot will not filter by host name).
+
+Also add shorthand `-H` for `backup` command.
+
+https://github.com/restic/restic/issues/2040

changelog/0.9.4_2019-01-06/issue-2089

@@ -0,0 +1,9 @@
+Enhancement: increase granularity of the "keep within" retention policy
+
+The `keep-within` option of the `forget` command now accepts time ranges with
+an hourly granularity. For example, running `restic forget --keep-within 3d12h`
+will keep all the snapshots made within three days and twelve hours from the
+time of the latest snapshot.
+
+https://github.com/restic/restic/issues/2089
+https://github.com/restic/restic/pull/2090

changelog/0.9.4_2019-01-06/issue-2097

@@ -0,0 +1,12 @@
+Enhancement: Add key hinting
+
+Added a new option `--key-hint` and corresponding environment variable
+`RESTIC_KEY_HINT`.  The key hint is a key ID to try decrypting first, before
+other keys in the repository.
+
+This change will benefit repositories with many keys; if the correct key hint
+is supplied then restic only needs to check one key.  If the key hint is
+incorrect (the key does not exist, or the password is incorrect) then restic
+will check all keys, as usual.
+
+https://github.com/restic/restic/issues/2097

changelog/0.9.4_2019-01-06/pull-2017

@@ -0,0 +1,11 @@
+Enhancement: mount: Enforce FUSE Unix permissions with allow-other
+
+The fuse mount (`restic mount`) now lets the kernel check the permissions of
+the files within snapshots (this is done through the `DefaultPermissions` FUSE
+option) when the option `--allow-other` is specified.
+
+To restore the old behavior, we've added the `--no-default-permissions` option.
+This allows all users that have access to the mount point to access all
+files within the snapshots.
+
+https://github.com/restic/restic/pull/2017

changelog/0.9.4_2019-01-06/pull-2068

@@ -0,0 +1,6 @@
+Bugfix: Correctly return error loading data
+
+In one case during `prune` and `check`, an error loading data from the backend is not returned properly. This is now corrected.
+
+https://github.com/restic/restic/pull/2068
+https://github.com/restic/restic/issues/1999#issuecomment-433737921

changelog/0.9.4_2019-01-06/pull-2070

@@ -0,0 +1,7 @@
+Enhancement: Make all commands display timestamps in local time
+
+Restic used to drop the timezone information from displayed timestamps, it now
+converts timestamps to local time before printing them so the times can be
+easily compared to.
+
+https://github.com/restic/restic/pull/2070

changelog/0.9.4_2019-01-06/pull-2086

@@ -0,0 +1,7 @@
+Enhancement: Allow --files-from to be specified multiple times
+
+Before, restic took only the last file specified with `--files-from` into
+account, this is now corrected.
+
+https://github.com/restic/restic/issues/2085
+https://github.com/restic/restic/pull/2086

changelog/0.9.4_2019-01-06/pull-2094

@@ -0,0 +1,8 @@
+Enhancement: Run command to get password
+
+We've added the `--password-command` option which allows specifying a command
+that restic runs every time the password for the repository is needed, so it
+can be integrated with a password manager or keyring. The option can also be
+set via the environment variable `$RESTIC_PASSWORD_COMMAND`.
+
+https://github.com/restic/restic/pull/2094

changelog/0.9.4_2019-01-06/pull-2095

@@ -0,0 +1,7 @@
+Bugfix: consistently use local time for snapshots times
+
+By default snapshots created with restic backup were set to local time,
+but when the --time flag was used the provided timestamp was parsed as
+UTC. With this change all snapshots times are set to local time.
+
+https://github.com/restic/restic/pull/2095

changelog/0.9.5_2019-04-23/issue-1895

@@ -0,0 +1,7 @@
+Enhancement: Add case insensitive include & exclude options
+
+The backup and restore commands now have --iexclude and --iinclude flags
+as case insensitive variants of --exclude and --include.
+
+https://github.com/restic/restic/issues/1895
+https://github.com/restic/restic/pull/2032

changelog/0.9.5_2019-04-23/issue-1937

@@ -0,0 +1,8 @@
+Enhancement: Support streaming JSON output for backup
+
+We've added support for getting machine-readable status output during backup,
+just pass the flag `--json` for `restic backup` and restic will output a stream
+of JSON objects which contain the current progress.
+
+https://github.com/restic/restic/issues/1937
+https://github.com/restic/restic/pull/1944

changelog/0.9.5_2019-04-23/issue-2135

@@ -0,0 +1,10 @@
+Bugfix: Return error when no bytes could be read from stdin
+
+We assume that users reading backup data from stdin want to know when no data
+could be read, so now restic returns an error when `backup --stdin` is called
+but no bytes could be read. Usually, this means that an earlier command in a
+pipe has failed. The documentation was amended and now recommends setting the
+`pipefail` option (`set -o pipefail`).
+
+https://github.com/restic/restic/pull/2135
+https://github.com/restic/restic/pull/2139

changelog/0.9.5_2019-04-23/issue-2155

@@ -0,0 +1,8 @@
+Enhancement: add Openstack application credential auth for Swift
+
+Since Openstack Queens Identity (auth V3) service supports an application
+credential auth method.  It allows to create a technical account with the
+limited roles. This commit adds an application credential authentication
+method for the Swift backend.
+
+https://github.com/restic/restic/issues/2155

changelog/0.9.5_2019-04-23/issue-2181

@@ -0,0 +1,3 @@
+Bugfix: Don't cancel timeout after 30 seconds for self-update
+
+https://github.com/restic/restic/issues/2181

changelog/0.9.5_2019-04-23/issue-2184

@@ -0,0 +1,8 @@
+Enhancement: Add --json support to forget command
+
+The forget command now supports the --json argument, outputting the
+information about what is (or would-be) kept and removed from the
+repository.
+
+https://github.com/restic/restic/issues/2184
+https://github.com/restic/restic/pull/2185

changelog/0.9.5_2019-04-23/issue-2203

@@ -0,0 +1,6 @@
+Bugfix: Fix reading passwords from stdin
+
+Passwords for the `init`, `key add`, and `key passwd` commands can now be read from
+non-terminal stdin.
+
+https://github.com/restic/restic/issues/2203

changelog/0.9.5_2019-04-23/issue-2224

@@ -0,0 +1,9 @@
+Bugfix: Don't abort the find command when a tree can't be loaded
+
+Change the find command so that missing trees don't result in a crash.
+Instead, the error is logged to the debug log, and the tree ID is displayed
+along with the snapshot it belongs to.  This makes it possible to recover
+repositories that are missing trees by forgetting the snapshots they are used
+in.
+
+https://github.com/restic/restic/issues/2224

changelog/0.9.5_2019-04-23/pull-2087

@@ -0,0 +1,10 @@
+Enhancement: Add group-by option to snapshots command
+
+We have added an option to group the output of the snapshots command, similar
+to the output of the forget command. The option has been called "--group-by"
+and accepts any combination of the values "host", "paths" and "tags", separated
+by commas. Default behavior (not specifying --group-by) has not been changed.
+We have added support of the grouping to the JSON output.
+
+https://github.com/restic/restic/issues/2037
+https://github.com/restic/restic/pull/2087

changelog/0.9.5_2019-04-23/pull-2124

@@ -0,0 +1,8 @@
+Enhancement: Ability to dump folders to tar via stdout
+
+We've added the ability to dump whole folders to stdout via the `dump` command.
+Restic now requires at least Go 1.10 due to a limitation of the standard
+library for Go <= 1.9.
+
+https://github.com/restic/restic/pull/2124
+https://github.com/restic/restic/issues/2123

changelog/0.9.5_2019-04-23/pull-2139

@@ -0,0 +1,8 @@
+Enhancement: Return error if no bytes could be read for `backup --stdin`
+
+When restic is used to backup the output of a program, like `mysqldump | restic
+backup --stdin`, it now returns an error if no bytes could be read at all. This
+catches the failure case when `mysqldump` failed for some reason and did not
+output any data to stdout.
+
+https://github.com/restic/restic/pull/2139

changelog/0.9.5_2019-04-23/pull-2205

@@ -0,0 +1,10 @@
+Enhancement: Add --ignore-inode option to backup cmd
+
+This option handles backup of virtual filesystems that do not keep fixed
+inodes for files, like Fuse-based, pCloud, etc. Ignoring inode changes allows
+to consider the file as unchanged if last modification date and size
+are unchanged.
+
+https://github.com/restic/restic/pull/2205
+https://github.com/restic/restic/pull/2047
+https://github.com/restic/restic/issues/1631

changelog/0.9.5_2019-04-23/pull-2220

@@ -0,0 +1,16 @@
+Enhancement: Add config option to set S3 storage class
+
+The `s3.storage-class` option can be passed to restic (using `-o`) to
+specify the storage class to be used for S3 objects created by restic.
+
+The storage class is passed as-is to S3, so it needs to be understood by
+the API. On AWS, it can be one of `STANDARD`, `STANDARD_IA`,
+`ONEZONE_IA`, `INTELLIGENT_TIERING` and `REDUCED_REDUNDANCY`. If
+unspecified, the default storage class is used (`STANDARD` on AWS).
+
+You can mix storage classes in the same bucket, and the setting isn't
+stored in the restic repository, so be sure to specify it with each
+command that writes to S3.
+
+https://github.com/restic/restic/pull/2220
+https://github.com/restic/restic/issues/706

cmd/restic/acl.go

@@ -0,0 +1,131 @@
+package main
+
+// Adapted from https://github.com/maxymania/go-system/blob/master/posix_acl/posix_acl.go
+
+import (
+	"bytes"
+	"encoding/binary"
+	"fmt"
+)
+
+const (
+	aclUserOwner  = 0x0001
+	aclUser       = 0x0002
+	aclGroupOwner = 0x0004
+	aclGroup      = 0x0008
+	aclMask       = 0x0010
+	aclOthers     = 0x0020
+)
+
+type aclSID uint64
+
+type aclElem struct {
+	Tag  uint16
+	Perm uint16
+	ID   uint32
+}
+
+type acl struct {
+	Version uint32
+	List    []aclElement
+}
+
+type aclElement struct {
+	aclSID
+	Perm uint16
+}
+
+func (a *aclSID) setUID(uid uint32) {
+	*a = aclSID(uid) | (aclUser << 32)
+}
+func (a *aclSID) setGID(gid uint32) {
+	*a = aclSID(gid) | (aclGroup << 32)
+}
+
+func (a *aclSID) setType(tp int) {
+	*a = aclSID(tp) << 32
+}
+
+func (a aclSID) getType() int {
+	return int(a >> 32)
+}
+func (a aclSID) getID() uint32 {
+	return uint32(a & 0xffffffff)
+}
+func (a aclSID) String() string {
+	switch a >> 32 {
+	case aclUserOwner:
+		return "user::"
+	case aclUser:
+		return fmt.Sprintf("user:%v:", a.getID())
+	case aclGroupOwner:
+		return "group::"
+	case aclGroup:
+		return fmt.Sprintf("group:%v:", a.getID())
+	case aclMask:
+		return "mask::"
+	case aclOthers:
+		return "other::"
+	}
+	return "?:"
+}
+
+func (a aclElement) String() string {
+	str := ""
+	if (a.Perm & 4) != 0 {
+		str += "r"
+	} else {
+		str += "-"
+	}
+	if (a.Perm & 2) != 0 {
+		str += "w"
+	} else {
+		str += "-"
+	}
+	if (a.Perm & 1) != 0 {
+		str += "x"
+	} else {
+		str += "-"
+	}
+	return fmt.Sprintf("%v%v", a.aclSID, str)
+}
+
+func (a *acl) decode(xattr []byte) {
+	var elem aclElement
+	ae := new(aclElem)
+	nr := bytes.NewReader(xattr)
+	e := binary.Read(nr, binary.LittleEndian, &a.Version)
+	if e != nil {
+		a.Version = 0
+		return
+	}
+	if len(a.List) > 0 {
+		a.List = a.List[:0]
+	}
+	for binary.Read(nr, binary.LittleEndian, ae) == nil {
+		elem.aclSID = (aclSID(ae.Tag) << 32) | aclSID(ae.ID)
+		elem.Perm = ae.Perm
+		a.List = append(a.List, elem)
+	}
+}
+
+func (a *acl) encode() []byte {
+	buf := new(bytes.Buffer)
+	ae := new(aclElem)
+	binary.Write(buf, binary.LittleEndian, &a.Version)
+	for _, elem := range a.List {
+		ae.Tag = uint16(elem.getType())
+		ae.Perm = elem.Perm
+		ae.ID = elem.getID()
+		binary.Write(buf, binary.LittleEndian, ae)
+	}
+	return buf.Bytes()
+}
+
+func (a *acl) String() string {
+	var finalacl string
+	for _, acl := range a.List {
+		finalacl += acl.String() + "\n"
+	}
+	return finalacl
+}

cmd/restic/acl_test.go

@@ -0,0 +1,96 @@
+package main
+
+import (
+	"reflect"
+	"testing"
+)
+
+func Test_acl_decode(t *testing.T) {
+	type args struct {
+		xattr []byte
+	}
+	tests := []struct {
+		name string
+		args args
+		want string
+	}{
+		{
+			name: "decode string",
+			args: args{
+				xattr: []byte{2, 0, 0, 0, 1, 0, 6, 0, 255, 255, 255, 255, 2, 0, 7, 0, 0, 0, 0, 0, 2, 0, 7, 0, 254, 255, 0, 0, 4, 0, 7, 0, 255, 255, 255, 255, 16, 0, 7, 0, 255, 255, 255, 255, 32, 0, 4, 0, 255, 255, 255, 255},
+			},
+			want: "user::rw-\nuser:0:rwx\nuser:65534:rwx\ngroup::rwx\nmask::rwx\nother::r--\n",
+		},
+		{
+			name: "decode fail",
+			args: args{
+				xattr: []byte("abctest"),
+			},
+			want: "",
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			a := &acl{}
+			a.decode(tt.args.xattr)
+			if tt.want != a.String() {
+				t.Errorf("acl.decode() = %v, want: %v", a.String(), tt.want)
+			}
+		})
+	}
+}
+
+func Test_acl_encode(t *testing.T) {
+	tests := []struct {
+		name string
+		want []byte
+		args []aclElement
+	}{
+		{
+			name: "encode values",
+			want: []byte{2, 0, 0, 0, 1, 0, 6, 0, 255, 255, 255, 255, 2, 0, 7, 0, 0, 0, 0, 0, 2, 0, 7, 0, 254, 255, 0, 0, 4, 0, 7, 0, 255, 255, 255, 255, 16, 0, 7, 0, 255, 255, 255, 255, 32, 0, 4, 0, 255, 255, 255, 255},
+			args: []aclElement{
+				{
+					aclSID: 8589934591,
+					Perm:   6,
+				},
+				{
+					aclSID: 8589934592,
+					Perm:   7,
+				},
+				{
+					aclSID: 8590000126,
+					Perm:   7,
+				},
+				{
+					aclSID: 21474836479,
+					Perm:   7,
+				},
+				{
+					aclSID: 73014444031,
+					Perm:   7,
+				},
+				{
+					aclSID: 141733920767,
+					Perm:   4,
+				},
+			},
+		},
+		{
+			name: "encode fail",
+			want: []byte{2, 0, 0, 0},
+			args: []aclElement{},
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			a := &acl{
+				Version: 2,
+				List:    tt.args,
+			}
+			if got := a.encode(); !reflect.DeepEqual(got, tt.want) {
+				t.Errorf("acl.encode() = %v, want %v", got, tt.want)
+			}
+		})
+	}
+}

cmd/restic/cmd_backup.go

@@ -4,8 +4,11 @@ import (
 	"bufio"
 	"bytes"
 	"context"
+	"fmt"
+	"io"
 	"io/ioutil"
 	"os"
+	"path/filepath"
 	"strconv"
 	"strings"
 	"time"
@@ -21,6 +24,7 @@ import (
 	"github.com/restic/restic/internal/restic"
 	"github.com/restic/restic/internal/textfile"
 	"github.com/restic/restic/internal/ui"
+	"github.com/restic/restic/internal/ui/jsonstatus"
 	"github.com/restic/restic/internal/ui/termstatus"
 )
 
@@ -32,19 +36,23 @@ The "backup" command creates a new snapshot and saves the files and directories
 given as the arguments.
 `,
 	PreRun: func(cmd *cobra.Command, args []string) {
-		if backupOptions.Hostname == "" {
+		if backupOptions.Host == "" {
 			hostname, err := os.Hostname()
 			if err != nil {
 				debug.Log("os.Hostname() returned err: %v", err)
 				return
 			}
-			backupOptions.Hostname = hostname
+			backupOptions.Host = hostname
 		}
 	},
 	DisableAutoGenTag: true,
 	RunE: func(cmd *cobra.Command, args []string) error {
-		if backupOptions.Stdin && backupOptions.FilesFrom == "-" {
-			return errors.Fatal("cannot use both `--stdin` and `--files-from -`")
+		if backupOptions.Stdin {
+			for _, filename := range backupOptions.FilesFrom {
+				if filename == "-" {
+					return errors.Fatal("cannot use both `--stdin` and `--files-from -`")
+				}
+			}
 		}
 
 		var t tomb.Tomb
@@ -62,20 +70,22 @@ given as the arguments.
 
 // BackupOptions bundles all options for the backup command.
 type BackupOptions struct {
-	Parent           string
-	Force            bool
-	Excludes         []string
-	ExcludeFiles     []string
-	ExcludeOtherFS   bool
-	ExcludeIfPresent []string
-	ExcludeCaches    bool
-	Stdin            bool
-	StdinFilename    string
-	Tags             []string
-	Hostname         string
-	FilesFrom        string
-	TimeStamp        string
-	WithAtime        bool
+	Parent              string
+	Force               bool
+	Excludes            []string
+	InsensitiveExcludes []string
+	ExcludeFiles        []string
+	ExcludeOtherFS      bool
+	ExcludeIfPresent    []string
+	ExcludeCaches       bool
+	Stdin               bool
+	StdinFilename       string
+	Tags                []string
+	Host                string
+	FilesFrom           []string
+	TimeStamp           string
+	WithAtime           bool
+	IgnoreInode         bool
 }
 
 var backupOptions BackupOptions
@@ -87,17 +97,23 @@ func init() {
 	f.StringVar(&backupOptions.Parent, "parent", "", "use this parent snapshot (default: last snapshot in the repo that has the same target files/directories)")
 	f.BoolVarP(&backupOptions.Force, "force", "f", false, `force re-reading the target files/directories (overrides the "parent" flag)`)
 	f.StringArrayVarP(&backupOptions.Excludes, "exclude", "e", nil, "exclude a `pattern` (can be specified multiple times)")
+	f.StringArrayVar(&backupOptions.InsensitiveExcludes, "iexclude", nil, "same as `--exclude` but ignores the casing of filenames")
 	f.StringArrayVar(&backupOptions.ExcludeFiles, "exclude-file", nil, "read exclude patterns from a `file` (can be specified multiple times)")
 	f.BoolVarP(&backupOptions.ExcludeOtherFS, "one-file-system", "x", false, "exclude other file systems")
 	f.StringArrayVar(&backupOptions.ExcludeIfPresent, "exclude-if-present", nil, "takes filename[:header], exclude contents of directories containing filename (except filename itself) if header of that file is as provided (can be specified multiple times)")
-	f.BoolVar(&backupOptions.ExcludeCaches, "exclude-caches", false, `excludes cache directories that are marked with a CACHEDIR.TAG file`)
+	f.BoolVar(&backupOptions.ExcludeCaches, "exclude-caches", false, `excludes cache directories that are marked with a CACHEDIR.TAG file. See http://bford.info/cachedir/spec.html for the Cache Directory Tagging Standard`)
 	f.BoolVar(&backupOptions.Stdin, "stdin", false, "read backup from stdin")
 	f.StringVar(&backupOptions.StdinFilename, "stdin-filename", "stdin", "file name to use when reading from stdin")
 	f.StringArrayVar(&backupOptions.Tags, "tag", nil, "add a `tag` for the new snapshot (can be specified multiple times)")
-	f.StringVar(&backupOptions.Hostname, "hostname", "", "set the `hostname` for the snapshot manually. To prevent an expensive rescan use the \"parent\" flag")
-	f.StringVar(&backupOptions.FilesFrom, "files-from", "", "read the files to backup from file (can be combined with file args)")
+
+	f.StringVarP(&backupOptions.Host, "host", "H", "", "set the `hostname` for the snapshot manually. To prevent an expensive rescan use the \"parent\" flag")
+	f.StringVar(&backupOptions.Host, "hostname", "", "set the `hostname` for the snapshot manually")
+	f.MarkDeprecated("hostname", "use --host")
+
+	f.StringArrayVar(&backupOptions.FilesFrom, "files-from", nil, "read the files to backup from file (can be combined with file args/can be specified multiple times)")
 	f.StringVar(&backupOptions.TimeStamp, "time", "", "time of the backup (ex. '2012-11-01 22:08:41') (default: now)")
 	f.BoolVar(&backupOptions.WithAtime, "with-atime", false, "store the atime for all files and directories")
+	f.BoolVar(&backupOptions.IgnoreInode, "ignore-inode", false, "ignore inode number changes when checking for modified files")
 }
 
 // filterExisting returns a slice of all existing items, or an error if no
@@ -169,12 +185,16 @@ func readLinesFromFile(filename string) ([]string, error) {
 
 // Check returns an error when an invalid combination of options was set.
 func (opts BackupOptions) Check(gopts GlobalOptions, args []string) error {
-	if opts.FilesFrom == "-" && gopts.password == "" {
-		return errors.Fatal("unable to read password from stdin when data is to be read from stdin, use --password-file or $RESTIC_PASSWORD")
+	if gopts.password == "" {
+		for _, filename := range opts.FilesFrom {
+			if filename == "-" {
+				return errors.Fatal("unable to read password from stdin when data is to be read from stdin, use --password-file or $RESTIC_PASSWORD")
+			}
+		}
 	}
 
 	if opts.Stdin {
-		if opts.FilesFrom != "" {
+		if len(opts.FilesFrom) > 0 {
 			return errors.Fatal("--stdin and --files-from cannot be used together")
 		}
 
@@ -186,18 +206,9 @@ func (opts BackupOptions) Check(gopts GlobalOptions, args []string) error {
 	return nil
 }
 
-// collectRejectFuncs returns a list of all functions which may reject data
-// from being saved in a snapshot
-func collectRejectFuncs(opts BackupOptions, repo *repository.Repository, targets []string) (fs []RejectFunc, err error) {
-	// allowed devices
-	if opts.ExcludeOtherFS {
-		f, err := rejectByDevice(targets)
-		if err != nil {
-			return nil, err
-		}
-		fs = append(fs, f)
-	}
-
+// collectRejectByNameFuncs returns a list of all functions which may reject data
+// from being saved in a snapshot based on path only
+func collectRejectByNameFuncs(opts BackupOptions, repo *repository.Repository, targets []string) (fs []RejectByNameFunc, err error) {
 	// exclude restic cache
 	if repo.Cache != nil {
 		f, err := rejectResticCache(repo)
@@ -210,7 +221,15 @@ func collectRejectFuncs(opts BackupOptions, repo *repository.Repository, targets
 
 	// add patterns from file
 	if len(opts.ExcludeFiles) > 0 {
-		opts.Excludes = append(opts.Excludes, readExcludePatternsFromFiles(opts.ExcludeFiles)...)
+		excludes, err := readExcludePatternsFromFiles(opts.ExcludeFiles)
+		if err != nil {
+			return nil, err
+		}
+		opts.Excludes = append(opts.Excludes, excludes...)
+	}
+
+	if len(opts.InsensitiveExcludes) > 0 {
+		fs = append(fs, rejectByInsensitivePattern(opts.InsensitiveExcludes))
 	}
 
 	if len(opts.Excludes) > 0 {
@@ -233,9 +252,34 @@ func collectRejectFuncs(opts BackupOptions, repo *repository.Repository, targets
 	return fs, nil
 }
 
+// collectRejectFuncs returns a list of all functions which may reject data
+// from being saved in a snapshot based on path and file info
+func collectRejectFuncs(opts BackupOptions, repo *repository.Repository, targets []string) (fs []RejectFunc, err error) {
+	// allowed devices
+	if opts.ExcludeOtherFS && !opts.Stdin {
+		f, err := rejectByDevice(targets)
+		if err != nil {
+			return nil, err
+		}
+		fs = append(fs, f)
+	}
+
+	return fs, nil
+}
+
 // readExcludePatternsFromFiles reads all exclude files and returns the list of
-// exclude patterns.
-func readExcludePatternsFromFiles(excludeFiles []string) []string {
+// exclude patterns. For each line, leading and trailing white space is removed
+// and comment lines are ignored. For each remaining pattern, environment
+// variables are resolved. For adding a literal dollar sign ($), write $$ to
+// the file.
+func readExcludePatternsFromFiles(excludeFiles []string) ([]string, error) {
+	getenvOrDollar := func(s string) string {
+		if s == "$" {
+			return "$"
+		}
+		return os.Getenv(s)
+	}
+
 	var excludes []string
 	for _, filename := range excludeFiles {
 		err := func() (err error) {
@@ -258,17 +302,16 @@ func readExcludePatternsFromFiles(excludeFiles []string) []string {
 					continue
 				}
 
-				line = os.ExpandEnv(line)
+				line = os.Expand(line, getenvOrDollar)
 				excludes = append(excludes, line)
 			}
 			return scanner.Err()
 		}()
 		if err != nil {
-			Warnf("error reading exclude patterns: %v:", err)
-			return nil
+			return nil, err
 		}
 	}
-	return excludes
+	return excludes, nil
 }
 
 // collectTargets returns a list of target files/dirs from several sources.
@@ -277,15 +320,31 @@ func collectTargets(opts BackupOptions, args []string) (targets []string, err er
 		return nil, nil
 	}
 
-	fromfile, err := readLinesFromFile(opts.FilesFrom)
-	if err != nil {
-		return nil, err
+	var lines []string
+	for _, file := range opts.FilesFrom {
+		fromfile, err := readLinesFromFile(file)
+		if err != nil {
+			return nil, err
+		}
+
+		// expand wildcards
+		for _, line := range fromfile {
+			var expanded []string
+			expanded, err := filepath.Glob(line)
+			if err != nil {
+				return nil, errors.WithMessage(err, fmt.Sprintf("pattern: %s", line))
+			}
+			if len(expanded) == 0 {
+				Warnf("pattern %q does not match any files, skipping\n", line)
+			}
+			lines = append(lines, expanded...)
+		}
 	}
 
 	// merge files from files-from into normal args so we can reuse the normal
 	// args checks and have the ability to use both files-from and args at the
 	// same time
-	args = append(args, fromfile...)
+	args = append(args, lines...)
 	if len(args) == 0 && !opts.Stdin {
 		return nil, errors.Fatal("nothing to backup, please specify target files/dirs")
 	}
@@ -314,7 +373,7 @@ func findParentSnapshot(ctx context.Context, repo restic.Repository, opts Backup
 
 	// Find last snapshot to set it as parent, if not already set
 	if !opts.Force && parentID == nil {
-		id, err := restic.FindLatestSnapshot(ctx, repo, targets, []restic.TagList{}, opts.Hostname)
+		id, err := restic.FindLatestSnapshot(ctx, repo, targets, []restic.TagList{}, opts.Host)
 		if err == nil {
 			parentID = &id
 		} else if err != restic.ErrNoSnapshotFound {
@@ -338,7 +397,7 @@ func runBackup(opts BackupOptions, gopts GlobalOptions, term *termstatus.Termina
 
 	timeStamp := time.Now()
 	if opts.TimeStamp != "" {
-		timeStamp, err = time.Parse(TimeFormat, opts.TimeStamp)
+		timeStamp, err = time.ParseInLocation(TimeFormat, opts.TimeStamp, time.Local)
 		if err != nil {
 			return errors.Fatalf("error in time option: %v\n", err)
 		}
@@ -346,7 +405,43 @@ func runBackup(opts BackupOptions, gopts GlobalOptions, term *termstatus.Termina
 
 	var t tomb.Tomb
 
-	p := ui.NewBackup(term, gopts.verbosity)
+	if gopts.verbosity >= 2 && !gopts.JSON {
+		term.Print("open repository\n")
+	}
+
+	repo, err := OpenRepository(gopts)
+	if err != nil {
+		return err
+	}
+
+	type ArchiveProgressReporter interface {
+		CompleteItem(item string, previous, current *restic.Node, s archiver.ItemStats, d time.Duration)
+		StartFile(filename string)
+		CompleteBlob(filename string, bytes uint64)
+		ScannerError(item string, fi os.FileInfo, err error) error
+		ReportTotal(item string, s archiver.ScanStats)
+		SetMinUpdatePause(d time.Duration)
+		Run(ctx context.Context) error
+		Error(item string, fi os.FileInfo, err error) error
+		Finish(snapshotID restic.ID)
+
+		// ui.StdioWrapper
+		Stdout() io.WriteCloser
+		Stderr() io.WriteCloser
+
+		// ui.Message
+		E(msg string, args ...interface{})
+		P(msg string, args ...interface{})
+		V(msg string, args ...interface{})
+		VV(msg string, args ...interface{})
+	}
+
+	var p ArchiveProgressReporter
+	if gopts.JSON {
+		p = jsonstatus.NewBackup(term, gopts.verbosity)
+	} else {
+		p = ui.NewBackup(term, gopts.verbosity)
+	}
 
 	// use the terminal for stdout/stderr
 	prevStdout, prevStderr := gopts.stdout, gopts.stderr
@@ -361,32 +456,36 @@ func runBackup(opts BackupOptions, gopts GlobalOptions, term *termstatus.Termina
 			if fps > 60 {
 				fps = 60
 			}
-			p.MinUpdatePause = time.Second / time.Duration(fps)
+			p.SetMinUpdatePause(time.Second / time.Duration(fps))
 		}
 	}
 
 	t.Go(func() error { return p.Run(t.Context(gopts.ctx)) })
 
-	p.V("open repository")
-	repo, err := OpenRepository(gopts)
-	if err != nil {
-		return err
+	if !gopts.JSON {
+		p.V("lock repository")
 	}
-
-	p.V("lock repository")
 	lock, err := lockRepo(repo)
 	defer unlockRepo(lock)
 	if err != nil {
 		return err
 	}
 
-	// rejectFuncs collect functions that can reject items from the backup
+	// rejectByNameFuncs collect functions that can reject items from the backup based on path only
+	rejectByNameFuncs, err := collectRejectByNameFuncs(opts, repo, targets)
+	if err != nil {
+		return err
+	}
+
+	// rejectFuncs collect functions that can reject items from the backup based on path and file info
 	rejectFuncs, err := collectRejectFuncs(opts, repo, targets)
 	if err != nil {
 		return err
 	}
 
-	p.V("load index files")
+	if !gopts.JSON {
+		p.V("load index files")
+	}
 	err = repo.LoadIndex(gopts.ctx)
 	if err != nil {
 		return err
@@ -397,10 +496,19 @@ func runBackup(opts BackupOptions, gopts GlobalOptions, term *termstatus.Termina
 		return err
 	}
 
-	if parentSnapshotID != nil {
+	if !gopts.JSON && parentSnapshotID != nil {
 		p.V("using parent snapshot %v\n", parentSnapshotID.Str())
 	}
 
+	selectByNameFilter := func(item string) bool {
+		for _, reject := range rejectByNameFuncs {
+			if reject(item) {
+				return false
+			}
+		}
+		return true
+	}
+
 	selectFilter := func(item string, fi os.FileInfo) bool {
 		for _, reject := range rejectFuncs {
 			if reject(item, fi) {
@@ -412,7 +520,9 @@ func runBackup(opts BackupOptions, gopts GlobalOptions, term *termstatus.Termina
 
 	var targetFS fs.FS = fs.Local{}
 	if opts.Stdin {
-		p.V("read data from stdin")
+		if !gopts.JSON {
+			p.V("read data from stdin")
+		}
 		targetFS = &fs.Reader{
 			ModTime:    timeStamp,
 			Name:       opts.StdinFilename,
@@ -423,20 +533,25 @@ func runBackup(opts BackupOptions, gopts GlobalOptions, term *termstatus.Termina
 	}
 
 	sc := archiver.NewScanner(targetFS)
+	sc.SelectByName = selectByNameFilter
 	sc.Select = selectFilter
 	sc.Error = p.ScannerError
 	sc.Result = p.ReportTotal
 
-	p.V("start scan on %v", targets)
+	if !gopts.JSON {
+		p.V("start scan on %v", targets)
+	}
 	t.Go(func() error { return sc.Scan(t.Context(gopts.ctx), targets) })
 
 	arch := archiver.New(repo, targetFS, archiver.Options{})
+	arch.SelectByName = selectByNameFilter
 	arch.Select = selectFilter
 	arch.WithAtime = opts.WithAtime
 	arch.Error = p.Error
-	arch.CompleteItem = p.CompleteItemFn
+	arch.CompleteItem = p.CompleteItem
 	arch.StartFile = p.StartFile
 	arch.CompleteBlob = p.CompleteBlob
+	arch.IgnoreInode = opts.IgnoreInode
 
 	if parentSnapshotID == nil {
 		parentSnapshotID = &restic.ID{}
@@ -446,17 +561,21 @@ func runBackup(opts BackupOptions, gopts GlobalOptions, term *termstatus.Termina
 		Excludes:       opts.Excludes,
 		Tags:           opts.Tags,
 		Time:           timeStamp,
-		Hostname:       opts.Hostname,
+		Hostname:       opts.Host,
 		ParentSnapshot: *parentSnapshotID,
 	}
 
 	uploader := archiver.IndexUploader{
 		Repository: repo,
 		Start: func() {
-			p.VV("uploading intermediate index")
+			if !gopts.JSON {
+				p.VV("uploading intermediate index")
+			}
 		},
 		Complete: func(id restic.ID) {
-			p.V("uploaded intermediate index %v", id.Str())
+			if !gopts.JSON {
+				p.V("uploaded intermediate index %v", id.Str())
+			}
 		},
 	}
 
@@ -464,14 +583,18 @@ func runBackup(opts BackupOptions, gopts GlobalOptions, term *termstatus.Termina
 		return uploader.Upload(gopts.ctx, t.Context(gopts.ctx), 30*time.Second)
 	})
 
-	p.V("start backup on %v", targets)
+	if !gopts.JSON {
+		p.V("start backup on %v", targets)
+	}
 	_, id, err := arch.Snapshot(gopts.ctx, targets, snapshotOpts)
 	if err != nil {
 		return errors.Fatalf("unable to save snapshot: %v", err)
 	}
 
-	p.Finish()
-	p.P("snapshot %s saved\n", id.Str())
+	p.Finish(id)
+	if !gopts.JSON {
+		p.P("snapshot %s saved\n", id.Str())
+	}
 
 	// cleanly shutdown all running goroutines
 	t.Kill(nil)

cmd/restic/cmd_cache.go

@@ -2,6 +2,7 @@ package main
 
 import (
 	"fmt"
+	"os"
 	"path/filepath"
 	"sort"
 	"time"
@@ -9,6 +10,7 @@ import (
 	"github.com/restic/restic/internal/cache"
 	"github.com/restic/restic/internal/errors"
 	"github.com/restic/restic/internal/fs"
+	"github.com/restic/restic/internal/ui/table"
 	"github.com/spf13/cobra"
 )
 
@@ -28,6 +30,7 @@ The "cache" command allows listing and cleaning local cache directories.
 type CacheOptions struct {
 	Cleanup bool
 	MaxAge  uint
+	NoSize  bool
 }
 
 var cacheOptions CacheOptions
@@ -38,6 +41,7 @@ func init() {
 	f := cmdCache.Flags()
 	f.BoolVar(&cacheOptions.Cleanup, "cleanup", false, "remove old cache directories")
 	f.UintVar(&cacheOptions.MaxAge, "max-age", 30, "max age in `days` for cache directories to be considered old")
+	f.BoolVar(&cacheOptions.NoSize, "no-size", false, "do not output the size of the cache directories")
 }
 
 func runCache(opts CacheOptions, gopts GlobalOptions, args []string) error {
@@ -85,9 +89,22 @@ func runCache(opts CacheOptions, gopts GlobalOptions, args []string) error {
 		return nil
 	}
 
-	tab := NewTable()
-	tab.Header = fmt.Sprintf("%-14s   %-16s %s", "Repository ID", "Last Used", "Old")
-	tab.RowFormat = "%-14s   %-16s %s"
+	tab := table.New()
+
+	type data struct {
+		ID   string
+		Last string
+		Old  string
+		Size string
+	}
+
+	tab.AddColumn("Repo ID", "{{ .ID }}")
+	tab.AddColumn("Last Used", "{{ .Last }}")
+	tab.AddColumn("Old", "{{ .Old }}")
+
+	if !opts.NoSize {
+		tab.AddColumn("Size", "{{ .Size }}")
+	}
 
 	dirs, err := cache.All(cachedir)
 	if err != nil {
@@ -109,14 +126,41 @@ func runCache(opts CacheOptions, gopts GlobalOptions, args []string) error {
 			old = "yes"
 		}
 
-		tab.Rows = append(tab.Rows, []interface{}{
+		var size string
+		if !opts.NoSize {
+			bytes, err := dirSize(filepath.Join(cachedir, entry.Name()))
+			if err != nil {
+				return err
+			}
+			size = fmt.Sprintf("%11s", formatBytes(uint64(bytes)))
+		}
+
+		tab.AddRow(data{
 			entry.Name()[:10],
 			fmt.Sprintf("%d days ago", uint(time.Since(entry.ModTime()).Hours()/24)),
 			old,
+			size,
 		})
 	}
 
 	tab.Write(gopts.stdout)
+	Printf("%d cache dirs in %s\n", len(dirs), cachedir)
 
 	return nil
 }
+
+func dirSize(path string) (int64, error) {
+	var size int64
+	err := filepath.Walk(path, func(_ string, info os.FileInfo, err error) error {
+		if err != nil || info == nil {
+			return err
+		}
+
+		if !info.IsDir() {
+			size += info.Size()
+		}
+
+		return nil
+	})
+	return size, err
+}

cmd/restic/cmd_cat.go

@@ -74,7 +74,7 @@ func runCat(gopts GlobalOptions, args []string) error {
 		fmt.Println(string(buf))
 		return nil
 	case "index":
-		buf, err := repo.LoadAndDecrypt(gopts.ctx, restic.IndexFile, id)
+		buf, err := repo.LoadAndDecrypt(gopts.ctx, nil, restic.IndexFile, id)
 		if err != nil {
 			return err
 		}
@@ -99,7 +99,7 @@ func runCat(gopts GlobalOptions, args []string) error {
 		return nil
 	case "key":
 		h := restic.Handle{Type: restic.KeyFile, Name: id.String()}
-		buf, err := backend.LoadAll(gopts.ctx, repo.Backend(), h)
+		buf, err := backend.LoadAll(gopts.ctx, nil, repo.Backend(), h)
 		if err != nil {
 			return err
 		}
@@ -150,7 +150,7 @@ func runCat(gopts GlobalOptions, args []string) error {
 	switch tpe {
 	case "pack":
 		h := restic.Handle{Type: restic.DataFile, Name: id.String()}
-		buf, err := backend.LoadAll(gopts.ctx, repo.Backend(), h)
+		buf, err := backend.LoadAll(gopts.ctx, nil, repo.Backend(), h)
 		if err != nil {
 			return err
 		}

cmd/restic/cmd_check.go

@@ -50,7 +50,7 @@ func init() {
 
 	f := cmdCheck.Flags()
 	f.BoolVar(&checkOptions.ReadData, "read-data", false, "read all data blobs")
-	f.StringVar(&checkOptions.ReadDataSubset, "read-data-subset", "", "read subset of data packs")
+	f.StringVar(&checkOptions.ReadDataSubset, "read-data-subset", "", "read subset n of m data packs (format: `n/m`)")
 	f.BoolVar(&checkOptions.CheckUnused, "check-unused", false, "find unused blobs")
 	f.BoolVar(&checkOptions.WithCache, "with-cache", false, "use the cache")
 }
@@ -123,6 +123,7 @@ func newReadProgress(gopts GlobalOptions, todo restic.Stat) *restic.Progress {
 //
 //  * if --with-cache is specified, the default cache is used
 //  * if the user explicitly requested --no-cache, we don't use any cache
+//  * if the user provides --cache-dir, we use a cache in a temporary sub-directory of the specified directory and the sub-directory is deleted after the check
 //  * by default, we use a cache in a temporary directory that is deleted after the check
 func prepareCheckCache(opts CheckOptions, gopts *GlobalOptions) (cleanup func()) {
 	cleanup = func() {}
@@ -136,8 +137,10 @@ func prepareCheckCache(opts CheckOptions, gopts *GlobalOptions) (cleanup func())
 		return cleanup
 	}
 
+	cachedir := gopts.CacheDir
+
 	// use a cache in a temporary directory
-	tempdir, err := ioutil.TempDir("", "restic-check-cache-")
+	tempdir, err := ioutil.TempDir(cachedir, "restic-check-cache-")
 	if err != nil {
 		// if an error occurs, don't use any cache
 		Warnf("unable to create temporary directory for cache during check, disabling cache: %v\n", err)

cmd/restic/cmd_diff.go

@@ -21,11 +21,11 @@ The "diff" command shows differences from the first to the second snapshot. The
 first characters in each line display what has happened to a particular file or
 directory:
 
- +  The item was added
- -  The item was removed
- U  The metadata (access mode, timestamps, ...) for the item was updated
- M  The file's content was modified
- T  The type was changed, e.g. a file was made a symlink
+* +  The item was added
+* -  The item was removed
+* U  The metadata (access mode, timestamps, ...) for the item was updated
+* M  The file's content was modified
+* T  The type was changed, e.g. a file was made a symlink
 `,
 	DisableAutoGenTag: true,
 	RunE: func(cmd *cobra.Command, args []string) error {

cmd/restic/cmd_dump.go

@@ -1,14 +1,19 @@
 package main
 
 import (
+	"archive/tar"
 	"context"
 	"fmt"
+	"io"
 	"os"
+	"path"
 	"path/filepath"
+	"strings"
 
 	"github.com/restic/restic/internal/debug"
 	"github.com/restic/restic/internal/errors"
 	"github.com/restic/restic/internal/restic"
+	"github.com/restic/restic/internal/walker"
 
 	"github.com/spf13/cobra"
 )
@@ -47,43 +52,20 @@ func init() {
 	flags.StringArrayVar(&dumpOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path` for snapshot ID \"latest\"")
 }
 
-func splitPath(path string) []string {
-	d, f := filepath.Split(path)
-	if d == "" || d == "/" {
+func splitPath(p string) []string {
+	d, f := path.Split(p)
+	if d == "" {
 		return []string{f}
 	}
-	s := splitPath(filepath.Clean(d))
+	if d == "/" {
+		return []string{d}
+	}
+	s := splitPath(path.Clean(d))
 	return append(s, f)
 }
 
-func dumpNode(ctx context.Context, repo restic.Repository, node *restic.Node) error {
-	var buf []byte
-	for _, id := range node.Content {
-		size, found := repo.LookupBlobSize(id, restic.DataBlob)
-		if !found {
-			return errors.Errorf("id %v not found in repository", id)
-		}
+func printFromTree(ctx context.Context, tree *restic.Tree, repo restic.Repository, prefix string, pathComponents []string, pathToPrint string) error {
 
-		buf = buf[:cap(buf)]
-		if len(buf) < restic.CiphertextLength(int(size)) {
-			buf = restic.NewBlobBuffer(int(size))
-		}
-
-		n, err := repo.LoadBlob(ctx, restic.DataBlob, id, buf)
-		if err != nil {
-			return err
-		}
-		buf = buf[:n]
-
-		_, err = os.Stdout.Write(buf)
-		if err != nil {
-			return errors.Wrap(err, "Write")
-		}
-	}
-	return nil
-}
-
-func printFromTree(ctx context.Context, tree *restic.Tree, repo restic.Repository, prefix string, pathComponents []string) error {
 	if tree == nil {
 		return fmt.Errorf("called with a nil tree")
 	}
@@ -96,16 +78,19 @@ func printFromTree(ctx context.Context, tree *restic.Tree, repo restic.Repositor
 	}
 	item := filepath.Join(prefix, pathComponents[0])
 	for _, node := range tree.Nodes {
-		if node.Name == pathComponents[0] {
+		if node.Name == pathComponents[0] || pathComponents[0] == "/" {
 			switch {
 			case l == 1 && node.Type == "file":
-				return dumpNode(ctx, repo, node)
+				return getNodeData(ctx, os.Stdout, repo, node)
 			case l > 1 && node.Type == "dir":
 				subtree, err := repo.LoadTree(ctx, *node.Subtree)
 				if err != nil {
 					return errors.Wrapf(err, "cannot load subtree for %q", item)
 				}
-				return printFromTree(ctx, subtree, repo, item, pathComponents[1:])
+				return printFromTree(ctx, subtree, repo, item, pathComponents[1:], pathToPrint)
+			case node.Type == "dir":
+				node.Path = pathToPrint
+				return tarTree(ctx, repo, node, pathToPrint)
 			case l > 1:
 				return fmt.Errorf("%q should be a dir, but s a %q", item, node.Type)
 			case node.Type != "file":
@@ -128,7 +113,7 @@ func runDump(opts DumpOptions, gopts GlobalOptions, args []string) error {
 
 	debug.Log("dump file %q from %q", pathToPrint, snapshotIDString)
 
-	splittedPath := splitPath(pathToPrint)
+	splittedPath := splitPath(path.Clean(pathToPrint))
 
 	repo, err := OpenRepository(gopts)
 	if err != nil {
@@ -172,10 +157,143 @@ func runDump(opts DumpOptions, gopts GlobalOptions, args []string) error {
 		Exitf(2, "loading tree for snapshot %q failed: %v", snapshotIDString, err)
 	}
 
-	err = printFromTree(ctx, tree, repo, "", splittedPath)
+	err = printFromTree(ctx, tree, repo, "", splittedPath, pathToPrint)
 	if err != nil {
 		Exitf(2, "cannot dump file: %v", err)
 	}
 
 	return nil
 }
+
+func getNodeData(ctx context.Context, output io.Writer, repo restic.Repository, node *restic.Node) error {
+	var buf []byte
+	for _, id := range node.Content {
+
+		size, found := repo.LookupBlobSize(id, restic.DataBlob)
+		if !found {
+			return errors.Errorf("id %v not found in repository", id)
+		}
+
+		buf = buf[:cap(buf)]
+		if len(buf) < restic.CiphertextLength(int(size)) {
+			buf = restic.NewBlobBuffer(int(size))
+		}
+
+		n, err := repo.LoadBlob(ctx, restic.DataBlob, id, buf)
+		if err != nil {
+			return err
+		}
+		buf = buf[:n]
+
+		_, err = output.Write(buf)
+		if err != nil {
+			return errors.Wrap(err, "Write")
+		}
+
+	}
+	return nil
+}
+
+func tarTree(ctx context.Context, repo restic.Repository, rootNode *restic.Node, rootPath string) error {
+
+	if stdoutIsTerminal() {
+		return fmt.Errorf("stdout is the terminal, please redirect output")
+	}
+
+	tw := tar.NewWriter(os.Stdout)
+	defer tw.Close()
+
+	// If we want to dump "/" we'll need to add the name of the first node, too
+	// as it would get lost otherwise.
+	if rootNode.Path == "/" {
+		rootNode.Path = path.Join(rootNode.Path, rootNode.Name)
+		rootPath = rootNode.Path
+	}
+
+	// we know that rootNode is a folder and walker.Walk will already process
+	// the next node, so we have to tar this one first, too
+	if err := tarNode(ctx, tw, rootNode, repo); err != nil {
+		return err
+	}
+
+	err := walker.Walk(ctx, repo, *rootNode.Subtree, nil, func(_ restic.ID, nodepath string, node *restic.Node, err error) (bool, error) {
+		if err != nil {
+			return false, err
+		}
+		if node == nil {
+			return false, nil
+		}
+
+		node.Path = path.Join(rootPath, nodepath)
+
+		if node.Type == "file" || node.Type == "symlink" || node.Type == "dir" {
+			err := tarNode(ctx, tw, node, repo)
+			if err != err {
+				return false, err
+			}
+		}
+
+		return false, nil
+	})
+
+	return err
+}
+
+func tarNode(ctx context.Context, tw *tar.Writer, node *restic.Node, repo restic.Repository) error {
+
+	header := &tar.Header{
+		Name:       node.Path,
+		Size:       int64(node.Size),
+		Mode:       int64(node.Mode),
+		Uid:        int(node.UID),
+		Gid:        int(node.GID),
+		ModTime:    node.ModTime,
+		AccessTime: node.AccessTime,
+		ChangeTime: node.ChangeTime,
+		PAXRecords: parseXattrs(node.ExtendedAttributes),
+	}
+
+	if node.Type == "symlink" {
+		header.Typeflag = tar.TypeSymlink
+		header.Linkname = node.LinkTarget
+	}
+
+	if node.Type == "dir" {
+		header.Typeflag = tar.TypeDir
+	}
+
+	err := tw.WriteHeader(header)
+
+	if err != nil {
+		return errors.Wrap(err, "TarHeader ")
+	}
+
+	return getNodeData(ctx, tw, repo, node)
+
+}
+
+func parseXattrs(xattrs []restic.ExtendedAttribute) map[string]string {
+	tmpMap := make(map[string]string)
+
+	for _, attr := range xattrs {
+		attrString := string(attr.Value)
+
+		if strings.HasPrefix(attr.Name, "system.posix_acl_") {
+			na := acl{}
+			na.decode(attr.Value)
+
+			if na.String() != "" {
+				if strings.Contains(attr.Name, "system.posix_acl_access") {
+					tmpMap["SCHILY.acl.access"] = na.String()
+				} else if strings.Contains(attr.Name, "system.posix_acl_default") {
+					tmpMap["SCHILY.acl.default"] = na.String()
+				}
+			}
+
+		} else {
+			tmpMap["SCHILY.xattr."+attr.Name] = attrString
+		}
+	}
+
+	return tmpMap
+}

cmd/restic/cmd_find.go

@@ -16,27 +16,38 @@ import (
 )
 
 var cmdFind = &cobra.Command{
-	Use:   "find [flags] PATTERN",
-	Short: "Find a file or directory",
+	Use:   "find [flags] PATTERN...",
+	Short: "Find a file, a directory or restic IDs",
 	Long: `
 The "find" command searches for files or directories in snapshots stored in the
-repo. `,
+repo.
+It can also be used to search for restic blobs or trees for troubleshooting.`,
+	Example: `restic find config.json
+restic find --json "*.yml" "*.json"
+restic find --json --blob 420f620f b46ebe8a ddd38656
+restic find --show-pack-id --blob 420f620f
+restic find --tree 577c2bc9 f81f2e22 a62827a9
+restic find --pack 025c1d06`,
 	DisableAutoGenTag: true,
 	RunE: func(cmd *cobra.Command, args []string) error {
 		return runFind(findOptions, globalOptions, args)
 	},
 }
 
+const shortStr = 8 // Length of short IDs: 4 bytes as hex strings
+
 // FindOptions bundles all options for the find command.
 type FindOptions struct {
-	Oldest          string
-	Newest          string
-	Snapshots       []string
-	CaseInsensitive bool
-	ListLong        bool
-	Host            string
-	Paths           []string
-	Tags            restic.TagLists
+	Oldest             string
+	Newest             string
+	Snapshots          []string
+	BlobID, TreeID     bool
+	PackID, ShowPackID bool
+	CaseInsensitive    bool
+	ListLong           bool
+	Host               string
+	Paths              []string
+	Tags               restic.TagLists
 }
 
 var findOptions FindOptions
@@ -48,6 +59,10 @@ func init() {
 	f.StringVarP(&findOptions.Oldest, "oldest", "O", "", "oldest modification date/time")
 	f.StringVarP(&findOptions.Newest, "newest", "N", "", "newest modification date/time")
 	f.StringArrayVarP(&findOptions.Snapshots, "snapshot", "s", nil, "snapshot `id` to search in (can be given multiple times)")
+	f.BoolVar(&findOptions.BlobID, "blob", false, "pattern is a blob-ID")
+	f.BoolVar(&findOptions.TreeID, "tree", false, "pattern is a tree-ID")
+	f.BoolVar(&findOptions.PackID, "pack", false, "pattern is a pack-ID")
+	f.BoolVar(&findOptions.ShowPackID, "show-pack-id", false, "display the pack-ID the blobs belong to (with --blob or --tree)")
 	f.BoolVarP(&findOptions.CaseInsensitive, "ignore-case", "i", false, "ignore case for pattern")
 	f.BoolVarP(&findOptions.ListLong, "long", "l", false, "use a long listing format showing size and mode")
 
@@ -58,7 +73,7 @@ func init() {
 
 type findPattern struct {
 	oldest, newest time.Time
-	pattern        string
+	pattern        []string
 	ignoreCase     bool
 }
 
@@ -95,7 +110,7 @@ type statefulOutput struct {
 	hits     int
 }
 
-func (s *statefulOutput) PrintJSON(path string, node *restic.Node) {
+func (s *statefulOutput) PrintPatternJSON(path string, node *restic.Node) {
 	type findNode restic.Node
 	b, err := json.Marshal(struct {
 		// Add these attributes
@@ -139,7 +154,7 @@ func (s *statefulOutput) PrintJSON(path string, node *restic.Node) {
 	s.hits++
 }
 
-func (s *statefulOutput) PrintNormal(path string, node *restic.Node) {
+func (s *statefulOutput) PrintPatternNormal(path string, node *restic.Node) {
 	if s.newsn != s.oldsn {
 		if s.oldsn != nil {
 			Verbosef("\n")
@@ -150,11 +165,62 @@ func (s *statefulOutput) PrintNormal(path string, node *restic.Node) {
 	Printf(formatNode(path, node, s.ListLong) + "\n")
 }
 
-func (s *statefulOutput) Print(path string, node *restic.Node) {
+func (s *statefulOutput) PrintPattern(path string, node *restic.Node) {
 	if s.JSON {
-		s.PrintJSON(path, node)
+		s.PrintPatternJSON(path, node)
 	} else {
-		s.PrintNormal(path, node)
+		s.PrintPatternNormal(path, node)
+	}
+}
+
+func (s *statefulOutput) PrintObjectJSON(kind, id, nodepath, treeID string, sn *restic.Snapshot) {
+	b, err := json.Marshal(struct {
+		// Add these attributes
+		ObjectType string    `json:"object_type"`
+		ID         string    `json:"id"`
+		Path       string    `json:"path"`
+		ParentTree string    `json:"parent_tree,omitempty"`
+		SnapshotID string    `json:"snapshot"`
+		Time       time.Time `json:"time,omitempty"`
+	}{
+		ObjectType: kind,
+		ID:         id,
+		Path:       nodepath,
+		SnapshotID: sn.ID().String(),
+		ParentTree: treeID,
+		Time:       sn.Time,
+	})
+	if err != nil {
+		Warnf("Marshall failed: %v\n", err)
+		return
+	}
+	if !s.inuse {
+		Printf("[")
+		s.inuse = true
+	}
+	if s.hits > 0 {
+		Printf(",")
+	}
+	Printf(string(b))
+	s.hits++
+}
+
+func (s *statefulOutput) PrintObjectNormal(kind, id, nodepath, treeID string, sn *restic.Snapshot) {
+	Printf("Found %s %s\n", kind, id)
+	if kind == "blob" {
+		Printf(" ... in file %s\n", nodepath)
+		Printf("     (tree %s)\n", treeID)
+	} else {
+		Printf(" ... path %s\n", nodepath)
+	}
+	Printf(" ... in snapshot %s (%s)\n", sn.ID().Str(), sn.Time.Local().Format(TimeFormat))
+}
+
+func (s *statefulOutput) PrintObject(kind, id, nodepath, treeID string, sn *restic.Snapshot) {
+	if s.JSON {
+		s.PrintObjectJSON(kind, id, nodepath, treeID, sn)
+	} else {
+		s.PrintObjectNormal(kind, id, nodepath, treeID, sn)
 	}
 }
 
@@ -179,6 +245,9 @@ type Finder struct {
 	pat         findPattern
 	out         statefulOutput
 	ignoreTrees restic.IDSet
+	blobIDs     map[string]struct{}
+	treeIDs     map[string]struct{}
+	itemsFound  int
 }
 
 func (f *Finder) findInSnapshot(ctx context.Context, sn *restic.Snapshot) error {
@@ -189,23 +258,35 @@ func (f *Finder) findInSnapshot(ctx context.Context, sn *restic.Snapshot) error
 	}
 
 	f.out.newsn = sn
-	return walker.Walk(ctx, f.repo, *sn.Tree, f.ignoreTrees, func(nodepath string, node *restic.Node, err error) (bool, error) {
+	return walker.Walk(ctx, f.repo, *sn.Tree, f.ignoreTrees, func(parentTreeID restic.ID, nodepath string, node *restic.Node, err error) (bool, error) {
 		if err != nil {
-			return false, err
+			debug.Log("Error loading tree %v: %v", parentTreeID, err)
+
+			Printf("Unable to load tree %s\n ... which belongs to snapshot %s.\n", parentTreeID, sn.ID())
+
+			return false, walker.SkipNode
 		}
 
 		if node == nil {
 			return false, nil
 		}
 
-		name := node.Name
+		normalizedNodepath := nodepath
 		if f.pat.ignoreCase {
-			name = strings.ToLower(name)
+			normalizedNodepath = strings.ToLower(nodepath)
 		}
 
-		foundMatch, err := filter.Match(f.pat.pattern, nodepath)
-		if err != nil {
-			return false, err
+		var foundMatch bool
+
+		for _, pat := range f.pat.pattern {
+			found, err := filter.Match(pat, normalizedNodepath)
+			if err != nil {
+				return false, err
+			}
+			if found {
+				foundMatch = true
+				break
+			}
 		}
 
 		var (
@@ -213,9 +294,16 @@ func (f *Finder) findInSnapshot(ctx context.Context, sn *restic.Snapshot) error
 			errIfNoMatch    error
 		)
 		if node.Type == "dir" {
-			childMayMatch, err := filter.ChildMatch(f.pat.pattern, nodepath)
-			if err != nil {
-				return false, err
+			var childMayMatch bool
+			for _, pat := range f.pat.pattern {
+				mayMatch, err := filter.ChildMatch(pat, normalizedNodepath)
+				if err != nil {
+					return false, err
+				}
+				if mayMatch {
+					childMayMatch = true
+					break
+				}
 			}
 
 			if !childMayMatch {
@@ -241,20 +329,171 @@ func (f *Finder) findInSnapshot(ctx context.Context, sn *restic.Snapshot) error
 		}
 
 		debug.Log("    found match\n")
-		f.out.Print(nodepath, node)
+		f.out.PrintPattern(nodepath, node)
 		return false, nil
 	})
 }
 
+func (f *Finder) findIDs(ctx context.Context, sn *restic.Snapshot) error {
+	debug.Log("searching IDs in snapshot %s", sn.ID())
+
+	if sn.Tree == nil {
+		return errors.Errorf("snapshot %v has no tree", sn.ID().Str())
+	}
+
+	f.out.newsn = sn
+	return walker.Walk(ctx, f.repo, *sn.Tree, f.ignoreTrees, func(parentTreeID restic.ID, nodepath string, node *restic.Node, err error) (bool, error) {
+		if err != nil {
+			debug.Log("Error loading tree %v: %v", parentTreeID, err)
+
+			Printf("Unable to load tree %s\n ... which belongs to snapshot %s.\n", parentTreeID, sn.ID())
+
+			return false, walker.SkipNode
+		}
+
+		if node == nil {
+			return false, nil
+		}
+
+		if node.Type == "dir" && f.treeIDs != nil {
+			treeID := node.Subtree
+			found := false
+			if _, ok := f.treeIDs[treeID.Str()]; ok {
+				found = true
+			} else if _, ok := f.treeIDs[treeID.String()]; ok {
+				found = true
+			}
+			if found {
+				f.out.PrintObject("tree", treeID.String(), nodepath, "", sn)
+				f.itemsFound++
+				// Terminate if we have found all trees (and we are not
+				// looking for blobs)
+				if f.itemsFound >= len(f.treeIDs) && f.blobIDs == nil {
+					// Return an error to terminate the Walk
+					return true, errors.New("OK")
+				}
+			}
+		}
+
+		if node.Type == "file" && f.blobIDs != nil {
+			for _, id := range node.Content {
+				idStr := id.String()
+				if _, ok := f.blobIDs[idStr]; !ok {
+					// Look for short ID form
+					if _, ok := f.blobIDs[idStr[:shortStr]]; !ok {
+						continue
+					}
+					// Replace the short ID with the long one
+					f.blobIDs[idStr] = struct{}{}
+					delete(f.blobIDs, idStr[:shortStr])
+				}
+				f.out.PrintObject("blob", idStr, nodepath, parentTreeID.String(), sn)
+				break
+			}
+		}
+
+		return false, nil
+	})
+}
+
+// packsToBlobs converts the list of pack IDs to a list of blob IDs that
+// belong to those packs.
+func (f *Finder) packsToBlobs(ctx context.Context, packs []string) error {
+	packIDs := make(map[string]struct{})
+	for _, p := range packs {
+		packIDs[p] = struct{}{}
+	}
+	if f.blobIDs == nil {
+		f.blobIDs = make(map[string]struct{})
+	}
+
+	allPacksFound := false
+	packsFound := 0
+
+	debug.Log("Looking for packs...")
+	err := f.repo.List(ctx, restic.DataFile, func(id restic.ID, size int64) error {
+		if allPacksFound {
+			return nil
+		}
+		idStr := id.String()
+		if _, ok := packIDs[idStr]; !ok {
+			// Look for short ID form
+			if _, ok := packIDs[idStr[:shortStr]]; !ok {
+				return nil
+			}
+		}
+		debug.Log("Found pack %s", idStr)
+		blobs, _, err := f.repo.ListPack(ctx, id, size)
+		if err != nil {
+			return err
+		}
+		for _, b := range blobs {
+			f.blobIDs[b.ID.String()] = struct{}{}
+		}
+		// Stop searching when all packs have been found
+		packsFound++
+		if packsFound >= len(packIDs) {
+			allPacksFound = true
+		}
+		return nil
+	})
+
+	if err != nil {
+		return err
+	}
+
+	if !allPacksFound {
+		return errors.Fatal("unable to find all specified pack(s)")
+	}
+
+	debug.Log("%d blobs found", len(f.blobIDs))
+	return nil
+}
+
+func (f *Finder) findObjectPack(ctx context.Context, id string, t restic.BlobType) {
+	idx := f.repo.Index()
+
+	rid, err := restic.ParseID(id)
+	if err != nil {
+		Printf("Note: cannot find pack for object '%s', unable to parse ID: %v\n", id, err)
+		return
+	}
+
+	blobs, found := idx.Lookup(rid, t)
+	if !found {
+		Printf("Object %s not found in the index\n", rid.Str())
+		return
+	}
+
+	for _, b := range blobs {
+		if b.ID.Equal(rid) {
+			Printf("Object belongs to pack %s\n ... Pack %s: %s\n", b.PackID, b.PackID.Str(), b.String())
+			break
+		}
+	}
+}
+
+func (f *Finder) findObjectsPacks(ctx context.Context) {
+	for i := range f.blobIDs {
+		f.findObjectPack(ctx, i, restic.DataBlob)
+	}
+
+	for i := range f.treeIDs {
+		f.findObjectPack(ctx, i, restic.TreeBlob)
+	}
+}
+
 func runFind(opts FindOptions, gopts GlobalOptions, args []string) error {
-	if len(args) != 1 {
+	if len(args) == 0 {
 		return errors.Fatal("wrong number of arguments")
 	}
 
 	var err error
-	pat := findPattern{pattern: args[0]}
+	pat := findPattern{pattern: args}
 	if opts.CaseInsensitive {
-		pat.pattern = strings.ToLower(pat.pattern)
+		for i := range pat.pattern {
+			pat.pattern[i] = strings.ToLower(pat.pattern[i])
+		}
 		pat.ignoreCase = true
 	}
 
@@ -270,6 +509,14 @@ func runFind(opts FindOptions, gopts GlobalOptions, args []string) error {
 		}
 	}
 
+	// Check at most only one kind of IDs is provided: currently we
+	// can't mix types
+	if (opts.BlobID && opts.TreeID) ||
+		(opts.BlobID && opts.PackID) ||
+		(opts.TreeID && opts.PackID) {
+		return errors.Fatal("cannot have several ID types")
+	}
+
 	repo, err := OpenRepository(gopts)
 	if err != nil {
 		return err
@@ -296,12 +543,40 @@ func runFind(opts FindOptions, gopts GlobalOptions, args []string) error {
 		out:         statefulOutput{ListLong: opts.ListLong, JSON: globalOptions.JSON},
 		ignoreTrees: restic.NewIDSet(),
 	}
+
+	if opts.BlobID {
+		f.blobIDs = make(map[string]struct{})
+		for _, pat := range f.pat.pattern {
+			f.blobIDs[pat] = struct{}{}
+		}
+	}
+	if opts.TreeID {
+		f.treeIDs = make(map[string]struct{})
+		for _, pat := range f.pat.pattern {
+			f.treeIDs[pat] = struct{}{}
+		}
+	}
+
+	if opts.PackID {
+		f.packsToBlobs(ctx, []string{f.pat.pattern[0]}) // TODO: support multiple packs
+	}
+
 	for sn := range FindFilteredSnapshots(ctx, repo, opts.Host, opts.Tags, opts.Paths, opts.Snapshots) {
+		if f.blobIDs != nil || f.treeIDs != nil {
+			if err = f.findIDs(ctx, sn); err != nil && err.Error() != "OK" {
+				return err
+			}
+			continue
+		}
 		if err = f.findInSnapshot(ctx, sn); err != nil {
 			return err
 		}
 	}
 	f.out.Finish()
 
+	if opts.ShowPackID && (f.blobIDs != nil || f.treeIDs != nil) {
+		f.findObjectsPacks(ctx)
+	}
+
 	return nil
 }

cmd/restic/cmd_forget.go

@@ -3,10 +3,8 @@ package main
 import (
 	"context"
 	"encoding/json"
-	"sort"
-	"strings"
+	"io"
 
-	"github.com/restic/restic/internal/errors"
 	"github.com/restic/restic/internal/restic"
 	"github.com/spf13/cobra"
 )
@@ -59,14 +57,15 @@ func init() {
 	f.IntVarP(&forgetOptions.Weekly, "keep-weekly", "w", 0, "keep the last `n` weekly snapshots")
 	f.IntVarP(&forgetOptions.Monthly, "keep-monthly", "m", 0, "keep the last `n` monthly snapshots")
 	f.IntVarP(&forgetOptions.Yearly, "keep-yearly", "y", 0, "keep the last `n` yearly snapshots")
-	f.VarP(&forgetOptions.Within, "keep-within", "", "keep snapshots that were created within `duration` before the newest (e.g. 1y5m7d)")
+	f.VarP(&forgetOptions.Within, "keep-within", "", "keep snapshots that are newer than `duration` (eg. 1y5m7d2h) relative to the latest snapshot")
 
 	f.Var(&forgetOptions.KeepTags, "keep-tag", "keep snapshots with this `taglist` (can be specified multiple times)")
-	// Sadly the commonly used shortcut `H` is already used.
 	f.StringVar(&forgetOptions.Host, "host", "", "only consider snapshots with the given `host`")
-	// Deprecated since 2017-03-07.
-	f.StringVar(&forgetOptions.Host, "hostname", "", "only consider snapshots with the given `hostname` (deprecated)")
+	f.StringVar(&forgetOptions.Host, "hostname", "", "only consider snapshots with the given `hostname`")
+	f.MarkDeprecated("hostname", "use --host")
+
 	f.Var(&forgetOptions.Tags, "tag", "only consider snapshots which include this `taglist` in the format `tag[,tag,...]` (can be specified multiple times)")
+
 	f.StringArrayVar(&forgetOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path` (can be specified multiple times)")
 	f.BoolVarP(&forgetOptions.Compact, "compact", "c", false, "use compact format")
 
@@ -89,153 +88,129 @@ func runForget(opts ForgetOptions, gopts GlobalOptions, args []string) error {
 		return err
 	}
 
-	// group by hostname and dirs
-	type key struct {
-		Hostname string
-		Paths    []string
-		Tags     []string
-	}
-	snapshotGroups := make(map[string]restic.Snapshots)
-
-	var GroupByTag bool
-	var GroupByHost bool
-	var GroupByPath bool
-	var GroupOptionList []string
-
-	GroupOptionList = strings.Split(opts.GroupBy, ",")
-
-	for _, option := range GroupOptionList {
-		switch option {
-		case "host":
-			GroupByHost = true
-		case "paths":
-			GroupByPath = true
-		case "tags":
-			GroupByTag = true
-		case "":
-		default:
-			return errors.Fatal("unknown grouping option: '" + option + "'")
-		}
-	}
-
 	removeSnapshots := 0
 
 	ctx, cancel := context.WithCancel(gopts.ctx)
 	defer cancel()
+
+	var snapshots restic.Snapshots
+
 	for sn := range FindFilteredSnapshots(ctx, repo, opts.Host, opts.Tags, opts.Paths, args) {
-		if len(args) > 0 {
-			// When explicit snapshots args are given, remove them immediately.
+		snapshots = append(snapshots, sn)
+	}
+
+	if len(args) > 0 {
+		// When explicit snapshots args are given, remove them immediately.
+		for _, sn := range snapshots {
 			if !opts.DryRun {
 				h := restic.Handle{Type: restic.SnapshotFile, Name: sn.ID().String()}
 				if err = repo.Backend().Remove(gopts.ctx, h); err != nil {
 					return err
 				}
-				Verbosef("removed snapshot %v\n", sn.ID().Str())
+				if !gopts.JSON {
+					Verbosef("removed snapshot %v\n", sn.ID().Str())
+				}
 				removeSnapshots++
 			} else {
-				Verbosef("would have removed snapshot %v\n", sn.ID().Str())
+				if !gopts.JSON {
+					Verbosef("would have removed snapshot %v\n", sn.ID().Str())
+				}
 			}
-		} else {
-			// Determining grouping-keys
-			var tags []string
-			var hostname string
-			var paths []string
-
-			if GroupByTag {
-				tags = sn.Tags
-				sort.StringSlice(tags).Sort()
-			}
-			if GroupByHost {
-				hostname = sn.Hostname
-			}
-			if GroupByPath {
-				paths = sn.Paths
-			}
-
-			sort.StringSlice(sn.Paths).Sort()
-			var k []byte
-			var err error
-
-			k, err = json.Marshal(key{Tags: tags, Hostname: hostname, Paths: paths})
-
-			if err != nil {
-				return err
-			}
-			snapshotGroups[string(k)] = append(snapshotGroups[string(k)], sn)
 		}
-	}
+	} else {
+		snapshotGroups, _, err := restic.GroupSnapshots(snapshots, opts.GroupBy)
+		if err != nil {
+			return err
+		}
 
-	policy := restic.ExpirePolicy{
-		Last:    opts.Last,
-		Hourly:  opts.Hourly,
-		Daily:   opts.Daily,
-		Weekly:  opts.Weekly,
-		Monthly: opts.Monthly,
-		Yearly:  opts.Yearly,
-		Within:  opts.Within,
-		Tags:    opts.KeepTags,
-	}
+		policy := restic.ExpirePolicy{
+			Last:    opts.Last,
+			Hourly:  opts.Hourly,
+			Daily:   opts.Daily,
+			Weekly:  opts.Weekly,
+			Monthly: opts.Monthly,
+			Yearly:  opts.Yearly,
+			Within:  opts.Within,
+			Tags:    opts.KeepTags,
+		}
 
-	if policy.Empty() && len(args) == 0 {
-		Verbosef("no policy was specified, no snapshots will be removed\n")
-	}
+		if policy.Empty() && len(args) == 0 {
+			if !gopts.JSON {
+				Verbosef("no policy was specified, no snapshots will be removed\n")
+			}
+		}
 
-	if !policy.Empty() {
-		Verbosef("Applying Policy: %v\n", policy)
-
-		for k, snapshotGroup := range snapshotGroups {
-			var key key
-			if json.Unmarshal([]byte(k), &key) != nil {
-				return err
+		if !policy.Empty() {
+			if !gopts.JSON {
+				Verbosef("Applying Policy: %v\n", policy)
 			}
 
-			// Info
-			Verbosef("snapshots")
-			var infoStrings []string
-			if GroupByTag {
-				infoStrings = append(infoStrings, "tags ["+strings.Join(key.Tags, ", ")+"]")
-			}
-			if GroupByHost {
-				infoStrings = append(infoStrings, "host ["+key.Hostname+"]")
-			}
-			if GroupByPath {
-				infoStrings = append(infoStrings, "paths ["+strings.Join(key.Paths, ", ")+"]")
-			}
-			if infoStrings != nil {
-				Verbosef(" for (" + strings.Join(infoStrings, ", ") + ")")
-			}
-			Verbosef(":\n\n")
+			var jsonGroups []*ForgetGroup
 
-			keep, remove := restic.ApplyPolicy(snapshotGroup, policy)
-
-			if len(keep) != 0 && !gopts.Quiet {
-				Printf("keep %d snapshots:\n", len(keep))
-				PrintSnapshots(globalOptions.stdout, keep, opts.Compact)
-				Printf("\n")
-			}
-
-			if len(remove) != 0 && !gopts.Quiet {
-				Printf("remove %d snapshots:\n", len(remove))
-				PrintSnapshots(globalOptions.stdout, remove, opts.Compact)
-				Printf("\n")
-			}
-
-			removeSnapshots += len(remove)
-
-			if !opts.DryRun {
-				for _, sn := range remove {
-					h := restic.Handle{Type: restic.SnapshotFile, Name: sn.ID().String()}
-					err = repo.Backend().Remove(gopts.ctx, h)
+			for k, snapshotGroup := range snapshotGroups {
+				if gopts.Verbose >= 1 && !gopts.JSON {
+					err = PrintSnapshotGroupHeader(gopts.stdout, k)
 					if err != nil {
 						return err
 					}
 				}
+
+				var key restic.SnapshotGroupKey
+				if json.Unmarshal([]byte(k), &key) != nil {
+					return err
+				}
+
+				var fg ForgetGroup
+				fg.Tags = key.Tags
+				fg.Host = key.Hostname
+				fg.Paths = key.Paths
+
+				keep, remove, reasons := restic.ApplyPolicy(snapshotGroup, policy)
+
+				if len(keep) != 0 && !gopts.Quiet && !gopts.JSON {
+					Printf("keep %d snapshots:\n", len(keep))
+					PrintSnapshots(globalOptions.stdout, keep, reasons, opts.Compact)
+					Printf("\n")
+				}
+				addJSONSnapshots(&fg.Keep, keep)
+
+				if len(remove) != 0 && !gopts.Quiet && !gopts.JSON {
+					Printf("remove %d snapshots:\n", len(remove))
+					PrintSnapshots(globalOptions.stdout, remove, nil, opts.Compact)
+					Printf("\n")
+				}
+				addJSONSnapshots(&fg.Remove, remove)
+
+				fg.Reasons = reasons
+
+				jsonGroups = append(jsonGroups, &fg)
+
+				removeSnapshots += len(remove)
+
+				if !opts.DryRun {
+					for _, sn := range remove {
+						h := restic.Handle{Type: restic.SnapshotFile, Name: sn.ID().String()}
+						err = repo.Backend().Remove(gopts.ctx, h)
+						if err != nil {
+							return err
+						}
+					}
+				}
+			}
+
+			if gopts.JSON {
+				err = printJSONForget(gopts.stdout, jsonGroups)
+				if err != nil {
+					return err
+				}
 			}
 		}
 	}
 
 	if removeSnapshots > 0 && opts.Prune {
-		Verbosef("%d snapshots have been removed, running prune\n", removeSnapshots)
+		if !gopts.JSON {
+			Verbosef("%d snapshots have been removed, running prune\n", removeSnapshots)
+		}
 		if !opts.DryRun {
 			return pruneRepository(gopts, repo)
 		}
@@ -243,3 +218,28 @@ func runForget(opts ForgetOptions, gopts GlobalOptions, args []string) error {
 
 	return nil
 }
+
+// ForgetGroup helps to print what is forgotten in JSON.
+type ForgetGroup struct {
+	Tags    []string            `json:"tags"`
+	Host    string              `json:"host"`
+	Paths   []string            `json:"paths"`
+	Keep    []Snapshot          `json:"keep"`
+	Remove  []Snapshot          `json:"remove"`
+	Reasons []restic.KeepReason `json:"reasons"`
+}
+
+func addJSONSnapshots(js *[]Snapshot, list restic.Snapshots) {
+	for _, sn := range list {
+		k := Snapshot{
+			Snapshot: sn,
+			ID:       sn.ID(),
+			ShortID:  sn.ID().Str(),
+		}
+		*js = append(*js, k)
+	}
+}
+
+func printJSONForget(stdout io.Writer, forgets []*ForgetGroup) error {
+	return json.NewEncoder(stdout).Encode(forgets)
+}

cmd/restic/cmd_generate.go

@@ -12,7 +12,7 @@ var cmdGenerate = &cobra.Command{
 	Use:   "generate [command]",
 	Short: "Generate manual pages and auto-completion files (bash, zsh)",
 	Long: `
-The "generate" command writes automatically generated files like the man pages
+The "generate" command writes automatically generated files (like the man pages
 and the auto-completion files for bash and zsh).
 `,
 	DisableAutoGenTag: true,

cmd/restic/cmd_key.go

@@ -2,7 +2,7 @@ package main
 
 import (
 	"context"
-	"fmt"
+	"encoding/json"
 	"io/ioutil"
 	"os"
 	"strings"
@@ -10,6 +10,7 @@ import (
 	"github.com/restic/restic/internal/errors"
 	"github.com/restic/restic/internal/repository"
 	"github.com/restic/restic/internal/restic"
+	"github.com/restic/restic/internal/ui/table"
 
 	"github.com/spf13/cobra"
 )
@@ -35,10 +36,16 @@ func init() {
 	flags.StringVarP(&newPasswordFile, "new-password-file", "", "", "the file from which to load a new password")
 }
 
-func listKeys(ctx context.Context, s *repository.Repository) error {
-	tab := NewTable()
-	tab.Header = fmt.Sprintf(" %-10s  %-10s  %-10s  %s", "ID", "User", "Host", "Created")
-	tab.RowFormat = "%s%-10s  %-10s  %-10s  %s"
+func listKeys(ctx context.Context, s *repository.Repository, gopts GlobalOptions) error {
+	type keyInfo struct {
+		Current  bool   `json:"current"`
+		ID       string `json:"id"`
+		UserName string `json:"userName"`
+		HostName string `json:"hostName"`
+		Created  string `json:"created"`
+	}
+
+	var keys []keyInfo
 
 	err := s.List(ctx, restic.KeyFile, func(id restic.ID, size int64) error {
 		k, err := repository.LoadKey(ctx, s, id.String())
@@ -47,20 +54,36 @@ func listKeys(ctx context.Context, s *repository.Repository) error {
 			return nil
 		}
 
-		var current string
-		if id.String() == s.KeyName() {
-			current = "*"
-		} else {
-			current = " "
+		key := keyInfo{
+			Current:  id.String() == s.KeyName(),
+			ID:       id.Str(),
+			UserName: k.Username,
+			HostName: k.Hostname,
+			Created:  k.Created.Local().Format(TimeFormat),
 		}
-		tab.Rows = append(tab.Rows, []interface{}{current, id.Str(),
-			k.Username, k.Hostname, k.Created.Format(TimeFormat)})
+
+		keys = append(keys, key)
 		return nil
 	})
+
 	if err != nil {
 		return err
 	}
 
+	if gopts.JSON {
+		return json.NewEncoder(globalOptions.stdout).Encode(keys)
+	}
+
+	tab := table.New()
+	tab.AddColumn(" ID", "{{if .Current}}*{{else}} {{end}}{{ .ID }}")
+	tab.AddColumn("User", "{{ .UserName }}")
+	tab.AddColumn("Host", "{{ .HostName }}")
+	tab.AddColumn("Created", "{{ .Created }}")
+
+	for _, key := range keys {
+		tab.AddRow(key)
+	}
+
 	return tab.Write(globalOptions.stdout)
 }
 
@@ -160,7 +183,7 @@ func runKey(gopts GlobalOptions, args []string) error {
 			return err
 		}
 
-		return listKeys(ctx, repo)
+		return listKeys(ctx, repo, gopts)
 	case "add":
 		lock, err := lockRepo(repo)
 		defer unlockRepo(lock)

cmd/restic/cmd_ls.go

@@ -2,21 +2,37 @@ package main
 
 import (
 	"context"
+	"encoding/json"
+	"os"
+	"strings"
+	"time"
 
 	"github.com/spf13/cobra"
 
 	"github.com/restic/restic/internal/errors"
+	"github.com/restic/restic/internal/fs"
 	"github.com/restic/restic/internal/restic"
 	"github.com/restic/restic/internal/walker"
 )
 
 var cmdLs = &cobra.Command{
-	Use:   "ls [flags] [snapshot-ID ...]",
+	Use:   "ls [flags] [snapshotID] [dir...]",
 	Short: "List files in a snapshot",
 	Long: `
-The "ls" command allows listing files and directories in a snapshot.
+The "ls" command lists files and directories in a snapshot.
 
-The special snapshot-ID "latest" can be used to list files and directories of the latest snapshot in the repository.
+The special snapshot ID "latest" can be used to list files and
+directories of the latest snapshot in the repository. The
+--host flag can be used in conjunction to select the latest
+snapshot originating from a certain host only.
+
+File listings can optionally be filtered by directories. Any
+positional arguments after the snapshot ID are interpreted as
+absolute directory paths, and only files inside those directories
+will be listed. If the --recursive flag is used, then the filter
+will allow traversing into matching directories' subfolders.
+Any directory paths specified must be absolute (starting with
+a path separator); paths use the forward slash '/' as separator.
 `,
 	DisableAutoGenTag: true,
 	RunE: func(cmd *cobra.Command, args []string) error {
@@ -26,10 +42,11 @@ The special snapshot-ID "latest" can be used to list files and directories of th
 
 // LsOptions collects all options for the ls command.
 type LsOptions struct {
-	ListLong bool
-	Host     string
-	Tags     restic.TagLists
-	Paths    []string
+	ListLong  bool
+	Host      string
+	Tags      restic.TagLists
+	Paths     []string
+	Recursive bool
 }
 
 var lsOptions LsOptions
@@ -39,10 +56,31 @@ func init() {
 
 	flags := cmdLs.Flags()
 	flags.BoolVarP(&lsOptions.ListLong, "long", "l", false, "use a long listing format showing size and mode")
-
 	flags.StringVarP(&lsOptions.Host, "host", "H", "", "only consider snapshots for this `host`, when no snapshot ID is given")
 	flags.Var(&lsOptions.Tags, "tag", "only consider snapshots which include this `taglist`, when no snapshot ID is given")
 	flags.StringArrayVar(&lsOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path`, when no snapshot ID is given")
+	flags.BoolVar(&lsOptions.Recursive, "recursive", false, "include files in subfolders of the listed directories")
+}
+
+type lsSnapshot struct {
+	*restic.Snapshot
+	ID         *restic.ID `json:"id"`
+	ShortID    string     `json:"short_id"`
+	StructType string     `json:"struct_type"` // "snapshot"
+}
+
+type lsNode struct {
+	Name       string      `json:"name"`
+	Type       string      `json:"type"`
+	Path       string      `json:"path"`
+	UID        uint32      `json:"uid"`
+	GID        uint32      `json:"gid"`
+	Size       uint64      `json:"size,omitempty"`
+	Mode       os.FileMode `json:"mode,omitempty"`
+	ModTime    time.Time   `json:"mtime,omitempty"`
+	AccessTime time.Time   `json:"atime,omitempty"`
+	ChangeTime time.Time   `json:"ctime,omitempty"`
+	StructType string      `json:"struct_type"` // "node"
 }
 
 func runLs(opts LsOptions, gopts GlobalOptions, args []string) error {
@@ -50,6 +88,51 @@ func runLs(opts LsOptions, gopts GlobalOptions, args []string) error {
 		return errors.Fatal("Invalid arguments, either give one or more snapshot IDs or set filters.")
 	}
 
+	// extract any specific directories to walk
+	var dirs []string
+	if len(args) > 1 {
+		dirs = args[1:]
+		for _, dir := range dirs {
+			if !strings.HasPrefix(dir, "/") {
+				return errors.Fatal("All path filters must be absolute, starting with a forward slash '/'")
+			}
+		}
+	}
+
+	withinDir := func(nodepath string) bool {
+		if len(dirs) == 0 {
+			return true
+		}
+
+		for _, dir := range dirs {
+			// we're within one of the selected dirs, example:
+			//   nodepath: "/test/foo"
+			//   dir:      "/test"
+			if fs.HasPathPrefix(dir, nodepath) {
+				return true
+			}
+		}
+		return false
+	}
+
+	approachingMatchingTree := func(nodepath string) bool {
+		if len(dirs) == 0 {
+			return true
+		}
+
+		for _, dir := range dirs {
+			// the current node path is a prefix for one of the
+			// directories, so we're interested in something deeper in the
+			// tree. Example:
+			//   nodepath: "/test"
+			//   dir:      "/test/foo"
+			if fs.HasPathPrefix(nodepath, dir) {
+				return true
+			}
+		}
+		return false
+	}
+
 	repo, err := OpenRepository(gopts)
 	if err != nil {
 		return err
@@ -61,23 +144,88 @@ func runLs(opts LsOptions, gopts GlobalOptions, args []string) error {
 
 	ctx, cancel := context.WithCancel(gopts.ctx)
 	defer cancel()
-	for sn := range FindFilteredSnapshots(ctx, repo, opts.Host, opts.Tags, opts.Paths, args) {
-		Verbosef("snapshot %s of %v at %s):\n", sn.ID().Str(), sn.Paths, sn.Time)
 
-		err := walker.Walk(ctx, repo, *sn.Tree, nil, func(nodepath string, node *restic.Node, err error) (bool, error) {
+	var (
+		printSnapshot func(sn *restic.Snapshot)
+		printNode     func(path string, node *restic.Node)
+	)
+
+	if gopts.JSON {
+		enc := json.NewEncoder(gopts.stdout)
+
+		printSnapshot = func(sn *restic.Snapshot) {
+			enc.Encode(lsSnapshot{
+				Snapshot:   sn,
+				ID:         sn.ID(),
+				ShortID:    sn.ID().Str(),
+				StructType: "snapshot",
+			})
+		}
+
+		printNode = func(path string, node *restic.Node) {
+			enc.Encode(lsNode{
+				Name:       node.Name,
+				Type:       node.Type,
+				Path:       path,
+				UID:        node.UID,
+				GID:        node.GID,
+				Size:       node.Size,
+				Mode:       node.Mode,
+				ModTime:    node.ModTime,
+				AccessTime: node.AccessTime,
+				ChangeTime: node.ChangeTime,
+				StructType: "node",
+			})
+		}
+	} else {
+		printSnapshot = func(sn *restic.Snapshot) {
+			Verbosef("snapshot %s of %v filtered by %v at %s):\n", sn.ID().Str(), sn.Paths, dirs, sn.Time)
+		}
+		printNode = func(path string, node *restic.Node) {
+			Printf("%s\n", formatNode(path, node, lsOptions.ListLong))
+		}
+	}
+
+	for sn := range FindFilteredSnapshots(ctx, repo, opts.Host, opts.Tags, opts.Paths, args[:1]) {
+		printSnapshot(sn)
+
+		err := walker.Walk(ctx, repo, *sn.Tree, nil, func(_ restic.ID, nodepath string, node *restic.Node, err error) (bool, error) {
 			if err != nil {
 				return false, err
 			}
-
 			if node == nil {
 				return false, nil
 			}
-			Printf("%s\n", formatNode(nodepath, node, lsOptions.ListLong))
+
+			if withinDir(nodepath) {
+				// if we're within a dir, print the node
+				printNode(nodepath, node)
+
+				// if recursive listing is requested, signal the walker that it
+				// should continue walking recursively
+				if opts.Recursive {
+					return false, nil
+				}
+			}
+
+			// if there's an upcoming match deeper in the tree (but we're not
+			// there yet), signal the walker to descend into any subdirs
+			if approachingMatchingTree(nodepath) {
+				return false, nil
+			}
+
+			// otherwise, signal the walker to not walk recursively into any
+			// subdirs
+			if node.Type == "dir" {
+				return false, walker.SkipNode
+			}
 			return false, nil
 		})
+
 		if err != nil {
 			return err
 		}
 	}
+
 	return nil
 }

cmd/restic/cmd_mount.go

@@ -1,3 +1,4 @@
+// +build !netbsd
 // +build !openbsd
 // +build !solaris
 // +build !windows
@@ -52,13 +53,14 @@ For details please see the documentation for time.Format() at:
 
 // MountOptions collects all options for the mount command.
 type MountOptions struct {
-	OwnerRoot        bool
-	AllowRoot        bool
-	AllowOther       bool
-	Host             string
-	Tags             restic.TagLists
-	Paths            []string
-	SnapshotTemplate string
+	OwnerRoot            bool
+	AllowRoot            bool
+	AllowOther           bool
+	NoDefaultPermissions bool
+	Host                 string
+	Tags                 restic.TagLists
+	Paths                []string
+	SnapshotTemplate     string
 }
 
 var mountOptions MountOptions
@@ -70,6 +72,7 @@ func init() {
 	mountFlags.BoolVar(&mountOptions.OwnerRoot, "owner-root", false, "use 'root' as the owner of files and dirs")
 	mountFlags.BoolVar(&mountOptions.AllowRoot, "allow-root", false, "allow root user to access the data in the mounted directory")
 	mountFlags.BoolVar(&mountOptions.AllowOther, "allow-other", false, "allow other users to access the data in the mounted directory")
+	mountFlags.BoolVar(&mountOptions.NoDefaultPermissions, "no-default-permissions", false, "for 'allow-other', ignore Unix permissions and allow users to read all snapshot files")
 
 	mountFlags.StringVarP(&mountOptions.Host, "host", "H", "", `only consider snapshots for this host`)
 	mountFlags.Var(&mountOptions.Tags, "tag", "only consider snapshots which include this `taglist`")
@@ -117,6 +120,11 @@ func mount(opts MountOptions, gopts GlobalOptions, mountpoint string) error {
 
 	if opts.AllowOther {
 		mountOptions = append(mountOptions, systemFuse.AllowOther())
+
+		// let the kernel check permissions unless it is explicitly disabled
+		if !opts.NoDefaultPermissions {
+			mountOptions = append(mountOptions, systemFuse.DefaultPermissions())
+		}
 	}
 
 	c, err := systemFuse.Mount(mountpoint, mountOptions...)
@@ -141,7 +149,7 @@ func mount(opts MountOptions, gopts GlobalOptions, mountpoint string) error {
 	}
 
 	Printf("Now serving the repository at %s\n", mountpoint)
-	Printf("Don't forget to umount after quitting!\n")
+	Printf("When finished, quit with Ctrl-c or umount the mountpoint.\n")
 
 	debug.Log("serving mount at %v", mountpoint)
 	err = fs.Serve(c, root)

cmd/restic/cmd_prune.go

@@ -149,8 +149,8 @@ func pruneRepository(gopts GlobalOptions, repo restic.Repository) error {
 		len(idx.Packs), blobs, formatBytes(uint64(stats.bytes)))
 
 	blobCount := make(map[restic.BlobHandle]int)
-	duplicateBlobs := 0
-	duplicateBytes := 0
+	var duplicateBlobs uint64
+	var duplicateBytes uint64
 
 	// find duplicate blobs
 	for _, p := range idx.Packs {
@@ -161,7 +161,7 @@ func pruneRepository(gopts GlobalOptions, repo restic.Repository) error {
 
 			if blobCount[h] > 1 {
 				duplicateBlobs++
-				duplicateBytes += int(entry.Length)
+				duplicateBytes += uint64(entry.Length)
 			}
 		}
 	}
@@ -252,7 +252,7 @@ func pruneRepository(gopts GlobalOptions, repo restic.Repository) error {
 				continue
 			}
 
-			removeBytes += int(blob.Length)
+			removeBytes += uint64(blob.Length)
 		}
 
 		if hasActiveBlob {

cmd/restic/cmd_recover.go

@@ -0,0 +1,148 @@
+package main
+
+import (
+	"os"
+	"time"
+
+	"github.com/restic/restic/internal/errors"
+	"github.com/restic/restic/internal/restic"
+	"github.com/spf13/cobra"
+)
+
+var cmdRecover = &cobra.Command{
+	Use:   "recover [flags]",
+	Short: "Recover data from the repository",
+	Long: `
+The "recover" command build a new snapshot from all directories it can find in
+the raw data of the repository. It can be used if, for example, a snapshot has
+been removed by accident with "forget".
+`,
+	DisableAutoGenTag: true,
+	RunE: func(cmd *cobra.Command, args []string) error {
+		return runRecover(globalOptions)
+	},
+}
+
+func init() {
+	cmdRoot.AddCommand(cmdRecover)
+}
+
+func runRecover(gopts GlobalOptions) error {
+	hostname, err := os.Hostname()
+	if err != nil {
+		return err
+	}
+
+	repo, err := OpenRepository(gopts)
+	if err != nil {
+		return err
+	}
+
+	lock, err := lockRepo(repo)
+	defer unlockRepo(lock)
+	if err != nil {
+		return err
+	}
+
+	Verbosef("load index files\n")
+	if err = repo.LoadIndex(gopts.ctx); err != nil {
+		return err
+	}
+
+	// trees maps a tree ID to whether or not it is referenced by a different
+	// tree. If it is not referenced, we have a root tree.
+	trees := make(map[restic.ID]bool)
+
+	for blob := range repo.Index().Each(gopts.ctx) {
+		if blob.Blob.Type != restic.TreeBlob {
+			continue
+		}
+		trees[blob.Blob.ID] = false
+	}
+
+	cur := 0
+	max := len(trees)
+	Verbosef("load %d trees\n\n", len(trees))
+
+	for id := range trees {
+		cur++
+		Verbosef("\rtree (%v/%v)", cur, max)
+
+		if !trees[id] {
+			trees[id] = false
+		}
+
+		tree, err := repo.LoadTree(gopts.ctx, id)
+		if err != nil {
+			Warnf("unable to load tree %v: %v\n", id.Str(), err)
+			continue
+		}
+
+		for _, node := range tree.Nodes {
+			if node.Type != "dir" || node.Subtree == nil {
+				continue
+			}
+
+			subtree := *node.Subtree
+			trees[subtree] = true
+		}
+	}
+	Verbosef("\ndone\n")
+
+	roots := restic.NewIDSet()
+	for id, seen := range trees {
+		if seen {
+			continue
+		}
+
+		roots.Insert(id)
+	}
+
+	Verbosef("found %d roots\n", len(roots))
+
+	tree := restic.NewTree()
+	for id := range roots {
+		var subtreeID = id
+		node := restic.Node{
+			Type:       "dir",
+			Name:       id.Str(),
+			Mode:       0755,
+			Subtree:    &subtreeID,
+			AccessTime: time.Now(),
+			ModTime:    time.Now(),
+			ChangeTime: time.Now(),
+		}
+		tree.Insert(&node)
+	}
+
+	treeID, err := repo.SaveTree(gopts.ctx, tree)
+	if err != nil {
+		return errors.Fatalf("unable to save new tree to the repo: %v", err)
+	}
+
+	err = repo.Flush(gopts.ctx)
+	if err != nil {
+		return errors.Fatalf("unable to save blobs to the repo: %v", err)
+	}
+
+	err = repo.SaveIndex(gopts.ctx)
+	if err != nil {
+		return errors.Fatalf("unable to save new index to the repo: %v", err)
+	}
+
+	sn, err := restic.NewSnapshot([]string{"/recover"}, []string{}, hostname, time.Now())
+	if err != nil {
+		return errors.Fatalf("unable to save snapshot: %v", err)
+	}
+
+	sn.Tree = &treeID
+
+	id, err := repo.SaveJSONUnpacked(gopts.ctx, restic.SnapshotFile, sn)
+	if err != nil {
+		return errors.Fatalf("unable to save snapshot: %v", err)
+	}
+
+	Printf("saved new snapshot %v\n", id.Str())
+
+	return nil
+}

cmd/restic/cmd_restore.go

@@ -5,6 +5,8 @@ import (
 	"github.com/restic/restic/internal/errors"
 	"github.com/restic/restic/internal/filter"
 	"github.com/restic/restic/internal/restic"
+	"github.com/restic/restic/internal/restorer"
+	"strings"
 
 	"github.com/spf13/cobra"
 )
@@ -27,12 +29,15 @@ repository.
 
 // RestoreOptions collects all options for the restore command.
 type RestoreOptions struct {
-	Exclude []string
-	Include []string
-	Target  string
-	Host    string
-	Paths   []string
-	Tags    restic.TagLists
+	Exclude            []string
+	InsensitiveExclude []string
+	Include            []string
+	InsensitiveInclude []string
+	Target             string
+	Host               string
+	Paths              []string
+	Tags               restic.TagLists
+	Verify             bool
 }
 
 var restoreOptions RestoreOptions
@@ -42,16 +47,29 @@ func init() {
 
 	flags := cmdRestore.Flags()
 	flags.StringArrayVarP(&restoreOptions.Exclude, "exclude", "e", nil, "exclude a `pattern` (can be specified multiple times)")
+	flags.StringArrayVar(&restoreOptions.InsensitiveExclude, "iexclude", nil, "same as `--exclude` but ignores the casing of filenames")
 	flags.StringArrayVarP(&restoreOptions.Include, "include", "i", nil, "include a `pattern`, exclude everything else (can be specified multiple times)")
+	flags.StringArrayVar(&restoreOptions.InsensitiveInclude, "iinclude", nil, "same as `--include` but ignores the casing of filenames")
 	flags.StringVarP(&restoreOptions.Target, "target", "t", "", "directory to extract data to")
 
 	flags.StringVarP(&restoreOptions.Host, "host", "H", "", `only consider snapshots for this host when the snapshot ID is "latest"`)
 	flags.Var(&restoreOptions.Tags, "tag", "only consider snapshots which include this `taglist` for snapshot ID \"latest\"")
 	flags.StringArrayVar(&restoreOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path` for snapshot ID \"latest\"")
+	flags.BoolVar(&restoreOptions.Verify, "verify", false, "verify restored files content")
 }
 
 func runRestore(opts RestoreOptions, gopts GlobalOptions, args []string) error {
 	ctx := gopts.ctx
+	hasExcludes := len(opts.Exclude) > 0 || len(opts.InsensitiveExclude) > 0
+	hasIncludes := len(opts.Include) > 0 || len(opts.InsensitiveInclude) > 0
+
+	for i, str := range opts.InsensitiveExclude {
+		opts.InsensitiveExclude[i] = strings.ToLower(str)
+	}
+
+	for i, str := range opts.InsensitiveInclude {
+		opts.InsensitiveInclude[i] = strings.ToLower(str)
+	}
 
 	switch {
 	case len(args) == 0:
@@ -64,7 +82,7 @@ func runRestore(opts RestoreOptions, gopts GlobalOptions, args []string) error {
 		return errors.Fatal("please specify a directory to restore to (--target)")
 	}
 
-	if len(opts.Exclude) > 0 && len(opts.Include) > 0 {
+	if hasExcludes && hasIncludes {
 		return errors.Fatal("exclude and include patterns are mutually exclusive")
 	}
 
@@ -104,14 +122,14 @@ func runRestore(opts RestoreOptions, gopts GlobalOptions, args []string) error {
 		}
 	}
 
-	res, err := restic.NewRestorer(repo, id)
+	res, err := restorer.NewRestorer(repo, id)
 	if err != nil {
 		Exitf(2, "creating restorer failed: %v\n", err)
 	}
 
 	totalErrors := 0
-	res.Error = func(dir string, node *restic.Node, err error) error {
-		Warnf("ignoring error for %s: %s\n", dir, err)
+	res.Error = func(location string, err error) error {
+		Warnf("ignoring error for %s: %s\n", location, err)
 		totalErrors++
 		return nil
 	}
@@ -122,11 +140,16 @@ func runRestore(opts RestoreOptions, gopts GlobalOptions, args []string) error {
 			Warnf("error for exclude pattern: %v", err)
 		}
 
+		matchedInsensitive, _, err := filter.List(opts.InsensitiveExclude, strings.ToLower(item))
+		if err != nil {
+			Warnf("error for iexclude pattern: %v", err)
+		}
+
 		// An exclude filter is basically a 'wildcard but foo',
 		// so even if a childMayMatch, other children of a dir may not,
 		// therefore childMayMatch does not matter, but we should not go down
 		// unless the dir is selected for restore
-		selectedForRestore = !matched
+		selectedForRestore = !matched && !matchedInsensitive
 		childMayBeSelected = selectedForRestore && node.Type == "dir"
 
 		return selectedForRestore, childMayBeSelected
@@ -138,21 +161,32 @@ func runRestore(opts RestoreOptions, gopts GlobalOptions, args []string) error {
 			Warnf("error for include pattern: %v", err)
 		}
 
-		selectedForRestore = matched
-		childMayBeSelected = childMayMatch && node.Type == "dir"
+		matchedInsensitive, childMayMatchInsensitive, err := filter.List(opts.InsensitiveInclude, strings.ToLower(item))
+		if err != nil {
+			Warnf("error for iexclude pattern: %v", err)
+		}
+
+		selectedForRestore = matched || matchedInsensitive
+		childMayBeSelected = (childMayMatch || childMayMatchInsensitive) && node.Type == "dir"
 
 		return selectedForRestore, childMayBeSelected
 	}
 
-	if len(opts.Exclude) > 0 {
+	if hasExcludes {
 		res.SelectFilter = selectExcludeFilter
-	} else if len(opts.Include) > 0 {
+	} else if hasIncludes {
 		res.SelectFilter = selectIncludeFilter
 	}
 
 	Verbosef("restoring %s to %s\n", res.Snapshot(), opts.Target)
 
 	err = res.RestoreTo(ctx, opts.Target)
+	if err == nil && opts.Verify {
+		Verbosef("verifying files in %s\n", opts.Target)
+		var count int
+		count, err = res.VerifyFiles(ctx, opts.Target)
+		Verbosef("finished verifying %d files in %s\n", count, opts.Target)
+	}
 	if totalErrors > 0 {
 		Printf("There were %d errors\n", totalErrors)
 	}

cmd/restic/cmd_self_update.go

@@ -0,0 +1,73 @@
+// xbuild selfupdate
+
+package main
+
+import (
+	"os"
+
+	"github.com/restic/restic/internal/errors"
+	"github.com/restic/restic/internal/selfupdate"
+	"github.com/spf13/cobra"
+)
+
+var cmdSelfUpdate = &cobra.Command{
+	Use:   "self-update [flags]",
+	Short: "Update the restic binary",
+	Long: `
+The command "self-update" downloads the latest stable release of restic from
+GitHub and replaces the currently running binary. After download, the
+authenticity of the binary is verified using the GPG signature on the release
+files.
+`,
+	DisableAutoGenTag: true,
+	RunE: func(cmd *cobra.Command, args []string) error {
+		return runSelfUpdate(selfUpdateOptions, globalOptions, args)
+	},
+}
+
+// SelfUpdateOptions collects all options for the update-restic command.
+type SelfUpdateOptions struct {
+	Output string
+}
+
+var selfUpdateOptions SelfUpdateOptions
+
+func init() {
+	cmdRoot.AddCommand(cmdSelfUpdate)
+
+	flags := cmdSelfUpdate.Flags()
+	flags.StringVar(&selfUpdateOptions.Output, "output", "", "Save the downloaded file as `filename` (default: running binary itself)")
+}
+
+func runSelfUpdate(opts SelfUpdateOptions, gopts GlobalOptions, args []string) error {
+	if opts.Output == "" {
+		file, err := os.Executable()
+		if err != nil {
+			return errors.Wrap(err, "unable to find executable")
+		}
+
+		opts.Output = file
+	}
+
+	fi, err := os.Lstat(opts.Output)
+	if err != nil {
+		return err
+	}
+
+	if !fi.Mode().IsRegular() {
+		return errors.Errorf("output file %v is not a normal file, use --output to specify a different file", opts.Output)
+	}
+
+	Printf("writing restic to %v\n", opts.Output)
+
+	v, err := selfupdate.DownloadLatestStableRelease(gopts.ctx, opts.Output, version, Verbosef)
+	if err != nil {
+		return errors.Fatalf("unable to update restic: %v", err)
+	}
+
+	if v != version {
+		Printf("successfully updated restic to version %v\n", v)
+	}
+
+	return nil
+}

cmd/restic/cmd_snapshots.go

@@ -9,6 +9,7 @@ import (
 	"strings"
 
 	"github.com/restic/restic/internal/restic"
+	"github.com/restic/restic/internal/ui/table"
 	"github.com/spf13/cobra"
 )
 
@@ -31,6 +32,7 @@ type SnapshotOptions struct {
 	Paths   []string
 	Compact bool
 	Last    bool
+	GroupBy string
 }
 
 var snapshotOptions SnapshotOptions
@@ -44,6 +46,7 @@ func init() {
 	f.StringArrayVar(&snapshotOptions.Paths, "path", nil, "only consider snapshots for this `path` (can be specified multiple times)")
 	f.BoolVarP(&snapshotOptions.Compact, "compact", "c", false, "use compact format")
 	f.BoolVar(&snapshotOptions.Last, "last", false, "only show the last snapshot for each host and path")
+	f.StringVarP(&snapshotOptions.GroupBy, "group-by", "g", "", "string for grouping snapshots by host,paths,tags")
 }
 
 func runSnapshots(opts SnapshotOptions, gopts GlobalOptions, args []string) error {
@@ -63,25 +66,41 @@ func runSnapshots(opts SnapshotOptions, gopts GlobalOptions, args []string) erro
 	ctx, cancel := context.WithCancel(gopts.ctx)
 	defer cancel()
 
-	var list restic.Snapshots
+	var snapshots restic.Snapshots
 	for sn := range FindFilteredSnapshots(ctx, repo, opts.Host, opts.Tags, opts.Paths, args) {
-		list = append(list, sn)
+		snapshots = append(snapshots, sn)
+	}
+	snapshotGroups, grouped, err := restic.GroupSnapshots(snapshots, opts.GroupBy)
+	if err != nil {
+		return err
 	}
 
-	if opts.Last {
-		list = FilterLastSnapshots(list)
+	for k, list := range snapshotGroups {
+		if opts.Last {
+			list = FilterLastSnapshots(list)
+		}
+		sort.Sort(sort.Reverse(list))
+		snapshotGroups[k] = list
 	}
 
-	sort.Sort(sort.Reverse(list))
-
 	if gopts.JSON {
-		err := printSnapshotsJSON(gopts.stdout, list)
+		err := printSnapshotGroupJSON(gopts.stdout, snapshotGroups, grouped)
 		if err != nil {
-			Warnf("error printing snapshot: %v\n", err)
+			Warnf("error printing snapshots: %v\n", err)
 		}
 		return nil
 	}
-	PrintSnapshots(gopts.stdout, list, opts.Compact)
+
+	for k, list := range snapshotGroups {
+		if grouped {
+			err := PrintSnapshotGroupHeader(gopts.stdout, k)
+			if err != nil {
+				Warnf("error printing snapshots: %v\n", err)
+				return nil
+			}
+		}
+		PrintSnapshots(gopts.stdout, list, nil, opts.Compact)
+	}
 
 	return nil
 }
@@ -123,7 +142,16 @@ func FilterLastSnapshots(list restic.Snapshots) restic.Snapshots {
 }
 
 // PrintSnapshots prints a text table of the snapshots in list to stdout.
-func PrintSnapshots(stdout io.Writer, list restic.Snapshots, compact bool) {
+func PrintSnapshots(stdout io.Writer, list restic.Snapshots, reasons []restic.KeepReason, compact bool) {
+	// keep the reasons a snasphot is being kept in a map, so that it doesn't
+	// get lost when the list of snapshots is sorted
+	keepReasons := make(map[restic.ID]restic.KeepReason, len(reasons))
+	if len(reasons) > 0 {
+		for i, sn := range list {
+			id := sn.ID()
+			keepReasons[*id] = reasons[i]
+		}
+	}
 
 	// always sort the snapshots so that the newer ones are listed last
 	sort.SliceStable(list, func(i, j int) bool {
@@ -143,75 +171,112 @@ func PrintSnapshots(stdout io.Writer, list restic.Snapshots, compact bool) {
 		}
 	}
 
-	tab := NewTable()
-	if !compact {
-		tab.Header = fmt.Sprintf("%-8s  %-19s  %-*s  %-*s  %-3s %s", "ID", "Date", -maxHost, "Host", -maxTag, "Tags", "", "Directory")
-		tab.RowFormat = fmt.Sprintf("%%-8s  %%-19s  %%%ds  %%%ds  %%-3s %%s", -maxHost, -maxTag)
+	tab := table.New()
+
+	if compact {
+		tab.AddColumn("ID", "{{ .ID }}")
+		tab.AddColumn("Time", "{{ .Timestamp }}")
+		tab.AddColumn("Host", "{{ .Hostname }}")
+		tab.AddColumn("Tags  ", `{{ join .Tags "\n" }}`)
 	} else {
-		tab.Header = fmt.Sprintf("%-8s  %-19s  %-*s  %-*s", "ID", "Date", -maxHost, "Host", -maxTag, "Tags")
-		tab.RowFormat = fmt.Sprintf("%%-8s  %%-19s  %%%ds  %%s", -maxHost)
+		tab.AddColumn("ID", "{{ .ID }}")
+		tab.AddColumn("Time", "{{ .Timestamp }}")
+		tab.AddColumn("Host      ", "{{ .Hostname }}")
+		tab.AddColumn("Tags      ", `{{ join .Tags "," }}`)
+		if len(reasons) > 0 {
+			tab.AddColumn("Reasons", `{{ join .Reasons "\n" }}`)
+		}
+		tab.AddColumn("Paths", `{{ join .Paths "\n" }}`)
 	}
 
+	type snapshot struct {
+		ID        string
+		Timestamp string
+		Hostname  string
+		Tags      []string
+		Reasons   []string
+		Paths     []string
+	}
+
+	var multiline bool
 	for _, sn := range list {
-		if len(sn.Paths) == 0 {
-			continue
+		data := snapshot{
+			ID:        sn.ID().Str(),
+			Timestamp: sn.Time.Local().Format(TimeFormat),
+			Hostname:  sn.Hostname,
+			Tags:      sn.Tags,
+			Paths:     sn.Paths,
 		}
 
-		firstTag := ""
-		if len(sn.Tags) > 0 {
-			firstTag = sn.Tags[0]
+		if len(reasons) > 0 {
+			id := sn.ID()
+			data.Reasons = keepReasons[*id].Matches
 		}
 
-		rows := len(sn.Paths)
-		if rows < len(sn.Tags) {
-			rows = len(sn.Tags)
+		if len(sn.Paths) > 1 && !compact {
+			multiline = true
 		}
 
-		treeElement := "   "
-		if rows != 1 {
-			treeElement = "┌──"
-		}
-
-		if !compact {
-			tab.Rows = append(tab.Rows, []interface{}{sn.ID().Str(), sn.Time.Format(TimeFormat), sn.Hostname, firstTag, treeElement, sn.Paths[0]})
-		} else {
-			allTags := ""
-			for _, tag := range sn.Tags {
-				allTags += tag + " "
-			}
-			tab.Rows = append(tab.Rows, []interface{}{sn.ID().Str(), sn.Time.Format(TimeFormat), sn.Hostname, allTags})
-			continue
-		}
-
-		if len(sn.Tags) > rows {
-			rows = len(sn.Tags)
-		}
-
-		for i := 1; i < rows; i++ {
-			path := ""
-			if len(sn.Paths) > i {
-				path = sn.Paths[i]
-			}
-
-			tag := ""
-			if len(sn.Tags) > i {
-				tag = sn.Tags[i]
-			}
-
-			treeElement := "│"
-			if i == (rows - 1) {
-				treeElement = "└──"
-			}
-
-			tab.Rows = append(tab.Rows, []interface{}{"", "", "", tag, treeElement, path})
-		}
+		tab.AddRow(data)
 	}
 
-	tab.Footer = fmt.Sprintf("%d snapshots", len(list))
+	tab.AddFooter(fmt.Sprintf("%d snapshots", len(list)))
+
+	if multiline {
+		// print an additional blank line between snapshots
+
+		var last int
+		tab.PrintData = func(w io.Writer, idx int, s string) error {
+			var err error
+			if idx == last {
+				_, err = fmt.Fprintf(w, "%s\n", s)
+			} else {
+				_, err = fmt.Fprintf(w, "\n%s\n", s)
+			}
+			last = idx
+			return err
+		}
+	}
 
 	tab.Write(stdout)
 }
 
+// PrintSnapshotGroupHeader prints which group of the group-by option the
+// following snapshots belong to.
+// Prints nothing, if we did not group at all.
+func PrintSnapshotGroupHeader(stdout io.Writer, groupKeyJSON string) error {
+	var key restic.SnapshotGroupKey
+	var err error
+
+	err = json.Unmarshal([]byte(groupKeyJSON), &key)
+	if err != nil {
+		return err
+	}
+
+	if key.Hostname == "" && key.Tags == nil && key.Paths == nil {
+		return nil
+	}
+
+	// Info
+	fmt.Fprintf(stdout, "snapshots")
+	var infoStrings []string
+	if key.Hostname != "" {
+		infoStrings = append(infoStrings, "host ["+key.Hostname+"]")
+	}
+	if key.Tags != nil {
+		infoStrings = append(infoStrings, "tags ["+strings.Join(key.Tags, ", ")+"]")
+	}
+	if key.Paths != nil {
+		infoStrings = append(infoStrings, "paths ["+strings.Join(key.Paths, ", ")+"]")
+	}
+	if infoStrings != nil {
+		fmt.Fprintf(stdout, " for (%s)", strings.Join(infoStrings, ", "))
+	}
+	fmt.Fprintf(stdout, ":\n")
+
+	return nil
+}
+
 // Snapshot helps to print Snaphots as JSON with their ID included.
 type Snapshot struct {
 	*restic.Snapshot
@@ -220,19 +285,58 @@ type Snapshot struct {
 	ShortID string     `json:"short_id"`
 }
 
-// printSnapshotsJSON writes the JSON representation of list to stdout.
-func printSnapshotsJSON(stdout io.Writer, list restic.Snapshots) error {
+// SnapshotGroup helps to print SnaphotGroups as JSON with their GroupReasons included.
+type SnapshotGroup struct {
+	GroupKey  restic.SnapshotGroupKey `json:"group_key"`
+	Snapshots []Snapshot              `json:"snapshots"`
+}
 
+// printSnapshotsJSON writes the JSON representation of list to stdout.
+func printSnapshotGroupJSON(stdout io.Writer, snGroups map[string]restic.Snapshots, grouped bool) error {
+	if grouped {
+		var snapshotGroups []SnapshotGroup
+
+		for k, list := range snGroups {
+			var key restic.SnapshotGroupKey
+			var err error
+			var snapshots []Snapshot
+
+			err = json.Unmarshal([]byte(k), &key)
+			if err != nil {
+				return err
+			}
+
+			for _, sn := range list {
+				k := Snapshot{
+					Snapshot: sn,
+					ID:       sn.ID(),
+					ShortID:  sn.ID().Str(),
+				}
+				snapshots = append(snapshots, k)
+			}
+
+			group := SnapshotGroup{
+				GroupKey:  key,
+				Snapshots: snapshots,
+			}
+			snapshotGroups = append(snapshotGroups, group)
+		}
+
+		return json.NewEncoder(stdout).Encode(snapshotGroups)
+	}
+
+	// Old behavior
 	var snapshots []Snapshot
 
-	for _, sn := range list {
-
-		k := Snapshot{
-			Snapshot: sn,
-			ID:       sn.ID(),
-			ShortID:  sn.ID().Str(),
+	for _, list := range snGroups {
+		for _, sn := range list {
+			k := Snapshot{
+				Snapshot: sn,
+				ID:       sn.ID(),
+				ShortID:  sn.ID().Str(),
+			}
+			snapshots = append(snapshots, k)
 		}
-		snapshots = append(snapshots, k)
 	}
 
 	return json.NewEncoder(stdout).Encode(snapshots)

cmd/restic/cmd_stats.go

@@ -0,0 +1,325 @@
+package main
+
+import (
+	"context"
+	"crypto/sha256"
+	"encoding/json"
+	"fmt"
+	"os"
+	"path/filepath"
+
+	"github.com/restic/restic/internal/errors"
+	"github.com/restic/restic/internal/restic"
+	"github.com/restic/restic/internal/walker"
+	"github.com/spf13/cobra"
+)
+
+var cmdStats = &cobra.Command{
+	Use:   "stats [flags] [snapshot-ID]",
+	Short: "Scan the repository and show basic statistics",
+	Long: `
+The "stats" command walks one or all snapshots in a repository and
+accumulates statistics about the data stored therein. It reports on
+the number of unique files and their sizes, according to one of
+the counting modes as given by the --mode flag.
+
+If no snapshot is specified, all snapshots will be considered. Some
+modes make more sense over just a single snapshot, while others
+are useful across all snapshots, depending on what you are trying
+to calculate.
+
+The modes are:
+
+* restore-size: (default) Counts the size of the restored files.
+* files-by-contents: Counts total size of files, where a file is
+   considered unique if it has unique contents.
+* raw-data: Counts the size of blobs in the repository, regardless of
+  how many files reference them.
+* blobs-per-file: A combination of files-by-contents and raw-data.
+
+Refer to the online manual for more details about each mode.
+`,
+	DisableAutoGenTag: true,
+	RunE: func(cmd *cobra.Command, args []string) error {
+		return runStats(globalOptions, args)
+	},
+}
+
+func init() {
+	cmdRoot.AddCommand(cmdStats)
+	f := cmdStats.Flags()
+	f.StringVar(&countMode, "mode", countModeRestoreSize, "counting mode: restore-size (default), files-by-contents, blobs-per-file, or raw-data")
+	f.StringVarP(&snapshotByHost, "host", "H", "", "filter latest snapshot by this hostname")
+}
+
+func runStats(gopts GlobalOptions, args []string) error {
+	err := verifyStatsInput(gopts, args)
+	if err != nil {
+		return err
+	}
+
+	ctx, cancel := context.WithCancel(gopts.ctx)
+	defer cancel()
+
+	repo, err := OpenRepository(gopts)
+	if err != nil {
+		return err
+	}
+
+	if err = repo.LoadIndex(ctx); err != nil {
+		return err
+	}
+
+	if !gopts.NoLock {
+		lock, err := lockRepo(repo)
+		defer unlockRepo(lock)
+		if err != nil {
+			return err
+		}
+	}
+
+	if !gopts.JSON {
+		Printf("scanning...\n")
+	}
+
+	// create a container for the stats (and other needed state)
+	stats := &statsContainer{
+		uniqueFiles: make(map[fileID]struct{}),
+		fileBlobs:   make(map[string]restic.IDSet),
+		blobs:       restic.NewBlobSet(),
+		blobsSeen:   restic.NewBlobSet(),
+	}
+
+	if snapshotIDString != "" {
+		// scan just a single snapshot
+
+		var sID restic.ID
+		if snapshotIDString == "latest" {
+			sID, err = restic.FindLatestSnapshot(ctx, repo, []string{}, []restic.TagList{}, snapshotByHost)
+			if err != nil {
+				return errors.Fatalf("latest snapshot for criteria not found: %v", err)
+			}
+		} else {
+			sID, err = restic.FindSnapshot(repo, snapshotIDString)
+			if err != nil {
+				return errors.Fatalf("error loading snapshot: %v", err)
+			}
+		}
+
+		snapshot, err := restic.LoadSnapshot(ctx, repo, sID)
+		if err != nil {
+			return errors.Fatalf("error loading snapshot from repo: %v", err)
+		}
+
+		err = statsWalkSnapshot(ctx, snapshot, repo, stats)
+	} else {
+		// iterate every snapshot in the repo
+		err = repo.List(ctx, restic.SnapshotFile, func(snapshotID restic.ID, size int64) error {
+			snapshot, err := restic.LoadSnapshot(ctx, repo, snapshotID)
+			if err != nil {
+				return fmt.Errorf("Error loading snapshot %s: %v", snapshotID.Str(), err)
+			}
+			return statsWalkSnapshot(ctx, snapshot, repo, stats)
+		})
+	}
+	if err != nil {
+		return err
+	}
+
+	if countMode == countModeRawData {
+		// the blob handles have been collected, but not yet counted
+		for blobHandle := range stats.blobs {
+			blobSize, found := repo.LookupBlobSize(blobHandle.ID, blobHandle.Type)
+			if !found {
+				return fmt.Errorf("blob %v not found", blobHandle)
+			}
+			stats.TotalSize += uint64(blobSize)
+			stats.TotalBlobCount++
+		}
+	}
+
+	if gopts.JSON {
+		err = json.NewEncoder(os.Stdout).Encode(stats)
+		if err != nil {
+			return fmt.Errorf("encoding output: %v", err)
+		}
+		return nil
+	}
+
+	// inform the user what was scanned and how it was scanned
+	snapshotsScanned := snapshotIDString
+	if snapshotsScanned == "latest" {
+		snapshotsScanned = "the latest snapshot"
+	} else if snapshotsScanned == "" {
+		snapshotsScanned = "all snapshots"
+	}
+	Printf("Stats for %s in %s mode:\n", snapshotsScanned, countMode)
+
+	if stats.TotalBlobCount > 0 {
+		Printf("  Total Blob Count:   %d\n", stats.TotalBlobCount)
+	}
+	if stats.TotalFileCount > 0 {
+		Printf("  Total File Count:   %d\n", stats.TotalFileCount)
+	}
+	Printf("        Total Size:   %-5s\n", formatBytes(stats.TotalSize))
+
+	return nil
+}
+
+func statsWalkSnapshot(ctx context.Context, snapshot *restic.Snapshot, repo restic.Repository, stats *statsContainer) error {
+	if snapshot.Tree == nil {
+		return fmt.Errorf("snapshot %s has nil tree", snapshot.ID().Str())
+	}
+
+	if countMode == countModeRawData {
+		// count just the sizes of unique blobs; we don't need to walk the tree
+		// ourselves in this case, since a nifty function does it for us
+		return restic.FindUsedBlobs(ctx, repo, *snapshot.Tree, stats.blobs, stats.blobsSeen)
+	}
+
+	err := walker.Walk(ctx, repo, *snapshot.Tree, restic.NewIDSet(), statsWalkTree(repo, stats))
+	if err != nil {
+		return fmt.Errorf("walking tree %s: %v", *snapshot.Tree, err)
+	}
+	return nil
+}
+
+func statsWalkTree(repo restic.Repository, stats *statsContainer) walker.WalkFunc {
+	return func(_ restic.ID, npath string, node *restic.Node, nodeErr error) (bool, error) {
+		if nodeErr != nil {
+			return true, nodeErr
+		}
+		if node == nil {
+			return true, nil
+		}
+
+		if countMode == countModeUniqueFilesByContents || countMode == countModeBlobsPerFile {
+			// only count this file if we haven't visited it before
+			fid := makeFileIDByContents(node)
+			if _, ok := stats.uniqueFiles[fid]; !ok {
+				// mark the file as visited
+				stats.uniqueFiles[fid] = struct{}{}
+
+				if countMode == countModeUniqueFilesByContents {
+					// simply count the size of each unique file (unique by contents only)
+					stats.TotalSize += node.Size
+					stats.TotalFileCount++
+				}
+				if countMode == countModeBlobsPerFile {
+					// count the size of each unique blob reference, which is
+					// by unique file (unique by contents and file path)
+					for _, blobID := range node.Content {
+						// ensure we have this file (by path) in our map; in this
+						// mode, a file is unique by both contents and path
+						nodePath := filepath.Join(npath, node.Name)
+						if _, ok := stats.fileBlobs[nodePath]; !ok {
+							stats.fileBlobs[nodePath] = restic.NewIDSet()
+							stats.TotalFileCount++
+						}
+						if _, ok := stats.fileBlobs[nodePath][blobID]; !ok {
+							// is always a data blob since we're accessing it via a file's Content array
+							blobSize, found := repo.LookupBlobSize(blobID, restic.DataBlob)
+							if !found {
+								return true, fmt.Errorf("blob %s not found for tree %s", blobID, *node.Subtree)
+							}
+
+							// count the blob's size, then add this blob by this
+							// file (path) so we don't double-count it
+							stats.TotalSize += uint64(blobSize)
+							stats.fileBlobs[nodePath].Insert(blobID)
+							// this mode also counts total unique blob _references_ per file
+							stats.TotalBlobCount++
+						}
+					}
+				}
+			}
+		}
+
+		if countMode == countModeRestoreSize {
+			// as this is a file in the snapshot, we can simply count its
+			// size without worrying about uniqueness, since duplicate files
+			// will still be restored
+			stats.TotalSize += node.Size
+			stats.TotalFileCount++
+		}
+
+		return true, nil
+	}
+}
+
+// makeFileIDByContents returns a hash of the blob IDs of the
+// node's Content in sequence.
+func makeFileIDByContents(node *restic.Node) fileID {
+	var bb []byte
+	for _, c := range node.Content {
+		bb = append(bb, []byte(c[:])...)
+	}
+	return sha256.Sum256(bb)
+}
+
+func verifyStatsInput(gopts GlobalOptions, args []string) error {
+	// require a recognized counting mode
+	switch countMode {
+	case countModeRestoreSize:
+	case countModeUniqueFilesByContents:
+	case countModeBlobsPerFile:
+	case countModeRawData:
+	default:
+		return fmt.Errorf("unknown counting mode: %s (use the -h flag to get a list of supported modes)", countMode)
+	}
+
+	// ensure at most one snapshot was specified
+	if len(args) > 1 {
+		return fmt.Errorf("only one snapshot may be specified")
+	}
+
+	// if a snapshot was specified, mark it as the one to scan
+	if len(args) == 1 {
+		snapshotIDString = args[0]
+	}
+
+	return nil
+}
+
+// statsContainer holds information during a walk of a repository
+// to collect information about it, as well as state needed
+// for a successful and efficient walk.
+type statsContainer struct {
+	TotalSize      uint64 `json:"total_size"`
+	TotalFileCount uint64 `json:"total_file_count"`
+	TotalBlobCount uint64 `json:"total_blob_count,omitempty"`
+
+	// uniqueFiles marks visited files according to their
+	// contents (hashed sequence of content blob IDs)
+	uniqueFiles map[fileID]struct{}
+
+	// fileBlobs maps a file name (path) to the set of
+	// blobs that have been seen as a part of the file
+	fileBlobs map[string]restic.IDSet
+
+	// blobs and blobsSeen are used to count individual
+	// unique blobs, independent of references to files
+	blobs, blobsSeen restic.BlobSet
+}
+
+// fileID is a 256-bit hash that distinguishes unique files.
+type fileID [32]byte
+
+var (
+	// the mode of counting to perform
+	countMode string
+
+	// the snapshot to scan, as given by the user
+	snapshotIDString string
+
+	// snapshotByHost is the host to filter latest
+	// snapshot by, if given by user
+	snapshotByHost string
+)
+
+const (
+	countModeRestoreSize           = "restore-size"
+	countModeUniqueFilesByContents = "files-by-contents"
+	countModeBlobsPerFile          = "blobs-per-file"
+	countModeRawData               = "raw-data"
+)

cmd/restic/exclude.go

@@ -60,15 +60,20 @@ func (rc *rejectionCache) Store(dir string, rejected bool) {
 	rc.m[dir] = rejected
 }
 
+// RejectByNameFunc is a function that takes a filename of a
+// file that would be included in the backup. The function returns true if it
+// should be excluded (rejected) from the backup.
+type RejectByNameFunc func(path string) bool
+
 // RejectFunc is a function that takes a filename and os.FileInfo of a
 // file that would be included in the backup. The function returns true if it
 // should be excluded (rejected) from the backup.
 type RejectFunc func(path string, fi os.FileInfo) bool
 
-// rejectByPattern returns a RejectFunc which rejects files that match
+// rejectByPattern returns a RejectByNameFunc which rejects files that match
 // one of the patterns.
-func rejectByPattern(patterns []string) RejectFunc {
-	return func(item string, fi os.FileInfo) bool {
+func rejectByPattern(patterns []string) RejectByNameFunc {
+	return func(item string) bool {
 		matched, _, err := filter.List(patterns, item)
 		if err != nil {
 			Warnf("error for exclude pattern: %v", err)
@@ -83,14 +88,26 @@ func rejectByPattern(patterns []string) RejectFunc {
 	}
 }
 
-// rejectIfPresent returns a RejectFunc which itself returns whether a path
-// should be excluded. The RejectFunc considers a file to be excluded when
+// Same as `rejectByPattern` but case insensitive.
+func rejectByInsensitivePattern(patterns []string) RejectByNameFunc {
+	for index, path := range patterns {
+		patterns[index] = strings.ToLower(path)
+	}
+
+	rejFunc := rejectByPattern(patterns)
+	return func(item string) bool {
+		return rejFunc(strings.ToLower(item))
+	}
+}
+
+// rejectIfPresent returns a RejectByNameFunc which itself returns whether a path
+// should be excluded. The RejectByNameFunc considers a file to be excluded when
 // it resides in a directory with an exclusion file, that is specified by
 // excludeFileSpec in the form "filename[:content]". The returned error is
 // non-nil if the filename component of excludeFileSpec is empty. If rc is
-// non-nil, it is going to be used in the RejectFunc to expedite the evaluation
+// non-nil, it is going to be used in the RejectByNameFunc to expedite the evaluation
 // of a directory based on previous visits.
-func rejectIfPresent(excludeFileSpec string) (RejectFunc, error) {
+func rejectIfPresent(excludeFileSpec string) (RejectByNameFunc, error) {
 	if excludeFileSpec == "" {
 		return nil, errors.New("name for exclusion tagfile is empty")
 	}
@@ -107,7 +124,7 @@ func rejectIfPresent(excludeFileSpec string) (RejectFunc, error) {
 	}
 	debug.Log("using %q as exclusion tagfile", tf)
 	rc := &rejectionCache{}
-	fn := func(filename string, _ os.FileInfo) bool {
+	fn := func(filename string) bool {
 		return isExcludedByFile(filename, tf, tc, rc)
 	}
 	return fn, nil
@@ -252,11 +269,11 @@ func rejectByDevice(samples []string) (RejectFunc, error) {
 	}, nil
 }
 
-// rejectResticCache returns a RejectFunc that rejects the restic cache
+// rejectResticCache returns a RejectByNameFunc that rejects the restic cache
 // directory (if set).
-func rejectResticCache(repo *repository.Repository) (RejectFunc, error) {
+func rejectResticCache(repo *repository.Repository) (RejectByNameFunc, error) {
 	if repo.Cache == nil {
-		return func(string, os.FileInfo) bool {
+		return func(string) bool {
 			return false
 		}, nil
 	}
@@ -266,7 +283,7 @@ func rejectResticCache(repo *repository.Repository) (RejectFunc, error) {
 		return nil, errors.New("cacheBase is empty string")
 	}
 
-	return func(item string, _ os.FileInfo) bool {
+	return func(item string) bool {
 		if fs.HasPathPrefix(cacheBase, item) {
 			debug.Log("rejecting restic cache directory %v", item)
 			return true

cmd/restic/exclude_test.go

@@ -27,7 +27,34 @@ func TestRejectByPattern(t *testing.T) {
 	for _, tc := range tests {
 		t.Run("", func(t *testing.T) {
 			reject := rejectByPattern(patterns)
-			res := reject(tc.filename, nil)
+			res := reject(tc.filename)
+			if res != tc.reject {
+				t.Fatalf("wrong result for filename %v: want %v, got %v",
+					tc.filename, tc.reject, res)
+			}
+		})
+	}
+}
+
+func TestRejectByInsensitivePattern(t *testing.T) {
+	var tests = []struct {
+		filename string
+		reject   bool
+	}{
+		{filename: "/home/user/foo.GO", reject: true},
+		{filename: "/home/user/foo.c", reject: false},
+		{filename: "/home/user/foobar", reject: false},
+		{filename: "/home/user/FOObar/x", reject: true},
+		{filename: "/home/user/README", reject: false},
+		{filename: "/home/user/readme.md", reject: true},
+	}
+
+	patterns := []string{"*.go", "README.md", "/home/user/foobar/*"}
+
+	for _, tc := range tests {
+		t.Run("", func(t *testing.T) {
+			reject := rejectByInsensitivePattern(patterns)
+			res := reject(tc.filename)
 			if res != tc.reject {
 				t.Fatalf("wrong result for filename %v: want %v, got %v",
 					tc.filename, tc.reject, res)
@@ -140,8 +167,8 @@ func TestMultipleIsExcludedByFile(t *testing.T) {
 		if err != nil {
 			return err
 		}
-		excludedByFoo := fooExclude(p, fi)
-		excludedByBar := barExclude(p, fi)
+		excludedByFoo := fooExclude(p)
+		excludedByBar := barExclude(p)
 		excluded := excludedByFoo || excludedByBar
 		// the log message helps debugging in case the test fails
 		t.Logf("%q: %v || %v = %v", p, excludedByFoo, excludedByBar, excluded)

cmd/restic/format.go

@@ -21,7 +21,7 @@ func formatBytes(c uint64) string {
 	case c > 1<<10:
 		return fmt.Sprintf("%.3f KiB", b/(1<<10))
 	default:
-		return fmt.Sprintf("%dB", c)
+		return fmt.Sprintf("%d B", c)
 	}
 }
 
@@ -90,6 +90,6 @@ func formatNode(path string, n *restic.Node, long bool) string {
 
 	return fmt.Sprintf("%s %5d %5d %6d %s %s%s",
 		mode|n.Mode, n.UID, n.GID, n.Size,
-		n.ModTime.Format(TimeFormat), path,
+		n.ModTime.Local().Format(TimeFormat), path,
 		target)
 }

cmd/restic/global.go

@@ -1,6 +1,7 @@
 package main
 
 import (
+	"bufio"
 	"context"
 	"fmt"
 	"io"
@@ -34,23 +35,29 @@ import (
 	"github.com/restic/restic/internal/errors"
 
 	"golang.org/x/crypto/ssh/terminal"
+	"os/exec"
 )
 
-var version = "compiled manually"
+var version = "0.9.5"
+
+// TimeFormat is the format used for all timestamps printed by restic.
+const TimeFormat = "2006-01-02 15:04:05"
 
 // GlobalOptions hold all global options for restic.
 type GlobalOptions struct {
-	Repo          string
-	PasswordFile  string
-	Quiet         bool
-	Verbose       int
-	NoLock        bool
-	JSON          bool
-	CacheDir      string
-	NoCache       bool
-	CACerts       []string
-	TLSClientCert string
-	CleanupCache  bool
+	Repo            string
+	PasswordFile    string
+	PasswordCommand string
+	KeyHint         string
+	Quiet           bool
+	Verbose         int
+	NoLock          bool
+	JSON            bool
+	CacheDir        string
+	NoCache         bool
+	CACerts         []string
+	TLSClientCert   string
+	CleanupCache    bool
 
 	LimitUploadKb   int
 	LimitDownloadKb int
@@ -64,7 +71,7 @@ type GlobalOptions struct {
 	//  0 means: don't print any messages except errors, this is used when --quiet is specified
 	//  1 is the default: print essential messages
 	//  2 means: print more messages, report minor things, this is used when --verbose is specified
-	//  3 means: print very detailed debug messages, this is used when --debug is specified
+	//  3 means: print very detailed debug messages, this is used when --verbose 2 is specified
 	verbosity uint
 
 	Options []string
@@ -88,11 +95,13 @@ func init() {
 	f := cmdRoot.PersistentFlags()
 	f.StringVarP(&globalOptions.Repo, "repo", "r", os.Getenv("RESTIC_REPOSITORY"), "repository to backup to or restore from (default: $RESTIC_REPOSITORY)")
 	f.StringVarP(&globalOptions.PasswordFile, "password-file", "p", os.Getenv("RESTIC_PASSWORD_FILE"), "read the repository password from a file (default: $RESTIC_PASSWORD_FILE)")
+	f.StringVarP(&globalOptions.KeyHint, "key-hint", "", os.Getenv("RESTIC_KEY_HINT"), "key ID of key to try decrypting first (default: $RESTIC_KEY_HINT)")
+	f.StringVarP(&globalOptions.PasswordCommand, "password-command", "", os.Getenv("RESTIC_PASSWORD_COMMAND"), "specify a shell command to obtain a password (default: $RESTIC_PASSWORD_COMMAND)")
 	f.BoolVarP(&globalOptions.Quiet, "quiet", "q", false, "do not output comprehensive progress report")
 	f.CountVarP(&globalOptions.Verbose, "verbose", "v", "be verbose (specify --verbose multiple times or level `n`)")
 	f.BoolVar(&globalOptions.NoLock, "no-lock", false, "do not lock the repo, this allows some operations on read-only repos")
 	f.BoolVarP(&globalOptions.JSON, "json", "", false, "set output mode to JSON for commands that support it")
-	f.StringVar(&globalOptions.CacheDir, "cache-dir", "", "set the cache directory")
+	f.StringVar(&globalOptions.CacheDir, "cache-dir", "", "set the cache directory. (default: use system default cache directory)")
 	f.BoolVar(&globalOptions.NoCache, "no-cache", false, "do not use a local cache")
 	f.StringSliceVar(&globalOptions.CACerts, "cacert", nil, "`file` to load root certificates from (default: use system certificates)")
 	f.StringVar(&globalOptions.TLSClientCert, "tls-client-cert", "", "path to a file containing PEM encoded TLS client certificate and private key")
@@ -176,7 +185,6 @@ func Printf(format string, args ...interface{}) {
 	_, err := fmt.Fprintf(globalOptions.stdout, format, args...)
 	if err != nil {
 		fmt.Fprintf(os.Stderr, "unable to write to stdout: %v\n", err)
-		Exit(100)
 	}
 }
 
@@ -217,7 +225,6 @@ func Warnf(format string, args ...interface{}) {
 	_, err := fmt.Fprintf(globalOptions.stderr, format, args...)
 	if err != nil {
 		fmt.Fprintf(os.Stderr, "unable to write to stderr: %v\n", err)
-		Exit(100)
 	}
 }
 
@@ -233,7 +240,23 @@ func Exitf(exitcode int, format string, args ...interface{}) {
 }
 
 // resolvePassword determines the password to be used for opening the repository.
-func resolvePassword(opts GlobalOptions, env string) (string, error) {
+func resolvePassword(opts GlobalOptions) (string, error) {
+	if opts.PasswordFile != "" && opts.PasswordCommand != "" {
+		return "", errors.Fatalf("Password file and command are mutually exclusive options")
+	}
+	if opts.PasswordCommand != "" {
+		args, err := backend.SplitShellStrings(opts.PasswordCommand)
+		if err != nil {
+			return "", err
+		}
+		cmd := exec.Command(args[0], args[1:]...)
+		cmd.Stderr = os.Stderr
+		output, err := cmd.Output()
+		if err != nil {
+			return "", err
+		}
+		return (strings.TrimSpace(string(output))), nil
+	}
 	if opts.PasswordFile != "" {
 		s, err := textfile.Read(opts.PasswordFile)
 		if os.IsNotExist(errors.Cause(err)) {
@@ -242,7 +265,7 @@ func resolvePassword(opts GlobalOptions, env string) (string, error) {
 		return strings.TrimSpace(string(s)), errors.Wrap(err, "Readfile")
 	}
 
-	if pwd := os.Getenv(env); pwd != "" {
+	if pwd := os.Getenv("RESTIC_PASSWORD"); pwd != "" {
 		return pwd, nil
 	}
 
@@ -251,15 +274,10 @@ func resolvePassword(opts GlobalOptions, env string) (string, error) {
 
 // readPassword reads the password from the given reader directly.
 func readPassword(in io.Reader) (password string, err error) {
-	buf := make([]byte, 1000)
-	n, err := io.ReadFull(in, buf)
-	buf = buf[:n]
+	sc := bufio.NewScanner(in)
+	sc.Scan()
 
-	if err != nil && errors.Cause(err) != io.ErrUnexpectedEOF {
-		return "", errors.Wrap(err, "ReadFull")
-	}
-
-	return strings.TrimRight(string(buf), "\r\n"), nil
+	return sc.Text(), errors.Wrap(err, "Scan")
 }
 
 // readPasswordTerminal reads the password from the given reader which must be a
@@ -293,6 +311,7 @@ func ReadPassword(opts GlobalOptions, prompt string) (string, error) {
 		password, err = readPasswordTerminal(os.Stdin, os.Stderr, prompt)
 	} else {
 		password, err = readPassword(os.Stdin)
+		Verbosef("read password from stdin\n")
 	}
 
 	if err != nil {
@@ -313,13 +332,15 @@ func ReadPasswordTwice(gopts GlobalOptions, prompt1, prompt2 string) (string, er
 	if err != nil {
 		return "", err
 	}
-	pw2, err := ReadPassword(gopts, prompt2)
-	if err != nil {
-		return "", err
-	}
+	if stdinIsTerminal() {
+		pw2, err := ReadPassword(gopts, prompt2)
+		if err != nil {
+			return "", err
+		}
 
-	if pw1 != pw2 {
-		return "", errors.Fatal("passwords do not match")
+		if pw1 != pw2 {
+			return "", errors.Fatal("passwords do not match")
+		}
 	}
 
 	return pw1, nil
@@ -349,17 +370,19 @@ func OpenRepository(opts GlobalOptions) (*repository.Repository, error) {
 		return nil, err
 	}
 
-	err = s.SearchKey(opts.ctx, opts.password, maxKeys)
+	err = s.SearchKey(opts.ctx, opts.password, maxKeys, opts.KeyHint)
 	if err != nil {
 		return nil, err
 	}
 
-	if stdoutIsTerminal() {
+	if stdoutIsTerminal() && !opts.JSON {
 		id := s.Config().ID
 		if len(id) > 8 {
 			id = id[:8]
 		}
-		Verbosef("repository %v opened successfully, password is correct\n", id)
+		if !opts.JSON {
+			Verbosef("repository %v opened successfully, password is correct\n", id)
+		}
 	}
 
 	if opts.NoCache {
@@ -372,6 +395,10 @@ func OpenRepository(opts GlobalOptions) (*repository.Repository, error) {
 		return s, nil
 	}
 
+	if c.Created && !opts.JSON {
+		Verbosef("created new cache in %v\n", c.Base)
+	}
+
 	// start using the cache
 	s.UseCache(c)
 

cmd/restic/integration_fuse_test.go

@@ -1,3 +1,4 @@
+// +build !netbsd
 // +build !openbsd
 // +build !solaris
 // +build !windows

cmd/restic/integration_helpers_test.go

@@ -66,7 +66,7 @@ func isSymlink(fi os.FileInfo) bool {
 
 func sameModTime(fi1, fi2 os.FileInfo) bool {
 	switch runtime.GOOS {
-	case "darwin", "freebsd", "openbsd":
+	case "darwin", "freebsd", "openbsd", "netbsd":
 		if isSymlink(fi1) && isSymlink(fi2) {
 			return true
 		}

cmd/restic/integration_test.go

@@ -219,6 +219,35 @@ func testRunForget(t testing.TB, gopts GlobalOptions, args ...string) {
 	rtest.OK(t, runForget(opts, gopts, args))
 }
 
+func testRunForgetJSON(t testing.TB, gopts GlobalOptions, args ...string) {
+	buf := bytes.NewBuffer(nil)
+	oldJSON := gopts.JSON
+	gopts.stdout = buf
+	gopts.JSON = true
+	defer func() {
+		gopts.stdout = os.Stdout
+		gopts.JSON = oldJSON
+	}()
+
+	opts := ForgetOptions{
+		DryRun: true,
+		Last:   1,
+	}
+
+	rtest.OK(t, runForget(opts, gopts, args))
+
+	var forgets []*ForgetGroup
+	rtest.OK(t, json.Unmarshal(buf.Bytes(), &forgets))
+
+	rtest.Assert(t, len(forgets) == 1,
+		"Expected 1 snapshot group, got %v", len(forgets))
+	rtest.Assert(t, len(forgets[0].Keep) == 1,
+		"Expected 1 snapshot to be kept, got %v", len(forgets[0].Keep))
+	rtest.Assert(t, len(forgets[0].Remove) == 2,
+		"Expected 2 snapshots to be removed, got %v", len(forgets[0].Remove))
+	return
+}
+
 func testRunPrune(t testing.TB, gopts GlobalOptions) {
 	rtest.OK(t, runPrune(gopts))
 }
@@ -875,9 +904,6 @@ func TestRestoreNoMetadataOnIgnoredIntermediateDirs(t *testing.T) {
 	fi, err := os.Stat(f1)
 	rtest.OK(t, err)
 
-	rtest.Assert(t, fi.ModTime() != time.Unix(0, 0),
-		"meta data of intermediate directory has been restore although it was ignored")
-
 	// restore with filter "*", this should restore meta data on everything.
 	testRunRestoreIncludes(t, env.gopts, filepath.Join(env.base, "restore1"), snapshotID, []string{"*"})
 
@@ -1054,6 +1080,7 @@ func TestPrune(t *testing.T) {
 	rtest.Assert(t, len(snapshotIDs) == 3,
 		"expected 3 snapshot, got %v", snapshotIDs)
 
+	testRunForgetJSON(t, env.gopts)
 	testRunForget(t, env.gopts, firstSnapshot[0].String())
 	testRunPrune(t, env.gopts)
 	testRunCheck(t, env.gopts)

cmd/restic/main.go

@@ -54,7 +54,7 @@ directories in an encrypted repository stored on different backends.
 		if c.Name() == "version" {
 			return nil
 		}
-		pwd, err := resolvePassword(globalOptions, "RESTIC_PASSWORD")
+		pwd, err := resolvePassword(globalOptions)
 		if err != nil {
 			fmt.Fprintf(os.Stderr, "Resolving password failed: %v\n", err)
 			Exit(1)

cmd/restic/table.go

@@ -1,63 +0,0 @@
-package main
-
-import (
-	"fmt"
-	"io"
-	"strings"
-)
-
-// Table contains data for a table to be printed.
-type Table struct {
-	Header string
-	Rows   [][]interface{}
-	Footer string
-
-	RowFormat string
-}
-
-// NewTable initializes a new Table.
-func NewTable() Table {
-	return Table{
-		Rows: [][]interface{}{},
-	}
-}
-
-func (t Table) printSeparationLine(w io.Writer) error {
-	_, err := fmt.Fprintln(w, strings.Repeat("-", 70))
-	return err
-}
-
-// Write prints the table to w.
-func (t Table) Write(w io.Writer) error {
-	_, err := fmt.Fprintln(w, t.Header)
-	if err != nil {
-		return err
-	}
-
-	err = t.printSeparationLine(w)
-	if err != nil {
-		return err
-	}
-
-	for _, row := range t.Rows {
-		_, err = fmt.Fprintf(w, t.RowFormat+"\n", row...)
-		if err != nil {
-			return err
-		}
-	}
-
-	err = t.printSeparationLine(w)
-	if err != nil {
-		return err
-	}
-
-	_, err = fmt.Fprintln(w, t.Footer)
-	if err != nil {
-		return err
-	}
-
-	return nil
-}
-
-// TimeFormat is the format used for all timestamps printed by restic.
-const TimeFormat = "2006-01-02 15:04:05"

contrib/restic.spec

@@ -0,0 +1,72 @@
+Name: restic	
+Version: 0.9.2git.20180812	
+Release: 4%{?dist}
+Summary: restic is a backup program that is fast, efficient and secure.
+
+%global debug_package %{nil}
+
+Group: Applications/Archiving
+License: BSD 2-Clause
+URL: https://restic.net/	
+Source0: %{name}-%{version}.tar.gz
+
+BuildRequires:	golang 
+Requires: /bin/bash	
+
+%description
+restic is a program that does backups right. The design goals are:
+
+Easy: Doing backups should be a frictionless process, otherwise you are tempted to skip it. Restic should be easy to configure and use, so that in the unlikely event of a data loss you can just restore it. Likewise, restoring data should not be complicated.
+
+Fast: Backing up your data with restic should only be limited by your network or hard disk bandwidth so that you can backup your files every day. Nobody does backups if it takes too much time. Restoring backups should only transfer data that is needed for the files that are to be restored, so that this process is also fast.
+
+Verifiable: Much more important than backup is restore, so restic enables you to easily verify that all data can be restored.
+
+Secure: Restic uses cryptography to guarantee confidentiality and integrity of your data. The location where the backup data is stored is assumed to be an untrusted environment (e.g. a shared space where others like system administrators are able to access your backups). Restic is built to secure your data against such attackers, by encrypting it with AES-256 in counter mode and authenticating it using Poly1305-AES.
+
+Efficient: With the growth of data, additional snapshots should only take the storage of the actual increment. Even more, duplicate data should be de-duplicated before it is actually written to the storage backend to save precious backup space.
+
+Versatile storage: Users can provide many different places to store the backups. Local, SFTP, Restics REST-Server, Amazon S3, Minio, Openstack Swift, Backblaze B2, Microsoft Azure Blob Storage, Google Cloud Storage and more by the usage of rclone.
+
+Free: restic is free software and licensed under the BSD 2-Clause License and actively developed on GitHub.
+
+%prep
+%setup -q
+
+
+%build
+make %{?_smp_mflags}
+
+
+%install
+mkdir -p %{buildroot}%{_bindir}
+mkdir -p %{buildroot}%{_mandir}/man1
+mkdir -p %{buildroot}%{_datarootdir}/zsh/site-functions
+mkdir -p %{buildroot}%{_datarootdir}/bash-completion/completions
+install -p -m 644 doc/man/* %{buildroot}%{_mandir}/man1/
+install -p -m 644 doc/zsh-completion.zsh %{buildroot}%{_datarootdir}/zsh/site-functions/_restic
+install -p -m 644 doc/bash-completion.sh %{buildroot}%{_datarootdir}/bash-completion/completions/restic
+install -p -m 755 %{name} %{buildroot}%{_bindir}
+
+%files
+%doc LICENSE
+%doc README.rst
+%{_bindir}/%{name}
+%dir %{_datadir}/zsh/site-functions
+%{_datadir}/zsh/site-functions/_restic
+%dir %{_datadir}/bash-completion/
+%dir %{_datadir}/bash-completion/completions
+%{_datadir}/bash-completion/completions/restic
+%{_mandir}/man1/restic*.*
+
+
+%changelog
+* Sun Aug 12 2018 Luc De Louw <luc@delouw.ch> - 0.9.2git.20180812-4
+- %license does not work with RHEL6, using %doc instead
+* Sun Aug 12 2018 Luc De Louw <luc@delouw.ch> - 0.9.2git.20180812-3
+- Better description
+* Sun Aug 12 2018 Luc De Louw <luc@delouw.ch> - 0.9.2git.20180812-2
+- Initial RPM build
+* Sun Aug 12 2018 Luc De Louw <luc@delouw.ch> - 0.9.2git.20180812-1
+- Initial RPM build
+

doc.go

@@ -0,0 +1,11 @@
+// Package restic gives a (very brief) introduction to the structure of source code.
+//
+// Overview
+//
+// The packages are structured so that cmd/ contains the main package for the
+// restic binary, and internal/ contains almost all code in library form. We've
+// chosen to use the internal/ path so that the packages cannot be imported by
+// other programs. This was done on purpose, at the moment restic is a
+// command-line program and not a library. This may be revisited at a later
+// point in time.
+package restic

doc/.gitignore

@@ -0,0 +1,2 @@
+_build
+.doctrees

doc/020_installation.rst

@@ -24,37 +24,18 @@ These are up to date binaries, built in a reproducible and verifiable way, that
 you can download and run without having to do additional installation work.
 
 Please see the :ref:`official_binaries` section below for various downloads.
-
-Mac OS X
-========
-
-If you are using Mac OS X, you can install restic using the
-`homebrew <http://brew.sh/>`__ package manager:
-
-.. code-block:: console
-
-    $ brew install restic
+Official binaries can be updated in place by using the ``restic self-update``
+command.
 
 Arch Linux
 ==========
 
-On `Arch Linux <https://www.archlinux.org/>`__, there is a package called ``restic-git``
-which can be installed from AUR, e.g. with ``pacaur``:
+On `Arch Linux <https://www.archlinux.org/>`__, there is a package called ``restic``
+installed from the official community repos, e.g. with ``pacman -S``:
 
 .. code-block:: console
 
-    $ pacaur -S restic-git
-
-Nix & NixOS
-===========
-
-If you are using `Nix <https://nixos.org/nix/>`__ or `NixOS <https://nixos.org/>`__
-there is a package available named ``restic``.
-It can be installed uisng ``nix-env``:
-
-.. code-block:: console
-
-    $ nix-env --install restic 
+    $ pacman -S restic
 
 Debian
 ======
@@ -70,7 +51,68 @@ installed from the official repos, e.g. with ``apt-get``:
 .. warning:: Please be aware that, at the time of writing, Debian *stable*
    has ``restic`` version 0.3.3 which is very old. The *testing* and *unstable*
    branches have recent versions of ``restic``.
-   
+
+Fedora
+======
+
+restic can be installed using ``dnf``:
+
+.. code-block:: console
+
+    $ dnf install restic
+
+If you used restic from copr previously, remove the copr repo as follows to
+avoid any conflicts:
+
+.. code-block:: console
+
+   $ dnf copr remove copart/restic
+
+macOS
+=====
+
+If you are using macOS, you can install restic using the
+`homebrew <https://brew.sh/>`__ package manager:
+
+.. code-block:: console
+
+    $ brew install restic
+
+You may also install it using `MacPorts <https://www.macports.org/>`__:
+
+.. code-block:: console
+
+    $ sudo port install restic
+
+Nix & NixOS
+===========
+
+If you are using `Nix <https://nixos.org/nix/>`__ or `NixOS <https://nixos.org/>`__
+there is a package available named ``restic``.
+It can be installed using ``nix-env``:
+
+.. code-block:: console
+
+    $ nix-env --install restic
+
+OpenBSD
+=======
+
+On OpenBSD 6.3 and greater, you can install restic using ``pkg_add``:
+
+.. code-block:: console
+
+    # pkg_add restic
+
+FreeBSD
+=======
+
+On FreeBSD (11 and probably later versions), you can install restic using ``pkg install``:
+
+.. code-block:: console
+
+    # pkg install restic
+
 RHEL & CentOS
 =============
 
@@ -94,17 +136,6 @@ For CentOS7 use:
 
     $ yum-config-manager --add-repo https://copr.fedorainfracloud.org/coprs/copart/restic/repo/epel-7/copart-restic-epel-7.repo
 
-Fedora
-======
-
-restic can be installed via copr repository.
-
-.. code-block:: console
-
-    $ dnf install dnf-plugin-core
-    $ dnf copr enable copart/restic
-    $ dnf install restic
-
 Solus
 =====
 
@@ -113,15 +144,19 @@ restic can be installed from the official repo of Solus via the ``eopkg`` packag
 .. code-block:: console
 
     $ eopkg install restic
-    
-OpenBSD
+
+Windows
 =======
 
-On OpenBSD 6.3 and greater, you can install restic using ``pkg_add``:
+restic can be installed using `Scoop <https://scoop.sh/>`__:
 
 .. code-block:: console
 
-    # pkg_add restic
+    scoop install restic
+
+Using this installation method, ``restic.exe`` will automatically be available
+in the ``PATH``. It can be called from cmd.exe or PowerShell by typing ``restic``.
+
 
 .. _official_binaries:
 
@@ -138,6 +173,38 @@ are considered stable and releases are made regularly in a controlled manner.
 There's both pre-compiled binaries for different platforms as well as the source
 code available for download. Just download and run the one matching your system.
 
+The official binaries can be updated in place using the ``restic self-update``
+command (needs restic 0.9.3 or later):
+
+.. code-block:: console
+
+    $ restic version
+    restic 0.9.3 compiled with go1.11.2 on linux/amd64
+
+    $ restic self-update
+    find latest release of restic at GitHub
+    latest version is 0.9.4
+    download file SHA256SUMS
+    download SHA256SUMS
+    download file SHA256SUMS
+    download SHA256SUMS.asc
+    GPG signature verification succeeded
+    download restic_0.9.4_linux_amd64.bz2
+    downloaded restic_0.9.4_linux_amd64.bz2
+    saved 12115904 bytes in ./restic
+    successfully updated restic to version 0.9.4
+
+    $ restic version
+    restic 0.9.4 compiled with go1.12.1 on linux/amd64
+
+The ``self-update`` command uses the GPG signature on the files uploaded to
+GitHub to verify their authenticity. No external programs are necessary.
+
+.. note:: Please be aware that the user executing the ``restic self-update``
+   command must have the permission to replace the restic binary.
+   If you want to save the downloaded restic binary into a different file, pass
+   the file name via the option ``--output``.
+
 Unstable Builds
 ===============
 
@@ -150,9 +217,9 @@ master branch.
 Windows
 =======
 
-On Windows, put the `restic.exe` into `%SystemRoot%\System32` to use restic
+On Windows, put the `restic.exe` binary into `%SystemRoot%\\System32` to use restic
 in scripts without the need for absolute paths to the binary. This requires
-Admin rights.
+administrator rights.
 
 Docker Container
 ****************
@@ -187,6 +254,13 @@ In order to build restic from source, execute the following steps:
 
     $ cd restic
 
+    $ go run -mod=vendor build.go
+
+For Go versions < 1.11, the option ``-mod=vendor`` needs to be removed, like
+this:
+
+.. code-block:: console
+
     $ go run build.go
 
 You can easily cross-compile restic for all supported platforms, just
@@ -195,12 +269,14 @@ supply the target OS and platform via the command-line options like this
 
 .. code-block:: console
 
-    $ go run build.go --goos windows --goarch amd64
+    $ go run -mod=vendor build.go --goos windows --goarch amd64
 
-    $ go run build.go --goos freebsd --goarch 386
+    $ go run -mod=vendor build.go --goos freebsd --goarch 386
+
+    $ go run -mod=vendor build.go --goos linux --goarch arm --goarm 6
+
+Again, for Go < 1.11 ``-mod=vendor`` needs to be removed.
 
-    $ go run build.go --goos linux --goarch arm --goarm 6
-    
 The resulting binary is statically linked and does not require any
 libraries.
 

doc/030_preparing_a_new_repo.rst

@@ -14,13 +14,26 @@
 Preparing a new repository
 ##########################
 
-The place where your backups will be saved at is called a "repository".
+The place where your backups will be saved is called a "repository".
 This chapter explains how to create ("init") such a repository. The repository
 can be stored locally, or on some remote server or service. We'll first cover
-using a local repository, the remaining sections of this chapter cover all the
+using a local repository; the remaining sections of this chapter cover all the
 other options. You can skip to the next chapter once you've read the relevant
 section here.
 
+For automated backups, restic accepts the repository location in the
+environment variable ``RESTIC_REPOSITORY``. For the password, several options
+exist:
+
+ * Setting the environment variable ``RESTIC_PASSWORD``
+
+ * Specifying the path to a file with the password via the option
+   ``--password-file`` or the environment variable ``RESTIC_PASSWORD_FILE``
+
+ * Configuring a program to be called when the password is needed via the
+   option ``--password-command`` or the environment variable
+   ``RESTIC_PASSWORD_COMMAND``
+
 Local
 *****
 
@@ -41,11 +54,6 @@ command and enter the same password twice:
    Remembering your password is important! If you lose it, you won't be
    able to access data stored in the repository.
 
-For automated backups, restic accepts the repository location in the
-environment variable ``RESTIC_REPOSITORY``. The password can be read
-from a file (via the option ``--password-file`` or the environment variable
-``RESTIC_PASSWORD_FILE``) or the environment variable ``RESTIC_PASSWORD``.
-
 SFTP
 ****
 
@@ -114,7 +122,17 @@ Last, if you'd like to use an entirely different program to create the
 SFTP connection, you can specify the command to be run with the option
 ``-o sftp.command="foobar"``.
 
+.. note:: Please be aware that sftp servers close connections when no data is
+          received by the client. This can happen when restic is processing huge
+          amounts of unchanged data. To avoid this issue add the following lines 
+          to the client’s .ssh/config file:
 
+::
+
+    ServerAliveInterval 60
+    ServerAliveCountMax 240
+          
+          
 REST Server
 ***********
 
@@ -129,8 +147,8 @@ scheme like this:
     $ restic -r rest:http://host:8000/
 
 Depending on your REST server setup, you can use HTTPS protocol,
-password protection, or multiple repositories. Or any combination of
-those features, as you see fit. TCP/IP port is also configurable. Here
+password protection, multiple repositories or any combination of
+those features. The TCP/IP port is also configurable. Here
 are some more examples:
 
 .. code-block:: console
@@ -167,7 +185,7 @@ while creating the bucket.
     $ export AWS_SECRET_ACCESS_KEY=<MY_SECRET_ACCESS_KEY>
 
 You can then easily initialize a repository that uses your Amazon S3 as
-a backend, if the bucket does not exist yet it will be created in the
+a backend. If the bucket does not exist it will be created in the
 default location:
 
 .. code-block:: console
@@ -209,7 +227,7 @@ written in Go and compatible with AWS S3 API.
    on installation and getting started on Minio Client and Minio Server.
 
 You must first setup the following environment variables with the
-credentials of your running Minio Server.
+credentials of your Minio Server.
 
 .. code-block:: console
 
@@ -234,7 +252,7 @@ OpenStack Swift
 Restic can backup data to an OpenStack Swift container. Because Swift supports
 various authentication methods, credentials are passed through environment
 variables. In order to help integration with existing OpenStack installations,
-the naming convention of those variables follows official python swift client:
+the naming convention of those variables follows the official Python Swift client:
 
 .. code-block:: console
 
@@ -260,17 +278,29 @@ the naming convention of those variables follows official python swift client:
    $ export OS_PROJECT_NAME=<MY_PROJECT_NAME>
    $ export OS_PROJECT_DOMAIN_NAME=<MY_PROJECT_DOMAIN_NAME>
 
+   # For keystone v3 application credential authentication (application credential id)
+   $ export OS_AUTH_URL=<MY_AUTH_URL>
+   $ export OS_APPLICATION_CREDENTIAL_ID=<MY_APPLICATION_CREDENTIAL_ID>
+   $ export OS_APPLICATION_CREDENTIAL_SECRET=<MY_APPLICATION_CREDENTIAL_SECRET>
+
+   # For keystone v3 application credential authentication (application credential name)
+   $ export OS_AUTH_URL=<MY_AUTH_URL>
+   $ export OS_USERNAME=<MY_USERNAME>
+   $ export OS_USER_DOMAIN_NAME=<MY_DOMAIN_NAME>
+   $ export OS_APPLICATION_CREDENTIAL_NAME=<MY_APPLICATION_CREDENTIAL_NAME>
+   $ export OS_APPLICATION_CREDENTIAL_SECRET=<MY_APPLICATION_CREDENTIAL_SECRET>
+
    # For authentication based on tokens
    $ export OS_STORAGE_URL=<MY_STORAGE_URL>
    $ export OS_AUTH_TOKEN=<MY_AUTH_TOKEN>
 
 
-Restic should be compatible with `OpenStack RC file
+Restic should be compatible with an `OpenStack RC file
 <https://docs.openstack.org/user-guide/common/cli-set-environment-variables-using-openstack-rc.html>`__
 in most cases.
 
 Once environment variables are set up, a new repository can be created. The
-name of swift container and optional path can be specified. If
+name of the Swift container and optional path can be specified. If
 the container does not exist, it will be created automatically:
 
 .. code-block:: console
@@ -282,7 +312,7 @@ the container does not exist, it will be created automatically:
    Please note that knowledge of your password is required to access the repository.
    Losing your password means that your data is irrecoverably lost.
 
-The policy of new container created by restic can be changed using environment variable:
+The policy of the new container created by restic can be changed using environment variable:
 
 .. code-block:: console
 
@@ -293,16 +323,19 @@ Backblaze B2
 ************
 
 Restic can backup data to any Backblaze B2 bucket. You need to first setup the
-following environment variables with the credentials you obtained when signed
-into your B2 account:
+following environment variables with the credentials you can find in the
+dashboard on the "Buckets" page when signed into your B2 account:
 
 .. code-block:: console
 
-    $ export B2_ACCOUNT_ID=<MY_ACCOUNT_ID>
+    $ export B2_ACCOUNT_ID=<MY_APPLICATION_KEY_ID>
     $ export B2_ACCOUNT_KEY=<MY_SECRET_ACCOUNT_KEY>
 
-You can then easily initialize a repository stored at Backblaze B2. If the
-bucket does not exist yet, it will be created:
+.. note:: In case you want to use Backblaze Application Keys replace <MY_APPLICATION_KEY_ID> and <MY_SECRET_ACCOUNT_KEY> with <applicationKeyId> and <applicationKey> respectively.
+
+You can then initialize a repository stored at Backblaze B2. If the
+bucket does not exist yet and the credentials you passed to restic have the
+privilege to create buckets, it will be created automatically:
 
 .. code-block:: console
 
@@ -313,8 +346,10 @@ bucket does not exist yet, it will be created:
     Please note that knowledge of your password is required to access the repository.
     Losing your password means that your data is irrecoverably lost.
 
+Note that the bucket name must be unique across all of B2.
+
 The number of concurrent connections to the B2 service can be set with the ``-o
-b2.connections=10``. By default, at most five parallel connections are
+b2.connections=10`` switch. By default, at most five parallel connections are
 established.
 
 Microsoft Azure Blob Storage
@@ -341,15 +376,13 @@ root path like this:
     [...]
 
 The number of concurrent connections to the Azure Blob Storage service can be set with the
-``-o azure.connections=10``. By default, at most five parallel connections are
+``-o azure.connections=10`` switch. By default, at most five parallel connections are
 established.
 
 Google Cloud Storage
 ********************
 
-Restic supports Google Cloud Storage as a backend.
-
-Restic connects to Google Cloud Storage via a `service account`_.
+Restic supports Google Cloud Storage as a backend and connects via a `service account`_.
 
 For normal restic operation, the service account must have the
 ``storage.objects.{create,delete,get,list}`` permissions for the bucket. These
@@ -369,10 +402,9 @@ key file and the project ID as follows:
     $ export GOOGLE_PROJECT_ID=123123123123
     $ export GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gs-secret-restic-key.json
 
-Restic uses  Google's client library to generate [default authentication
-material](https://developers.google.com/identity/protocols/application-default-credentials),
+Restic uses  Google's client library to generate `default authentication material`_,
 which means if you're running in Google Container Engine or are otherwise
-located on an instance with default service accounts then these should work out
+located on an instance with default service accounts then these should work out of 
 the box.
 
 Once authenticated, you can use the ``gs:`` backend type to create a new
@@ -388,11 +420,12 @@ repository in the bucket ``foo`` at the root path:
     [...]
 
 The number of concurrent connections to the GCS service can be set with the
-``-o gs.connections=10``. By default, at most five parallel connections are
+``-o gs.connections=10`` switch. By default, at most five parallel connections are
 established.
 
 .. _service account: https://cloud.google.com/storage/docs/authentication#service_accounts
 .. _create a service account key: https://cloud.google.com/storage/docs/authentication#generating-a-private-key
+.. _default authentication material: https://developers.google.com/identity/protocols/application-default-credentials
 
 Other Services via rclone
 *************************
@@ -506,10 +539,10 @@ At the moment, restic only supports the default Windows console
 interaction. If you use emulation environments like
 `MSYS2 <https://msys2.github.io/>`__ or
 `Cygwin <https://www.cygwin.com/>`__, which use terminals like
-``Mintty`` or ``rxvt``, you may get a password error:
+``Mintty`` or ``rxvt``, you may get a password error.
 
 You can workaround this by using a special tool called ``winpty`` (look
-`here <https://sourceforge.net/p/msys2/wiki/Porting/>`__ and
+`here <https://github.com/msys2/msys2/wiki/Porting>`__ and
 `here <https://github.com/rprichard/winpty>`__ for detail information).
 On MSYS2, you can install ``winpty`` as follows:
 

doc/040_backup.rst

@@ -33,19 +33,26 @@ again:
     processed 1.720 GiB in 0:12
     Files:        5307 new,     0 changed,     0 unmodified
     Dirs:         1867 new,     0 changed,     0 unmodified
-    Added:      1.700 GiB
+    Added:      1.200 GiB
     snapshot 40dc1520 saved
 
 As you can see, restic created a backup of the directory and was pretty
 fast! The specific snapshot just created is identified by a sequence of
 hexadecimal characters, ``40dc1520`` in this case.
 
-If you don't pass the ``--verbose`` option, restic will print less data (but
-you'll still get a nice live status display).
+You can see that restic tells us it processed 1.720 GiB of data, this is the
+size of the files and directories in ``~/work`` on the local file system. It
+also tells us that only 1.200 GiB was added to the repository. This means that
+some of the data was duplicate and restic was able to efficiently reduce it.
+
+If you don't pass the ``--verbose`` option, restic will print less data. You'll
+still get a nice live status display. Be aware that the live status shows the
+processed files and not the transferred data. Transferred volume might be lower
+(due to de-duplication) or higher.
 
 If you run the command again, restic will create another snapshot of
-your data, but this time it's even faster. This is de-duplication at
-work!
+your data, but this time it's even faster and no new data was added to the
+repository (since all data is already there). This is de-duplication at work!
 
 .. code-block:: console
 
@@ -114,23 +121,28 @@ can compute which parts of the files need to be saved. When you backup
 the same directory again (maybe with new or changed files) restic will
 find the old snapshot in the repo and by default only reads those files
 that are new or have been modified since the last snapshot. This is
-decided based on the modify date of the file in the file system.
+decided based on the following attributes of the file in the file system:
+
+ * Type (file, symlink, or directory?)
+ * Modification time
+ * Size
+ * Inode number (internal number used to reference a file in a file system)
 
 Now is a good time to run ``restic check`` to verify that all data
 is properly stored in the repository. You should run this command regularly
 to make sure the internal structure of the repository is free of errors.
 
-Including and Excluding Files 
+Including and Excluding Files
 *****************************
 
 You can exclude folders and files by specifying exclude patterns, currently
 the exclude options are:
 
 -  ``--exclude`` Specified one or more times to exclude one or more items
+-  ``--iexclude`` Same as ``--exclude`` but ignores the case of paths
 -  ``--exclude-caches`` Specified once to exclude folders containing a special file
 -  ``--exclude-file`` Specified one or more times to exclude items listed in a given file
--  ``--exclude-if-present`` Specified one or more times to exclude a folders content
-   if it contains a given file (optionally having a given header)
+-  ``--exclude-if-present foo`` Specified one or more times to exclude a folder's content if it contains a file called ``foo``` (optionally having a given header, no wildcards for the file name supported)
 
  Let's say we have a file called ``excludes.txt`` with the following content:
 
@@ -159,7 +171,9 @@ Patterns use `filepath.Glob <https://golang.org/pkg/path/filepath/#Glob>`__ inte
 see `filepath.Match <https://golang.org/pkg/path/filepath/#Match>`__ for
 syntax. Patterns are tested against the full path of a file/dir to be saved,
 even if restic is passed a relative path to save. Environment-variables in
-exclude-files are expanded with `os.ExpandEnv <https://golang.org/pkg/os/#ExpandEnv>`__.
+exclude-files are expanded with `os.ExpandEnv <https://golang.org/pkg/os/#ExpandEnv>`__,
+so `/home/$USER/foo` will be expanded to `/home/bob/foo` for the user `bob`. To
+get a literal dollar sign, write `$$` to the file.
 
 Patterns need to match on complete path components. For example, the pattern ``foo``:
 
@@ -178,7 +192,7 @@ For this, the special wildcard ``**`` can be used to match arbitrary
 sub-directories: The pattern ``foo/**/bar`` matches:
 
  * ``/dir1/foo/dir2/bar/file``
- * ``/foo/bar/file`` 
+ * ``/foo/bar/file``
  * ``/tmp/foo/bar``
 
 By specifying the option ``--one-file-system`` you can instruct restic
@@ -190,9 +204,12 @@ backup ``/sys`` or ``/dev`` on a Linux system:
 
     $ restic -r /srv/restic-repo backup --one-file-system /
 
+.. note:: ``--one-file-system`` is currently unsupported on Windows, and will
+    cause the backup to immediately fail with an error.
+
 By using the ``--files-from`` option you can read the files you want to
-backup from a file. This is especially useful if a lot of files have to
-be backed up that are not in the same folder or are maybe pre-filtered
+backup from one or more files. This is especially useful if a lot of files have
+to be backed up that are not in the same folder or are maybe pre-filtered
 by other software.
 
 For example maybe you want to backup files which have a name that matches a
@@ -262,6 +279,10 @@ written, and the next backup needs to write new metadata again. If you really
 want to save the access time for files and directories, you can pass the
 ``--with-atime`` option to the ``backup`` command.
 
+In filesystems that do not support inode consistency, like FUSE-based ones and pCloud, it is
+possible to ignore inode on changed files comparison by passing ``--ignore-inode`` to
+``backup`` command.
+
 Reading data from stdin
 ***********************
 
@@ -272,6 +293,7 @@ this mode of operation, just supply the option ``--stdin`` to the
 
 .. code-block:: console
 
+    $ set -o pipefail
     $ mysqldump [...] | restic -r /srv/restic-repo backup --stdin
 
 This creates a new snapshot of the output of ``mysqldump``. You can then
@@ -285,6 +307,13 @@ specified with ``--stdin-filename``, e.g. like this:
 
     $ mysqldump [...] | restic -r /srv/restic-repo backup --stdin --stdin-filename production.sql
 
+The option ``pipefail`` is highly recommended so that a non-zero exit code from
+one of the programs in the pipe (e.g. ``mysqldump`` here) makes the whole chain
+return a non-zero exit code. Refer to the `Use the Unofficial Bash Strict Mode
+<http://redsymbol.net/articles/unofficial-bash-strict-mode/>`__ for more
+details on this.
+
+
 Tags for backup
 ***************
 
@@ -299,3 +328,69 @@ information. Just specify the tags for a snapshot one by one with ``--tag``:
 The tags can later be used to keep (or forget) snapshots with the ``forget``
 command. The command ``tag`` can be used to modify tags on an existing
 snapshot.
+
+Space requirements
+******************
+
+Restic currently assumes that your backup repository has sufficient space
+for the backup operation you are about to perform. This is a realistic
+assumption for many cloud providers, but may not be true when backing up
+to local disks.
+
+Should you run out of space during the middle of a backup, there will be
+some additional data in the repository, but the snapshot will never be
+created as it would only be written at the very (successful) end of
+the backup operation.  Previous snapshots will still be there and will still
+work.
+
+
+Environment Variables
+*********************
+
+In addition to command-line options, restic supports passing various options in
+environment variables. The following list of environment variables:
+
+.. code-block:: console
+
+    RESTIC_REPOSITORY                   Location of repository (replaces -r)
+    RESTIC_PASSWORD_FILE                Location of password file (replaces --password-file)
+    RESTIC_PASSWORD                     The actual password for the repository
+
+    AWS_ACCESS_KEY_ID                   Amazon S3 access key ID
+    AWS_SECRET_ACCESS_KEY               Amazon S3 secret access key
+
+    ST_AUTH                             Auth URL for keystone v1 authentication
+    ST_USER                             Username for keystone v1 authentication
+    ST_KEY                              Password for keystone v1 authentication
+
+    OS_AUTH_URL                         Auth URL for keystone authentication
+    OS_REGION_NAME                      Region name for keystone authentication
+    OS_USERNAME                         Username for keystone authentication
+    OS_PASSWORD                         Password for keystone authentication
+    OS_TENANT_ID                        Tenant ID for keystone v2 authentication
+    OS_TENANT_NAME                      Tenant name for keystone v2 authentication
+
+    OS_USER_DOMAIN_NAME                 User domain name for keystone authentication
+    OS_PROJECT_NAME                     Project name for keystone authentication
+    OS_PROJECT_DOMAIN_NAME              Project domain name for keystone authentication
+
+    OS_APPLICATION_CREDENTIAL_ID        Application Credential ID (keystone v3)
+    OS_APPLICATION_CREDENTIAL_NAME      Application Credential Name (keystone v3)
+    OS_APPLICATION_CREDENTIAL_SECRET    Application Credential Secret (keystone v3)
+
+    OS_STORAGE_URL                      Storage URL for token authentication
+    OS_AUTH_TOKEN                       Auth token for token authentication
+
+    B2_ACCOUNT_ID                       Account ID or applicationKeyId for Backblaze B2
+    B2_ACCOUNT_KEY                      Account Key or applicationKey for Backblaze B2
+
+    AZURE_ACCOUNT_NAME                  Account name for Azure
+    AZURE_ACCOUNT_KEY                   Account key for Azure
+
+    GOOGLE_PROJECT_ID                   Project ID for Google Cloud Storage
+    GOOGLE_APPLICATION_CREDENTIALS      Application Credentials for Google Cloud Storage (e.g. $HOME/.config/gs-secret-restic-key.json)
+
+    RCLONE_BWLIMIT                      rclone bandwidth limit
+
+
+

doc/045_working_with_repos.rst

@@ -56,6 +56,31 @@ Or filter by host:
 
 Combining filters is also possible.
 
+Furthermore you can group the output by the same filters (host, paths, tags):
+
+.. code-block:: console
+
+    $ restic -r /srv/restic-repo snapshots --group-by host
+
+    enter password for repository:
+    snapshots for (host [kasimir])
+    ID        Date                 Host    Tags   Directory
+    ----------------------------------------------------------------------
+    40dc1520  2015-05-08 21:38:30  kasimir        /home/user/work
+    79766175  2015-05-08 21:40:19  kasimir        /home/user/work
+    2 snapshots
+    snapshots for (host [luigi])
+    ID        Date                 Host    Tags   Directory
+    ----------------------------------------------------------------------
+    bdbd3439  2015-05-08 21:45:17  luigi          /home/art
+    9f0bc19e  2015-05-08 21:46:11  luigi          /srv
+    2 snapshots
+    snapshots for (host [kazik])
+    ID        Date                 Host    Tags   Directory
+    ----------------------------------------------------------------------
+    590c8fc8  2015-05-08 21:47:38  kazik          /srv
+    1 snapshots
+
 
 Checking a repo's integrity and consistency
 ===========================================
@@ -66,7 +91,7 @@ backup with the intention to make you restore malicious data:
 
 .. code-block:: console
 
-    $ sudo echo "boom" >> backup/index/d795ffa99a8ab8f8e42cec1f814df4e48b8f49129360fb57613df93739faee97
+    $ echo "boom" >> backup/index/d795ffa99a8ab8f8e42cec1f814df4e48b8f49129360fb57613df93739faee97
 
 In order to detect these things, it is a good idea to regularly use the
 ``check`` command to test whether everything is alright, your precious
@@ -101,7 +126,7 @@ data files:
 
 Use ``--read-data-subset=n/t`` parameter to check subset of repository data
 files. The parameter takes two values, ``n`` and ``t``. All repository data 
-files are logically devided in ``t`` roughly equal groups and only files that
+files are logically divided in ``t`` roughly equal groups and only files that
 belong to the group number ``n`` are checked. For example, the following 
 commands check all repository data files over 5 separate invocations: