Add VERSION file for 0.7.1

Set minio-go to v3.0.0

CHANGELOG.md

@@ -1,6 +1,116 @@
 This file describes changes relevant to all users that are made in each
 released version of restic from the perspective of the user.
 
+Important Changes in 0.7.1
+==========================
+
+ * The `migrate` command for chaning the `s3legacy` layout to the `default`
+   layout for s3 backends has been improved: It can now be restarted with
+   `restic migrate --force s3_layout` and automatically retries operations on
+   error.
+   https://github.com/restic/restic/issues/1073
+   https://github.com/restic/restic/pull/1075
+
+Small changes
+-------------
+
+ * The local and sftp backends now create the subdirs below `data/` on
+   open/init. This way, restic makes sure that they always exist. This is
+   connected to an issue for the sftp server:
+   https://github.com/restic/rest-server/pull/11#issuecomment-309879710
+   https://github.com/restic/restic/issues/1055
+   https://github.com/restic/restic/pull/1077
+   https://github.com/restic/restic/pull/1105
+
+ * When no S3 credentials are specified in the environment variables, restic
+   now tries to load credentials from an IAM instance profile when the s3
+   backend is used.
+   https://github.com/restic/restic/issues/1067
+   https://github.com/restic/restic/pull/1086
+
+ * On Darwin and FreeBSD, restic now prints stats when SIGINFO is received
+   (usually when ctrl+t is pressed).
+   https://github.com/restic/restic/pull/1082
+
+ * The dependencies have been updated.
+   https://github.com/restic/restic/pull/1108
+   https://github.com/restic/restic/pull/1124
+
+ * A bug was found (and corrected) in the index rebuilding after prune, which
+   led to indexes which include blobs that were not present in the repo any
+   more. There were already checks in place which detected this situation and
+   aborted with an error message. A new run of either `prune` or
+   `rebuild-index` corrected the index files. This is now fixed and a test has
+   been added to detect this.
+   https://github.com/restic/restic/pull/1115
+
+ * Errors for chmod() on Unix for filesystems which do not support it (e.g. smb
+   mounted via gvfs) are now ignored.
+   https://github.com/restic/restic/pull/1080
+   https://github.com/restic/restic/pull/1112
+
+ * The semantic for the `--tags` option to `forget` and `snapshots` was
+   clarified:
+   https://github.com/restic/restic/issues/1081
+   https://github.com/restic/restic/pull/1090
+
+Important Changes in 0.7.0
+==========================
+
+ * New "swift" backend: A new backend for the OpenStack Swift cloud storage
+   protocol has been added, https://wiki.openstack.org/wiki/Swift
+   https://github.com/restic/restic/pull/975
+   https://github.com/restic/restic/pull/648
+
+ * New "b2" backend: A new backend for Backblaze B2 cloud storage
+   service has been added, https://www.backblaze.com
+   https://github.com/restic/restic/issues/512
+   https://github.com/restic/restic/pull/978
+
+ * Improved performance for the `find` command: Restic recognizes paths it has
+   already checked for the files in question, so the number of backend requests
+   is reduced a lot.
+   https://github.com/restic/restic/issues/989
+   https://github.com/restic/restic/pull/993
+
+ * Improved performance for the fuse mount: Listing directories which contain
+   large files now is significantly faster.
+   https://github.com/restic/restic/pull/998
+
+ * The default layout for the s3 backend is now `default` (instead of
+   `s3legacy`). Also, there's a new `migrate` command to convert an existing
+   repo, it can be run like this: `restic migrate s3_layout`
+   https://github.com/restic/restic/issues/965
+   https://github.com/restic/restic/pull/1004
+
+ * The fuse mount now has two more directories: `tags` contains a subdir for
+   each tag, which in turn contains only the snapshots that have this tag. The
+   subdir `hosts` contains a subdir for each host that has a snapshot, and the
+   subdir contains the snapshots for that host.
+   https://github.com/restic/restic/issues/636
+   https://github.com/restic/restic/pull/1050
+
+Small changes
+-------------
+
+ * For the s3 backend we're back to using the high-level API the s3 client
+   library for uploading data, a few users reported dropped connections (which
+   the library will automatically retry now).
+   https://github.com/restic/restic/issues/1013
+   https://github.com/restic/restic/issues/1023
+   https://github.com/restic/restic/pull/1025
+
+ * The `prune` command has been improved and will now remove invalid pack
+   files, for example files that have not been uploaded completely because a
+   backup was interrupted.
+   https://github.com/restic/restic/issues/1029
+   https://github.com/restic/restic/pull/1036
+
+ * restic now tries to detect when an invalid/unknown backend is used and
+   returns an error message.
+   https://github.com/restic/restic/issues/1021
+   https://github.com/restic/restic/pull/1070
+
 Important Changes in 0.6.1
 ==========================
 

README.rst

@@ -72,10 +72,10 @@ Reproducible Builds
 -------------------
 
 The binaries released with each restic version starting at 0.6.1 are
-[reproducible](https://reproducible-builds.org/), which means that you can
+`reproducible <https://reproducible-builds.org/>`__, which means that you can
 easily reproduce a byte identical version from the source code for that
 release. Instructions on how to do that are contained in the
-[build repository](https://github.com/restic/build).
+`builder repository <https://github.com/restic/builder>`__.
 
 News
 ----
@@ -95,7 +95,7 @@ complete text in ``LICENSE``.
    :target: https://travis-ci.org/restic/restic
 .. |Build status| image:: https://ci.appveyor.com/api/projects/status/nuy4lfbgfbytw92q/branch/master?svg=true
    :target: https://ci.appveyor.com/project/fd0/restic/branch/master
-.. |Report Card| image:: http://goreportcard.com/badge/github.com/restic/restic
-   :target: http://goreportcard.com/report/github.com/restic/restic
+.. |Report Card| image:: https://goreportcard.com/badge/github.com/restic/restic
+   :target: https://goreportcard.com/report/github.com/restic/restic
 .. |Say Thanks| image:: https://img.shields.io/badge/Say%20Thanks-!-1EAEDB.svg
    :target: https://saythanks.io/to/restic

VERSION

@@ -1 +1 @@
-0.6.1
+0.7.1

appveyor.yml

@@ -17,8 +17,8 @@ init:
 
 install:
   - rmdir c:\go /s /q
-  - appveyor DownloadFile https://storage.googleapis.com/golang/go1.8.1.windows-amd64.msi
-  - msiexec /i go1.8.1.windows-amd64.msi /q
+  - appveyor DownloadFile https://storage.googleapis.com/golang/go1.8.3.windows-amd64.msi
+  - msiexec /i go1.8.3.windows-amd64.msi /q
   - go version
   - go env
   - appveyor DownloadFile http://sourceforge.netcologne.de/project/gnuwin32/tar/1.13-1/tar-1.13-1-bin.zip -FileName tar.zip

build.go

@@ -401,7 +401,10 @@ func main() {
 	if err != nil {
 		die("Getwd() returned %v\n", err)
 	}
-	output := filepath.Join(cwd, outputFilename)
+	output := outputFilename
+	if !filepath.IsAbs(output) {
+		output = filepath.Join(cwd, output)
+	}
 
 	version := getVersion()
 	constants := Constants{}

codecov.yml

@@ -1,2 +0,0 @@
-codecov:
-  disable_default_path_fixes: true

doc/development.rst

@@ -9,7 +9,7 @@ new feature. This way, duplicate work is prevented and we can discuss
 your ideas and design first.
 
 More information and a description of the development environment can be
-found in `CONTRIBUTING.md <CONTRIBUTING.md>`__.
+found in `CONTRIBUTING.md <https://github.com/restic/restic/blob/master/CONTRIBUTING.md>`__.
 A document describing the design of restic and the data structures stored on the
 back end is contained in `Design <https://restic.readthedocs.io/en/latest/design.html>`__.
 

doc/manual.rst

@@ -282,6 +282,96 @@ this command.
     Please note that knowledge of your password is required to access
     the repository. Losing your password means that your data is irrecoverably lost.
 
+OpenStack Swift
+~~~~~~~~~~~~~~~
+
+Restic can backup data to an OpenStack Swift container. Because Swift supports
+various authentication methods, credentials are passed through environment
+variables. In order to help integration with existing OpenStack installations,
+the naming convention of those variables follows official python swift client:
+
+.. code-block:: console
+
+   # For keystone v1 authentication
+   $ export ST_AUTH=<MY_AUTH_URL>
+   $ export ST_USER=<MY_USER_NAME>
+   $ export ST_KEY=<MY_USER_PASSWORD>
+
+   # For keystone v2 authentication (some variables are optional)
+   $ export OS_AUTH_URL=<MY_AUTH_URL>
+   $ export OS_REGION_NAME=<MY_REGION_NAME>
+   $ export OS_USERNAME=<MY_USERNAME>
+   $ export OS_PASSWORD=<MY_PASSWORD>
+   $ export OS_TENANT_ID=<MY_TENANT_ID>
+   $ export OS_TENANT_NAME=<MY_TENANT_NAME>
+
+   # For keystone v3 authentication (some variables are optional)
+   $ export OS_AUTH_URL=<MY_AUTH_URL>
+   $ export OS_REGION_NAME=<MY_REGION_NAME>
+   $ export OS_USERNAME=<MY_USERNAME>
+   $ export OS_PASSWORD=<MY_PASSWORD>
+   $ export OS_USER_DOMAIN_NAME=<MY_DOMAIN_NAME>
+   $ export OS_PROJECT_NAME=<MY_PROJECT_NAME>
+   $ export OS_PROJECT_DOMAIN_NAME=<MY_PROJECT_DOMAIN_NAME>
+
+   # For authentication based on tokens
+   $ export OS_STORAGE_URL=<MY_STORAGE_URL>
+   $ export OS_AUTH_TOKEN=<MY_AUTH_TOKEN>
+
+
+Restic should be compatible with [OpenStack RC
+file](https://docs.openstack.org/user-guide/common/cli-set-environment-variables-using-openstack-rc.html)
+in most cases.
+
+Once environment variables are set up, a new repository can be created. The
+name of swift container and optional path can be specified. If
+the container does not exist, it will be created automatically:
+
+.. code-block:: console
+
+   $ restic -r swift:container_name:/path init   # path is optional
+   enter password for new backend:
+   enter password again:
+   created restic backend eefee03bbd at swift:container_name:/path
+   Please note that knowledge of your password is required to access the repository.
+   Losing your password means that your data is irrecoverably lost.
+
+The policy of new container created by restic can be changed using environment variable:
+
+.. code-block:: console
+
+   $ export SWIFT_DEFAULT_CONTAINER_POLICY=<MY_CONTAINER_POLICY>
+
+
+Backblaze B2
+~~~~~~~~~~~~
+
+Restic can backup data to any Backblaze B2 bucket. You need to first setup the
+following environment variables with the credentials you obtained when signed
+into your B2 account:
+
+.. code-block:: console
+
+    $ export B2_ACCOUNT_ID=<MY_ACCOUNT_ID>
+    $ export B2_ACCOUNT_KEY=<MY_SECRET_ACCOUNT_KEY>
+
+You can then easily initialize a repository stored at Backblaze B2. If the
+bucket does not exist yet, it will be created:
+
+.. code-block:: console
+
+    $ restic -r b2:bucketname:path/to/repo init
+    enter password for new backend:
+    enter password again:
+    created restic backend eefee03bbd at b2:bucketname:path/to/repo
+    Please note that knowledge of your password is required to access the repository.
+    Losing your password means that your data is irrecoverably lost.
+
+The number of concurrent connections to the B2 service can be set with the `-o
+b2.connections=10`. By default, at most five parallel connections are
+established.
+
+
 Password prompt on Windows
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -436,18 +526,20 @@ specified with ``--stdin-filename``, e.g. like this:
 
     $ mysqldump [...] | restic -r /tmp/backup backup --stdin --stdin-filename production.sql
 
-Tags
-~~~~
+Tags for backup
+~~~~~~~~~~~~~~~
 
 Snapshots can have one or more tags, short strings which add identifying
-information. Just specify the tags for a snapshot with ``--tag``:
+information. Just specify the tags for a snapshot one by one with ``--tag``:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup backup --tag projectX ~/shared/work/web
+    $ restic -r /tmp/backup backup --tag projectX -tag foo --tag bar ~/shared/work/web
     [...]
 
-The tags can later be used to keep (or forget) snapshots.
+The tags can later be used to keep (or forget) snapshots with the ``forget``
+command. The command ``tag`` can be used to modify tags on an existing
+snapshot.
 
 List all snapshots
 ------------------
@@ -554,7 +646,7 @@ command does that:
 
 .. code-block:: console
 
-    $ restic -r /tmp/backup tag --set NL,CH 590c8fc8
+    $ restic -r /tmp/backup tag --set NL --set CH 590c8fc8
     Create exclusive lock for repository
     Modified tags on 1 snapshots
 
@@ -624,7 +716,7 @@ command to serve the repository with FUSE:
     $ mkdir /mnt/restic
     $ restic -r /tmp/backup mount /mnt/restic
     enter password for repository:
-    Now serving /tmp/backup at /tmp/restic
+    Now serving /tmp/backup at /mnt/restic
     Don't forget to umount after quitting!
 
 Mounting repositories via FUSE is not possible on Windows and OpenBSD.
@@ -782,7 +874,26 @@ The ``forget`` command accepts the following parameters:
 Additionally, you can restrict removing snapshots to those which have a
 particular hostname with the ``--hostname`` parameter, or tags with the
 ``--tag`` option. When multiple tags are specified, only the snapshots
-which have all the tags are considered.
+which have all the tags are considered. For example, the following command
+removes all but the latest snapshot of all snapshots that have the tag ``foo``:
+
+.. code-block:: console
+
+   $ restic forget --tag foo --keep-last 1
+
+This command removes all but the last snapshot of all snapshots that have
+either the ``foo`` or ``bar`` tag set:
+
+.. code-block:: console
+
+   $ restic forget --tag foo --tag bar --keep-last 1
+
+To only keep the last snapshot of all snapshots with both the tag ``foo`` and
+``bar`` set use:
+
+.. code-block:: console
+
+   $ restic forget --tag foo,tag bar --keep-last 1
 
 All the ``--keep-*`` options above only count
 hours/days/weeks/months/years which have a snapshot, so those without a
@@ -896,7 +1007,7 @@ Under the hood: Browse repository objects
 
 Internally, a repository stores data of several different types
 described in the `design
-documentation <https://github.com/restic/restic/blob/master/doc/Design.md>`__.
+documentation <https://github.com/restic/restic/blob/master/doc/Design.rst>`__.
 You can ``list`` objects such as blobs, packs, index, snapshots, keys or
 locks with the following command:
 

doc/tutorial_aws_s3.rst

@@ -64,7 +64,7 @@ changes:
 .. image:: images/aws_s3/05_bucket_create_review.png
    :alt: Review Bucket Creation
 
-The newly created ``restic-demo`` bucket will no appear on the list of S3
+The newly created ``restic-demo`` bucket will now appear on the list of S3
 buckets:
 
 .. image:: images/aws_s3/06_buckets_list_after.png

run_integration_tests.go

@@ -91,7 +91,7 @@ func (env *TravisEnvironment) Prepare() error {
 		"golang.org/x/tools/cmd/cover",
 		"github.com/pierrre/gotestcover",
 		"github.com/NebulousLabs/glyphcheck",
-		"github.com/restic/rest-server",
+		"github.com/restic/rest-server/cmd/rest-server",
 	}
 
 	for _, pkg := range pkgs {
@@ -164,6 +164,20 @@ func (env *TravisEnvironment) RunTests() error {
 		msg("S3 repository not available\n")
 	}
 
+	// if the test swift service is available, make sure that the test is not skipped
+	if os.Getenv("RESTIC_TEST_SWIFT") != "" {
+		ensureTests = append(ensureTests, "restic/backend/swift.TestBackendSwift")
+	} else {
+		msg("Swift service not available\n")
+	}
+
+	// if the test b2 repository is available, make sure that the test is not skipped
+	if os.Getenv("RESTIC_TEST_B2_REPOSITORY") != "" {
+		ensureTests = append(ensureTests, "restic/backend/b2.TestBackendB2")
+	} else {
+		msg("B2 repository not available\n")
+	}
+
 	env.env["RESTIC_TEST_DISALLOW_SKIP"] = strings.Join(ensureTests, ",")
 
 	if *runCrossCompile {

src/cmds/restic/cmd_backup.go

@@ -2,6 +2,7 @@ package main
 
 import (
 	"bufio"
+	"context"
 	"fmt"
 	"io"
 	"os"
@@ -67,12 +68,12 @@ func init() {
 	f := cmdBackup.Flags()
 	f.StringVar(&backupOptions.Parent, "parent", "", "use this parent snapshot (default: last snapshot in the repo that has the same target files/directories)")
 	f.BoolVarP(&backupOptions.Force, "force", "f", false, `force re-reading the target files/directories (overrides the "parent" flag)`)
-	f.StringSliceVarP(&backupOptions.Excludes, "exclude", "e", nil, "exclude a `pattern` (can be specified multiple times)")
-	f.StringSliceVar(&backupOptions.ExcludeFiles, "exclude-file", nil, "read exclude patterns from a `file` (can be specified multiple times)")
+	f.StringArrayVarP(&backupOptions.Excludes, "exclude", "e", nil, "exclude a `pattern` (can be specified multiple times)")
+	f.StringArrayVar(&backupOptions.ExcludeFiles, "exclude-file", nil, "read exclude patterns from a `file` (can be specified multiple times)")
 	f.BoolVarP(&backupOptions.ExcludeOtherFS, "one-file-system", "x", false, "exclude other file systems")
 	f.BoolVar(&backupOptions.Stdin, "stdin", false, "read backup from stdin")
 	f.StringVar(&backupOptions.StdinFilename, "stdin-filename", "stdin", "file name to use when reading from stdin")
-	f.StringSliceVar(&backupOptions.Tags, "tag", nil, "add a `tag` for the new snapshot (can be specified multiple times)")
+	f.StringArrayVar(&backupOptions.Tags, "tag", nil, "add a `tag` for the new snapshot (can be specified multiple times)")
 	f.StringVar(&backupOptions.Hostname, "hostname", hostname, "set the `hostname` for the snapshot manually")
 	f.StringVar(&backupOptions.FilesFrom, "files-from", "", "read the files to backup from file (can be combined with file args)")
 }
@@ -263,7 +264,7 @@ func readBackupFromStdin(opts BackupOptions, gopts GlobalOptions, args []string)
 		return err
 	}
 
-	err = repo.LoadIndex()
+	err = repo.LoadIndex(context.TODO())
 	if err != nil {
 		return err
 	}
@@ -274,7 +275,7 @@ func readBackupFromStdin(opts BackupOptions, gopts GlobalOptions, args []string)
 		Hostname:   opts.Hostname,
 	}
 
-	_, id, err := r.Archive(opts.StdinFilename, os.Stdin, newArchiveStdinProgress(gopts))
+	_, id, err := r.Archive(context.TODO(), opts.StdinFilename, os.Stdin, newArchiveStdinProgress(gopts))
 	if err != nil {
 		return err
 	}
@@ -372,7 +373,7 @@ func runBackup(opts BackupOptions, gopts GlobalOptions, args []string) error {
 		return err
 	}
 
-	err = repo.LoadIndex()
+	err = repo.LoadIndex(context.TODO())
 	if err != nil {
 		return err
 	}
@@ -391,7 +392,7 @@ func runBackup(opts BackupOptions, gopts GlobalOptions, args []string) error {
 
 	// Find last snapshot to set it as parent, if not already set
 	if !opts.Force && parentSnapshotID == nil {
-		id, err := restic.FindLatestSnapshot(repo, target, opts.Tags, opts.Hostname)
+		id, err := restic.FindLatestSnapshot(context.TODO(), repo, target, []restic.TagList{opts.Tags}, opts.Hostname)
 		if err == nil {
 			parentSnapshotID = &id
 		} else if err != restic.ErrNoSnapshotFound {
@@ -489,7 +490,7 @@ func runBackup(opts BackupOptions, gopts GlobalOptions, args []string) error {
 		Warnf("%s\rwarning for %s: %v\n", ClearLine(), dir, err)
 	}
 
-	_, id, err := arch.Snapshot(newArchiveProgress(gopts, stat), target, opts.Tags, opts.Hostname, parentSnapshotID)
+	_, id, err := arch.Snapshot(context.TODO(), newArchiveProgress(gopts, stat), target, opts.Tags, opts.Hostname, parentSnapshotID)
 	if err != nil {
 		return err
 	}

src/cmds/restic/cmd_cat.go

@@ -1,6 +1,7 @@
 package main
 
 import (
+	"context"
 	"encoding/json"
 	"fmt"
 	"os"
@@ -73,7 +74,7 @@ func runCat(gopts GlobalOptions, args []string) error {
 		fmt.Println(string(buf))
 		return nil
 	case "index":
-		buf, err := repo.LoadAndDecrypt(restic.IndexFile, id)
+		buf, err := repo.LoadAndDecrypt(context.TODO(), restic.IndexFile, id)
 		if err != nil {
 			return err
 		}
@@ -83,7 +84,7 @@ func runCat(gopts GlobalOptions, args []string) error {
 
 	case "snapshot":
 		sn := &restic.Snapshot{}
-		err = repo.LoadJSONUnpacked(restic.SnapshotFile, id, sn)
+		err = repo.LoadJSONUnpacked(context.TODO(), restic.SnapshotFile, id, sn)
 		if err != nil {
 			return err
 		}
@@ -98,7 +99,7 @@ func runCat(gopts GlobalOptions, args []string) error {
 		return nil
 	case "key":
 		h := restic.Handle{Type: restic.KeyFile, Name: id.String()}
-		buf, err := backend.LoadAll(repo.Backend(), h)
+		buf, err := backend.LoadAll(context.TODO(), repo.Backend(), h)
 		if err != nil {
 			return err
 		}
@@ -125,7 +126,7 @@ func runCat(gopts GlobalOptions, args []string) error {
 		fmt.Println(string(buf))
 		return nil
 	case "lock":
-		lock, err := restic.LoadLock(repo, id)
+		lock, err := restic.LoadLock(context.TODO(), repo, id)
 		if err != nil {
 			return err
 		}
@@ -141,7 +142,7 @@ func runCat(gopts GlobalOptions, args []string) error {
 	}
 
 	// load index, handle all the other types
-	err = repo.LoadIndex()
+	err = repo.LoadIndex(context.TODO())
 	if err != nil {
 		return err
 	}
@@ -149,7 +150,7 @@ func runCat(gopts GlobalOptions, args []string) error {
 	switch tpe {
 	case "pack":
 		h := restic.Handle{Type: restic.DataFile, Name: id.String()}
-		buf, err := backend.LoadAll(repo.Backend(), h)
+		buf, err := backend.LoadAll(context.TODO(), repo.Backend(), h)
 		if err != nil {
 			return err
 		}
@@ -171,7 +172,7 @@ func runCat(gopts GlobalOptions, args []string) error {
 			blob := list[0]
 
 			buf := make([]byte, blob.Length)
-			n, err := repo.LoadBlob(t, id, buf)
+			n, err := repo.LoadBlob(context.TODO(), t, id, buf)
 			if err != nil {
 				return err
 			}

src/cmds/restic/cmd_check.go

@@ -1,6 +1,7 @@
 package main
 
 import (
+	"context"
 	"fmt"
 	"os"
 	"time"
@@ -92,7 +93,7 @@ func runCheck(opts CheckOptions, gopts GlobalOptions, args []string) error {
 	chkr := checker.New(repo)
 
 	Verbosef("Load indexes\n")
-	hints, errs := chkr.LoadIndex()
+	hints, errs := chkr.LoadIndex(context.TODO())
 
 	dupFound := false
 	for _, hint := range hints {
@@ -113,14 +114,11 @@ func runCheck(opts CheckOptions, gopts GlobalOptions, args []string) error {
 		return errors.Fatal("LoadIndex returned errors")
 	}
 
-	done := make(chan struct{})
-	defer close(done)
-
 	errorsFound := false
 	errChan := make(chan error)
 
 	Verbosef("Check all packs\n")
-	go chkr.Packs(errChan, done)
+	go chkr.Packs(context.TODO(), errChan)
 
 	for err := range errChan {
 		errorsFound = true
@@ -129,7 +127,7 @@ func runCheck(opts CheckOptions, gopts GlobalOptions, args []string) error {
 
 	Verbosef("Check snapshots, trees and blobs\n")
 	errChan = make(chan error)
-	go chkr.Structure(errChan, done)
+	go chkr.Structure(context.TODO(), errChan)
 
 	for err := range errChan {
 		errorsFound = true
@@ -156,7 +154,7 @@ func runCheck(opts CheckOptions, gopts GlobalOptions, args []string) error {
 		p := newReadProgress(gopts, restic.Stat{Blobs: chkr.CountPacks()})
 		errChan := make(chan error)
 
-		go chkr.ReadData(p, errChan, done)
+		go chkr.ReadData(context.TODO(), p, errChan)
 
 		for err := range errChan {
 			errorsFound = true

src/cmds/restic/cmd_dump.go

@@ -1,8 +1,9 @@
-// +build debug
+// xbuild debug
 
 package main
 
 import (
+	"context"
 	"encoding/json"
 	"fmt"
 	"io"
@@ -44,11 +45,8 @@ func prettyPrintJSON(wr io.Writer, item interface{}) error {
 }
 
 func debugPrintSnapshots(repo *repository.Repository, wr io.Writer) error {
-	done := make(chan struct{})
-	defer close(done)
-
-	for id := range repo.List(restic.SnapshotFile, done) {
-		snapshot, err := restic.LoadSnapshot(repo, id)
+	for id := range repo.List(context.TODO(), restic.SnapshotFile) {
+		snapshot, err := restic.LoadSnapshot(context.TODO(), repo, id)
 		if err != nil {
 			fmt.Fprintf(os.Stderr, "LoadSnapshot(%v): %v", id.Str(), err)
 			continue
@@ -83,15 +81,12 @@ type Blob struct {
 }
 
 func printPacks(repo *repository.Repository, wr io.Writer) error {
-	done := make(chan struct{})
-	defer close(done)
-
-	f := func(job worker.Job, done <-chan struct{}) (interface{}, error) {
+	f := func(ctx context.Context, job worker.Job) (interface{}, error) {
 		name := job.Data.(string)
 
 		h := restic.Handle{Type: restic.DataFile, Name: name}
 
-		blobInfo, err := repo.Backend().Stat(h)
+		blobInfo, err := repo.Backend().Stat(ctx, h)
 		if err != nil {
 			return nil, err
 		}
@@ -106,10 +101,10 @@ func printPacks(repo *repository.Repository, wr io.Writer) error {
 
 	jobCh := make(chan worker.Job)
 	resCh := make(chan worker.Job)
-	wp := worker.New(dumpPackWorkers, f, jobCh, resCh)
+	wp := worker.New(context.TODO(), dumpPackWorkers, f, jobCh, resCh)
 
 	go func() {
-		for name := range repo.Backend().List(restic.DataFile, done) {
+		for name := range repo.Backend().List(context.TODO(), restic.DataFile) {
 			jobCh <- worker.Job{Data: name}
 		}
 		close(jobCh)
@@ -146,13 +141,10 @@ func printPacks(repo *repository.Repository, wr io.Writer) error {
 }
 
 func dumpIndexes(repo restic.Repository) error {
-	done := make(chan struct{})
-	defer close(done)
-
-	for id := range repo.List(restic.IndexFile, done) {
+	for id := range repo.List(context.TODO(), restic.IndexFile) {
 		fmt.Printf("index_id: %v\n", id)
 
-		idx, err := repository.LoadIndex(repo, id)
+		idx, err := repository.LoadIndex(context.TODO(), repo, id)
 		if err != nil {
 			return err
 		}
@@ -184,7 +176,7 @@ func runDump(gopts GlobalOptions, args []string) error {
 		}
 	}
 
-	err = repo.LoadIndex()
+	err = repo.LoadIndex(context.TODO())
 	if err != nil {
 		return err
 	}

src/cmds/restic/cmd_find.go

@@ -12,7 +12,6 @@ import (
 	"restic"
 	"restic/debug"
 	"restic/errors"
-	"restic/repository"
 )
 
 var cmdFind = &cobra.Command{
@@ -35,7 +34,7 @@ type FindOptions struct {
 	ListLong        bool
 	Host            string
 	Paths           []string
-	Tags            []string
+	Tags            restic.TagLists
 }
 
 var findOptions FindOptions
@@ -46,13 +45,13 @@ func init() {
 	f := cmdFind.Flags()
 	f.StringVarP(&findOptions.Oldest, "oldest", "O", "", "oldest modification date/time")
 	f.StringVarP(&findOptions.Newest, "newest", "N", "", "newest modification date/time")
-	f.StringSliceVarP(&findOptions.Snapshots, "snapshot", "s", nil, "snapshot `id` to search in (can be given multiple times)")
+	f.StringArrayVarP(&findOptions.Snapshots, "snapshot", "s", nil, "snapshot `id` to search in (can be given multiple times)")
 	f.BoolVarP(&findOptions.CaseInsensitive, "ignore-case", "i", false, "ignore case for pattern")
 	f.BoolVarP(&findOptions.ListLong, "long", "l", false, "use a long listing format showing size and mode")
 
 	f.StringVarP(&findOptions.Host, "host", "H", "", "only consider snapshots for this `host`, when no snapshot ID is given")
-	f.StringSliceVar(&findOptions.Tags, "tag", nil, "only consider snapshots which include this `tag`, when no snapshot-ID is given")
-	f.StringSliceVar(&findOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path`, when no snapshot-ID is given")
+	f.Var(&findOptions.Tags, "tag", "only consider snapshots which include this `taglist`, when no snapshot-ID is given")
+	f.StringArrayVar(&findOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path`, when no snapshot-ID is given")
 }
 
 type findPattern struct {
@@ -172,59 +171,76 @@ func (s *statefulOutput) Finish() {
 	}
 }
 
-func findInTree(repo *repository.Repository, pat *findPattern, id restic.ID, prefix string, state *statefulOutput) error {
-	debug.Log("checking tree %v\n", id)
+// Finder bundles information needed to find a file or directory.
+type Finder struct {
+	repo     restic.Repository
+	pat      findPattern
+	out      statefulOutput
+	notfound restic.IDSet
+}
 
-	tree, err := repo.LoadTree(id)
+func (f *Finder) findInTree(treeID restic.ID, prefix string) error {
+	if f.notfound.Has(treeID) {
+		debug.Log("%v skipping tree %v, has already been checked", prefix, treeID.Str())
+		return nil
+	}
+
+	debug.Log("%v checking tree %v\n", prefix, treeID.Str())
+
+	tree, err := f.repo.LoadTree(context.TODO(), treeID)
 	if err != nil {
 		return err
 	}
 
+	var found bool
 	for _, node := range tree.Nodes {
 		debug.Log("  testing entry %q\n", node.Name)
 
 		name := node.Name
-		if pat.ignoreCase {
+		if f.pat.ignoreCase {
 			name = strings.ToLower(name)
 		}
 
-		m, err := filepath.Match(pat.pattern, name)
+		m, err := filepath.Match(f.pat.pattern, name)
 		if err != nil {
 			return err
 		}
 
 		if m {
-			debug.Log("    pattern matches\n")
-			if !pat.oldest.IsZero() && node.ModTime.Before(pat.oldest) {
-				debug.Log("    ModTime is older than %s\n", pat.oldest)
+			if !f.pat.oldest.IsZero() && node.ModTime.Before(f.pat.oldest) {
+				debug.Log("    ModTime is older than %s\n", f.pat.oldest)
 				continue
 			}
 
-			if !pat.newest.IsZero() && node.ModTime.After(pat.newest) {
-				debug.Log("    ModTime is newer than %s\n", pat.newest)
+			if !f.pat.newest.IsZero() && node.ModTime.After(f.pat.newest) {
+				debug.Log("    ModTime is newer than %s\n", f.pat.newest)
 				continue
 			}
 
-			state.Print(prefix, node)
-		} else {
-			debug.Log("    pattern does not match\n")
+			debug.Log("    found match\n")
+			found = true
+			f.out.Print(prefix, node)
 		}
 
 		if node.Type == "dir" {
-			if err := findInTree(repo, pat, *node.Subtree, filepath.Join(prefix, node.Name), state); err != nil {
+			if err := f.findInTree(*node.Subtree, filepath.Join(prefix, node.Name)); err != nil {
 				return err
 			}
 		}
 	}
 
+	if !found {
+		f.notfound.Insert(treeID)
+	}
+
 	return nil
 }
 
-func findInSnapshot(repo *repository.Repository, sn *restic.Snapshot, pat findPattern, state *statefulOutput) error {
-	debug.Log("searching in snapshot %s\n  for entries within [%s %s]", sn.ID(), pat.oldest, pat.newest)
+func (f *Finder) findInSnapshot(sn *restic.Snapshot) error {
+	debug.Log("searching in snapshot %s\n  for entries within [%s %s]", sn.ID(), f.pat.oldest, f.pat.newest)
 
-	state.newsn = sn
-	if err := findInTree(repo, &pat, *sn.Tree, string(filepath.Separator), state); err != nil {
+	f.out.newsn = sn
+	if err := f.findInTree(*sn.Tree, string(filepath.Separator)); err != nil {
 		return err
 	}
 	return nil
@@ -267,19 +283,25 @@ func runFind(opts FindOptions, gopts GlobalOptions, args []string) error {
 		}
 	}
 
-	if err = repo.LoadIndex(); err != nil {
+	if err = repo.LoadIndex(context.TODO()); err != nil {
 		return err
 	}
 
 	ctx, cancel := context.WithCancel(gopts.ctx)
 	defer cancel()
-	state := statefulOutput{ListLong: opts.ListLong, JSON: globalOptions.JSON}
+
+	f := &Finder{
+		repo:     repo,
+		pat:      pat,
+		out:      statefulOutput{ListLong: opts.ListLong, JSON: globalOptions.JSON},
+		notfound: restic.NewIDSet(),
+	}
 	for sn := range FindFilteredSnapshots(ctx, repo, opts.Host, opts.Tags, opts.Paths, opts.Snapshots) {
-		if err = findInSnapshot(repo, sn, pat, &state); err != nil {
+		if err = f.findInSnapshot(sn); err != nil {
 			return err
 		}
 	}
-	state.Finish()
+	f.out.Finish()
 
 	return nil
 }

src/cmds/restic/cmd_forget.go

@@ -31,10 +31,10 @@ type ForgetOptions struct {
 	Weekly   int
 	Monthly  int
 	Yearly   int
-	KeepTags []string
+	KeepTags restic.TagLists
 
 	Host  string
-	Tags  []string
+	Tags  restic.TagLists
 	Paths []string
 
 	GroupByTags bool
@@ -55,14 +55,14 @@ func init() {
 	f.IntVarP(&forgetOptions.Monthly, "keep-monthly", "m", 0, "keep the last `n` monthly snapshots")
 	f.IntVarP(&forgetOptions.Yearly, "keep-yearly", "y", 0, "keep the last `n` yearly snapshots")
 
-	f.StringSliceVar(&forgetOptions.KeepTags, "keep-tag", []string{}, "keep snapshots with this `tag` (can be specified multiple times)")
+	f.Var(&forgetOptions.KeepTags, "keep-tag", "keep snapshots with this `taglist` (can be specified multiple times)")
 	f.BoolVarP(&forgetOptions.GroupByTags, "group-by-tags", "G", false, "Group by host,paths,tags instead of just host,paths")
 	// Sadly the commonly used shortcut `H` is already used.
 	f.StringVar(&forgetOptions.Host, "host", "", "only consider snapshots with the given `host`")
 	// Deprecated since 2017-03-07.
 	f.StringVar(&forgetOptions.Host, "hostname", "", "only consider snapshots with the given `hostname` (deprecated)")
-	f.StringSliceVar(&forgetOptions.Tags, "tag", nil, "only consider snapshots which include this `tag` (can be specified multiple times)")
-	f.StringSliceVar(&forgetOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path` (can be specified multiple times)")
+	f.Var(&forgetOptions.Tags, "tag", "only consider snapshots which include this `taglist` in the format `tag[,tag,...]` (can be specified multiple times)")
+	f.StringArrayVar(&forgetOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path` (can be specified multiple times)")
 
 	f.BoolVarP(&forgetOptions.DryRun, "dry-run", "n", false, "do not delete anything, just print what would be done")
 	f.BoolVar(&forgetOptions.Prune, "prune", false, "automatically run the 'prune' command if snapshots have been removed")
@@ -97,7 +97,7 @@ func runForget(opts ForgetOptions, gopts GlobalOptions, args []string) error {
 			// When explicit snapshots args are given, remove them immediately.
 			if !opts.DryRun {
 				h := restic.Handle{Type: restic.SnapshotFile, Name: sn.ID().String()}
-				if err = repo.Backend().Remove(h); err != nil {
+				if err = repo.Backend().Remove(context.TODO(), h); err != nil {
 					return err
 				}
 				Verbosef("removed snapshot %v\n", sn.ID().Str())
@@ -167,7 +167,7 @@ func runForget(opts ForgetOptions, gopts GlobalOptions, args []string) error {
 		if !opts.DryRun {
 			for _, sn := range remove {
 				h := restic.Handle{Type: restic.SnapshotFile, Name: sn.ID().String()}
-				err = repo.Backend().Remove(h)
+				err = repo.Backend().Remove(context.TODO(), h)
 				if err != nil {
 					return err
 				}

src/cmds/restic/cmd_init.go

@@ -1,6 +1,7 @@
 package main
 
 import (
+	"context"
 	"restic/errors"
 	"restic/repository"
 
@@ -43,7 +44,7 @@ func runInit(gopts GlobalOptions, args []string) error {
 
 	s := repository.New(be)
 
-	err = s.Init(gopts.password)
+	err = s.Init(context.TODO(), gopts.password)
 	if err != nil {
 		return errors.Fatalf("create key in backend at %s failed: %v\n", gopts.Repo, err)
 	}

src/cmds/restic/cmd_key.go

@@ -30,8 +30,8 @@ func listKeys(ctx context.Context, s *repository.Repository) error {
 	tab.Header = fmt.Sprintf(" %-10s  %-10s  %-10s  %s", "ID", "User", "Host", "Created")
 	tab.RowFormat = "%s%-10s  %-10s  %-10s  %s"
 
-	for id := range s.List(restic.KeyFile, ctx.Done()) {
-		k, err := repository.LoadKey(s, id.String())
+	for id := range s.List(ctx, restic.KeyFile) {
+		k, err := repository.LoadKey(ctx, s, id.String())
 		if err != nil {
 			Warnf("LoadKey() failed: %v\n", err)
 			continue
@@ -69,7 +69,7 @@ func addKey(gopts GlobalOptions, repo *repository.Repository) error {
 		return err
 	}
 
-	id, err := repository.AddKey(repo, pw, repo.Key())
+	id, err := repository.AddKey(context.TODO(), repo, pw, repo.Key())
 	if err != nil {
 		return errors.Fatalf("creating new key failed: %v\n", err)
 	}
@@ -85,7 +85,7 @@ func deleteKey(repo *repository.Repository, name string) error {
 	}
 
 	h := restic.Handle{Type: restic.KeyFile, Name: name}
-	err := repo.Backend().Remove(h)
+	err := repo.Backend().Remove(context.TODO(), h)
 	if err != nil {
 		return err
 	}
@@ -100,13 +100,13 @@ func changePassword(gopts GlobalOptions, repo *repository.Repository) error {
 		return err
 	}
 
-	id, err := repository.AddKey(repo, pw, repo.Key())
+	id, err := repository.AddKey(context.TODO(), repo, pw, repo.Key())
 	if err != nil {
 		return errors.Fatalf("creating new key failed: %v\n", err)
 	}
 
 	h := restic.Handle{Type: restic.KeyFile, Name: repo.KeyName()}
-	err = repo.Backend().Remove(h)
+	err = repo.Backend().Remove(context.TODO(), h)
 	if err != nil {
 		return err
 	}

src/cmds/restic/cmd_list.go

@@ -1,6 +1,7 @@
 package main
 
 import (
+	"context"
 	"fmt"
 	"restic"
 	"restic/errors"
@@ -55,7 +56,7 @@ func runList(opts GlobalOptions, args []string) error {
 	case "locks":
 		t = restic.LockFile
 	case "blobs":
-		idx, err := index.Load(repo, nil)
+		idx, err := index.Load(context.TODO(), repo, nil)
 		if err != nil {
 			return err
 		}
@@ -71,7 +72,7 @@ func runList(opts GlobalOptions, args []string) error {
 		return errors.Fatal("invalid type")
 	}
 
-	for id := range repo.List(t, nil) {
+	for id := range repo.List(context.TODO(), t) {
 		Printf("%s\n", id)
 	}
 

src/cmds/restic/cmd_ls.go

@@ -28,7 +28,7 @@ The special snapshot-ID "latest" can be used to list files and directories of th
 type LsOptions struct {
 	ListLong bool
 	Host     string
-	Tags     []string
+	Tags     restic.TagLists
 	Paths    []string
 }
 
@@ -41,18 +41,18 @@ func init() {
 	flags.BoolVarP(&lsOptions.ListLong, "long", "l", false, "use a long listing format showing size and mode")
 
 	flags.StringVarP(&lsOptions.Host, "host", "H", "", "only consider snapshots for this `host`, when no snapshot ID is given")
-	flags.StringSliceVar(&lsOptions.Tags, "tag", nil, "only consider snapshots which include this `tag`, when no snapshot ID is given")
-	flags.StringSliceVar(&lsOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path`, when no snapshot ID is given")
+	flags.Var(&lsOptions.Tags, "tag", "only consider snapshots which include this `taglist`, when no snapshot ID is given")
+	flags.StringArrayVar(&lsOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path`, when no snapshot ID is given")
 }
 
 func printTree(repo *repository.Repository, id *restic.ID, prefix string) error {
-	tree, err := repo.LoadTree(*id)
+	tree, err := repo.LoadTree(context.TODO(), *id)
 	if err != nil {
 		return err
 	}
 
 	for _, entry := range tree.Nodes {
-		Printf(formatNode(prefix, entry, lsOptions.ListLong) + "\n")
+		Printf("%s\n", formatNode(prefix, entry, lsOptions.ListLong))
 
 		if entry.Type == "dir" && entry.Subtree != nil {
 			if err = printTree(repo, entry.Subtree, filepath.Join(prefix, entry.Name)); err != nil {
@@ -74,7 +74,7 @@ func runLs(opts LsOptions, gopts GlobalOptions, args []string) error {
 		return err
 	}
 
-	if err = repo.LoadIndex(); err != nil {
+	if err = repo.LoadIndex(context.TODO()); err != nil {
 		return err
 	}
 

src/cmds/restic/cmd_migrate.go

@@ -0,0 +1,107 @@
+package main
+
+import (
+	"restic"
+	"restic/migrations"
+
+	"github.com/spf13/cobra"
+)
+
+var cmdMigrate = &cobra.Command{
+	Use:   "migrate [name]",
+	Short: "apply migrations",
+	Long: `
+The "migrate" command applies migrations to a repository. When no migration
+name is explicitely given, a list of migrations that can be applied is printed.
+`,
+	RunE: func(cmd *cobra.Command, args []string) error {
+		return runMigrate(migrateOptions, globalOptions, args)
+	},
+}
+
+// MigrateOptions bundles all options for the 'check' command.
+type MigrateOptions struct {
+	Force bool
+}
+
+var migrateOptions MigrateOptions
+
+func init() {
+	cmdRoot.AddCommand(cmdMigrate)
+	f := cmdMigrate.Flags()
+	f.BoolVarP(&migrateOptions.Force, "force", "f", false, `apply a migration a second time`)
+}
+
+func checkMigrations(opts MigrateOptions, gopts GlobalOptions, repo restic.Repository) error {
+	ctx := gopts.ctx
+	Printf("available migrations:\n")
+	for _, m := range migrations.All {
+		ok, err := m.Check(ctx, repo)
+		if err != nil {
+			return err
+		}
+
+		if ok {
+			Printf("  %v: %v\n", m.Name(), m.Desc())
+		}
+	}
+
+	return nil
+}
+
+func applyMigrations(opts MigrateOptions, gopts GlobalOptions, repo restic.Repository, args []string) error {
+	ctx := gopts.ctx
+
+	var firsterr error
+	for _, name := range args {
+		for _, m := range migrations.All {
+			if m.Name() == name {
+				ok, err := m.Check(ctx, repo)
+				if err != nil {
+					return err
+				}
+
+				if !ok {
+					if !opts.Force {
+						Warnf("migration %v cannot be applied: check failed\nIf you want to apply this migration anyway, re-run with option --force\n", m.Name())
+						continue
+					}
+
+					Warnf("check for migration %v failed, continuing anyway\n", m.Name())
+				}
+
+				Printf("applying migration %v...\n", m.Name())
+				if err = m.Apply(ctx, repo); err != nil {
+					Warnf("migration %v failed: %v\n", m.Name(), err)
+					if firsterr == nil {
+						firsterr = err
+					}
+					continue
+				}
+
+				Printf("migration %v: success\n", m.Name())
+			}
+		}
+	}
+
+	return firsterr
+}
+
+func runMigrate(opts MigrateOptions, gopts GlobalOptions, args []string) error {
+	repo, err := OpenRepository(gopts)
+	if err != nil {
+		return err
+	}
+
+	lock, err := lockRepoExclusive(repo)
+	defer unlockRepo(lock)
+	if err != nil {
+		return err
+	}
+
+	if len(args) == 0 {
+		return checkMigrations(opts, gopts, repo)
+	}
+
+	return applyMigrations(opts, gopts, repo, args)
+}

src/cmds/restic/cmd_mount.go

@@ -4,7 +4,9 @@
 package main
 
 import (
+	"context"
 	"os"
+	"restic"
 
 	"github.com/spf13/cobra"
 
@@ -36,7 +38,7 @@ type MountOptions struct {
 	AllowRoot  bool
 	AllowOther bool
 	Host       string
-	Tags       []string
+	Tags       restic.TagLists
 	Paths      []string
 }
 
@@ -51,8 +53,8 @@ func init() {
 	mountFlags.BoolVar(&mountOptions.AllowOther, "allow-other", false, "allow other users to access the data in the mounted directory")
 
 	mountFlags.StringVarP(&mountOptions.Host, "host", "H", "", `only consider snapshots for this host`)
-	mountFlags.StringSliceVar(&mountOptions.Tags, "tag", nil, "only consider snapshots which include this `tag`")
-	mountFlags.StringSliceVar(&mountOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path`")
+	mountFlags.Var(&mountOptions.Tags, "tag", "only consider snapshots which include this `taglist`")
+	mountFlags.StringArrayVar(&mountOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path`")
 }
 
 func mount(opts MountOptions, gopts GlobalOptions, mountpoint string) error {
@@ -64,7 +66,7 @@ func mount(opts MountOptions, gopts GlobalOptions, mountpoint string) error {
 		return err
 	}
 
-	err = repo.LoadIndex()
+	err = repo.LoadIndex(context.TODO())
 	if err != nil {
 		return err
 	}
@@ -95,14 +97,26 @@ func mount(opts MountOptions, gopts GlobalOptions, mountpoint string) error {
 		return err
 	}
 
+	systemFuse.Debug = func(msg interface{}) {
+		debug.Log("fuse: %v", msg)
+	}
+
+	cfg := fuse.Config{
+		OwnerIsRoot: opts.OwnerRoot,
+		Host:        opts.Host,
+		Tags:        opts.Tags,
+		Paths:       opts.Paths,
+	}
+	root, err := fuse.NewRoot(context.TODO(), repo, cfg)
+	if err != nil {
+		return err
+	}
+
 	Printf("Now serving the repository at %s\n", mountpoint)
 	Printf("Don't forget to umount after quitting!\n")
 
-	root := fs.Tree{}
-	root.Add("snapshots", fuse.NewSnapshotsDir(repo, opts.OwnerRoot, opts.Paths, opts.Tags, opts.Host))
-
 	debug.Log("serving mount at %v", mountpoint)
-	err = fs.Serve(c, &root)
+	err = fs.Serve(c, root)
 	if err != nil {
 		return err
 	}

src/cmds/restic/cmd_prune.go

@@ -1,7 +1,6 @@
 package main
 
 import (
-	"context"
 	"fmt"
 	"restic"
 	"restic/debug"
@@ -29,6 +28,18 @@ func init() {
 	cmdRoot.AddCommand(cmdPrune)
 }
 
+func shortenStatus(maxLength int, s string) string {
+	if len(s) <= maxLength {
+		return s
+	}
+
+	if maxLength < 3 {
+		return s[:maxLength]
+	}
+
+	return s[:maxLength-3] + "..."
+}
+
 // newProgressMax returns a progress that counts blobs.
 func newProgressMax(show bool, max uint64, description string) *restic.Progress {
 	if !show {
@@ -44,10 +55,7 @@ func newProgressMax(show bool, max uint64, description string) *restic.Progress
 			s.Blobs, max, description)
 
 		if w := stdoutTerminalWidth(); w > 0 {
-			if len(status) > w {
-				max := w - len(status) - 4
-				status = status[:max] + "... "
-			}
+			status = shortenStatus(w, status)
 		}
 
 		PrintProgress("%s", status)
@@ -76,14 +84,13 @@ func runPrune(gopts GlobalOptions) error {
 }
 
 func pruneRepository(gopts GlobalOptions, repo restic.Repository) error {
-	err := repo.LoadIndex()
+	ctx := gopts.ctx
+
+	err := repo.LoadIndex(ctx)
 	if err != nil {
 		return err
 	}
 
-	ctx, cancel := context.WithCancel(gopts.ctx)
-	defer cancel()
-
 	var stats struct {
 		blobs     int
 		packs     int
@@ -92,18 +99,22 @@ func pruneRepository(gopts GlobalOptions, repo restic.Repository) error {
 	}
 
 	Verbosef("counting files in repo\n")
-	for range repo.List(restic.DataFile, ctx.Done()) {
+	for range repo.List(ctx, restic.DataFile) {
 		stats.packs++
 	}
 
 	Verbosef("building new index for repo\n")
 
 	bar := newProgressMax(!gopts.Quiet, uint64(stats.packs), "packs")
-	idx, err := index.New(repo, bar)
+	idx, invalidFiles, err := index.New(ctx, repo, restic.NewIDSet(), bar)
 	if err != nil {
 		return err
 	}
 
+	for _, id := range invalidFiles {
+		Warnf("incomplete pack file (will be removed): %v\n", id)
+	}
+
 	blobs := 0
 	for _, pack := range idx.Packs {
 		stats.bytes += pack.Size
@@ -135,7 +146,7 @@ func pruneRepository(gopts GlobalOptions, repo restic.Repository) error {
 	Verbosef("load all snapshots\n")
 
 	// find referenced blobs
-	snapshots, err := restic.LoadAllSnapshots(repo)
+	snapshots, err := restic.LoadAllSnapshots(ctx, repo)
 	if err != nil {
 		return err
 	}
@@ -152,12 +163,16 @@ func pruneRepository(gopts GlobalOptions, repo restic.Repository) error {
 	for _, sn := range snapshots {
 		debug.Log("process snapshot %v", sn.ID().Str())
 
-		err = restic.FindUsedBlobs(repo, *sn.Tree, usedBlobs, seenBlobs)
+		err = restic.FindUsedBlobs(ctx, repo, *sn.Tree, usedBlobs, seenBlobs)
 		if err != nil {
+			if repo.Backend().IsNotExist(err) {
+				return errors.Fatal("unable to load a tree from the repo: " + err.Error())
+			}
+
 			return err
 		}
 
-		debug.Log("found %v blobs for snapshot %v", sn.ID().Str())
+		debug.Log("processed snapshot %v", sn.ID().Str())
 		bar.Report(restic.Stat{Blobs: 1})
 	}
 	bar.Done()
@@ -185,6 +200,12 @@ func pruneRepository(gopts GlobalOptions, repo restic.Repository) error {
 
 	// find packs that are unneeded
 	removePacks := restic.NewIDSet()
+
+	Verbosef("will remove %d invalid files\n", len(invalidFiles))
+	for _, id := range invalidFiles {
+		removePacks.Insert(id)
+	}
+
 	for packID, p := range idx.Packs {
 
 		hasActiveBlob := false
@@ -214,22 +235,29 @@ func pruneRepository(gopts GlobalOptions, repo restic.Repository) error {
 	Verbosef("will delete %d packs and rewrite %d packs, this frees %s\n",
 		len(removePacks), len(rewritePacks), formatBytes(uint64(removeBytes)))
 
+	var obsoletePacks restic.IDSet
 	if len(rewritePacks) != 0 {
 		bar = newProgressMax(!gopts.Quiet, uint64(len(rewritePacks)), "packs rewritten")
 		bar.Start()
-		err = repository.Repack(repo, rewritePacks, usedBlobs, bar)
+		obsoletePacks, err = repository.Repack(ctx, repo, rewritePacks, usedBlobs, bar)
 		if err != nil {
 			return err
 		}
 		bar.Done()
 	}
 
+	removePacks.Merge(obsoletePacks)
+
+	if err = rebuildIndex(ctx, repo, removePacks); err != nil {
+		return err
+	}
+
 	if len(removePacks) != 0 {
 		bar = newProgressMax(!gopts.Quiet, uint64(len(removePacks)), "packs deleted")
 		bar.Start()
 		for packID := range removePacks {
 			h := restic.Handle{Type: restic.DataFile, Name: packID.String()}
-			err = repo.Backend().Remove(h)
+			err = repo.Backend().Remove(ctx, h)
 			if err != nil {
 				Warnf("unable to remove file %v from the repository\n", packID.Str())
 			}
@@ -238,10 +266,6 @@ func pruneRepository(gopts GlobalOptions, repo restic.Repository) error {
 		bar.Done()
 	}
 
-	if err = rebuildIndex(ctx, repo); err != nil {
-		return err
-	}
-
 	Verbosef("done\n")
 	return nil
 }

src/cmds/restic/cmd_rebuild_index.go

@@ -38,19 +38,19 @@ func runRebuildIndex(gopts GlobalOptions) error {
 
 	ctx, cancel := context.WithCancel(gopts.ctx)
 	defer cancel()
-	return rebuildIndex(ctx, repo)
+	return rebuildIndex(ctx, repo, restic.NewIDSet())
 }
 
-func rebuildIndex(ctx context.Context, repo restic.Repository) error {
+func rebuildIndex(ctx context.Context, repo restic.Repository, ignorePacks restic.IDSet) error {
 	Verbosef("counting files in repo\n")
 
 	var packs uint64
-	for range repo.List(restic.DataFile, ctx.Done()) {
+	for range repo.List(ctx, restic.DataFile) {
 		packs++
 	}
 
-	bar := newProgressMax(!globalOptions.Quiet, packs, "packs")
-	idx, err := index.New(repo, bar)
+	bar := newProgressMax(!globalOptions.Quiet, packs-uint64(len(ignorePacks)), "packs")
+	idx, _, err := index.New(ctx, repo, ignorePacks, bar)
 	if err != nil {
 		return err
 	}
@@ -58,11 +58,11 @@ func rebuildIndex(ctx context.Context, repo restic.Repository) error {
 	Verbosef("finding old index files\n")
 
 	var supersedes restic.IDs
-	for id := range repo.List(restic.IndexFile, ctx.Done()) {
+	for id := range repo.List(ctx, restic.IndexFile) {
 		supersedes = append(supersedes, id)
 	}
 
-	id, err := idx.Save(repo, supersedes)
+	id, err := idx.Save(ctx, repo, supersedes)
 	if err != nil {
 		return err
 	}
@@ -72,7 +72,7 @@ func rebuildIndex(ctx context.Context, repo restic.Repository) error {
 	Verbosef("remove %d old index files\n", len(supersedes))
 
 	for _, id := range supersedes {
-		if err := repo.Backend().Remove(restic.Handle{
+		if err := repo.Backend().Remove(ctx, restic.Handle{
 			Type: restic.IndexFile,
 			Name: id.String(),
 		}); err != nil {

src/cmds/restic/cmd_restore.go

@@ -31,7 +31,7 @@ type RestoreOptions struct {
 	Target  string
 	Host    string
 	Paths   []string
-	Tags    []string
+	Tags    restic.TagLists
 }
 
 var restoreOptions RestoreOptions
@@ -40,16 +40,18 @@ func init() {
 	cmdRoot.AddCommand(cmdRestore)
 
 	flags := cmdRestore.Flags()
-	flags.StringSliceVarP(&restoreOptions.Exclude, "exclude", "e", nil, "exclude a `pattern` (can be specified multiple times)")
-	flags.StringSliceVarP(&restoreOptions.Include, "include", "i", nil, "include a `pattern`, exclude everything else (can be specified multiple times)")
+	flags.StringArrayVarP(&restoreOptions.Exclude, "exclude", "e", nil, "exclude a `pattern` (can be specified multiple times)")
+	flags.StringArrayVarP(&restoreOptions.Include, "include", "i", nil, "include a `pattern`, exclude everything else (can be specified multiple times)")
 	flags.StringVarP(&restoreOptions.Target, "target", "t", "", "directory to extract data to")
 
 	flags.StringVarP(&restoreOptions.Host, "host", "H", "", `only consider snapshots for this host when the snapshot ID is "latest"`)
-	flags.StringSliceVar(&restoreOptions.Tags, "tag", nil, "only consider snapshots which include this `tag` for snapshot ID \"latest\"")
-	flags.StringSliceVar(&restoreOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path` for snapshot ID \"latest\"")
+	flags.Var(&restoreOptions.Tags, "tag", "only consider snapshots which include this `taglist` for snapshot ID \"latest\"")
+	flags.StringArrayVar(&restoreOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path` for snapshot ID \"latest\"")
 }
 
 func runRestore(opts RestoreOptions, gopts GlobalOptions, args []string) error {
+	ctx := gopts.ctx
+
 	if len(args) != 1 {
 		return errors.Fatal("no snapshot ID specified")
 	}
@@ -79,7 +81,7 @@ func runRestore(opts RestoreOptions, gopts GlobalOptions, args []string) error {
 		}
 	}
 
-	err = repo.LoadIndex()
+	err = repo.LoadIndex(ctx)
 	if err != nil {
 		return err
 	}
@@ -87,7 +89,7 @@ func runRestore(opts RestoreOptions, gopts GlobalOptions, args []string) error {
 	var id restic.ID
 
 	if snapshotIDString == "latest" {
-		id, err = restic.FindLatestSnapshot(repo, opts.Paths, opts.Tags, opts.Host)
+		id, err = restic.FindLatestSnapshot(ctx, repo, opts.Paths, opts.Tags, opts.Host)
 		if err != nil {
 			Exitf(1, "latest snapshot for criteria not found: %v Paths:%v Host:%v", err, opts.Paths, opts.Host)
 		}
@@ -136,7 +138,7 @@ func runRestore(opts RestoreOptions, gopts GlobalOptions, args []string) error {
 
 	Verbosef("restoring %s to %s\n", res.Snapshot(), opts.Target)
 
-	err = res.RestoreTo(opts.Target)
+	err = res.RestoreTo(ctx, opts.Target)
 	if totalErrors > 0 {
 		Printf("There were %d errors\n", totalErrors)
 	}

src/cmds/restic/cmd_snapshots.go

@@ -26,7 +26,7 @@ The "snapshots" command lists all snapshots stored in the repository.
 // SnapshotOptions bundles all options for the snapshots command.
 type SnapshotOptions struct {
 	Host  string
-	Tags  []string
+	Tags  restic.TagLists
 	Paths []string
 }
 
@@ -37,8 +37,8 @@ func init() {
 
 	f := cmdSnapshots.Flags()
 	f.StringVarP(&snapshotOptions.Host, "host", "H", "", "only consider snapshots for this `host`")
-	f.StringSliceVar(&snapshotOptions.Tags, "tag", nil, "only consider snapshots which include this `tag` (can be specified multiple times)")
-	f.StringSliceVar(&snapshotOptions.Paths, "path", nil, "only consider snapshots for this `path` (can be specified multiple times)")
+	f.Var(&snapshotOptions.Tags, "tag", "only consider snapshots which include this `taglist` (can be specified multiple times)")
+	f.StringArrayVar(&snapshotOptions.Paths, "path", nil, "only consider snapshots for this `path` (can be specified multiple times)")
 }
 
 func runSnapshots(opts SnapshotOptions, gopts GlobalOptions, args []string) error {

src/cmds/restic/cmd_tag.go

@@ -31,7 +31,7 @@ When no snapshot-ID is given, all snapshots matching the host, tag and path filt
 type TagOptions struct {
 	Host       string
 	Paths      []string
-	Tags       []string
+	Tags       restic.TagLists
 	SetTags    []string
 	AddTags    []string
 	RemoveTags []string
@@ -48,8 +48,8 @@ func init() {
 	tagFlags.StringSliceVar(&tagOptions.RemoveTags, "remove", nil, "`tag` which will be removed from the existing tags (can be given multiple times)")
 
 	tagFlags.StringVarP(&tagOptions.Host, "host", "H", "", "only consider snapshots for this `host`, when no snapshot ID is given")
-	tagFlags.StringSliceVar(&tagOptions.Tags, "tag", nil, "only consider snapshots which include this `tag`, when no snapshot-ID is given")
-	tagFlags.StringSliceVar(&tagOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path`, when no snapshot-ID is given")
+	tagFlags.Var(&tagOptions.Tags, "tag", "only consider snapshots which include this `taglist`, when no snapshot-ID is given")
+	tagFlags.StringArrayVar(&tagOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path`, when no snapshot-ID is given")
 }
 
 func changeTags(repo *repository.Repository, sn *restic.Snapshot, setTags, addTags, removeTags []string) (bool, error) {
@@ -76,7 +76,7 @@ func changeTags(repo *repository.Repository, sn *restic.Snapshot, setTags, addTa
 		}
 
 		// Save the new snapshot.
-		id, err := repo.SaveJSONUnpacked(restic.SnapshotFile, sn)
+		id, err := repo.SaveJSONUnpacked(context.TODO(), restic.SnapshotFile, sn)
 		if err != nil {
 			return false, err
 		}
@@ -89,7 +89,7 @@ func changeTags(repo *repository.Repository, sn *restic.Snapshot, setTags, addTa
 
 		// Remove the old snapshot.
 		h := restic.Handle{Type: restic.SnapshotFile, Name: sn.ID().String()}
-		if err = repo.Backend().Remove(h); err != nil {
+		if err = repo.Backend().Remove(context.TODO(), h); err != nil {
 			return false, err
 		}
 

src/cmds/restic/cmd_unlock.go

@@ -1,6 +1,7 @@
 package main
 
 import (
+	"context"
 	"restic"
 
 	"github.com/spf13/cobra"
@@ -41,7 +42,7 @@ func runUnlock(opts UnlockOptions, gopts GlobalOptions) error {
 		fn = restic.RemoveAllLocks
 	}
 
-	err = fn(repo)
+	err = fn(context.TODO(), repo)
 	if err != nil {
 		return err
 	}

src/cmds/restic/find.go

@@ -8,7 +8,7 @@ import (
 )
 
 // FindFilteredSnapshots yields Snapshots, either given explicitly by `snapshotIDs` or filtered from the list of all snapshots.
-func FindFilteredSnapshots(ctx context.Context, repo *repository.Repository, host string, tags []string, paths []string, snapshotIDs []string) <-chan *restic.Snapshot {
+func FindFilteredSnapshots(ctx context.Context, repo *repository.Repository, host string, tags []restic.TagList, paths []string, snapshotIDs []string) <-chan *restic.Snapshot {
 	out := make(chan *restic.Snapshot)
 	go func() {
 		defer close(out)
@@ -22,7 +22,7 @@ func FindFilteredSnapshots(ctx context.Context, repo *repository.Repository, hos
 			// Process all snapshot IDs given as arguments.
 			for _, s := range snapshotIDs {
 				if s == "latest" {
-					id, err = restic.FindLatestSnapshot(repo, paths, tags, host)
+					id, err = restic.FindLatestSnapshot(ctx, repo, paths, tags, host)
 					if err != nil {
 						Warnf("Ignoring %q, no snapshot matched given filter (Paths:%v Tags:%v Host:%v)\n", s, paths, tags, host)
 						usedFilter = true
@@ -44,7 +44,7 @@ func FindFilteredSnapshots(ctx context.Context, repo *repository.Repository, hos
 			}
 
 			for _, id := range ids.Uniq() {
-				sn, err := restic.LoadSnapshot(repo, id)
+				sn, err := restic.LoadSnapshot(ctx, repo, id)
 				if err != nil {
 					Warnf("Ignoring %q, could not load snapshot: %v\n", id, err)
 					continue
@@ -58,15 +58,7 @@ func FindFilteredSnapshots(ctx context.Context, repo *repository.Repository, hos
 			return
 		}
 
-		for id := range repo.List(restic.SnapshotFile, ctx.Done()) {
-			sn, err := restic.LoadSnapshot(repo, id)
-			if err != nil {
-				Warnf("Ignoring %q, could not load snapshot: %v\n", id, err)
-				continue
-			}
-			if (host != "" && host != sn.Hostname) || !sn.HasTags(tags) || !sn.HasPaths(paths) {
-				continue
-			}
+		for _, sn := range restic.FindFilteredSnapshots(ctx, repo, host, tags, paths) {
 			select {
 			case <-ctx.Done():
 				return

src/cmds/restic/flags_test.go

@@ -8,10 +8,6 @@ import (
 // TestFlags checks for double defined flags, the commands will panic on
 // ParseFlags() when a shorthand flag is defined twice.
 func TestFlags(t *testing.T) {
-	type FlagParser interface {
-		ParseFlags([]string) error
-	}
-
 	for _, cmd := range cmdRoot.Commands() {
 		t.Run(cmd.Name(), func(t *testing.T) {
 			cmd.Flags().SetOutput(ioutil.Discard)

src/cmds/restic/global.go

@@ -11,11 +11,13 @@ import (
 	"strings"
 	"syscall"
 
+	"restic/backend/b2"
 	"restic/backend/local"
 	"restic/backend/location"
 	"restic/backend/rest"
 	"restic/backend/s3"
 	"restic/backend/sftp"
+	"restic/backend/swift"
 	"restic/debug"
 	"restic/options"
 	"restic/repository"
@@ -65,7 +67,7 @@ func init() {
 
 	f := cmdRoot.PersistentFlags()
 	f.StringVarP(&globalOptions.Repo, "repo", "r", os.Getenv("RESTIC_REPOSITORY"), "repository to backup to or restore from (default: $RESTIC_REPOSITORY)")
-	f.StringVarP(&globalOptions.PasswordFile, "password-file", "p", "", "read the repository password from a file")
+	f.StringVarP(&globalOptions.PasswordFile, "password-file", "p", os.Getenv("RESTIC_PASSWORD_FILE"), "read the repository password from a file (default: $RESTIC_PASSWORD_FILE)")
 	f.BoolVarP(&globalOptions.Quiet, "quiet", "q", false, "do not output comprehensive progress report")
 	f.BoolVar(&globalOptions.NoLock, "no-lock", false, "do not lock the repo, this allows some operations on read-only repos")
 	f.BoolVarP(&globalOptions.JSON, "json", "", false, "set output mode to JSON for commands that support it")
@@ -308,7 +310,7 @@ func OpenRepository(opts GlobalOptions) (*repository.Repository, error) {
 		}
 	}
 
-	err = s.SearchKey(opts.password, maxKeys)
+	err = s.SearchKey(context.TODO(), opts.password, maxKeys)
 	if err != nil {
 		return nil, errors.Fatalf("unable to open repo: %v", err)
 	}
@@ -356,6 +358,37 @@ func parseConfig(loc location.Location, opts options.Options) (interface{}, erro
 		debug.Log("opening s3 repository at %#v", cfg)
 		return cfg, nil
 
+	case "swift":
+		cfg := loc.Config.(swift.Config)
+
+		if err := swift.ApplyEnvironment("", &cfg); err != nil {
+			return nil, err
+		}
+
+		if err := opts.Apply(loc.Scheme, &cfg); err != nil {
+			return nil, err
+		}
+
+		debug.Log("opening swift repository at %#v", cfg)
+		return cfg, nil
+
+	case "b2":
+		cfg := loc.Config.(b2.Config)
+
+		if cfg.AccountID == "" {
+			cfg.AccountID = os.Getenv("B2_ACCOUNT_ID")
+		}
+
+		if cfg.Key == "" {
+			cfg.Key = os.Getenv("B2_ACCOUNT_KEY")
+		}
+
+		if err := opts.Apply(loc.Scheme, &cfg); err != nil {
+			return nil, err
+		}
+
+		debug.Log("opening b2 repository at %#v", cfg)
+		return cfg, nil
 	case "rest":
 		cfg := loc.Config.(rest.Config)
 		if err := opts.Apply(loc.Scheme, &cfg); err != nil {
@@ -391,6 +424,10 @@ func open(s string, opts options.Options) (restic.Backend, error) {
 		be, err = sftp.Open(cfg.(sftp.Config))
 	case "s3":
 		be, err = s3.Open(cfg.(s3.Config))
+	case "swift":
+		be, err = swift.Open(cfg.(swift.Config))
+	case "b2":
+		be, err = b2.Open(cfg.(b2.Config))
 	case "rest":
 		be, err = rest.Open(cfg.(rest.Config))
 
@@ -403,7 +440,7 @@ func open(s string, opts options.Options) (restic.Backend, error) {
 	}
 
 	// check if config is there
-	fi, err := be.Stat(restic.Handle{Type: restic.ConfigFile})
+	fi, err := be.Stat(context.TODO(), restic.Handle{Type: restic.ConfigFile})
 	if err != nil {
 		return nil, errors.Fatalf("unable to open config file: %v\nIs there a repository at the following location?\n%v", err, s)
 	}
@@ -434,7 +471,11 @@ func create(s string, opts options.Options) (restic.Backend, error) {
 	case "sftp":
 		return sftp.Create(cfg.(sftp.Config))
 	case "s3":
-		return s3.Open(cfg.(s3.Config))
+		return s3.Create(cfg.(s3.Config))
+	case "swift":
+		return swift.Open(cfg.(swift.Config))
+	case "b2":
+		return b2.Create(cfg.(b2.Config))
 	case "rest":
 		return rest.Create(cfg.(rest.Config))
 	}

src/cmds/restic/integration_fuse_test.go

@@ -1,10 +1,10 @@
-// +build ignore
 // +build !openbsd
 // +build !windows
 
 package main
 
 import (
+	"context"
 	"fmt"
 	"io/ioutil"
 	"os"
@@ -55,17 +55,15 @@ func waitForMount(t testing.TB, dir string) {
 	t.Errorf("subdir %q of dir %s never appeared", mountTestSubdir, dir)
 }
 
-func mount(t testing.TB, global GlobalOptions, dir string) {
-	cmd := &CmdMount{global: &global}
-	OK(t, cmd.Mount(dir))
+func testRunMount(t testing.TB, gopts GlobalOptions, dir string) {
+	opts := MountOptions{}
+	OK(t, runMount(opts, gopts, []string{dir}))
 }
 
-func umount(t testing.TB, global GlobalOptions, dir string) {
-	cmd := &CmdMount{global: &global}
-
+func testRunUmount(t testing.TB, gopts GlobalOptions, dir string) {
 	var err error
 	for i := 0; i < mountWait; i++ {
-		if err = cmd.Umount(dir); err == nil {
+		if err = umount(dir); err == nil {
 			t.Logf("directory %v umounted", dir)
 			return
 		}
@@ -87,9 +85,10 @@ func listSnapshots(t testing.TB, dir string) []string {
 
 func checkSnapshots(t testing.TB, global GlobalOptions, repo *repository.Repository, mountpoint, repodir string, snapshotIDs restic.IDs) {
 	t.Logf("checking for %d snapshots: %v", len(snapshotIDs), snapshotIDs)
-	go mount(t, global, mountpoint)
+
+	go testRunMount(t, global, mountpoint)
 	waitForMount(t, mountpoint)
-	defer umount(t, global, mountpoint)
+	defer testRunUmount(t, global, mountpoint)
 
 	if !snapshotsDirExists(t, mountpoint) {
 		t.Fatal(`virtual directory "snapshots" doesn't exist`)
@@ -110,7 +109,7 @@ func checkSnapshots(t testing.TB, global GlobalOptions, repo *repository.Reposit
 	}
 
 	for _, id := range snapshotIDs {
-		snapshot, err := restic.LoadSnapshot(repo, id)
+		snapshot, err := restic.LoadSnapshot(context.TODO(), repo, id)
 		OK(t, err)
 
 		ts := snapshot.Time.Format(time.RFC3339)
@@ -144,45 +143,46 @@ func TestMount(t *testing.T) {
 		t.Skip("Skipping fuse tests")
 	}
 
-	withTestEnvironment(t, func(env *testEnvironment, global GlobalOptions) {
-
-		cmdInit(t, global)
-		repo, err := global.OpenRepository()
+	withTestEnvironment(t, func(env *testEnvironment, gopts GlobalOptions) {
+		mountpoint, err := ioutil.TempDir(TestTempDir, "restic-test-mount-")
 		OK(t, err)
 
-		mountpoint, err := ioutil.TempDir(TestTempDir, "restic-test-mount-")
+		testRunInit(t, gopts)
+
+		repo, err := OpenRepository(gopts)
 		OK(t, err)
 
 		// We remove the mountpoint now to check that cmdMount creates it
 		RemoveAll(t, mountpoint)
 
-		checkSnapshots(t, global, repo, mountpoint, env.repo, []restic.ID{})
+		checkSnapshots(t, gopts, repo, mountpoint, env.repo, []restic.ID{})
 
 		SetupTarTestFixture(t, env.testdata, filepath.Join("testdata", "backup-data.tar.gz"))
 
 		// first backup
-		cmdBackup(t, global, []string{env.testdata}, nil)
-		snapshotIDs := cmdList(t, global, "snapshots")
+		testRunBackup(t, []string{env.testdata}, BackupOptions{}, gopts)
+		snapshotIDs := testRunList(t, "snapshots", gopts)
 		Assert(t, len(snapshotIDs) == 1,
 			"expected one snapshot, got %v", snapshotIDs)
 
-		checkSnapshots(t, global, repo, mountpoint, env.repo, snapshotIDs)
+		checkSnapshots(t, gopts, repo, mountpoint, env.repo, snapshotIDs)
 
 		// second backup, implicit incremental
-		cmdBackup(t, global, []string{env.testdata}, nil)
-		snapshotIDs = cmdList(t, global, "snapshots")
+		testRunBackup(t, []string{env.testdata}, BackupOptions{}, gopts)
+		snapshotIDs = testRunList(t, "snapshots", gopts)
 		Assert(t, len(snapshotIDs) == 2,
 			"expected two snapshots, got %v", snapshotIDs)
 
-		checkSnapshots(t, global, repo, mountpoint, env.repo, snapshotIDs)
+		checkSnapshots(t, gopts, repo, mountpoint, env.repo, snapshotIDs)
 
 		// third backup, explicit incremental
-		cmdBackup(t, global, []string{env.testdata}, &snapshotIDs[0])
-		snapshotIDs = cmdList(t, global, "snapshots")
+		bopts := BackupOptions{Parent: snapshotIDs[0].String()}
+		testRunBackup(t, []string{env.testdata}, bopts, gopts)
+		snapshotIDs = testRunList(t, "snapshots", gopts)
 		Assert(t, len(snapshotIDs) == 3,
 			"expected three snapshots, got %v", snapshotIDs)
 
-		checkSnapshots(t, global, repo, mountpoint, env.repo, snapshotIDs)
+		checkSnapshots(t, gopts, repo, mountpoint, env.repo, snapshotIDs)
 	})
 }
 
@@ -191,10 +191,10 @@ func TestMountSameTimestamps(t *testing.T) {
 		t.Skip("Skipping fuse tests")
 	}
 
-	withTestEnvironment(t, func(env *testEnvironment, global GlobalOptions) {
+	withTestEnvironment(t, func(env *testEnvironment, gopts GlobalOptions) {
 		SetupTarTestFixture(t, env.base, filepath.Join("testdata", "repo-same-timestamps.tar.gz"))
 
-		repo, err := global.OpenRepository()
+		repo, err := OpenRepository(gopts)
 		OK(t, err)
 
 		mountpoint, err := ioutil.TempDir(TestTempDir, "restic-test-mount-")
@@ -206,6 +206,6 @@ func TestMountSameTimestamps(t *testing.T) {
 			restic.TestParseID("5fd0d8b2ef0fa5d23e58f1e460188abb0f525c0f0c4af8365a1280c807a80a1b"),
 		}
 
-		checkSnapshots(t, global, repo, mountpoint, env.repo, ids)
+		checkSnapshots(t, gopts, repo, mountpoint, env.repo, ids)
 	})
 }

src/cmds/restic/integration_helpers_unix_test.go

@@ -52,11 +52,6 @@ func nlink(info os.FileInfo) uint64 {
 	return uint64(stat.Nlink)
 }
 
-func inode(info os.FileInfo) uint64 {
-	stat, _ := info.Sys().(*syscall.Stat_t)
-	return uint64(stat.Ino)
-}
-
 func createFileSetPerHardlink(dir string) map[uint64][]string {
 	var stat syscall.Stat_t
 	linkTests := make(map[uint64][]string)

src/cmds/restic/integration_test.go

@@ -1175,15 +1175,19 @@ func TestPrune(t *testing.T) {
 		SetupTarTestFixture(t, env.testdata, datafile)
 		opts := BackupOptions{}
 
-		testRunBackup(t, []string{filepath.Join(env.testdata, "0", "0", "1")}, opts, gopts)
+		testRunBackup(t, []string{filepath.Join(env.testdata, "0", "0")}, opts, gopts)
+		firstSnapshot := testRunList(t, "snapshots", gopts)
+		Assert(t, len(firstSnapshot) == 1,
+			"expected one snapshot, got %v", firstSnapshot)
+
 		testRunBackup(t, []string{filepath.Join(env.testdata, "0", "0", "2")}, opts, gopts)
 		testRunBackup(t, []string{filepath.Join(env.testdata, "0", "0", "3")}, opts, gopts)
 
 		snapshotIDs := testRunList(t, "snapshots", gopts)
 		Assert(t, len(snapshotIDs) == 3,
-			"expected one snapshot, got %v", snapshotIDs)
+			"expected 3 snapshot, got %v", snapshotIDs)
 
-		testRunForget(t, gopts, snapshotIDs[0].String())
+		testRunForget(t, gopts, firstSnapshot[0].String())
 		testRunPrune(t, gopts)
 		testRunCheck(t, gopts)
 	})

src/cmds/restic/lock.go

@@ -1,6 +1,7 @@
 package main
 
 import (
+	"context"
 	"fmt"
 	"os"
 	"sync"
@@ -32,7 +33,7 @@ func lockRepository(repo *repository.Repository, exclusive bool) (*restic.Lock,
 		lockFn = restic.NewExclusiveLock
 	}
 
-	lock, err := lockFn(repo)
+	lock, err := lockFn(context.TODO(), repo)
 	if err != nil {
 		return nil, err
 	}
@@ -75,7 +76,7 @@ func refreshLocks(wg *sync.WaitGroup, done <-chan struct{}) {
 			debug.Log("refreshing locks")
 			globalLocks.Lock()
 			for _, lock := range globalLocks.locks {
-				err := lock.Refresh()
+				err := lock.Refresh(context.TODO())
 				if err != nil {
 					fmt.Fprintf(os.Stderr, "unable to refresh lock: %v\n", err)
 				}

src/cmds/restic/main.go

@@ -9,6 +9,7 @@ import (
 	"restic"
 	"restic/debug"
 	"restic/options"
+	"runtime"
 
 	"github.com/spf13/cobra"
 
@@ -57,6 +58,8 @@ func init() {
 
 func main() {
 	debug.Log("main %#v", os.Args)
+	debug.Log("restic %s, compiled with %v on %v/%v",
+		version, runtime.Version(), runtime.GOOS, runtime.GOARCH)
 	err := cmdRoot.Execute()
 
 	switch {

src/restic/archiver/archive_reader.go

@@ -1,6 +1,7 @@
 package archiver
 
 import (
+	"context"
 	"io"
 	"restic"
 	"restic/debug"
@@ -20,7 +21,7 @@ type Reader struct {
 }
 
 // Archive reads data from the reader and saves it to the repo.
-func (r *Reader) Archive(name string, rd io.Reader, p *restic.Progress) (*restic.Snapshot, restic.ID, error) {
+func (r *Reader) Archive(ctx context.Context, name string, rd io.Reader, p *restic.Progress) (*restic.Snapshot, restic.ID, error) {
 	if name == "" {
 		return nil, restic.ID{}, errors.New("no filename given")
 	}
@@ -53,7 +54,7 @@ func (r *Reader) Archive(name string, rd io.Reader, p *restic.Progress) (*restic
 		id := restic.Hash(chunk.Data)
 
 		if !repo.Index().Has(id, restic.DataBlob) {
-			_, err := repo.SaveBlob(restic.DataBlob, chunk.Data, id)
+			_, err := repo.SaveBlob(ctx, restic.DataBlob, chunk.Data, id)
 			if err != nil {
 				return nil, restic.ID{}, err
 			}
@@ -87,14 +88,14 @@ func (r *Reader) Archive(name string, rd io.Reader, p *restic.Progress) (*restic
 		},
 	}
 
-	treeID, err := repo.SaveTree(tree)
+	treeID, err := repo.SaveTree(ctx, tree)
 	if err != nil {
 		return nil, restic.ID{}, err
 	}
 	sn.Tree = &treeID
 	debug.Log("tree saved as %v", treeID.Str())
 
-	id, err := repo.SaveJSONUnpacked(restic.SnapshotFile, sn)
+	id, err := repo.SaveJSONUnpacked(ctx, restic.SnapshotFile, sn)
 	if err != nil {
 		return nil, restic.ID{}, err
 	}
@@ -106,7 +107,7 @@ func (r *Reader) Archive(name string, rd io.Reader, p *restic.Progress) (*restic
 		return nil, restic.ID{}, err
 	}
 
-	err = repo.SaveIndex()
+	err = repo.SaveIndex(ctx)
 	if err != nil {
 		return nil, restic.ID{}, err
 	}

src/restic/archiver/archive_reader_test.go

@@ -2,6 +2,7 @@ package archiver
 
 import (
 	"bytes"
+	"context"
 	"errors"
 	"io"
 	"math/rand"
@@ -12,7 +13,7 @@ import (
 )
 
 func loadBlob(t *testing.T, repo restic.Repository, id restic.ID, buf []byte) int {
-	n, err := repo.LoadBlob(restic.DataBlob, id, buf)
+	n, err := repo.LoadBlob(context.TODO(), restic.DataBlob, id, buf)
 	if err != nil {
 		t.Fatalf("LoadBlob(%v) returned error %v", id, err)
 	}
@@ -21,7 +22,7 @@ func loadBlob(t *testing.T, repo restic.Repository, id restic.ID, buf []byte) in
 }
 
 func checkSavedFile(t *testing.T, repo restic.Repository, treeID restic.ID, name string, rd io.Reader) {
-	tree, err := repo.LoadTree(treeID)
+	tree, err := repo.LoadTree(context.TODO(), treeID)
 	if err != nil {
 		t.Fatalf("LoadTree() returned error %v", err)
 	}
@@ -85,7 +86,7 @@ func TestArchiveReader(t *testing.T) {
 		Tags:       []string{"test"},
 	}
 
-	sn, id, err := r.Archive("fakefile", f, nil)
+	sn, id, err := r.Archive(context.TODO(), "fakefile", f, nil)
 	if err != nil {
 		t.Fatalf("ArchiveReader() returned error %v", err)
 	}
@@ -111,7 +112,7 @@ func TestArchiveReaderNull(t *testing.T) {
 		Tags:       []string{"test"},
 	}
 
-	sn, id, err := r.Archive("fakefile", bytes.NewReader(nil), nil)
+	sn, id, err := r.Archive(context.TODO(), "fakefile", bytes.NewReader(nil), nil)
 	if err != nil {
 		t.Fatalf("ArchiveReader() returned error %v", err)
 	}
@@ -132,11 +133,8 @@ func (e errReader) Read([]byte) (int, error) {
 }
 
 func countSnapshots(t testing.TB, repo restic.Repository) int {
-	done := make(chan struct{})
-	defer close(done)
-
 	snapshots := 0
-	for range repo.List(restic.SnapshotFile, done) {
+	for range repo.List(context.TODO(), restic.SnapshotFile) {
 		snapshots++
 	}
 	return snapshots
@@ -152,7 +150,7 @@ func TestArchiveReaderError(t *testing.T) {
 		Tags:       []string{"test"},
 	}
 
-	sn, id, err := r.Archive("fakefile", errReader("error returned by reading stdin"), nil)
+	sn, id, err := r.Archive(context.TODO(), "fakefile", errReader("error returned by reading stdin"), nil)
 	if err == nil {
 		t.Errorf("expected error not returned")
 	}
@@ -195,7 +193,7 @@ func BenchmarkArchiveReader(t *testing.B) {
 	t.ResetTimer()
 
 	for i := 0; i < t.N; i++ {
-		_, _, err := r.Archive("fakefile", bytes.NewReader(buf), nil)
+		_, _, err := r.Archive(context.TODO(), "fakefile", bytes.NewReader(buf), nil)
 		if err != nil {
 			t.Fatal(err)
 		}

src/restic/archiver/archiver.go

@@ -1,6 +1,7 @@
 package archiver
 
 import (
+	"context"
 	"encoding/json"
 	"fmt"
 	"io"
@@ -92,7 +93,7 @@ func (arch *Archiver) isKnownBlob(id restic.ID, t restic.BlobType) bool {
 }
 
 // Save stores a blob read from rd in the repository.
-func (arch *Archiver) Save(t restic.BlobType, data []byte, id restic.ID) error {
+func (arch *Archiver) Save(ctx context.Context, t restic.BlobType, data []byte, id restic.ID) error {
 	debug.Log("Save(%v, %v)\n", t, id.Str())
 
 	if arch.isKnownBlob(id, restic.DataBlob) {
@@ -100,7 +101,7 @@ func (arch *Archiver) Save(t restic.BlobType, data []byte, id restic.ID) error {
 		return nil
 	}
 
-	_, err := arch.repo.SaveBlob(t, data, id)
+	_, err := arch.repo.SaveBlob(ctx, t, data, id)
 	if err != nil {
 		debug.Log("Save(%v, %v): error %v\n", t, id.Str(), err)
 		return err
@@ -111,7 +112,7 @@ func (arch *Archiver) Save(t restic.BlobType, data []byte, id restic.ID) error {
 }
 
 // SaveTreeJSON stores a tree in the repository.
-func (arch *Archiver) SaveTreeJSON(tree *restic.Tree) (restic.ID, error) {
+func (arch *Archiver) SaveTreeJSON(ctx context.Context, tree *restic.Tree) (restic.ID, error) {
 	data, err := json.Marshal(tree)
 	if err != nil {
 		return restic.ID{}, errors.Wrap(err, "Marshal")
@@ -124,7 +125,7 @@ func (arch *Archiver) SaveTreeJSON(tree *restic.Tree) (restic.ID, error) {
 		return id, nil
 	}
 
-	return arch.repo.SaveBlob(restic.TreeBlob, data, id)
+	return arch.repo.SaveBlob(ctx, restic.TreeBlob, data, id)
 }
 
 func (arch *Archiver) reloadFileIfChanged(node *restic.Node, file fs.File) (*restic.Node, error) {
@@ -153,13 +154,14 @@ type saveResult struct {
 	bytes uint64
 }
 
-func (arch *Archiver) saveChunk(chunk chunker.Chunk, p *restic.Progress, token struct{}, file fs.File, resultChannel chan<- saveResult) {
+func (arch *Archiver) saveChunk(ctx context.Context, chunk chunker.Chunk, p *restic.Progress, token struct{}, file fs.File, resultChannel chan<- saveResult) {
 	defer freeBuf(chunk.Data)
 
 	id := restic.Hash(chunk.Data)
-	err := arch.Save(restic.DataBlob, chunk.Data, id)
+	err := arch.Save(ctx, restic.DataBlob, chunk.Data, id)
 	// TODO handle error
 	if err != nil {
+		debug.Log("Save(%v) failed: %v", id.Str(), err)
 		panic(err)
 	}
 
@@ -206,7 +208,7 @@ func updateNodeContent(node *restic.Node, results []saveResult) error {
 
 // SaveFile stores the content of the file on the backend as a Blob by calling
 // Save for each chunk.
-func (arch *Archiver) SaveFile(p *restic.Progress, node *restic.Node) (*restic.Node, error) {
+func (arch *Archiver) SaveFile(ctx context.Context, p *restic.Progress, node *restic.Node) (*restic.Node, error) {
 	file, err := fs.Open(node.Path)
 	defer file.Close()
 	if err != nil {
@@ -234,7 +236,7 @@ func (arch *Archiver) SaveFile(p *restic.Progress, node *restic.Node) (*restic.N
 		}
 
 		resCh := make(chan saveResult, 1)
-		go arch.saveChunk(chunk, p, <-arch.blobToken, file, resCh)
+		go arch.saveChunk(ctx, chunk, p, <-arch.blobToken, file, resCh)
 		resultChannels = append(resultChannels, resCh)
 	}
 
@@ -247,7 +249,7 @@ func (arch *Archiver) SaveFile(p *restic.Progress, node *restic.Node) (*restic.N
 	return node, err
 }
 
-func (arch *Archiver) fileWorker(wg *sync.WaitGroup, p *restic.Progress, done <-chan struct{}, entCh <-chan pipe.Entry) {
+func (arch *Archiver) fileWorker(ctx context.Context, wg *sync.WaitGroup, p *restic.Progress, entCh <-chan pipe.Entry) {
 	defer func() {
 		debug.Log("done")
 		wg.Done()
@@ -305,7 +307,7 @@ func (arch *Archiver) fileWorker(wg *sync.WaitGroup, p *restic.Progress, done <-
 			// otherwise read file normally
 			if node.Type == "file" && len(node.Content) == 0 {
 				debug.Log("   read and save %v", e.Path())
-				node, err = arch.SaveFile(p, node)
+				node, err = arch.SaveFile(ctx, p, node)
 				if err != nil {
 					fmt.Fprintf(os.Stderr, "error for %v: %v\n", node.Path, err)
 					arch.Warn(e.Path(), nil, err)
@@ -322,14 +324,14 @@ func (arch *Archiver) fileWorker(wg *sync.WaitGroup, p *restic.Progress, done <-
 			debug.Log("   processed %v, %d blobs", e.Path(), len(node.Content))
 			e.Result() <- node
 			p.Report(restic.Stat{Files: 1})
-		case <-done:
+		case <-ctx.Done():
 			// pipeline was cancelled
 			return
 		}
 	}
 }
 
-func (arch *Archiver) dirWorker(wg *sync.WaitGroup, p *restic.Progress, done <-chan struct{}, dirCh <-chan pipe.Dir) {
+func (arch *Archiver) dirWorker(ctx context.Context, wg *sync.WaitGroup, p *restic.Progress, dirCh <-chan pipe.Dir) {
 	debug.Log("start")
 	defer func() {
 		debug.Log("done")
@@ -398,7 +400,7 @@ func (arch *Archiver) dirWorker(wg *sync.WaitGroup, p *restic.Progress, done <-c
 				node.Error = err.Error()
 			}
 
-			id, err := arch.SaveTreeJSON(tree)
+			id, err := arch.SaveTreeJSON(ctx, tree)
 			if err != nil {
 				panic(err)
 			}
@@ -415,7 +417,7 @@ func (arch *Archiver) dirWorker(wg *sync.WaitGroup, p *restic.Progress, done <-c
 			if dir.Path() != "" {
 				p.Report(restic.Stat{Dirs: 1})
 			}
-		case <-done:
+		case <-ctx.Done():
 			// pipeline was cancelled
 			return
 		}
@@ -427,7 +429,7 @@ type archivePipe struct {
 	New <-chan pipe.Job
 }
 
-func copyJobs(done <-chan struct{}, in <-chan pipe.Job, out chan<- pipe.Job) {
+func copyJobs(ctx context.Context, in <-chan pipe.Job, out chan<- pipe.Job) {
 	var (
 		// disable sending on the outCh until we received a job
 		outCh chan<- pipe.Job
@@ -439,7 +441,7 @@ func copyJobs(done <-chan struct{}, in <-chan pipe.Job, out chan<- pipe.Job) {
 
 	for {
 		select {
-		case <-done:
+		case <-ctx.Done():
 			return
 		case job, ok = <-inCh:
 			if !ok {
@@ -462,7 +464,7 @@ type archiveJob struct {
 	new    pipe.Job
 }
 
-func (a *archivePipe) compare(done <-chan struct{}, out chan<- pipe.Job) {
+func (a *archivePipe) compare(ctx context.Context, out chan<- pipe.Job) {
 	defer func() {
 		close(out)
 		debug.Log("done")
@@ -488,7 +490,7 @@ func (a *archivePipe) compare(done <-chan struct{}, out chan<- pipe.Job) {
 					out <- archiveJob{new: newJob}.Copy()
 				}
 
-				copyJobs(done, a.New, out)
+				copyJobs(ctx, a.New, out)
 				return
 			}
 
@@ -585,7 +587,7 @@ func (j archiveJob) Copy() pipe.Job {
 const saveIndexTime = 30 * time.Second
 
 // saveIndexes regularly queries the master index for full indexes and saves them.
-func (arch *Archiver) saveIndexes(wg *sync.WaitGroup, done <-chan struct{}) {
+func (arch *Archiver) saveIndexes(ctx context.Context, wg *sync.WaitGroup) {
 	defer wg.Done()
 
 	ticker := time.NewTicker(saveIndexTime)
@@ -593,11 +595,11 @@ func (arch *Archiver) saveIndexes(wg *sync.WaitGroup, done <-chan struct{}) {
 
 	for {
 		select {
-		case <-done:
+		case <-ctx.Done():
 			return
 		case <-ticker.C:
 			debug.Log("saving full indexes")
-			err := arch.repo.SaveFullIndex()
+			err := arch.repo.SaveFullIndex(ctx)
 			if err != nil {
 				debug.Log("save indexes returned an error: %v", err)
 				fmt.Fprintf(os.Stderr, "error saving preliminary index: %v\n", err)
@@ -634,7 +636,7 @@ func (p baseNameSlice) Swap(i, j int)      { p[i], p[j] = p[j], p[i] }
 // Snapshot creates a snapshot of the given paths. If parentrestic.ID is set, this is
 // used to compare the files to the ones archived at the time this snapshot was
 // taken.
-func (arch *Archiver) Snapshot(p *restic.Progress, paths, tags []string, hostname string, parentID *restic.ID) (*restic.Snapshot, restic.ID, error) {
+func (arch *Archiver) Snapshot(ctx context.Context, p *restic.Progress, paths, tags []string, hostname string, parentID *restic.ID) (*restic.Snapshot, restic.ID, error) {
 	paths = unique(paths)
 	sort.Sort(baseNameSlice(paths))
 
@@ -643,7 +645,6 @@ func (arch *Archiver) Snapshot(p *restic.Progress, paths, tags []string, hostnam
 	debug.RunHook("Archiver.Snapshot", nil)
 
 	// signal the whole pipeline to stop
-	done := make(chan struct{})
 	var err error
 
 	p.Start()
@@ -663,14 +664,14 @@ func (arch *Archiver) Snapshot(p *restic.Progress, paths, tags []string, hostnam
 		sn.Parent = parentID
 
 		// load parent snapshot
-		parent, err := restic.LoadSnapshot(arch.repo, *parentID)
+		parent, err := restic.LoadSnapshot(ctx, arch.repo, *parentID)
 		if err != nil {
 			return nil, restic.ID{}, err
 		}
 
 		// start walker on old tree
 		ch := make(chan walk.TreeJob)
-		go walk.Tree(arch.repo, *parent.Tree, done, ch)
+		go walk.Tree(ctx, arch.repo, *parent.Tree, ch)
 		jobs.Old = ch
 	} else {
 		// use closed channel
@@ -683,13 +684,13 @@ func (arch *Archiver) Snapshot(p *restic.Progress, paths, tags []string, hostnam
 	pipeCh := make(chan pipe.Job)
 	resCh := make(chan pipe.Result, 1)
 	go func() {
-		pipe.Walk(paths, arch.SelectFilter, done, pipeCh, resCh)
+		pipe.Walk(ctx, paths, arch.SelectFilter, pipeCh, resCh)
 		debug.Log("pipe.Walk done")
 	}()
 	jobs.New = pipeCh
 
 	ch := make(chan pipe.Job)
-	go jobs.compare(done, ch)
+	go jobs.compare(ctx, ch)
 
 	var wg sync.WaitGroup
 	entCh := make(chan pipe.Entry)
@@ -708,22 +709,22 @@ func (arch *Archiver) Snapshot(p *restic.Progress, paths, tags []string, hostnam
 	// run workers
 	for i := 0; i < maxConcurrency; i++ {
 		wg.Add(2)
-		go arch.fileWorker(&wg, p, done, entCh)
-		go arch.dirWorker(&wg, p, done, dirCh)
+		go arch.fileWorker(ctx, &wg, p, entCh)
+		go arch.dirWorker(ctx, &wg, p, dirCh)
 	}
 
 	// run index saver
 	var wgIndexSaver sync.WaitGroup
-	stopIndexSaver := make(chan struct{})
+	indexCtx, indexCancel := context.WithCancel(ctx)
 	wgIndexSaver.Add(1)
-	go arch.saveIndexes(&wgIndexSaver, stopIndexSaver)
+	go arch.saveIndexes(indexCtx, &wgIndexSaver)
 
 	// wait for all workers to terminate
 	debug.Log("wait for workers")
 	wg.Wait()
 
 	// stop index saver
-	close(stopIndexSaver)
+	indexCancel()
 	wgIndexSaver.Wait()
 
 	debug.Log("workers terminated")
@@ -740,7 +741,7 @@ func (arch *Archiver) Snapshot(p *restic.Progress, paths, tags []string, hostnam
 	sn.Tree = root.Subtree
 
 	// load top-level tree again to see if it is empty
-	toptree, err := arch.repo.LoadTree(*root.Subtree)
+	toptree, err := arch.repo.LoadTree(ctx, *root.Subtree)
 	if err != nil {
 		return nil, restic.ID{}, err
 	}
@@ -750,7 +751,7 @@ func (arch *Archiver) Snapshot(p *restic.Progress, paths, tags []string, hostnam
 	}
 
 	// save index
-	err = arch.repo.SaveIndex()
+	err = arch.repo.SaveIndex(ctx)
 	if err != nil {
 		debug.Log("error saving index: %v", err)
 		return nil, restic.ID{}, err
@@ -759,7 +760,7 @@ func (arch *Archiver) Snapshot(p *restic.Progress, paths, tags []string, hostnam
 	debug.Log("saved indexes")
 
 	// save snapshot
-	id, err := arch.repo.SaveJSONUnpacked(restic.SnapshotFile, sn)
+	id, err := arch.repo.SaveJSONUnpacked(ctx, restic.SnapshotFile, sn)
 	if err != nil {
 		return nil, restic.ID{}, err
 	}

src/restic/archiver/archiver_duplication_test.go

@@ -1,6 +1,7 @@
 package archiver_test
 
 import (
+	"context"
 	"crypto/rand"
 	"io"
 	mrand "math/rand"
@@ -39,33 +40,33 @@ func randomID() restic.ID {
 func forgetfulBackend() restic.Backend {
 	be := &mock.Backend{}
 
-	be.TestFn = func(h restic.Handle) (bool, error) {
+	be.TestFn = func(ctx context.Context, h restic.Handle) (bool, error) {
 		return false, nil
 	}
 
-	be.LoadFn = func(h restic.Handle, length int, offset int64) (io.ReadCloser, error) {
+	be.LoadFn = func(ctx context.Context, h restic.Handle, length int, offset int64) (io.ReadCloser, error) {
 		return nil, errors.New("not found")
 	}
 
-	be.SaveFn = func(h restic.Handle, rd io.Reader) error {
+	be.SaveFn = func(ctx context.Context, h restic.Handle, rd io.Reader) error {
 		return nil
 	}
 
-	be.StatFn = func(h restic.Handle) (restic.FileInfo, error) {
+	be.StatFn = func(ctx context.Context, h restic.Handle) (restic.FileInfo, error) {
 		return restic.FileInfo{}, errors.New("not found")
 	}
 
-	be.RemoveFn = func(h restic.Handle) error {
+	be.RemoveFn = func(ctx context.Context, h restic.Handle) error {
 		return nil
 	}
 
-	be.ListFn = func(t restic.FileType, done <-chan struct{}) <-chan string {
+	be.ListFn = func(ctx context.Context, t restic.FileType) <-chan string {
 		ch := make(chan string)
 		close(ch)
 		return ch
 	}
 
-	be.DeleteFn = func() error {
+	be.DeleteFn = func(ctx context.Context) error {
 		return nil
 	}
 
@@ -80,7 +81,7 @@ func testArchiverDuplication(t *testing.T) {
 
 	repo := repository.New(forgetfulBackend())
 
-	err = repo.Init("foo")
+	err = repo.Init(context.TODO(), "foo")
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -108,7 +109,7 @@ func testArchiverDuplication(t *testing.T) {
 
 				buf := make([]byte, 50)
 
-				err := arch.Save(restic.DataBlob, buf, id)
+				err := arch.Save(context.TODO(), restic.DataBlob, buf, id)
 				if err != nil {
 					t.Fatal(err)
 				}
@@ -127,7 +128,7 @@ func testArchiverDuplication(t *testing.T) {
 			case <-done:
 				return
 			case <-ticker.C:
-				err := repo.SaveFullIndex()
+				err := repo.SaveFullIndex(context.TODO())
 				if err != nil {
 					t.Fatal(err)
 				}

src/restic/archiver/archiver_int_test.go

@@ -1,6 +1,7 @@
 package archiver
 
 import (
+	"context"
 	"os"
 	"testing"
 
@@ -83,10 +84,10 @@ func (j testPipeJob) Error() error               { return j.err }
 func (j testPipeJob) Info() os.FileInfo          { return j.fi }
 func (j testPipeJob) Result() chan<- pipe.Result { return j.res }
 
-func testTreeWalker(done <-chan struct{}, out chan<- walk.TreeJob) {
+func testTreeWalker(ctx context.Context, out chan<- walk.TreeJob) {
 	for _, e := range treeJobs {
 		select {
-		case <-done:
+		case <-ctx.Done():
 			return
 		case out <- walk.TreeJob{Path: e}:
 		}
@@ -95,10 +96,10 @@ func testTreeWalker(done <-chan struct{}, out chan<- walk.TreeJob) {
 	close(out)
 }
 
-func testPipeWalker(done <-chan struct{}, out chan<- pipe.Job) {
+func testPipeWalker(ctx context.Context, out chan<- pipe.Job) {
 	for _, e := range pipeJobs {
 		select {
-		case <-done:
+		case <-ctx.Done():
 			return
 		case out <- testPipeJob{path: e}:
 		}
@@ -108,19 +109,19 @@ func testPipeWalker(done <-chan struct{}, out chan<- pipe.Job) {
 }
 
 func TestArchivePipe(t *testing.T) {
-	done := make(chan struct{})
+	ctx := context.TODO()
 
 	treeCh := make(chan walk.TreeJob)
 	pipeCh := make(chan pipe.Job)
 
-	go testTreeWalker(done, treeCh)
-	go testPipeWalker(done, pipeCh)
+	go testTreeWalker(ctx, treeCh)
+	go testPipeWalker(ctx, pipeCh)
 
 	p := archivePipe{Old: treeCh, New: pipeCh}
 
 	ch := make(chan pipe.Job)
 
-	go p.compare(done, ch)
+	go p.compare(ctx, ch)
 
 	i := 0
 	for job := range ch {

src/restic/archiver/archiver_test.go

@@ -2,6 +2,7 @@ package archiver_test
 
 import (
 	"bytes"
+	"context"
 	"io"
 	"testing"
 	"time"
@@ -42,7 +43,7 @@ func benchmarkChunkEncrypt(b testing.TB, buf, buf2 []byte, rd Rdr, key *crypto.K
 		Assert(b, uint(len(chunk.Data)) == chunk.Length,
 			"invalid length: got %d, expected %d", len(chunk.Data), chunk.Length)
 
-		_, err = crypto.Encrypt(key, buf2, chunk.Data)
+		_, err = key.Encrypt(buf2, chunk.Data)
 		OK(b, err)
 	}
 }
@@ -75,7 +76,7 @@ func benchmarkChunkEncryptP(b *testing.PB, buf []byte, rd Rdr, key *crypto.Key)
 		}
 
 		// reduce length of chunkBuf
-		crypto.Encrypt(key, chunk.Data, chunk.Data)
+		key.Encrypt(chunk.Data, chunk.Data)
 	}
 }
 
@@ -104,7 +105,7 @@ func archiveDirectory(b testing.TB) {
 
 	arch := archiver.New(repo)
 
-	_, id, err := arch.Snapshot(nil, []string{BenchArchiveDirectory}, nil, "localhost", nil)
+	_, id, err := arch.Snapshot(context.TODO(), nil, []string{BenchArchiveDirectory}, nil, "localhost", nil)
 	OK(b, err)
 
 	b.Logf("snapshot archived as %v", id)
@@ -129,7 +130,7 @@ func BenchmarkArchiveDirectory(b *testing.B) {
 }
 
 func countPacks(repo restic.Repository, t restic.FileType) (n uint) {
-	for range repo.Backend().List(t, nil) {
+	for range repo.Backend().List(context.TODO(), t) {
 		n++
 	}
 
@@ -234,7 +235,7 @@ func testParallelSaveWithDuplication(t *testing.T, seed int) {
 
 				id := restic.Hash(c.Data)
 				time.Sleep(time.Duration(id[0]))
-				err := arch.Save(restic.DataBlob, c.Data, id)
+				err := arch.Save(context.TODO(), restic.DataBlob, c.Data, id)
 				<-barrier
 				errChan <- err
 			}(c, errChan)
@@ -246,7 +247,7 @@ func testParallelSaveWithDuplication(t *testing.T, seed int) {
 	}
 
 	OK(t, repo.Flush())
-	OK(t, repo.SaveIndex())
+	OK(t, repo.SaveIndex(context.TODO()))
 
 	chkr := createAndInitChecker(t, repo)
 	assertNoUnreferencedPacks(t, chkr)
@@ -271,7 +272,7 @@ func getRandomData(seed int, size int) []chunker.Chunk {
 func createAndInitChecker(t *testing.T, repo restic.Repository) *checker.Checker {
 	chkr := checker.New(repo)
 
-	hints, errs := chkr.LoadIndex()
+	hints, errs := chkr.LoadIndex(context.TODO())
 	if len(errs) > 0 {
 		t.Fatalf("expected no errors, got %v: %v", len(errs), errs)
 	}
@@ -284,11 +285,8 @@ func createAndInitChecker(t *testing.T, repo restic.Repository) *checker.Checker
 }
 
 func assertNoUnreferencedPacks(t *testing.T, chkr *checker.Checker) {
-	done := make(chan struct{})
-	defer close(done)
-
 	errChan := make(chan error)
-	go chkr.Packs(errChan, done)
+	go chkr.Packs(context.TODO(), errChan)
 
 	for err := range errChan {
 		OK(t, err)
@@ -301,7 +299,7 @@ func TestArchiveEmptySnapshot(t *testing.T) {
 
 	arch := archiver.New(repo)
 
-	sn, id, err := arch.Snapshot(nil, []string{"file-does-not-exist-123123213123", "file2-does-not-exist-too-123123123"}, nil, "localhost", nil)
+	sn, id, err := arch.Snapshot(context.TODO(), nil, []string{"file-does-not-exist-123123213123", "file2-does-not-exist-too-123123123"}, nil, "localhost", nil)
 	if err == nil {
 		t.Errorf("expected error for empty snapshot, got nil")
 	}

src/restic/archiver/testing.go

@@ -1,6 +1,7 @@
 package archiver
 
 import (
+	"context"
 	"restic"
 	"testing"
 )
@@ -8,7 +9,7 @@ import (
 // TestSnapshot creates a new snapshot of path.
 func TestSnapshot(t testing.TB, repo restic.Repository, path string, parent *restic.ID) *restic.Snapshot {
 	arch := New(repo)
-	sn, _, err := arch.Snapshot(nil, []string{path}, []string{"test"}, "localhost", parent)
+	sn, _, err := arch.Snapshot(context.TODO(), nil, []string{path}, []string{"test"}, "localhost", parent)
 	if err != nil {
 		t.Fatal(err)
 	}

src/restic/backend.go

@@ -1,6 +1,9 @@
 package restic
 
-import "io"
+import (
+	"context"
+	"io"
+)
 
 // Backend is used to store and access data.
 type Backend interface {
@@ -9,30 +12,34 @@ type Backend interface {
 	Location() string
 
 	// Test a boolean value whether a File with the name and type exists.
-	Test(h Handle) (bool, error)
+	Test(ctx context.Context, h Handle) (bool, error)
 
 	// Remove removes a File with type t and name.
-	Remove(h Handle) error
+	Remove(ctx context.Context, h Handle) error
 
 	// Close the backend
 	Close() error
 
 	// Save stores the data in the backend under the given handle.
-	Save(h Handle, rd io.Reader) error
+	Save(ctx context.Context, h Handle, rd io.Reader) error
 
 	// Load returns a reader that yields the contents of the file at h at the
 	// given offset. If length is larger than zero, only a portion of the file
 	// is returned. rd must be closed after use. If an error is returned, the
 	// ReadCloser must be nil.
-	Load(h Handle, length int, offset int64) (io.ReadCloser, error)
+	Load(ctx context.Context, h Handle, length int, offset int64) (io.ReadCloser, error)
 
 	// Stat returns information about the File identified by h.
-	Stat(h Handle) (FileInfo, error)
+	Stat(ctx context.Context, h Handle) (FileInfo, error)
 
 	// List returns a channel that yields all names of files of type t in an
-	// arbitrary order. A goroutine is started for this. If the channel done is
-	// closed, sending stops.
-	List(t FileType, done <-chan struct{}) <-chan string
+	// arbitrary order. A goroutine is started for this, which is stopped when
+	// ctx is cancelled.
+	List(ctx context.Context, t FileType) <-chan string
+
+	// IsNotExist returns true if the error was caused by a non-existing file
+	// in the backend.
+	IsNotExist(err error) bool
 }
 
 // FileInfo is returned by Stat() and contains information about a file in the

src/restic/backend/b2/b2.go

@@ -0,0 +1,377 @@
+package b2
+
+import (
+	"context"
+	"io"
+	"path"
+	"restic"
+	"strings"
+
+	"restic/backend"
+	"restic/debug"
+	"restic/errors"
+
+	"github.com/kurin/blazer/b2"
+)
+
+// b2Backend is a backend which stores its data on Backblaze B2.
+type b2Backend struct {
+	client *b2.Client
+	bucket *b2.Bucket
+	cfg    Config
+	backend.Layout
+	sem *backend.Semaphore
+}
+
+// ensure statically that *b2Backend implements restic.Backend.
+var _ restic.Backend = &b2Backend{}
+
+func newClient(ctx context.Context, cfg Config) (*b2.Client, error) {
+	opts := []b2.ClientOption{b2.Transport(backend.Transport())}
+
+	c, err := b2.NewClient(ctx, cfg.AccountID, cfg.Key, opts...)
+	if err != nil {
+		return nil, errors.Wrap(err, "b2.NewClient")
+	}
+	return c, nil
+}
+
+// Open opens a connection to the B2 service.
+func Open(cfg Config) (restic.Backend, error) {
+	debug.Log("cfg %#v", cfg)
+
+	ctx, cancel := context.WithCancel(context.TODO())
+	defer cancel()
+
+	client, err := newClient(ctx, cfg)
+	if err != nil {
+		return nil, err
+	}
+
+	bucket, err := client.Bucket(ctx, cfg.Bucket)
+	if err != nil {
+		return nil, errors.Wrap(err, "Bucket")
+	}
+
+	sem, err := backend.NewSemaphore(cfg.Connections)
+	if err != nil {
+		return nil, err
+	}
+
+	be := &b2Backend{
+		client: client,
+		bucket: bucket,
+		cfg:    cfg,
+		Layout: &backend.DefaultLayout{
+			Join: path.Join,
+			Path: cfg.Prefix,
+		},
+		sem: sem,
+	}
+
+	return be, nil
+}
+
+// Create opens a connection to the B2 service. If the bucket does not exist yet,
+// it is created.
+func Create(cfg Config) (restic.Backend, error) {
+	debug.Log("cfg %#v", cfg)
+
+	ctx, cancel := context.WithCancel(context.TODO())
+	defer cancel()
+
+	client, err := newClient(ctx, cfg)
+	if err != nil {
+		return nil, err
+	}
+
+	attr := b2.BucketAttrs{
+		Type: b2.Private,
+	}
+	bucket, err := client.NewBucket(ctx, cfg.Bucket, &attr)
+	if err != nil {
+		return nil, errors.Wrap(err, "NewBucket")
+	}
+
+	sem, err := backend.NewSemaphore(cfg.Connections)
+	if err != nil {
+		return nil, err
+	}
+
+	be := &b2Backend{
+		client: client,
+		bucket: bucket,
+		cfg:    cfg,
+		Layout: &backend.DefaultLayout{
+			Join: path.Join,
+			Path: cfg.Prefix,
+		},
+		sem: sem,
+	}
+
+	present, err := be.Test(context.TODO(), restic.Handle{Type: restic.ConfigFile})
+	if err != nil {
+		return nil, err
+	}
+
+	if present {
+		return nil, errors.New("config already exists")
+	}
+
+	return be, nil
+}
+
+// Location returns the location for the backend.
+func (be *b2Backend) Location() string {
+	return be.cfg.Bucket
+}
+
+// wrapReader wraps an io.ReadCloser to run an additional function on Close.
+type wrapReader struct {
+	io.ReadCloser
+	eofSeen bool
+	f       func()
+}
+
+func (wr *wrapReader) Read(p []byte) (int, error) {
+	if wr.eofSeen {
+		return 0, io.EOF
+	}
+
+	n, err := wr.ReadCloser.Read(p)
+	if err == io.EOF {
+		wr.eofSeen = true
+	}
+	return n, err
+}
+
+func (wr *wrapReader) Close() error {
+	err := wr.ReadCloser.Close()
+	wr.f()
+	return err
+}
+
+// IsNotExist returns true if the error is caused by a non-existing file.
+func (be *b2Backend) IsNotExist(err error) bool {
+	return b2.IsNotExist(errors.Cause(err))
+}
+
+// Load returns the data stored in the backend for h at the given offset
+// and saves it in p. Load has the same semantics as io.ReaderAt.
+func (be *b2Backend) Load(ctx context.Context, h restic.Handle, length int, offset int64) (io.ReadCloser, error) {
+	debug.Log("Load %v, length %v, offset %v from %v", h, length, offset, be.Filename(h))
+	if err := h.Valid(); err != nil {
+		return nil, err
+	}
+
+	if offset < 0 {
+		return nil, errors.New("offset is negative")
+	}
+
+	if length < 0 {
+		return nil, errors.Errorf("invalid length %d", length)
+	}
+
+	ctx, cancel := context.WithCancel(ctx)
+
+	be.sem.GetToken()
+
+	name := be.Layout.Filename(h)
+	obj := be.bucket.Object(name)
+
+	if offset == 0 && length == 0 {
+		rd := obj.NewReader(ctx)
+		wrapper := &wrapReader{
+			ReadCloser: rd,
+			f: func() {
+				cancel()
+				be.sem.ReleaseToken()
+			},
+		}
+		return wrapper, nil
+	}
+
+	// pass a negative length to NewRangeReader so that the remainder of the
+	// file is read.
+	if length == 0 {
+		length = -1
+	}
+
+	rd := obj.NewRangeReader(ctx, offset, int64(length))
+	wrapper := &wrapReader{
+		ReadCloser: rd,
+		f: func() {
+			cancel()
+			be.sem.ReleaseToken()
+		},
+	}
+	return wrapper, nil
+}
+
+// Save stores data in the backend at the handle.
+func (be *b2Backend) Save(ctx context.Context, h restic.Handle, rd io.Reader) (err error) {
+	ctx, cancel := context.WithCancel(ctx)
+	defer cancel()
+
+	if err := h.Valid(); err != nil {
+		return err
+	}
+
+	be.sem.GetToken()
+	defer be.sem.ReleaseToken()
+
+	name := be.Filename(h)
+	debug.Log("Save %v, name %v", h, name)
+	obj := be.bucket.Object(name)
+
+	_, err = obj.Attrs(ctx)
+	if err == nil {
+		debug.Log("  %v already exists", h)
+		return errors.New("key already exists")
+	}
+
+	w := obj.NewWriter(ctx)
+	n, err := io.Copy(w, rd)
+	debug.Log("  saved %d bytes, err %v", n, err)
+
+	if err != nil {
+		_ = w.Close()
+		return errors.Wrap(err, "Copy")
+	}
+
+	return errors.Wrap(w.Close(), "Close")
+}
+
+// Stat returns information about a blob.
+func (be *b2Backend) Stat(ctx context.Context, h restic.Handle) (bi restic.FileInfo, err error) {
+	debug.Log("Stat %v", h)
+
+	be.sem.GetToken()
+	defer be.sem.ReleaseToken()
+
+	name := be.Filename(h)
+	obj := be.bucket.Object(name)
+	info, err := obj.Attrs(ctx)
+	if err != nil {
+		debug.Log("Attrs() err %v", err)
+		return restic.FileInfo{}, errors.Wrap(err, "Stat")
+	}
+	return restic.FileInfo{Size: info.Size}, nil
+}
+
+// Test returns true if a blob of the given type and name exists in the backend.
+func (be *b2Backend) Test(ctx context.Context, h restic.Handle) (bool, error) {
+	debug.Log("Test %v", h)
+
+	be.sem.GetToken()
+	defer be.sem.ReleaseToken()
+
+	found := false
+	name := be.Filename(h)
+	obj := be.bucket.Object(name)
+	info, err := obj.Attrs(ctx)
+	if err == nil && info != nil && info.Status == b2.Uploaded {
+		found = true
+	}
+	return found, nil
+}
+
+// Remove removes the blob with the given name and type.
+func (be *b2Backend) Remove(ctx context.Context, h restic.Handle) error {
+	debug.Log("Remove %v", h)
+
+	be.sem.GetToken()
+	defer be.sem.ReleaseToken()
+
+	obj := be.bucket.Object(be.Filename(h))
+	return errors.Wrap(obj.Delete(ctx), "Delete")
+}
+
+// List returns a channel that yields all names of blobs of type t. A
+// goroutine is started for this. If the channel done is closed, sending
+// stops.
+func (be *b2Backend) List(ctx context.Context, t restic.FileType) <-chan string {
+	debug.Log("List %v", t)
+	ch := make(chan string)
+
+	ctx, cancel := context.WithCancel(ctx)
+
+	be.sem.GetToken()
+
+	go func() {
+		defer close(ch)
+		defer cancel()
+		defer be.sem.ReleaseToken()
+
+		prefix := be.Dirname(restic.Handle{Type: t})
+		cur := &b2.Cursor{Prefix: prefix}
+
+		for {
+			objs, c, err := be.bucket.ListCurrentObjects(ctx, 1000, cur)
+			if err != nil && err != io.EOF {
+				return
+			}
+			for _, obj := range objs {
+				// Skip objects returned that do not have the specified prefix.
+				if !strings.HasPrefix(obj.Name(), prefix) {
+					continue
+				}
+
+				m := path.Base(obj.Name())
+				if m == "" {
+					continue
+				}
+
+				select {
+				case ch <- m:
+				case <-ctx.Done():
+					return
+				}
+			}
+			if err == io.EOF {
+				return
+			}
+			cur = c
+		}
+	}()
+
+	return ch
+}
+
+// Remove keys for a specified backend type.
+func (be *b2Backend) removeKeys(ctx context.Context, t restic.FileType) error {
+	debug.Log("removeKeys %v", t)
+	for key := range be.List(ctx, t) {
+		err := be.Remove(ctx, restic.Handle{Type: t, Name: key})
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// Delete removes all restic keys in the bucket. It will not remove the bucket itself.
+func (be *b2Backend) Delete(ctx context.Context) error {
+	alltypes := []restic.FileType{
+		restic.DataFile,
+		restic.KeyFile,
+		restic.LockFile,
+		restic.SnapshotFile,
+		restic.IndexFile}
+
+	for _, t := range alltypes {
+		err := be.removeKeys(ctx, t)
+		if err != nil {
+			return nil
+		}
+	}
+	err := be.Remove(ctx, restic.Handle{Type: restic.ConfigFile})
+	if err != nil && b2.IsNotExist(errors.Cause(err)) {
+		err = nil
+	}
+
+	return err
+}
+
+// Close does nothing
+func (be *b2Backend) Close() error { return nil }

src/restic/backend/b2/b2_test.go

@@ -0,0 +1,97 @@
+package b2_test
+
+import (
+	"context"
+	"fmt"
+	"os"
+	"testing"
+	"time"
+
+	"restic"
+	"restic/backend/b2"
+	"restic/backend/test"
+
+	. "restic/test"
+)
+
+func newB2TestSuite(t testing.TB) *test.Suite {
+	return &test.Suite{
+		// do not use excessive data
+		MinimalData: true,
+
+		// wait for at most 10 seconds for removed files to disappear
+		WaitForDelayedRemoval: 10 * time.Second,
+
+		// NewConfig returns a config for a new temporary backend that will be used in tests.
+		NewConfig: func() (interface{}, error) {
+			b2cfg, err := b2.ParseConfig(os.Getenv("RESTIC_TEST_B2_REPOSITORY"))
+			if err != nil {
+				return nil, err
+			}
+
+			cfg := b2cfg.(b2.Config)
+			cfg.AccountID = os.Getenv("RESTIC_TEST_B2_ACCOUNT_ID")
+			cfg.Key = os.Getenv("RESTIC_TEST_B2_ACCOUNT_KEY")
+			cfg.Prefix = fmt.Sprintf("test-%d", time.Now().UnixNano())
+			return cfg, nil
+		},
+
+		// CreateFn is a function that creates a temporary repository for the tests.
+		Create: func(config interface{}) (restic.Backend, error) {
+			cfg := config.(b2.Config)
+			return b2.Create(cfg)
+		},
+
+		// OpenFn is a function that opens a previously created temporary repository.
+		Open: func(config interface{}) (restic.Backend, error) {
+			cfg := config.(b2.Config)
+			return b2.Open(cfg)
+		},
+
+		// CleanupFn removes data created during the tests.
+		Cleanup: func(config interface{}) error {
+			cfg := config.(b2.Config)
+			be, err := b2.Open(cfg)
+			if err != nil {
+				return err
+			}
+
+			if err := be.(restic.Deleter).Delete(context.TODO()); err != nil {
+				return err
+			}
+
+			return nil
+		},
+	}
+}
+
+func testVars(t testing.TB) {
+	vars := []string{
+		"RESTIC_TEST_B2_ACCOUNT_ID",
+		"RESTIC_TEST_B2_ACCOUNT_KEY",
+		"RESTIC_TEST_B2_REPOSITORY",
+	}
+
+	for _, v := range vars {
+		if os.Getenv(v) == "" {
+			t.Skipf("environment variable %v not set", v)
+			return
+		}
+	}
+}
+
+func TestBackendB2(t *testing.T) {
+	defer func() {
+		if t.Skipped() {
+			SkipDisallowed(t, "restic/backend/b2.TestBackendB2")
+		}
+	}()
+
+	testVars(t)
+	newB2TestSuite(t).RunTests(t)
+}
+
+func BenchmarkBackendb2(t *testing.B) {
+	testVars(t)
+	newB2TestSuite(t).RunBenchmarks(t)
+}

src/restic/backend/b2/config.go

@@ -0,0 +1,93 @@
+package b2
+
+import (
+	"path"
+	"regexp"
+	"strings"
+
+	"restic/errors"
+	"restic/options"
+)
+
+// Config contains all configuration necessary to connect to an b2 compatible
+// server.
+type Config struct {
+	AccountID string
+	Key       string
+	Bucket    string
+	Prefix    string
+
+	Connections uint `option:"connections" help:"set a limit for the number of concurrent connections (default: 5)"`
+}
+
+// NewConfig returns a new config with default options applied.
+func NewConfig() Config {
+	return Config{
+		Connections: 5,
+	}
+}
+
+func init() {
+	options.Register("b2", Config{})
+}
+
+var bucketName = regexp.MustCompile("^[a-zA-Z0-9-]+$")
+
+// checkBucketName tests the bucket name against the rules at
+// https://help.backblaze.com/hc/en-us/articles/217666908-What-you-need-to-know-about-B2-Bucket-names
+func checkBucketName(name string) error {
+	if name == "" {
+		return errors.New("bucket name is empty")
+	}
+
+	if len(name) < 6 {
+		return errors.New("bucket name is too short")
+	}
+
+	if len(name) > 50 {
+		return errors.New("bucket name is too long")
+	}
+
+	if !bucketName.MatchString(name) {
+		return errors.New("bucket name contains invalid characters, allowed are: a-z, 0-9, dash (-)")
+	}
+
+	return nil
+}
+
+// ParseConfig parses the string s and extracts the b2 config. The supported
+// configuration format is b2:bucketname/prefix. If no prefix is given the
+// prefix "restic" will be used.
+func ParseConfig(s string) (interface{}, error) {
+	if !strings.HasPrefix(s, "b2:") {
+		return nil, errors.New("invalid format, want: b2:bucket-name[:path]")
+	}
+
+	s = s[3:]
+	data := strings.SplitN(s, ":", 2)
+	if len(data) == 0 || len(data[0]) == 0 {
+		return nil, errors.New("bucket name not found")
+	}
+
+	cfg := NewConfig()
+	cfg.Bucket = data[0]
+
+	if err := checkBucketName(cfg.Bucket); err != nil {
+		return nil, err
+	}
+
+	if len(data) == 2 {
+		p := data[1]
+		if len(p) > 0 {
+			p = path.Clean(p)
+		}
+
+		if len(p) > 0 && path.IsAbs(p) {
+			p = p[1:]
+		}
+
+		cfg.Prefix = p
+	}
+
+	return cfg, nil
+}

src/restic/backend/b2/config_test.go

@@ -0,0 +1,92 @@
+package b2
+
+import "testing"
+
+var configTests = []struct {
+	s   string
+	cfg Config
+}{
+	{"b2:bucketname", Config{
+		Bucket:      "bucketname",
+		Prefix:      "",
+		Connections: 5,
+	}},
+	{"b2:bucketname:", Config{
+		Bucket:      "bucketname",
+		Prefix:      "",
+		Connections: 5,
+	}},
+	{"b2:bucketname:/prefix/directory", Config{
+		Bucket:      "bucketname",
+		Prefix:      "prefix/directory",
+		Connections: 5,
+	}},
+	{"b2:foobar", Config{
+		Bucket:      "foobar",
+		Prefix:      "",
+		Connections: 5,
+	}},
+	{"b2:foobar:", Config{
+		Bucket:      "foobar",
+		Prefix:      "",
+		Connections: 5,
+	}},
+	{"b2:foobar:/", Config{
+		Bucket:      "foobar",
+		Prefix:      "",
+		Connections: 5,
+	}},
+}
+
+func TestParseConfig(t *testing.T) {
+	for _, test := range configTests {
+		t.Run("", func(t *testing.T) {
+			cfg, err := ParseConfig(test.s)
+			if err != nil {
+				t.Fatalf("%s failed: %v", test.s, err)
+			}
+
+			if cfg != test.cfg {
+				t.Fatalf("input: %s\n wrong config, want:\n  %#v\ngot:\n  %#v",
+					test.s, test.cfg, cfg)
+			}
+		})
+	}
+}
+
+var invalidConfigTests = []struct {
+	s   string
+	err string
+}{
+	{
+		"b2",
+		"invalid format, want: b2:bucket-name[:path]",
+	},
+	{
+		"b2:",
+		"bucket name not found",
+	},
+	{
+		"b2:bucket_name",
+		"bucket name contains invalid characters, allowed are: a-z, 0-9, dash (-)",
+	},
+	{
+		"b2:bucketname/prefix/directory/",
+		"bucket name contains invalid characters, allowed are: a-z, 0-9, dash (-)",
+	},
+}
+
+func TestInvalidConfig(t *testing.T) {
+	for _, test := range invalidConfigTests {
+		t.Run("", func(t *testing.T) {
+			cfg, err := ParseConfig(test.s)
+			if err == nil {
+				t.Fatalf("expected error not found for invalid config: %v, cfg is:\n%#v", test.s, cfg)
+			}
+
+			if err.Error() != test.err {
+				t.Fatalf("unexpected error found, want:\n  %v\ngot:\n  %v", test.err, err.Error())
+			}
+		})
+	}
+}

src/restic/backend/layout.go

@@ -17,6 +17,7 @@ type Layout interface {
 	Dirname(restic.Handle) string
 	Basedir(restic.FileType) string
 	Paths() []string
+	Name() string
 }
 
 // Filesystem is the abstraction of a file system used for a backend.

src/restic/backend/layout_default.go

@@ -1,6 +1,9 @@
 package backend
 
-import "restic"
+import (
+	"encoding/hex"
+	"restic"
+)
 
 // DefaultLayout implements the default layout for local and sftp backends, as
 // described in the Design document. The `data` directory has one level of
@@ -19,6 +22,15 @@ var defaultLayoutPaths = map[restic.FileType]string{
 	restic.KeyFile:      "keys",
 }
 
+func (l *DefaultLayout) String() string {
+	return "<DefaultLayout>"
+}
+
+// Name returns the name for this layout.
+func (l *DefaultLayout) Name() string {
+	return "default"
+}
+
 // Dirname returns the directory path for a given file type and name.
 func (l *DefaultLayout) Dirname(h restic.Handle) string {
 	p := defaultLayoutPaths[h.Type]
@@ -40,11 +52,18 @@ func (l *DefaultLayout) Filename(h restic.Handle) string {
 	return l.Join(l.Dirname(h), name)
 }
 
-// Paths returns all directory names
+// Paths returns all directory names needed for a repo.
 func (l *DefaultLayout) Paths() (dirs []string) {
 	for _, p := range defaultLayoutPaths {
 		dirs = append(dirs, l.Join(l.Path, p))
 	}
+
+	// also add subdirs
+	for i := 0; i < 256; i++ {
+		subdir := hex.EncodeToString([]byte{byte(i)})
+		dirs = append(dirs, l.Join(l.Path, defaultLayoutPaths[restic.DataFile], subdir))
+	}
+
 	return dirs
 }
 

src/restic/backend/layout_rest.go

@@ -11,6 +11,15 @@ type RESTLayout struct {
 
 var restLayoutPaths = defaultLayoutPaths
 
+func (l *RESTLayout) String() string {
+	return "<RESTLayout>"
+}
+
+// Name returns the name for this layout.
+func (l *RESTLayout) Name() string {
+	return "rest"
+}
+
 // Dirname returns the directory path for a given file type and name.
 func (l *RESTLayout) Dirname(h restic.Handle) string {
 	if h.Type == restic.ConfigFile {

src/restic/backend/layout_s3legacy.go

@@ -18,6 +18,15 @@ var s3LayoutPaths = map[restic.FileType]string{
 	restic.KeyFile:      "key",
 }
 
+func (l *S3LegacyLayout) String() string {
+	return "<S3LegacyLayout>"
+}
+
+// Name returns the name for this layout.
+func (l *S3LegacyLayout) Name() string {
+	return "s3legacy"
+}
+
 // join calls Join with the first empty elements removed.
 func (l *S3LegacyLayout) join(url string, items ...string) string {
 	for len(items) > 0 && items[0] == "" {

src/restic/backend/layout_test.go

@@ -111,6 +111,10 @@ func TestDefaultLayout(t *testing.T) {
 			filepath.Join(tempdir, "keys"),
 		}
 
+		for i := 0; i < 256; i++ {
+			want = append(want, filepath.Join(tempdir, "data", fmt.Sprintf("%02x", i)))
+		}
+
 		sort.Sort(sort.StringSlice(want))
 		sort.Sort(sort.StringSlice(dirs))
 

src/restic/backend/local/layout_test.go

@@ -1,6 +1,7 @@
 package local
 
 import (
+	"context"
 	"path/filepath"
 	"restic"
 	. "restic/test"
@@ -47,7 +48,7 @@ func TestLayout(t *testing.T) {
 			}
 
 			datafiles := make(map[string]bool)
-			for id := range be.List(restic.DataFile, nil) {
+			for id := range be.List(context.TODO(), restic.DataFile) {
 				datafiles[id] = false
 			}
 

src/restic/backend/local/local.go

@@ -1,6 +1,7 @@
 package local
 
 import (
+	"context"
 	"io"
 	"os"
 	"path/filepath"
@@ -34,6 +35,14 @@ func Open(cfg Config) (*Local, error) {
 
 	be := &Local{Config: cfg, Layout: l}
 
+	// create paths for data and refs. MkdirAll does nothing if the directory already exists.
+	for _, d := range be.Paths() {
+		err := fs.MkdirAll(d, backend.Modes.Dir)
+		if err != nil {
+			return nil, errors.Wrap(err, "MkdirAll")
+		}
+	}
+
 	return be, nil
 }
 
@@ -74,8 +83,13 @@ func (b *Local) Location() string {
 	return b.Path
 }
 
+// IsNotExist returns true if the error is caused by a non existing file.
+func (b *Local) IsNotExist(err error) bool {
+	return os.IsNotExist(errors.Cause(err))
+}
+
 // Save stores data in the backend at the handle.
-func (b *Local) Save(h restic.Handle, rd io.Reader) (err error) {
+func (b *Local) Save(ctx context.Context, h restic.Handle, rd io.Reader) (err error) {
 	debug.Log("Save %v", h)
 	if err := h.Valid(); err != nil {
 		return err
@@ -83,26 +97,8 @@ func (b *Local) Save(h restic.Handle, rd io.Reader) (err error) {
 
 	filename := b.Filename(h)
 
-	// create directories if necessary, ignore errors
-	if h.Type == restic.DataFile {
-		err = fs.MkdirAll(filepath.Dir(filename), backend.Modes.Dir)
-		if err != nil {
-			return errors.Wrap(err, "MkdirAll")
-		}
-	}
-
 	// create new file
 	f, err := fs.OpenFile(filename, os.O_CREATE|os.O_EXCL|os.O_WRONLY, backend.Modes.File)
-	if os.IsNotExist(errors.Cause(err)) {
-		// create the locks dir, then try again
-		err = fs.MkdirAll(b.Dirname(h), backend.Modes.Dir)
-		if err != nil {
-			return errors.Wrap(err, "MkdirAll")
-		}
-
-		return b.Save(h, rd)
-	}
-
 	if err != nil {
 		return errors.Wrap(err, "OpenFile")
 	}
@@ -110,12 +106,12 @@ func (b *Local) Save(h restic.Handle, rd io.Reader) (err error) {
 	// save data, then sync
 	_, err = io.Copy(f, rd)
 	if err != nil {
-		f.Close()
+		_ = f.Close()
 		return errors.Wrap(err, "Write")
 	}
 
 	if err = f.Sync(); err != nil {
-		f.Close()
+		_ = f.Close()
 		return errors.Wrap(err, "Sync")
 	}
 
@@ -136,7 +132,7 @@ func (b *Local) Save(h restic.Handle, rd io.Reader) (err error) {
 // Load returns a reader that yields the contents of the file at h at the
 // given offset. If length is nonzero, only a portion of the file is
 // returned. rd must be closed after use.
-func (b *Local) Load(h restic.Handle, length int, offset int64) (io.ReadCloser, error) {
+func (b *Local) Load(ctx context.Context, h restic.Handle, length int, offset int64) (io.ReadCloser, error) {
 	debug.Log("Load %v, length %v, offset %v", h, length, offset)
 	if err := h.Valid(); err != nil {
 		return nil, err
@@ -154,7 +150,7 @@ func (b *Local) Load(h restic.Handle, length int, offset int64) (io.ReadCloser,
 	if offset > 0 {
 		_, err = f.Seek(offset, 0)
 		if err != nil {
-			f.Close()
+			_ = f.Close()
 			return nil, err
 		}
 	}
@@ -167,7 +163,7 @@ func (b *Local) Load(h restic.Handle, length int, offset int64) (io.ReadCloser,
 }
 
 // Stat returns information about a blob.
-func (b *Local) Stat(h restic.Handle) (restic.FileInfo, error) {
+func (b *Local) Stat(ctx context.Context, h restic.Handle) (restic.FileInfo, error) {
 	debug.Log("Stat %v", h)
 	if err := h.Valid(); err != nil {
 		return restic.FileInfo{}, err
@@ -182,7 +178,7 @@ func (b *Local) Stat(h restic.Handle) (restic.FileInfo, error) {
 }
 
 // Test returns true if a blob of the given type and name exists in the backend.
-func (b *Local) Test(h restic.Handle) (bool, error) {
+func (b *Local) Test(ctx context.Context, h restic.Handle) (bool, error) {
 	debug.Log("Test %v", h)
 	_, err := fs.Stat(b.Filename(h))
 	if err != nil {
@@ -196,7 +192,7 @@ func (b *Local) Test(h restic.Handle) (bool, error) {
 }
 
 // Remove removes the blob with the given name and type.
-func (b *Local) Remove(h restic.Handle) error {
+func (b *Local) Remove(ctx context.Context, h restic.Handle) error {
 	debug.Log("Remove %v", h)
 	fn := b.Filename(h)
 
@@ -214,9 +210,8 @@ func isFile(fi os.FileInfo) bool {
 }
 
 // List returns a channel that yields all names of blobs of type t. A
-// goroutine is started for this. If the channel done is closed, sending
-// stops.
-func (b *Local) List(t restic.FileType, done <-chan struct{}) <-chan string {
+// goroutine is started for this.
+func (b *Local) List(ctx context.Context, t restic.FileType) <-chan string {
 	debug.Log("List %v", t)
 
 	ch := make(chan string)
@@ -235,7 +230,7 @@ func (b *Local) List(t restic.FileType, done <-chan struct{}) <-chan string {
 
 			select {
 			case ch <- filepath.Base(path):
-			case <-done:
+			case <-ctx.Done():
 				return err
 			}
 

src/restic/backend/location/location.go

@@ -4,10 +4,13 @@ package location
 import (
 	"strings"
 
+	"restic/backend/b2"
 	"restic/backend/local"
 	"restic/backend/rest"
 	"restic/backend/s3"
 	"restic/backend/sftp"
+	"restic/backend/swift"
+	"restic/errors"
 )
 
 // Location specifies the location of a repository, including the method of
@@ -25,12 +28,44 @@ type parser struct {
 // parsers is a list of valid config parsers for the backends. The first parser
 // is the fallback and should always be set to the local backend.
 var parsers = []parser{
+	{"b2", b2.ParseConfig},
 	{"local", local.ParseConfig},
 	{"sftp", sftp.ParseConfig},
 	{"s3", s3.ParseConfig},
+	{"swift", swift.ParseConfig},
 	{"rest", rest.ParseConfig},
 }
 
+func isPath(s string) bool {
+	if strings.HasPrefix(s, "../") || strings.HasPrefix(s, `..\`) {
+		return true
+	}
+
+	if strings.HasPrefix(s, "/") || strings.HasPrefix(s, `\`) {
+		return true
+	}
+
+	if len(s) < 3 {
+		return false
+	}
+
+	// check for drive paths
+	drive := s[0]
+	if !(drive >= 'a' && drive <= 'z') && !(drive >= 'A' && drive <= 'Z') {
+		return false
+	}
+
+	if s[1] != ':' {
+		return false
+	}
+
+	if s[2] != '\\' && s[2] != '/' {
+		return false
+	}
+
+	return true
+}
+
 // Parse extracts repository location information from the string s. If s
 // starts with a backend name followed by a colon, that backend's Parse()
 // function is called. Otherwise, the local backend is used which interprets s
@@ -52,7 +87,11 @@ func Parse(s string) (u Location, err error) {
 		return u, nil
 	}
 
-	// try again, with the local parser and the prefix "local:"
+	// if s is not a path or contains ":", it's ambiguous
+	if !isPath(s) && strings.ContainsRune(s, ':') {
+		return Location{}, errors.New("invalid backend\nIf the repo is in a local directory, you need to add a `local:` prefix")
+	}
+
 	u.Scheme = "local"
 	u.Config, err = local.ParseConfig("local:" + s)
 	if err != nil {

src/restic/backend/location/location_test.go

@@ -5,10 +5,12 @@ import (
 	"reflect"
 	"testing"
 
+	"restic/backend/b2"
 	"restic/backend/local"
 	"restic/backend/rest"
 	"restic/backend/s3"
 	"restic/backend/sftp"
+	"restic/backend/swift"
 )
 
 func parseURL(s string) *url.URL {
@@ -56,6 +58,14 @@ var parseTests = []struct {
 			},
 		},
 	},
+	{
+		"/dir1/dir2",
+		Location{Scheme: "local",
+			Config: local.Config{
+				Path: "/dir1/dir2",
+			},
+		},
+	},
 	{
 		"local:../dir1/dir2",
 		Location{Scheme: "local",
@@ -72,7 +82,46 @@ var parseTests = []struct {
 			},
 		},
 	},
-
+	{
+		"/dir1:foobar/dir2",
+		Location{Scheme: "local",
+			Config: local.Config{
+				Path: "/dir1:foobar/dir2",
+			},
+		},
+	},
+	{
+		`\dir1\foobar\dir2`,
+		Location{Scheme: "local",
+			Config: local.Config{
+				Path: `\dir1\foobar\dir2`,
+			},
+		},
+	},
+	{
+		`c:\dir1\foobar\dir2`,
+		Location{Scheme: "local",
+			Config: local.Config{
+				Path: `c:\dir1\foobar\dir2`,
+			},
+		},
+	},
+	{
+		`C:\Users\appveyor\AppData\Local\Temp\1\restic-test-879453535\repo`,
+		Location{Scheme: "local",
+			Config: local.Config{
+				Path: `C:\Users\appveyor\AppData\Local\Temp\1\restic-test-879453535\repo`,
+			},
+		},
+	},
+	{
+		`c:/dir1/foobar/dir2`,
+		Location{Scheme: "local",
+			Config: local.Config{
+				Path: `c:/dir1/foobar/dir2`,
+			},
+		},
+	},
 	{
 		"sftp:user@host:/srv/repo",
 		Location{Scheme: "sftp",
@@ -118,9 +167,10 @@ var parseTests = []struct {
 		"s3://eu-central-1/bucketname",
 		Location{Scheme: "s3",
 			Config: s3.Config{
-				Endpoint: "eu-central-1",
-				Bucket:   "bucketname",
-				Prefix:   "restic",
+				Endpoint:    "eu-central-1",
+				Bucket:      "bucketname",
+				Prefix:      "restic",
+				Connections: 5,
 			},
 		},
 	},
@@ -128,9 +178,10 @@ var parseTests = []struct {
 		"s3://hostname.foo/bucketname",
 		Location{Scheme: "s3",
 			Config: s3.Config{
-				Endpoint: "hostname.foo",
-				Bucket:   "bucketname",
-				Prefix:   "restic",
+				Endpoint:    "hostname.foo",
+				Bucket:      "bucketname",
+				Prefix:      "restic",
+				Connections: 5,
 			},
 		},
 	},
@@ -138,9 +189,10 @@ var parseTests = []struct {
 		"s3://hostname.foo/bucketname/prefix/directory",
 		Location{Scheme: "s3",
 			Config: s3.Config{
-				Endpoint: "hostname.foo",
-				Bucket:   "bucketname",
-				Prefix:   "prefix/directory",
+				Endpoint:    "hostname.foo",
+				Bucket:      "bucketname",
+				Prefix:      "prefix/directory",
+				Connections: 5,
 			},
 		},
 	},
@@ -148,9 +200,10 @@ var parseTests = []struct {
 		"s3:eu-central-1/repo",
 		Location{Scheme: "s3",
 			Config: s3.Config{
-				Endpoint: "eu-central-1",
-				Bucket:   "repo",
-				Prefix:   "restic",
+				Endpoint:    "eu-central-1",
+				Bucket:      "repo",
+				Prefix:      "restic",
+				Connections: 5,
 			},
 		},
 	},
@@ -158,9 +211,10 @@ var parseTests = []struct {
 		"s3:eu-central-1/repo/prefix/directory",
 		Location{Scheme: "s3",
 			Config: s3.Config{
-				Endpoint: "eu-central-1",
-				Bucket:   "repo",
-				Prefix:   "prefix/directory",
+				Endpoint:    "eu-central-1",
+				Bucket:      "repo",
+				Prefix:      "prefix/directory",
+				Connections: 5,
 			},
 		},
 	},
@@ -168,9 +222,10 @@ var parseTests = []struct {
 		"s3:https://hostname.foo/repo",
 		Location{Scheme: "s3",
 			Config: s3.Config{
-				Endpoint: "hostname.foo",
-				Bucket:   "repo",
-				Prefix:   "restic",
+				Endpoint:    "hostname.foo",
+				Bucket:      "repo",
+				Prefix:      "restic",
+				Connections: 5,
 			},
 		},
 	},
@@ -178,9 +233,10 @@ var parseTests = []struct {
 		"s3:https://hostname.foo/repo/prefix/directory",
 		Location{Scheme: "s3",
 			Config: s3.Config{
-				Endpoint: "hostname.foo",
-				Bucket:   "repo",
-				Prefix:   "prefix/directory",
+				Endpoint:    "hostname.foo",
+				Bucket:      "repo",
+				Prefix:      "prefix/directory",
+				Connections: 5,
 			},
 		},
 	},
@@ -188,10 +244,31 @@ var parseTests = []struct {
 		"s3:http://hostname.foo/repo",
 		Location{Scheme: "s3",
 			Config: s3.Config{
-				Endpoint: "hostname.foo",
-				Bucket:   "repo",
-				Prefix:   "restic",
-				UseHTTP:  true,
+				Endpoint:    "hostname.foo",
+				Bucket:      "repo",
+				Prefix:      "restic",
+				UseHTTP:     true,
+				Connections: 5,
+			},
+		},
+	},
+	{
+		"swift:container17:/",
+		Location{Scheme: "swift",
+			Config: swift.Config{
+				Container:   "container17",
+				Prefix:      "",
+				Connections: 5,
+			},
+		},
+	},
+	{
+		"swift:container17:/prefix97",
+		Location{Scheme: "swift",
+			Config: swift.Config{
+				Container:   "container17",
+				Prefix:      "prefix97",
+				Connections: 5,
 			},
 		},
 	},
@@ -199,7 +276,26 @@ var parseTests = []struct {
 		"rest:http://hostname.foo:1234/",
 		Location{Scheme: "rest",
 			Config: rest.Config{
-				URL: parseURL("http://hostname.foo:1234/"),
+				URL:         parseURL("http://hostname.foo:1234/"),
+				Connections: 5,
+			},
+		},
+	},
+	{
+		"b2:bucketname:/prefix", Location{Scheme: "b2",
+			Config: b2.Config{
+				Bucket:      "bucketname",
+				Prefix:      "prefix",
+				Connections: 5,
+			},
+		},
+	},
+	{
+		"b2:bucketname", Location{Scheme: "b2",
+			Config: b2.Config{
+				Bucket:      "bucketname",
+				Prefix:      "",
+				Connections: 5,
 			},
 		},
 	},
@@ -225,3 +321,19 @@ func TestParse(t *testing.T) {
 		})
 	}
 }
+
+func TestInvalidScheme(t *testing.T) {
+	var invalidSchemes = []string{
+		"foobar:xxx",
+		"foobar:/dir/dir2",
+	}
+
+	for _, s := range invalidSchemes {
+		t.Run(s, func(t *testing.T) {
+			_, err := Parse(s)
+			if err == nil {
+				t.Fatalf("error for invalid location %q not found", s)
+			}
+		})
+	}
+}

src/restic/backend/mem/mem_backend.go

@@ -2,12 +2,12 @@ package mem
 
 import (
 	"bytes"
+	"context"
 	"io"
 	"io/ioutil"
 	"restic"
 	"sync"
 
-	"restic/backend"
 	"restic/errors"
 
 	"restic/debug"
@@ -18,6 +18,8 @@ type memMap map[restic.Handle][]byte
 // make sure that MemoryBackend implements backend.Backend
 var _ restic.Backend = &MemoryBackend{}
 
+var errNotFound = errors.New("not found")
+
 // MemoryBackend is a mock backend that uses a map for storing all data in
 // memory. This should only be used for tests.
 type MemoryBackend struct {
@@ -37,7 +39,7 @@ func New() *MemoryBackend {
 }
 
 // Test returns whether a file exists.
-func (be *MemoryBackend) Test(h restic.Handle) (bool, error) {
+func (be *MemoryBackend) Test(ctx context.Context, h restic.Handle) (bool, error) {
 	be.m.Lock()
 	defer be.m.Unlock()
 
@@ -50,8 +52,13 @@ func (be *MemoryBackend) Test(h restic.Handle) (bool, error) {
 	return false, nil
 }
 
+// IsNotExist returns true if the file does not exist.
+func (be *MemoryBackend) IsNotExist(err error) bool {
+	return errors.Cause(err) == errNotFound
+}
+
 // Save adds new Data to the backend.
-func (be *MemoryBackend) Save(h restic.Handle, rd io.Reader) error {
+func (be *MemoryBackend) Save(ctx context.Context, h restic.Handle, rd io.Reader) error {
 	if err := h.Valid(); err != nil {
 		return err
 	}
@@ -81,7 +88,7 @@ func (be *MemoryBackend) Save(h restic.Handle, rd io.Reader) error {
 // Load returns a reader that yields the contents of the file at h at the
 // given offset. If length is nonzero, only a portion of the file is
 // returned. rd must be closed after use.
-func (be *MemoryBackend) Load(h restic.Handle, length int, offset int64) (io.ReadCloser, error) {
+func (be *MemoryBackend) Load(ctx context.Context, h restic.Handle, length int, offset int64) (io.ReadCloser, error) {
 	if err := h.Valid(); err != nil {
 		return nil, err
 	}
@@ -100,7 +107,7 @@ func (be *MemoryBackend) Load(h restic.Handle, length int, offset int64) (io.Rea
 	}
 
 	if _, ok := be.data[h]; !ok {
-		return nil, errors.New("no such data")
+		return nil, errNotFound
 	}
 
 	buf := be.data[h]
@@ -113,11 +120,11 @@ func (be *MemoryBackend) Load(h restic.Handle, length int, offset int64) (io.Rea
 		buf = buf[:length]
 	}
 
-	return backend.Closer{Reader: bytes.NewReader(buf)}, nil
+	return ioutil.NopCloser(bytes.NewReader(buf)), nil
 }
 
 // Stat returns information about a file in the backend.
-func (be *MemoryBackend) Stat(h restic.Handle) (restic.FileInfo, error) {
+func (be *MemoryBackend) Stat(ctx context.Context, h restic.Handle) (restic.FileInfo, error) {
 	be.m.Lock()
 	defer be.m.Unlock()
 
@@ -133,21 +140,21 @@ func (be *MemoryBackend) Stat(h restic.Handle) (restic.FileInfo, error) {
 
 	e, ok := be.data[h]
 	if !ok {
-		return restic.FileInfo{}, errors.New("no such data")
+		return restic.FileInfo{}, errNotFound
 	}
 
 	return restic.FileInfo{Size: int64(len(e))}, nil
 }
 
 // Remove deletes a file from the backend.
-func (be *MemoryBackend) Remove(h restic.Handle) error {
+func (be *MemoryBackend) Remove(ctx context.Context, h restic.Handle) error {
 	be.m.Lock()
 	defer be.m.Unlock()
 
 	debug.Log("Remove %v", h)
 
 	if _, ok := be.data[h]; !ok {
-		return errors.New("no such data")
+		return errNotFound
 	}
 
 	delete(be.data, h)
@@ -156,7 +163,7 @@ func (be *MemoryBackend) Remove(h restic.Handle) error {
 }
 
 // List returns a channel which yields entries from the backend.
-func (be *MemoryBackend) List(t restic.FileType, done <-chan struct{}) <-chan string {
+func (be *MemoryBackend) List(ctx context.Context, t restic.FileType) <-chan string {
 	be.m.Lock()
 	defer be.m.Unlock()
 
@@ -177,7 +184,7 @@ func (be *MemoryBackend) List(t restic.FileType, done <-chan struct{}) <-chan st
 		for _, id := range ids {
 			select {
 			case ch <- id:
-			case <-done:
+			case <-ctx.Done():
 				return
 			}
 		}
@@ -192,7 +199,7 @@ func (be *MemoryBackend) Location() string {
 }
 
 // Delete removes all data in the backend.
-func (be *MemoryBackend) Delete() error {
+func (be *MemoryBackend) Delete(ctx context.Context) error {
 	be.m.Lock()
 	defer be.m.Unlock()
 

src/restic/backend/mem/mem_backend_test.go

@@ -1,6 +1,7 @@
 package mem_test
 
 import (
+	"context"
 	"restic"
 	"testing"
 
@@ -25,7 +26,7 @@ func newTestSuite() *test.Suite {
 		Create: func(cfg interface{}) (restic.Backend, error) {
 			c := cfg.(*memConfig)
 			if c.be != nil {
-				ok, err := c.be.Test(restic.Handle{Type: restic.ConfigFile})
+				ok, err := c.be.Test(context.TODO(), restic.Handle{Type: restic.ConfigFile})
 				if err != nil {
 					return nil, err
 				}

src/restic/backend/rest/config.go

@@ -5,11 +5,24 @@ import (
 	"strings"
 
 	"restic/errors"
+	"restic/options"
 )
 
 // Config contains all configuration necessary to connect to a REST server.
 type Config struct {
-	URL *url.URL
+	URL         *url.URL
+	Connections uint `option:"connections" help:"set a limit for the number of concurrent connections (default: 5)"`
+}
+
+func init() {
+	options.Register("rest", Config{})
+}
+
+// NewConfig returns a new Config with the default values filled in.
+func NewConfig() Config {
+	return Config{
+		Connections: 5,
+	}
 }
 
 // ParseConfig parses the string s and extracts the REST server URL.
@@ -25,6 +38,7 @@ func ParseConfig(s string) (interface{}, error) {
 		return nil, errors.Wrap(err, "url.Parse")
 	}
 
-	cfg := Config{URL: u}
+	cfg := NewConfig()
+	cfg.URL = u
 	return cfg, nil
 }

src/restic/backend/rest/config_test.go

@@ -20,7 +20,8 @@ var configTests = []struct {
 	cfg Config
 }{
 	{"rest:http://localhost:1234", Config{
-		URL: parseURL("http://localhost:1234"),
+		URL:         parseURL("http://localhost:1234"),
+		Connections: 5,
 	}},
 }
 

src/restic/backend/rest/rest.go

@@ -1,6 +1,7 @@
 package rest
 
 import (
+	"context"
 	"encoding/json"
 	"fmt"
 	"io"
@@ -11,32 +12,32 @@ import (
 	"restic"
 	"strings"
 
+	"golang.org/x/net/context/ctxhttp"
+
 	"restic/debug"
 	"restic/errors"
 
 	"restic/backend"
 )
 
-const connLimit = 40
-
 // make sure the rest backend implements restic.Backend
 var _ restic.Backend = &restBackend{}
 
 type restBackend struct {
-	url      *url.URL
-	connChan chan struct{}
-	client   http.Client
+	url    *url.URL
+	sem    *backend.Semaphore
+	client *http.Client
 	backend.Layout
 }
 
 // Open opens the REST backend with the given config.
 func Open(cfg Config) (restic.Backend, error) {
-	connChan := make(chan struct{}, connLimit)
-	for i := 0; i < connLimit; i++ {
-		connChan <- struct{}{}
-	}
+	client := &http.Client{Transport: backend.Transport()}
 
-	client := http.Client{Transport: backend.Transport()}
+	sem, err := backend.NewSemaphore(cfg.Connections)
+	if err != nil {
+		return nil, err
+	}
 
 	// use url without trailing slash for layout
 	url := cfg.URL.String()
@@ -45,10 +46,10 @@ func Open(cfg Config) (restic.Backend, error) {
 	}
 
 	be := &restBackend{
-		url:      cfg.URL,
-		connChan: connChan,
-		client:   client,
-		Layout:   &backend.RESTLayout{URL: url, Join: path.Join},
+		url:    cfg.URL,
+		client: client,
+		Layout: &backend.RESTLayout{URL: url, Join: path.Join},
+		sem:    sem,
 	}
 
 	return be, nil
@@ -61,7 +62,7 @@ func Create(cfg Config) (restic.Backend, error) {
 		return nil, err
 	}
 
-	_, err = be.Stat(restic.Handle{Type: restic.ConfigFile})
+	_, err = be.Stat(context.TODO(), restic.Handle{Type: restic.ConfigFile})
 	if err == nil {
 		return nil, errors.Fatal("config file already exists")
 	}
@@ -99,22 +100,24 @@ func (b *restBackend) Location() string {
 }
 
 // Save stores data in the backend at the handle.
-func (b *restBackend) Save(h restic.Handle, rd io.Reader) (err error) {
+func (b *restBackend) Save(ctx context.Context, h restic.Handle, rd io.Reader) (err error) {
 	if err := h.Valid(); err != nil {
 		return err
 	}
 
-	// make sure that client.Post() cannot close the reader by wrapping it in
-	// backend.Closer, which has a noop method.
-	rd = backend.Closer{Reader: rd}
+	ctx, cancel := context.WithCancel(ctx)
+	defer cancel()
 
-	<-b.connChan
-	resp, err := b.client.Post(b.Filename(h), "binary/octet-stream", rd)
-	b.connChan <- struct{}{}
+	// make sure that client.Post() cannot close the reader by wrapping it
+	rd = ioutil.NopCloser(rd)
+
+	b.sem.GetToken()
+	resp, err := ctxhttp.Post(ctx, b.client, b.Filename(h), "binary/octet-stream", rd)
+	b.sem.ReleaseToken()
 
 	if resp != nil {
 		defer func() {
-			io.Copy(ioutil.Discard, resp.Body)
+			_, _ = io.Copy(ioutil.Discard, resp.Body)
 			e := resp.Body.Close()
 
 			if err == nil {
@@ -134,10 +137,27 @@ func (b *restBackend) Save(h restic.Handle, rd io.Reader) (err error) {
 	return nil
 }
 
+// ErrIsNotExist is returned whenever the requested file does not exist on the
+// server.
+type ErrIsNotExist struct {
+	restic.Handle
+}
+
+func (e ErrIsNotExist) Error() string {
+	return fmt.Sprintf("%v does not exist", e.Handle)
+}
+
+// IsNotExist returns true if the error was caused by a non-existing file.
+func (b *restBackend) IsNotExist(err error) bool {
+	err = errors.Cause(err)
+	_, ok := err.(ErrIsNotExist)
+	return ok
+}
+
 // Load returns a reader that yields the contents of the file at h at the
 // given offset. If length is nonzero, only a portion of the file is
 // returned. rd must be closed after use.
-func (b *restBackend) Load(h restic.Handle, length int, offset int64) (io.ReadCloser, error) {
+func (b *restBackend) Load(ctx context.Context, h restic.Handle, length int, offset int64) (io.ReadCloser, error) {
 	debug.Log("Load %v, length %v, offset %v", h, length, offset)
 	if err := h.Valid(); err != nil {
 		return nil, err
@@ -163,21 +183,25 @@ func (b *restBackend) Load(h restic.Handle, length int, offset int64) (io.ReadCl
 	req.Header.Add("Range", byteRange)
 	debug.Log("Load(%v) send range %v", h, byteRange)
 
-	<-b.connChan
-	resp, err := b.client.Do(req)
-	b.connChan <- struct{}{}
+	b.sem.GetToken()
+	resp, err := ctxhttp.Do(ctx, b.client, req)
+	b.sem.ReleaseToken()
 
 	if err != nil {
 		if resp != nil {
-			io.Copy(ioutil.Discard, resp.Body)
-			resp.Body.Close()
+			_, _ = io.Copy(ioutil.Discard, resp.Body)
+			_ = resp.Body.Close()
 		}
 		return nil, errors.Wrap(err, "client.Do")
 	}
 
+	if resp.StatusCode == http.StatusNotFound {
+		_ = resp.Body.Close()
+		return nil, ErrIsNotExist{h}
+	}
+
 	if resp.StatusCode != 200 && resp.StatusCode != 206 {
-		io.Copy(ioutil.Discard, resp.Body)
-		resp.Body.Close()
+		_ = resp.Body.Close()
 		return nil, errors.Errorf("unexpected HTTP response (%v): %v", resp.StatusCode, resp.Status)
 	}
 
@@ -185,23 +209,28 @@ func (b *restBackend) Load(h restic.Handle, length int, offset int64) (io.ReadCl
 }
 
 // Stat returns information about a blob.
-func (b *restBackend) Stat(h restic.Handle) (restic.FileInfo, error) {
+func (b *restBackend) Stat(ctx context.Context, h restic.Handle) (restic.FileInfo, error) {
 	if err := h.Valid(); err != nil {
 		return restic.FileInfo{}, err
 	}
 
-	<-b.connChan
-	resp, err := b.client.Head(b.Filename(h))
-	b.connChan <- struct{}{}
+	b.sem.GetToken()
+	resp, err := ctxhttp.Head(ctx, b.client, b.Filename(h))
+	b.sem.ReleaseToken()
 	if err != nil {
 		return restic.FileInfo{}, errors.Wrap(err, "client.Head")
 	}
 
-	io.Copy(ioutil.Discard, resp.Body)
+	_, _ = io.Copy(ioutil.Discard, resp.Body)
 	if err = resp.Body.Close(); err != nil {
 		return restic.FileInfo{}, errors.Wrap(err, "Close")
 	}
 
+	if resp.StatusCode == http.StatusNotFound {
+		_ = resp.Body.Close()
+		return restic.FileInfo{}, ErrIsNotExist{h}
+	}
+
 	if resp.StatusCode != 200 {
 		return restic.FileInfo{}, errors.Errorf("unexpected HTTP response (%v): %v", resp.StatusCode, resp.Status)
 	}
@@ -218,8 +247,8 @@ func (b *restBackend) Stat(h restic.Handle) (restic.FileInfo, error) {
 }
 
 // Test returns true if a blob of the given type and name exists in the backend.
-func (b *restBackend) Test(h restic.Handle) (bool, error) {
-	_, err := b.Stat(h)
+func (b *restBackend) Test(ctx context.Context, h restic.Handle) (bool, error) {
+	_, err := b.Stat(ctx, h)
 	if err != nil {
 		return false, nil
 	}
@@ -228,7 +257,7 @@ func (b *restBackend) Test(h restic.Handle) (bool, error) {
 }
 
 // Remove removes the blob with the given name and type.
-func (b *restBackend) Remove(h restic.Handle) error {
+func (b *restBackend) Remove(ctx context.Context, h restic.Handle) error {
 	if err := h.Valid(); err != nil {
 		return err
 	}
@@ -237,26 +266,35 @@ func (b *restBackend) Remove(h restic.Handle) error {
 	if err != nil {
 		return errors.Wrap(err, "http.NewRequest")
 	}
-	<-b.connChan
-	resp, err := b.client.Do(req)
-	b.connChan <- struct{}{}
+	b.sem.GetToken()
+	resp, err := ctxhttp.Do(ctx, b.client, req)
+	b.sem.ReleaseToken()
 
 	if err != nil {
 		return errors.Wrap(err, "client.Do")
 	}
 
+	if resp.StatusCode == http.StatusNotFound {
+		_ = resp.Body.Close()
+		return ErrIsNotExist{h}
+	}
+
 	if resp.StatusCode != 200 {
 		return errors.Errorf("blob not removed, server response: %v (%v)", resp.Status, resp.StatusCode)
 	}
 
-	io.Copy(ioutil.Discard, resp.Body)
-	return resp.Body.Close()
+	_, err = io.Copy(ioutil.Discard, resp.Body)
+	if err != nil {
+		return errors.Wrap(err, "Copy")
+	}
+
+	return errors.Wrap(resp.Body.Close(), "Close")
 }
 
 // List returns a channel that yields all names of blobs of type t. A
 // goroutine is started for this. If the channel done is closed, sending
 // stops.
-func (b *restBackend) List(t restic.FileType, done <-chan struct{}) <-chan string {
+func (b *restBackend) List(ctx context.Context, t restic.FileType) <-chan string {
 	ch := make(chan string)
 
 	url := b.Dirname(restic.Handle{Type: t})
@@ -264,13 +302,13 @@ func (b *restBackend) List(t restic.FileType, done <-chan struct{}) <-chan strin
 		url += "/"
 	}
 
-	<-b.connChan
-	resp, err := b.client.Get(url)
-	b.connChan <- struct{}{}
+	b.sem.GetToken()
+	resp, err := ctxhttp.Get(ctx, b.client, url)
+	b.sem.ReleaseToken()
 
 	if resp != nil {
 		defer func() {
-			io.Copy(ioutil.Discard, resp.Body)
+			_, _ = io.Copy(ioutil.Discard, resp.Body)
 			e := resp.Body.Close()
 
 			if err == nil {
@@ -296,7 +334,7 @@ func (b *restBackend) List(t restic.FileType, done <-chan struct{}) <-chan strin
 		for _, m := range list {
 			select {
 			case ch <- m:
-			case <-done:
+			case <-ctx.Done():
 				return
 			}
 		}

src/restic/backend/rest/rest_test.go

@@ -76,9 +76,8 @@ func newTestSuite(ctx context.Context, t testing.TB) *test.Suite {
 				t.Fatal(err)
 			}
 
-			cfg := rest.Config{
-				URL: url,
-			}
+			cfg := rest.NewConfig()
+			cfg.URL = url
 			return cfg, nil
 		},
 

src/restic/backend/s3/config.go

@@ -18,6 +18,16 @@ type Config struct {
 	Bucket        string
 	Prefix        string
 	Layout        string `option:"layout" help:"use this backend layout (default: auto-detect)"`
+
+	Connections uint `option:"connections" help:"set a limit for the number of concurrent connections (default: 5)"`
+	MaxRetries  uint `option:"retries" help:"set the number of retries attempted"`
+}
+
+// NewConfig returns a new Config with the default values filled in.
+func NewConfig() Config {
+	return Config{
+		Connections: 5,
+	}
 }
 
 func init() {
@@ -70,10 +80,10 @@ func createConfig(endpoint string, p []string, useHTTP bool) (interface{}, error
 	default:
 		prefix = path.Clean(p[1])
 	}
-	return Config{
-		Endpoint: endpoint,
-		UseHTTP:  useHTTP,
-		Bucket:   p[0],
-		Prefix:   prefix,
-	}, nil
+	cfg := NewConfig()
+	cfg.Endpoint = endpoint
+	cfg.UseHTTP = useHTTP
+	cfg.Bucket = p[0]
+	cfg.Prefix = prefix
+	return cfg, nil
 }

src/restic/backend/s3/config_test.go

@@ -7,78 +7,92 @@ var configTests = []struct {
 	cfg Config
 }{
 	{"s3://eu-central-1/bucketname", Config{
-		Endpoint: "eu-central-1",
-		Bucket:   "bucketname",
-		Prefix:   "restic",
+		Endpoint:    "eu-central-1",
+		Bucket:      "bucketname",
+		Prefix:      "restic",
+		Connections: 5,
 	}},
 	{"s3://eu-central-1/bucketname/", Config{
-		Endpoint: "eu-central-1",
-		Bucket:   "bucketname",
-		Prefix:   "restic",
+		Endpoint:    "eu-central-1",
+		Bucket:      "bucketname",
+		Prefix:      "restic",
+		Connections: 5,
 	}},
 	{"s3://eu-central-1/bucketname/prefix/directory", Config{
-		Endpoint: "eu-central-1",
-		Bucket:   "bucketname",
-		Prefix:   "prefix/directory",
+		Endpoint:    "eu-central-1",
+		Bucket:      "bucketname",
+		Prefix:      "prefix/directory",
+		Connections: 5,
 	}},
 	{"s3://eu-central-1/bucketname/prefix/directory/", Config{
-		Endpoint: "eu-central-1",
-		Bucket:   "bucketname",
-		Prefix:   "prefix/directory",
+		Endpoint:    "eu-central-1",
+		Bucket:      "bucketname",
+		Prefix:      "prefix/directory",
+		Connections: 5,
 	}},
 	{"s3:eu-central-1/foobar", Config{
-		Endpoint: "eu-central-1",
-		Bucket:   "foobar",
-		Prefix:   "restic",
+		Endpoint:    "eu-central-1",
+		Bucket:      "foobar",
+		Prefix:      "restic",
+		Connections: 5,
 	}},
 	{"s3:eu-central-1/foobar/", Config{
-		Endpoint: "eu-central-1",
-		Bucket:   "foobar",
-		Prefix:   "restic",
+		Endpoint:    "eu-central-1",
+		Bucket:      "foobar",
+		Prefix:      "restic",
+		Connections: 5,
 	}},
 	{"s3:eu-central-1/foobar/prefix/directory", Config{
-		Endpoint: "eu-central-1",
-		Bucket:   "foobar",
-		Prefix:   "prefix/directory",
+		Endpoint:    "eu-central-1",
+		Bucket:      "foobar",
+		Prefix:      "prefix/directory",
+		Connections: 5,
 	}},
 	{"s3:eu-central-1/foobar/prefix/directory/", Config{
-		Endpoint: "eu-central-1",
-		Bucket:   "foobar",
-		Prefix:   "prefix/directory",
+		Endpoint:    "eu-central-1",
+		Bucket:      "foobar",
+		Prefix:      "prefix/directory",
+		Connections: 5,
 	}},
 	{"s3:https://hostname:9999/foobar", Config{
-		Endpoint: "hostname:9999",
-		Bucket:   "foobar",
-		Prefix:   "restic",
+		Endpoint:    "hostname:9999",
+		Bucket:      "foobar",
+		Prefix:      "restic",
+		Connections: 5,
 	}},
 	{"s3:https://hostname:9999/foobar/", Config{
-		Endpoint: "hostname:9999",
-		Bucket:   "foobar",
-		Prefix:   "restic",
+		Endpoint:    "hostname:9999",
+		Bucket:      "foobar",
+		Prefix:      "restic",
+		Connections: 5,
 	}},
 	{"s3:http://hostname:9999/foobar", Config{
-		Endpoint: "hostname:9999",
-		Bucket:   "foobar",
-		Prefix:   "restic",
-		UseHTTP:  true,
+		Endpoint:    "hostname:9999",
+		Bucket:      "foobar",
+		Prefix:      "restic",
+		UseHTTP:     true,
+		Connections: 5,
 	}},
 	{"s3:http://hostname:9999/foobar/", Config{
-		Endpoint: "hostname:9999",
-		Bucket:   "foobar",
-		Prefix:   "restic",
-		UseHTTP:  true,
+		Endpoint:    "hostname:9999",
+		Bucket:      "foobar",
+		Prefix:      "restic",
+		UseHTTP:     true,
+		Connections: 5,
 	}},
 	{"s3:http://hostname:9999/bucket/prefix/directory", Config{
-		Endpoint: "hostname:9999",
-		Bucket:   "bucket",
-		Prefix:   "prefix/directory",
-		UseHTTP:  true,
+		Endpoint:    "hostname:9999",
+		Bucket:      "bucket",
+		Prefix:      "prefix/directory",
+		UseHTTP:     true,
+		Connections: 5,
 	}},
 	{"s3:http://hostname:9999/bucket/prefix/directory/", Config{
-		Endpoint: "hostname:9999",
-		Bucket:   "bucket",
-		Prefix:   "prefix/directory",
-		UseHTTP:  true,
+		Endpoint:    "hostname:9999",
+		Bucket:      "bucket",
+		Prefix:      "prefix/directory",
+		UseHTTP:     true,
+		Connections: 5,
 	}},
 }
 

src/restic/backend/s3/s3.go

@@ -1,53 +1,74 @@
 package s3
 
 import (
+	"context"
 	"fmt"
 	"io"
+	"io/ioutil"
 	"os"
 	"path"
 	"restic"
 	"strings"
-	"sync"
 	"time"
 
 	"restic/backend"
 	"restic/errors"
 
 	"github.com/minio/minio-go"
+	"github.com/minio/minio-go/pkg/credentials"
 
 	"restic/debug"
 )
 
-const connLimit = 10
-
-// s3 is a backend which stores the data on an S3 endpoint.
-type s3 struct {
-	client       *minio.Client
-	connChan     chan struct{}
-	bucketname   string
-	prefix       string
-	cacheMutex   sync.RWMutex
-	cacheObjSize map[string]int64
+// Backend stores data on an S3 endpoint.
+type Backend struct {
+	client *minio.Client
+	sem    *backend.Semaphore
+	cfg    Config
 	backend.Layout
 }
 
-const defaultLayout = "s3legacy"
+// make sure that *Backend implements backend.Backend
+var _ restic.Backend = &Backend{}
 
-// Open opens the S3 backend at bucket and region. The bucket is created if it
-// does not exist yet.
-func Open(cfg Config) (restic.Backend, error) {
+const defaultLayout = "default"
+
+func open(cfg Config) (*Backend, error) {
 	debug.Log("open, config %#v", cfg)
 
-	client, err := minio.New(cfg.Endpoint, cfg.KeyID, cfg.Secret, !cfg.UseHTTP)
-	if err != nil {
-		return nil, errors.Wrap(err, "minio.New")
+	if cfg.MaxRetries > 0 {
+		minio.MaxRetry = int(cfg.MaxRetries)
 	}
 
-	be := &s3{
-		client:       client,
-		bucketname:   cfg.Bucket,
-		prefix:       cfg.Prefix,
-		cacheObjSize: make(map[string]int64),
+	var client *minio.Client
+	var err error
+
+	if cfg.KeyID == "" || cfg.Secret == "" {
+		debug.Log("key/secret not found, trying to get them from IAM")
+		creds := credentials.NewIAM("")
+		client, err = minio.NewWithCredentials(cfg.Endpoint, creds, !cfg.UseHTTP, "")
+
+		if err != nil {
+			return nil, errors.Wrap(err, "minio.NewWithCredentials")
+		}
+	} else {
+		debug.Log("key/secret found")
+		client, err = minio.New(cfg.Endpoint, cfg.KeyID, cfg.Secret, !cfg.UseHTTP)
+
+		if err != nil {
+			return nil, errors.Wrap(err, "minio.New")
+		}
+	}
+
+	sem, err := backend.NewSemaphore(cfg.Connections)
+	if err != nil {
+		return nil, err
+	}
+
+	be := &Backend{
+		client: client,
+		sem:    sem,
+		cfg:    cfg,
 	}
 
 	client.SetCustomTransport(backend.Transport())
@@ -59,9 +80,23 @@ func Open(cfg Config) (restic.Backend, error) {
 
 	be.Layout = l
 
-	be.createConnections()
+	return be, nil
+}
 
-	found, err := client.BucketExists(cfg.Bucket)
+// Open opens the S3 backend at bucket and region. The bucket is created if it
+// does not exist yet.
+func Open(cfg Config) (restic.Backend, error) {
+	return open(cfg)
+}
+
+// Create opens the S3 backend at bucket and region and creates the bucket if
+// it does not exist yet.
+func Create(cfg Config) (restic.Backend, error) {
+	be, err := open(cfg)
+	if err != nil {
+		return nil, errors.Wrap(err, "open")
+	}
+	found, err := be.client.BucketExists(cfg.Bucket)
 	if err != nil {
 		debug.Log("BucketExists(%v) returned err %v", cfg.Bucket, err)
 		return nil, errors.Wrap(err, "client.BucketExists")
@@ -69,7 +104,7 @@ func Open(cfg Config) (restic.Backend, error) {
 
 	if !found {
 		// create new bucket with default ACL in default region
-		err = client.MakeBucket(cfg.Bucket, "")
+		err = be.client.MakeBucket(cfg.Bucket, "")
 		if err != nil {
 			return nil, errors.Wrap(err, "client.MakeBucket")
 		}
@@ -78,21 +113,22 @@ func Open(cfg Config) (restic.Backend, error) {
 	return be, nil
 }
 
-func (be *s3) createConnections() {
-	be.connChan = make(chan struct{}, connLimit)
-	for i := 0; i < connLimit; i++ {
-		be.connChan <- struct{}{}
-	}
-}
-
 // IsNotExist returns true if the error is caused by a not existing file.
-func (be *s3) IsNotExist(err error) bool {
+func (be *Backend) IsNotExist(err error) bool {
 	debug.Log("IsNotExist(%T, %#v)", err, err)
-	return os.IsNotExist(err)
+	if os.IsNotExist(errors.Cause(err)) {
+		return true
+	}
+
+	if e, ok := errors.Cause(err).(minio.ErrorResponse); ok && e.Code == "NoSuchKey" {
+		return true
+	}
+
+	return false
 }
 
 // Join combines path components with slashes.
-func (be *s3) Join(p ...string) string {
+func (be *Backend) Join(p ...string) string {
 	return path.Join(p...)
 }
 
@@ -112,7 +148,7 @@ func (fi fileInfo) IsDir() bool        { return fi.isDir }   // abbreviation for
 func (fi fileInfo) Sys() interface{}   { return nil }        // underlying data source (can return nil)
 
 // ReadDir returns the entries for a directory.
-func (be *s3) ReadDir(dir string) (list []os.FileInfo, err error) {
+func (be *Backend) ReadDir(dir string) (list []os.FileInfo, err error) {
 	debug.Log("ReadDir(%v)", dir)
 
 	// make sure dir ends with a slash
@@ -123,7 +159,7 @@ func (be *s3) ReadDir(dir string) (list []os.FileInfo, err error) {
 	done := make(chan struct{})
 	defer close(done)
 
-	for obj := range be.client.ListObjects(be.bucketname, dir, false, done) {
+	for obj := range be.client.ListObjects(be.cfg.Bucket, dir, false, done) {
 		if obj.Key == "" {
 			continue
 		}
@@ -153,93 +189,41 @@ func (be *s3) ReadDir(dir string) (list []os.FileInfo, err error) {
 }
 
 // Location returns this backend's location (the bucket name).
-func (be *s3) Location() string {
-	return be.bucketname
+func (be *Backend) Location() string {
+	return be.Join(be.cfg.Bucket, be.cfg.Prefix)
 }
 
-// getRemainingSize returns number of bytes remaining. If it is not possible to
-// determine the size, panic() is called.
-func getRemainingSize(rd io.Reader) (size int64, err error) {
-	type Sizer interface {
-		Size() int64
-	}
-
-	type Lenner interface {
-		Len() int
-	}
-
-	if r, ok := rd.(Lenner); ok {
-		size = int64(r.Len())
-	} else if r, ok := rd.(Sizer); ok {
-		size = r.Size()
-	} else if f, ok := rd.(*os.File); ok {
-		fi, err := f.Stat()
-		if err != nil {
-			return 0, err
-		}
-
-		pos, err := f.Seek(0, io.SeekCurrent)
-		if err != nil {
-			return 0, err
-		}
-
-		size = fi.Size() - pos
-	} else {
-		panic(fmt.Sprintf("Save() got passed a reader without a method to determine the data size, type is %T", rd))
-	}
-	return size, nil
-}
-
-// preventCloser wraps an io.Reader to run a function instead of the original Close() function.
-type preventCloser struct {
-	io.Reader
-	f func()
-}
-
-func (wr preventCloser) Close() error {
-	wr.f()
-	return nil
+// Path returns the path in the bucket that is used for this backend.
+func (be *Backend) Path() string {
+	return be.cfg.Prefix
 }
 
 // Save stores data in the backend at the handle.
-func (be *s3) Save(h restic.Handle, rd io.Reader) (err error) {
+func (be *Backend) Save(ctx context.Context, h restic.Handle, rd io.Reader) (err error) {
+	debug.Log("Save %v", h)
+
 	if err := h.Valid(); err != nil {
 		return err
 	}
 
 	objName := be.Filename(h)
-	size, err := getRemainingSize(rd)
-	if err != nil {
-		return err
-	}
-
-	debug.Log("Save %v at %v", h, objName)
 
 	// Check key does not already exist
-	_, err = be.client.StatObject(be.bucketname, objName)
+	_, err = be.client.StatObject(be.cfg.Bucket, objName)
 	if err == nil {
 		debug.Log("%v already exists", h)
 		return errors.New("key already exists")
 	}
 
-	<-be.connChan
+	// prevent the HTTP client from closing a file
+	rd = ioutil.NopCloser(rd)
 
-	// wrap the reader so that net/http client cannot close the reader, return
-	// the token instead.
-	rd = preventCloser{
-		Reader: rd,
-		f: func() {
-			debug.Log("Close()")
-		},
-	}
+	be.sem.GetToken()
+	debug.Log("PutObject(%v, %v)", be.cfg.Bucket, objName)
+	n, err := be.client.PutObject(be.cfg.Bucket, objName, rd, "application/octet-stream")
+	be.sem.ReleaseToken()
 
-	debug.Log("PutObject(%v, %v)", be.bucketname, objName)
-	coreClient := minio.Core{be.client}
-	info, err := coreClient.PutObject(be.bucketname, objName, size, rd, nil, nil, nil)
-
-	// return token
-	be.connChan <- struct{}{}
-	debug.Log("%v -> %v bytes, err %#v", objName, info.Size, err)
+	debug.Log("%v -> %v bytes, err %#v: %v", objName, n, err, err)
 
 	return errors.Wrap(err, "client.PutObject")
 }
@@ -259,7 +243,7 @@ func (wr wrapReader) Close() error {
 // Load returns a reader that yields the contents of the file at h at the
 // given offset. If length is nonzero, only a portion of the file is
 // returned. rd must be closed after use.
-func (be *s3) Load(h restic.Handle, length int, offset int64) (io.ReadCloser, error) {
+func (be *Backend) Load(ctx context.Context, h restic.Handle, length int, offset int64) (io.ReadCloser, error) {
 	debug.Log("Load %v, length %v, offset %v from %v", h, length, offset, be.Filename(h))
 	if err := h.Valid(); err != nil {
 		return nil, err
@@ -275,22 +259,20 @@ func (be *s3) Load(h restic.Handle, length int, offset int64) (io.ReadCloser, er
 
 	objName := be.Filename(h)
 
-	// get token for connection
-	<-be.connChan
-
 	byteRange := fmt.Sprintf("bytes=%d-", offset)
 	if length > 0 {
 		byteRange = fmt.Sprintf("bytes=%d-%d", offset, offset+int64(length)-1)
 	}
 	headers := minio.NewGetReqHeaders()
 	headers.Add("Range", byteRange)
+
+	be.sem.GetToken()
 	debug.Log("Load(%v) send range %v", h, byteRange)
 
-	coreClient := minio.Core{be.client}
-	rd, _, err := coreClient.GetObject(be.bucketname, objName, headers)
+	coreClient := minio.Core{Client: be.client}
+	rd, _, err := coreClient.GetObject(be.cfg.Bucket, objName, headers)
 	if err != nil {
-		// return token
-		be.connChan <- struct{}{}
+		be.sem.ReleaseToken()
 		return nil, err
 	}
 
@@ -298,8 +280,7 @@ func (be *s3) Load(h restic.Handle, length int, offset int64) (io.ReadCloser, er
 		ReadCloser: rd,
 		f: func() {
 			debug.Log("Close()")
-			// return token
-			be.connChan <- struct{}{}
+			be.sem.ReleaseToken()
 		},
 	}
 
@@ -307,13 +288,13 @@ func (be *s3) Load(h restic.Handle, length int, offset int64) (io.ReadCloser, er
 }
 
 // Stat returns information about a blob.
-func (be *s3) Stat(h restic.Handle) (bi restic.FileInfo, err error) {
+func (be *Backend) Stat(ctx context.Context, h restic.Handle) (bi restic.FileInfo, err error) {
 	debug.Log("%v", h)
 
 	objName := be.Filename(h)
 	var obj *minio.Object
 
-	obj, err = be.client.GetObject(be.bucketname, objName)
+	obj, err = be.client.GetObject(be.cfg.Bucket, objName)
 	if err != nil {
 		debug.Log("GetObject() err %v", err)
 		return restic.FileInfo{}, errors.Wrap(err, "client.GetObject")
@@ -337,10 +318,10 @@ func (be *s3) Stat(h restic.Handle) (bi restic.FileInfo, err error) {
 }
 
 // Test returns true if a blob of the given type and name exists in the backend.
-func (be *s3) Test(h restic.Handle) (bool, error) {
+func (be *Backend) Test(ctx context.Context, h restic.Handle) (bool, error) {
 	found := false
 	objName := be.Filename(h)
-	_, err := be.client.StatObject(be.bucketname, objName)
+	_, err := be.client.StatObject(be.cfg.Bucket, objName)
 	if err == nil {
 		found = true
 	}
@@ -350,17 +331,22 @@ func (be *s3) Test(h restic.Handle) (bool, error) {
 }
 
 // Remove removes the blob with the given name and type.
-func (be *s3) Remove(h restic.Handle) error {
+func (be *Backend) Remove(ctx context.Context, h restic.Handle) error {
 	objName := be.Filename(h)
-	err := be.client.RemoveObject(be.bucketname, objName)
+	err := be.client.RemoveObject(be.cfg.Bucket, objName)
 	debug.Log("Remove(%v) at %v -> err %v", h, objName, err)
+
+	if be.IsNotExist(err) {
+		err = nil
+	}
+
 	return errors.Wrap(err, "client.RemoveObject")
 }
 
 // List returns a channel that yields all names of blobs of type t. A
 // goroutine is started for this. If the channel done is closed, sending
 // stops.
-func (be *s3) List(t restic.FileType, done <-chan struct{}) <-chan string {
+func (be *Backend) List(ctx context.Context, t restic.FileType) <-chan string {
 	debug.Log("listing %v", t)
 	ch := make(chan string)
 
@@ -371,7 +357,7 @@ func (be *s3) List(t restic.FileType, done <-chan struct{}) <-chan string {
 		prefix += "/"
 	}
 
-	listresp := be.client.ListObjects(be.bucketname, prefix, true, done)
+	listresp := be.client.ListObjects(be.cfg.Bucket, prefix, true, ctx.Done())
 
 	go func() {
 		defer close(ch)
@@ -383,7 +369,7 @@ func (be *s3) List(t restic.FileType, done <-chan struct{}) <-chan string {
 
 			select {
 			case ch <- path.Base(m):
-			case <-done:
+			case <-ctx.Done():
 				return
 			}
 		}
@@ -393,11 +379,9 @@ func (be *s3) List(t restic.FileType, done <-chan struct{}) <-chan string {
 }
 
 // Remove keys for a specified backend type.
-func (be *s3) removeKeys(t restic.FileType) error {
-	done := make(chan struct{})
-	defer close(done)
-	for key := range be.List(restic.DataFile, done) {
-		err := be.Remove(restic.Handle{Type: restic.DataFile, Name: key})
+func (be *Backend) removeKeys(ctx context.Context, t restic.FileType) error {
+	for key := range be.List(ctx, restic.DataFile) {
+		err := be.Remove(ctx, restic.Handle{Type: restic.DataFile, Name: key})
 		if err != nil {
 			return err
 		}
@@ -407,7 +391,7 @@ func (be *s3) removeKeys(t restic.FileType) error {
 }
 
 // Delete removes all restic keys in the bucket. It will not remove the bucket itself.
-func (be *s3) Delete() error {
+func (be *Backend) Delete(ctx context.Context) error {
 	alltypes := []restic.FileType{
 		restic.DataFile,
 		restic.KeyFile,
@@ -416,14 +400,48 @@ func (be *s3) Delete() error {
 		restic.IndexFile}
 
 	for _, t := range alltypes {
-		err := be.removeKeys(t)
+		err := be.removeKeys(ctx, t)
 		if err != nil {
 			return nil
 		}
 	}
 
-	return be.Remove(restic.Handle{Type: restic.ConfigFile})
+	return be.Remove(ctx, restic.Handle{Type: restic.ConfigFile})
 }
 
 // Close does nothing
-func (be *s3) Close() error { return nil }
+func (be *Backend) Close() error { return nil }
+
+// Rename moves a file based on the new layout l.
+func (be *Backend) Rename(h restic.Handle, l backend.Layout) error {
+	debug.Log("Rename %v to %v", h, l)
+	oldname := be.Filename(h)
+	newname := l.Filename(h)
+
+	if oldname == newname {
+		debug.Log("  %v is already renamed", newname)
+		return nil
+	}
+
+	debug.Log("  %v -> %v", oldname, newname)
+
+	src := minio.NewSourceInfo(be.cfg.Bucket, oldname, nil)
+
+	dst, err := minio.NewDestinationInfo(be.cfg.Bucket, newname, nil, nil)
+	if err != nil {
+		return errors.Wrap(err, "NewDestinationInfo")
+	}
+
+	err = be.client.CopyObject(dst, src)
+	if err != nil && be.IsNotExist(err) {
+		debug.Log("copy failed: %v, seems to already have been renamed", err)
+		return nil
+	}
+
+	if err != nil {
+		debug.Log("copy failed: %v", err)
+		return err
+	}
+
+	return be.client.RemoveObject(be.cfg.Bucket, oldname)
+}

src/restic/backend/s3/s3_internal_test.go

@@ -1,71 +0,0 @@
-package s3
-
-import (
-	"bytes"
-	"io"
-	"io/ioutil"
-	"os"
-	"restic/test"
-	"testing"
-)
-
-func writeFile(t testing.TB, data []byte, offset int64) *os.File {
-	tempfile, err := ioutil.TempFile("", "restic-test-")
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	if _, err = tempfile.Write(data); err != nil {
-		t.Fatal(err)
-	}
-
-	if _, err = tempfile.Seek(offset, io.SeekStart); err != nil {
-		t.Fatal(err)
-	}
-
-	return tempfile
-}
-
-func TestGetRemainingSize(t *testing.T) {
-	length := 18 * 1123
-	partialRead := 1005
-
-	data := test.Random(23, length)
-
-	partReader := bytes.NewReader(data)
-	buf := make([]byte, partialRead)
-	_, _ = io.ReadFull(partReader, buf)
-
-	partFileReader := writeFile(t, data, int64(partialRead))
-	defer func() {
-		if err := partFileReader.Close(); err != nil {
-			t.Fatal(err)
-		}
-
-		if err := os.Remove(partFileReader.Name()); err != nil {
-			t.Fatal(err)
-		}
-	}()
-
-	var tests = []struct {
-		io.Reader
-		size int64
-	}{
-		{bytes.NewReader([]byte("foobar test")), 11},
-		{partReader, int64(length - partialRead)},
-		{partFileReader, int64(length - partialRead)},
-	}
-
-	for _, test := range tests {
-		t.Run("", func(t *testing.T) {
-			size, err := getRemainingSize(test.Reader)
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			if size != test.size {
-				t.Fatalf("invalid size returned, want %v, got %v", test.size, size)
-			}
-		})
-	}
-}

src/restic/backend/s3/s3_test.go

@@ -49,17 +49,16 @@ func runMinio(ctx context.Context, t testing.TB, dir, key, secret string) func()
 
 	// wait until the TCP port is reachable
 	var success bool
-	for i := 0; i < 10; i++ {
+	for i := 0; i < 100; i++ {
 		time.Sleep(200 * time.Millisecond)
 
 		c, err := net.Dial("tcp", "localhost:9000")
-		if err != nil {
-			continue
-		}
-
-		success = true
-		if err := c.Close(); err != nil {
-			t.Fatal(err)
+		if err == nil {
+			success = true
+			if err := c.Close(); err != nil {
+				t.Fatal(err)
+			}
+			break
 		}
 	}
 
@@ -104,6 +103,21 @@ type MinioTestConfig struct {
 	stopServer    func()
 }
 
+func createS3(t testing.TB, cfg MinioTestConfig) (be restic.Backend, err error) {
+	for i := 0; i < 10; i++ {
+		be, err = s3.Create(cfg.Config)
+		if err != nil {
+			t.Logf("s3 open: try %d: error %v", i, err)
+			time.Sleep(500 * time.Millisecond)
+			continue
+		}
+
+		break
+	}
+
+	return be, err
+}
+
 func newMinioTestSuite(ctx context.Context, t testing.TB) *test.Suite {
 	return &test.Suite{
 		// NewConfig returns a config for a new temporary backend that will be used in tests.
@@ -114,14 +128,13 @@ func newMinioTestSuite(ctx context.Context, t testing.TB) *test.Suite {
 			key, secret := newRandomCredentials(t)
 			cfg.stopServer = runMinio(ctx, t, cfg.tempdir, key, secret)
 
-			cfg.Config = s3.Config{
-				Endpoint: "localhost:9000",
-				Bucket:   "restictestbucket",
-				Prefix:   fmt.Sprintf("test-%d", time.Now().UnixNano()),
-				UseHTTP:  true,
-				KeyID:    key,
-				Secret:   secret,
-			}
+			cfg.Config = s3.NewConfig()
+			cfg.Config.Endpoint = "localhost:9000"
+			cfg.Config.Bucket = "restictestbucket"
+			cfg.Config.Prefix = fmt.Sprintf("test-%d", time.Now().UnixNano())
+			cfg.Config.UseHTTP = true
+			cfg.Config.KeyID = key
+			cfg.Config.Secret = secret
 			return cfg, nil
 		},
 
@@ -129,12 +142,12 @@ func newMinioTestSuite(ctx context.Context, t testing.TB) *test.Suite {
 		Create: func(config interface{}) (restic.Backend, error) {
 			cfg := config.(MinioTestConfig)
 
-			be, err := s3.Open(cfg.Config)
+			be, err := createS3(t, cfg)
 			if err != nil {
 				return nil, err
 			}
 
-			exists, err := be.Test(restic.Handle{Type: restic.ConfigFile})
+			exists, err := be.Test(context.TODO(), restic.Handle{Type: restic.ConfigFile})
 			if err != nil {
 				return nil, err
 			}
@@ -223,12 +236,12 @@ func newS3TestSuite(t testing.TB) *test.Suite {
 		Create: func(config interface{}) (restic.Backend, error) {
 			cfg := config.(s3.Config)
 
-			be, err := s3.Open(cfg)
+			be, err := s3.Create(cfg)
 			if err != nil {
 				return nil, err
 			}
 
-			exists, err := be.Test(restic.Handle{Type: restic.ConfigFile})
+			exists, err := be.Test(context.TODO(), restic.Handle{Type: restic.ConfigFile})
 			if err != nil {
 				return nil, err
 			}
@@ -255,7 +268,7 @@ func newS3TestSuite(t testing.TB) *test.Suite {
 				return err
 			}
 
-			if err := be.(restic.Deleter).Delete(); err != nil {
+			if err := be.(restic.Deleter).Delete(context.TODO()); err != nil {
 				return err
 			}
 

src/restic/backend/semaphore.go

@@ -0,0 +1,28 @@
+package backend
+
+import "restic/errors"
+
+// Semaphore limits access to a restricted resource.
+type Semaphore struct {
+	ch chan struct{}
+}
+
+// NewSemaphore returns a new semaphore with capacity n.
+func NewSemaphore(n uint) (*Semaphore, error) {
+	if n <= 0 {
+		return nil, errors.New("must be a positive number")
+	}
+	return &Semaphore{
+		ch: make(chan struct{}, n),
+	}, nil
+}
+
+// GetToken blocks until a Token is available.
+func (s *Semaphore) GetToken() {
+	s.ch <- struct{}{}
+}
+
+// ReleaseToken returns a token.
+func (s *Semaphore) ReleaseToken() {
+	<-s.ch
+}

src/restic/backend/sftp/layout_test.go

@@ -1,6 +1,7 @@
 package sftp_test
 
 import (
+	"context"
 	"fmt"
 	"path/filepath"
 	"restic"
@@ -54,7 +55,7 @@ func TestLayout(t *testing.T) {
 			}
 
 			datafiles := make(map[string]bool)
-			for id := range be.List(restic.DataFile, nil) {
+			for id := range be.List(context.TODO(), restic.DataFile) {
 				datafiles[id] = false
 			}
 

src/restic/backend/sftp/sftp.go

@@ -2,6 +2,7 @@ package sftp
 
 import (
 	"bufio"
+	"context"
 	"fmt"
 	"io"
 	"os"
@@ -125,11 +126,56 @@ func Open(cfg Config) (*SFTP, error) {
 
 	debug.Log("layout: %v\n", sftp.Layout)
 
+	if err := sftp.checkDataSubdirs(); err != nil {
+		debug.Log("checkDataSubdirs returned %v", err)
+		return nil, err
+	}
+
 	sftp.Config = cfg
 	sftp.p = cfg.Path
 	return sftp, nil
 }
 
+func (r *SFTP) checkDataSubdirs() error {
+	datadir := r.Dirname(restic.Handle{Type: restic.DataFile})
+
+	// check if all paths for data/ exist
+	entries, err := r.c.ReadDir(datadir)
+	if err != nil {
+		return err
+	}
+
+	subdirs := make(map[string]struct{}, len(entries))
+	for _, entry := range entries {
+		subdirs[entry.Name()] = struct{}{}
+	}
+
+	for i := 0; i < 256; i++ {
+		subdir := fmt.Sprintf("%02x", i)
+		if _, ok := subdirs[subdir]; !ok {
+			debug.Log("subdir %v is missing, creating", subdir)
+			err := r.mkdirAll(path.Join(datadir, subdir), backend.Modes.Dir)
+			if err != nil {
+				return err
+			}
+		}
+	}
+
+	return nil
+}
+
+func (r *SFTP) mkdirAllDataSubdirs() error {
+	for _, d := range r.Paths() {
+		err := r.mkdirAll(d, backend.Modes.Dir)
+		debug.Log("mkdirAll %v -> %v", d, err)
+		if err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
 // Join combines path components with slashes (according to the sftp spec).
 func (r *SFTP) Join(p ...string) string {
 	return path.Join(p...)
@@ -201,12 +247,8 @@ func Create(cfg Config) (*SFTP, error) {
 	}
 
 	// create paths for data and refs
-	for _, d := range sftp.Paths() {
-		err = sftp.mkdirAll(d, backend.Modes.Dir)
-		debug.Log("mkdirAll %v -> %v", d, err)
-		if err != nil {
-			return nil, err
-		}
+	if err = sftp.mkdirAllDataSubdirs(); err != nil {
+		return nil, err
 	}
 
 	err = sftp.Close()
@@ -262,7 +304,7 @@ func Join(parts ...string) string {
 }
 
 // Save stores data in the backend at the handle.
-func (r *SFTP) Save(h restic.Handle, rd io.Reader) (err error) {
+func (r *SFTP) Save(ctx context.Context, h restic.Handle, rd io.Reader) (err error) {
 	debug.Log("Save %v", h)
 	if err := r.clientError(); err != nil {
 		return err
@@ -283,7 +325,7 @@ func (r *SFTP) Save(h restic.Handle, rd io.Reader) (err error) {
 			return errors.Wrap(err, "MkdirAll")
 		}
 
-		return r.Save(h, rd)
+		return r.Save(ctx, h, rd)
 	}
 
 	if err != nil {
@@ -315,7 +357,7 @@ func (r *SFTP) Save(h restic.Handle, rd io.Reader) (err error) {
 // Load returns a reader that yields the contents of the file at h at the
 // given offset. If length is nonzero, only a portion of the file is
 // returned. rd must be closed after use.
-func (r *SFTP) Load(h restic.Handle, length int, offset int64) (io.ReadCloser, error) {
+func (r *SFTP) Load(ctx context.Context, h restic.Handle, length int, offset int64) (io.ReadCloser, error) {
 	debug.Log("Load %v, length %v, offset %v", h, length, offset)
 	if err := h.Valid(); err != nil {
 		return nil, err
@@ -346,7 +388,7 @@ func (r *SFTP) Load(h restic.Handle, length int, offset int64) (io.ReadCloser, e
 }
 
 // Stat returns information about a blob.
-func (r *SFTP) Stat(h restic.Handle) (restic.FileInfo, error) {
+func (r *SFTP) Stat(ctx context.Context, h restic.Handle) (restic.FileInfo, error) {
 	debug.Log("Stat(%v)", h)
 	if err := r.clientError(); err != nil {
 		return restic.FileInfo{}, err
@@ -365,7 +407,7 @@ func (r *SFTP) Stat(h restic.Handle) (restic.FileInfo, error) {
 }
 
 // Test returns true if a blob of the given type and name exists in the backend.
-func (r *SFTP) Test(h restic.Handle) (bool, error) {
+func (r *SFTP) Test(ctx context.Context, h restic.Handle) (bool, error) {
 	debug.Log("Test(%v)", h)
 	if err := r.clientError(); err != nil {
 		return false, err
@@ -384,7 +426,7 @@ func (r *SFTP) Test(h restic.Handle) (bool, error) {
 }
 
 // Remove removes the content stored at name.
-func (r *SFTP) Remove(h restic.Handle) error {
+func (r *SFTP) Remove(ctx context.Context, h restic.Handle) error {
 	debug.Log("Remove(%v)", h)
 	if err := r.clientError(); err != nil {
 		return err
@@ -396,7 +438,7 @@ func (r *SFTP) Remove(h restic.Handle) error {
 // List returns a channel that yields all names of blobs of type t. A
 // goroutine is started for this. If the channel done is closed, sending
 // stops.
-func (r *SFTP) List(t restic.FileType, done <-chan struct{}) <-chan string {
+func (r *SFTP) List(ctx context.Context, t restic.FileType) <-chan string {
 	debug.Log("List %v", t)
 
 	ch := make(chan string)
@@ -416,7 +458,7 @@ func (r *SFTP) List(t restic.FileType, done <-chan struct{}) <-chan string {
 
 			select {
 			case ch <- path.Base(walker.Path()):
-			case <-done:
+			case <-ctx.Done():
 				return
 			}
 		}

src/restic/backend/swift/config.go

@@ -0,0 +1,109 @@
+package swift
+
+import (
+	"os"
+	"restic/errors"
+	"restic/options"
+	"strings"
+)
+
+// Config contains basic configuration needed to specify swift location for a swift server
+type Config struct {
+	UserName     string
+	Domain       string
+	APIKey       string
+	AuthURL      string
+	Region       string
+	Tenant       string
+	TenantID     string
+	TenantDomain string
+	TrustID      string
+
+	StorageURL string
+	AuthToken  string
+
+	Container              string
+	Prefix                 string
+	DefaultContainerPolicy string
+
+	Connections uint `option:"connections" help:"set a limit for the number of concurrent connections (default: 5)"`
+}
+
+func init() {
+	options.Register("swift", Config{})
+}
+
+// NewConfig returns a new config with the default values filled in.
+func NewConfig() Config {
+	return Config{
+		Connections: 5,
+	}
+}
+
+// ParseConfig parses the string s and extract swift's container name and prefix.
+func ParseConfig(s string) (interface{}, error) {
+	data := strings.SplitN(s, ":", 3)
+	if len(data) != 3 {
+		return nil, errors.New("invalid URL, expected: swift:container-name:/[prefix]")
+	}
+
+	scheme, container, prefix := data[0], data[1], data[2]
+	if scheme != "swift" {
+		return nil, errors.Errorf("unexpected prefix: %s", data[0])
+	}
+
+	if len(prefix) == 0 {
+		return nil, errors.Errorf("prefix is empty")
+	}
+
+	if prefix[0] != '/' {
+		return nil, errors.Errorf("prefix does not start with slash (/)")
+	}
+	prefix = prefix[1:]
+
+	cfg := NewConfig()
+	cfg.Container = container
+	cfg.Prefix = prefix
+
+	return cfg, nil
+}
+
+// ApplyEnvironment saves values from the environment to the config.
+func ApplyEnvironment(prefix string, cfg interface{}) error {
+	c := cfg.(*Config)
+	for _, val := range []struct {
+		s   *string
+		env string
+	}{
+		// v2/v3 specific
+		{&c.UserName, prefix + "OS_USERNAME"},
+		{&c.APIKey, prefix + "OS_PASSWORD"},
+		{&c.Region, prefix + "OS_REGION_NAME"},
+		{&c.AuthURL, prefix + "OS_AUTH_URL"},
+
+		// v3 specific
+		{&c.Domain, prefix + "OS_USER_DOMAIN_NAME"},
+		{&c.Tenant, prefix + "OS_PROJECT_NAME"},
+		{&c.TenantDomain, prefix + "OS_PROJECT_DOMAIN_NAME"},
+
+		// v2 specific
+		{&c.TenantID, prefix + "OS_TENANT_ID"},
+		{&c.Tenant, prefix + "OS_TENANT_NAME"},
+
+		// v1 specific
+		{&c.AuthURL, prefix + "ST_AUTH"},
+		{&c.UserName, prefix + "ST_USER"},
+		{&c.APIKey, prefix + "ST_KEY"},
+
+		// Manual authentication
+		{&c.StorageURL, prefix + "OS_STORAGE_URL"},
+		{&c.AuthToken, prefix + "OS_AUTH_TOKEN"},
+
+		{&c.DefaultContainerPolicy, prefix + "SWIFT_DEFAULT_CONTAINER_POLICY"},
+	} {
+		if *val.s == "" {
+			*val.s = os.Getenv(val.env)
+		}
+	}
+	return nil
+}

src/restic/backend/swift/config_test.go

@@ -0,0 +1,72 @@
+package swift
+
+import "testing"
+
+var configTests = []struct {
+	s   string
+	cfg Config
+}{
+	{
+		"swift:cnt1:/",
+		Config{
+			Container:   "cnt1",
+			Prefix:      "",
+			Connections: 5,
+		},
+	},
+	{
+		"swift:cnt2:/prefix",
+		Config{Container: "cnt2",
+			Prefix:      "prefix",
+			Connections: 5,
+		},
+	},
+	{
+		"swift:cnt3:/prefix/longer",
+		Config{Container: "cnt3",
+			Prefix:      "prefix/longer",
+			Connections: 5,
+		},
+	},
+}
+
+func TestParseConfig(t *testing.T) {
+	for _, test := range configTests {
+		t.Run("", func(t *testing.T) {
+			v, err := ParseConfig(test.s)
+			if err != nil {
+				t.Fatalf("parsing %q failed: %v", test.s, err)
+			}
+
+			cfg, ok := v.(Config)
+			if !ok {
+				t.Fatalf("wrong type returned, want Config, got %T", cfg)
+			}
+
+			if cfg != test.cfg {
+				t.Fatalf("wrong output for %q, want:\n  %#v\ngot:\n  %#v",
+					test.s, test.cfg, cfg)
+			}
+		})
+	}
+}
+
+var configTestsInvalid = []string{
+	"swift://hostname/container",
+	"swift:////",
+	"swift://",
+	"swift:////prefix",
+	"swift:container",
+	"swift:container:",
+	"swift:container/prefix",
+}
+
+func TestParseConfigInvalid(t *testing.T) {
+	for i, test := range configTestsInvalid {
+		_, err := ParseConfig(test)
+		if err == nil {
+			t.Errorf("test %d: invalid config %s did not return an error", i, test)
+			continue
+		}
+	}
+}

src/restic/backend/swift/swift.go

@@ -0,0 +1,321 @@
+package swift
+
+import (
+	"context"
+	"fmt"
+	"io"
+	"net/http"
+	"path"
+	"path/filepath"
+	"restic"
+	"restic/backend"
+	"restic/debug"
+	"restic/errors"
+	"strings"
+	"time"
+
+	"github.com/ncw/swift"
+)
+
+const connLimit = 10
+
+// beSwift is a backend which stores the data on a swift endpoint.
+type beSwift struct {
+	conn      *swift.Connection
+	sem       *backend.Semaphore
+	container string // Container name
+	prefix    string // Prefix of object names in the container
+	backend.Layout
+}
+
+// ensure statically that *beSwift implements restic.Backend.
+var _ restic.Backend = &beSwift{}
+
+// Open opens the swift backend at a container in region. The container is
+// created if it does not exist yet.
+func Open(cfg Config) (restic.Backend, error) {
+	debug.Log("config %#v", cfg)
+
+	sem, err := backend.NewSemaphore(cfg.Connections)
+	if err != nil {
+		return nil, err
+	}
+
+	be := &beSwift{
+		conn: &swift.Connection{
+			UserName:       cfg.UserName,
+			Domain:         cfg.Domain,
+			ApiKey:         cfg.APIKey,
+			AuthUrl:        cfg.AuthURL,
+			Region:         cfg.Region,
+			Tenant:         cfg.Tenant,
+			TenantId:       cfg.TenantID,
+			TenantDomain:   cfg.TenantDomain,
+			TrustId:        cfg.TrustID,
+			StorageUrl:     cfg.StorageURL,
+			AuthToken:      cfg.AuthToken,
+			ConnectTimeout: time.Minute,
+			Timeout:        time.Minute,
+
+			Transport: backend.Transport(),
+		},
+		sem:       sem,
+		container: cfg.Container,
+		prefix:    cfg.Prefix,
+		Layout: &backend.DefaultLayout{
+			Path: cfg.Prefix,
+			Join: path.Join,
+		},
+	}
+
+	// Authenticate if needed
+	if !be.conn.Authenticated() {
+		if err := be.conn.Authenticate(); err != nil {
+			return nil, errors.Wrap(err, "conn.Authenticate")
+		}
+	}
+
+	// Ensure container exists
+	switch _, _, err := be.conn.Container(be.container); err {
+	case nil:
+		// Container exists
+
+	case swift.ContainerNotFound:
+		err = be.createContainer(cfg.DefaultContainerPolicy)
+		if err != nil {
+			return nil, errors.Wrap(err, "beSwift.createContainer")
+		}
+
+	default:
+		return nil, errors.Wrap(err, "conn.Container")
+	}
+
+	return be, nil
+}
+
+func (be *beSwift) createContainer(policy string) error {
+	var h swift.Headers
+	if policy != "" {
+		h = swift.Headers{
+			"X-Storage-Policy": policy,
+		}
+	}
+
+	return be.conn.ContainerCreate(be.container, h)
+}
+
+// Location returns this backend's location (the container name).
+func (be *beSwift) Location() string {
+	return be.container
+}
+
+// Load returns a reader that yields the contents of the file at h at the
+// given offset. If length is nonzero, only a portion of the file is
+// returned. rd must be closed after use.
+func (be *beSwift) Load(ctx context.Context, h restic.Handle, length int, offset int64) (io.ReadCloser, error) {
+	debug.Log("Load %v, length %v, offset %v", h, length, offset)
+	if err := h.Valid(); err != nil {
+		return nil, err
+	}
+
+	if offset < 0 {
+		return nil, errors.New("offset is negative")
+	}
+
+	if length < 0 {
+		return nil, errors.Errorf("invalid length %d", length)
+	}
+
+	objName := be.Filename(h)
+
+	be.sem.GetToken()
+	defer func() {
+		be.sem.ReleaseToken()
+	}()
+
+	headers := swift.Headers{}
+	if offset > 0 {
+		headers["Range"] = fmt.Sprintf("bytes=%d-", offset)
+	}
+
+	if length > 0 {
+		headers["Range"] = fmt.Sprintf("bytes=%d-%d", offset, offset+int64(length)-1)
+	}
+
+	if _, ok := headers["Range"]; ok {
+		debug.Log("Load(%v) send range %v", h, headers["Range"])
+	}
+
+	obj, _, err := be.conn.ObjectOpen(be.container, objName, false, headers)
+	if err != nil {
+		debug.Log("  err %v", err)
+		return nil, errors.Wrap(err, "conn.ObjectOpen")
+	}
+
+	return obj, nil
+}
+
+// Save stores data in the backend at the handle.
+func (be *beSwift) Save(ctx context.Context, h restic.Handle, rd io.Reader) (err error) {
+	if err = h.Valid(); err != nil {
+		return err
+	}
+
+	objName := be.Filename(h)
+
+	debug.Log("Save %v at %v", h, objName)
+
+	// Check key does not already exist
+	switch _, _, err = be.conn.Object(be.container, objName); err {
+	case nil:
+		debug.Log("%v already exists", h)
+		return errors.New("key already exists")
+
+	case swift.ObjectNotFound:
+		// Ok, that's what we want
+
+	default:
+		return errors.Wrap(err, "conn.Object")
+	}
+
+	be.sem.GetToken()
+	defer func() {
+		be.sem.ReleaseToken()
+	}()
+
+	encoding := "binary/octet-stream"
+
+	debug.Log("PutObject(%v, %v, %v)", be.container, objName, encoding)
+	_, err = be.conn.ObjectPut(be.container, objName, rd, true, "", encoding, nil)
+	debug.Log("%v, err %#v", objName, err)
+
+	return errors.Wrap(err, "client.PutObject")
+}
+
+// Stat returns information about a blob.
+func (be *beSwift) Stat(ctx context.Context, h restic.Handle) (bi restic.FileInfo, err error) {
+	debug.Log("%v", h)
+
+	objName := be.Filename(h)
+
+	obj, _, err := be.conn.Object(be.container, objName)
+	if err != nil {
+		debug.Log("Object() err %v", err)
+		return restic.FileInfo{}, errors.Wrap(err, "conn.Object")
+	}
+
+	return restic.FileInfo{Size: obj.Bytes}, nil
+}
+
+// Test returns true if a blob of the given type and name exists in the backend.
+func (be *beSwift) Test(ctx context.Context, h restic.Handle) (bool, error) {
+	objName := be.Filename(h)
+	switch _, _, err := be.conn.Object(be.container, objName); err {
+	case nil:
+		return true, nil
+
+	case swift.ObjectNotFound:
+		return false, nil
+
+	default:
+		return false, errors.Wrap(err, "conn.Object")
+	}
+}
+
+// Remove removes the blob with the given name and type.
+func (be *beSwift) Remove(ctx context.Context, h restic.Handle) error {
+	objName := be.Filename(h)
+	err := be.conn.ObjectDelete(be.container, objName)
+	debug.Log("Remove(%v) -> err %v", h, err)
+	return errors.Wrap(err, "conn.ObjectDelete")
+}
+
+// List returns a channel that yields all names of blobs of type t. A
+// goroutine is started for this. If the channel done is closed, sending
+// stops.
+func (be *beSwift) List(ctx context.Context, t restic.FileType) <-chan string {
+	debug.Log("listing %v", t)
+	ch := make(chan string)
+
+	prefix := be.Filename(restic.Handle{Type: t}) + "/"
+
+	go func() {
+		defer close(ch)
+
+		err := be.conn.ObjectsWalk(be.container, &swift.ObjectsOpts{Prefix: prefix},
+			func(opts *swift.ObjectsOpts) (interface{}, error) {
+				newObjects, err := be.conn.ObjectNames(be.container, opts)
+				if err != nil {
+					return nil, errors.Wrap(err, "conn.ObjectNames")
+				}
+				for _, obj := range newObjects {
+					m := filepath.Base(strings.TrimPrefix(obj, prefix))
+					if m == "" {
+						continue
+					}
+
+					select {
+					case ch <- m:
+					case <-ctx.Done():
+						return nil, io.EOF
+					}
+				}
+				return newObjects, nil
+			})
+
+		if err != nil {
+			debug.Log("ObjectsWalk returned error: %v", err)
+		}
+	}()
+
+	return ch
+}
+
+// Remove keys for a specified backend type.
+func (be *beSwift) removeKeys(ctx context.Context, t restic.FileType) error {
+	for key := range be.List(ctx, t) {
+		err := be.Remove(ctx, restic.Handle{Type: t, Name: key})
+		if err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+// IsNotExist returns true if the error is caused by a not existing file.
+func (be *beSwift) IsNotExist(err error) bool {
+	if e, ok := errors.Cause(err).(*swift.Error); ok {
+		return e.StatusCode == http.StatusNotFound
+	}
+
+	return false
+}
+
+// Delete removes all restic objects in the container.
+// It will not remove the container itself.
+func (be *beSwift) Delete(ctx context.Context) error {
+	alltypes := []restic.FileType{
+		restic.DataFile,
+		restic.KeyFile,
+		restic.LockFile,
+		restic.SnapshotFile,
+		restic.IndexFile}
+
+	for _, t := range alltypes {
+		err := be.removeKeys(ctx, t)
+		if err != nil {
+			return nil
+		}
+	}
+
+	err := be.Remove(ctx, restic.Handle{Type: restic.ConfigFile})
+	if err != nil && !be.IsNotExist(err) {
+		return err
+	}
+
+	return nil
+}
+
+// Close does nothing
+func (be *beSwift) Close() error { return nil }

src/restic/backend/swift/swift_test.go

@@ -0,0 +1,111 @@
+package swift_test
+
+import (
+	"context"
+	"fmt"
+	"os"
+	"restic"
+	"testing"
+	"time"
+
+	"restic/errors"
+	. "restic/test"
+
+	"restic/backend/swift"
+	"restic/backend/test"
+)
+
+func newSwiftTestSuite(t testing.TB) *test.Suite {
+	return &test.Suite{
+		// do not use excessive data
+		MinimalData: true,
+
+		// wait for removals for at least 60s
+		WaitForDelayedRemoval: 60 * time.Second,
+
+		// NewConfig returns a config for a new temporary backend that will be used in tests.
+		NewConfig: func() (interface{}, error) {
+			swiftcfg, err := swift.ParseConfig(os.Getenv("RESTIC_TEST_SWIFT"))
+			if err != nil {
+				return nil, err
+			}
+
+			cfg := swiftcfg.(swift.Config)
+			if err = swift.ApplyEnvironment("RESTIC_TEST_", &cfg); err != nil {
+				return nil, err
+			}
+			cfg.Prefix += fmt.Sprintf("/test-%d", time.Now().UnixNano())
+			t.Logf("using prefix %v", cfg.Prefix)
+			return cfg, nil
+		},
+
+		// CreateFn is a function that creates a temporary repository for the tests.
+		Create: func(config interface{}) (restic.Backend, error) {
+			cfg := config.(swift.Config)
+
+			be, err := swift.Open(cfg)
+			if err != nil {
+				return nil, err
+			}
+
+			exists, err := be.Test(context.TODO(), restic.Handle{Type: restic.ConfigFile})
+			if err != nil {
+				return nil, err
+			}
+
+			if exists {
+				return nil, errors.New("config already exists")
+			}
+
+			return be, nil
+		},
+
+		// OpenFn is a function that opens a previously created temporary repository.
+		Open: func(config interface{}) (restic.Backend, error) {
+			cfg := config.(swift.Config)
+			return swift.Open(cfg)
+		},
+
+		// CleanupFn removes data created during the tests.
+		Cleanup: func(config interface{}) error {
+			cfg := config.(swift.Config)
+
+			be, err := swift.Open(cfg)
+			if err != nil {
+				return err
+			}
+
+			if err := be.(restic.Deleter).Delete(context.TODO()); err != nil {
+				return err
+			}
+
+			return nil
+		},
+	}
+}
+
+func TestBackendSwift(t *testing.T) {
+	defer func() {
+		if t.Skipped() {
+			SkipDisallowed(t, "restic/backend/swift.TestBackendSwift")
+		}
+	}()
+
+	if os.Getenv("RESTIC_TEST_SWIFT") == "" {
+		t.Skip("RESTIC_TEST_SWIFT unset, skipping test")
+		return
+	}
+
+	t.Logf("run tests")
+	newSwiftTestSuite(t).RunTests(t)
+}
+
+func BenchmarkBackendSwift(t *testing.B) {
+	if os.Getenv("RESTIC_TEST_SWIFT") == "" {
+		t.Skip("RESTIC_TEST_SWIFT unset, skipping test")
+		return
+	}
+
+	t.Logf("run tests")
+	newSwiftTestSuite(t).RunBenchmarks(t)
+}

src/restic/backend/test/benchmarks.go

@@ -2,6 +2,7 @@ package test
 
 import (
 	"bytes"
+	"context"
 	"io"
 	"restic"
 	"restic/test"
@@ -12,14 +13,14 @@ func saveRandomFile(t testing.TB, be restic.Backend, length int) ([]byte, restic
 	data := test.Random(23, length)
 	id := restic.Hash(data)
 	handle := restic.Handle{Type: restic.DataFile, Name: id.String()}
-	if err := be.Save(handle, bytes.NewReader(data)); err != nil {
+	if err := be.Save(context.TODO(), handle, bytes.NewReader(data)); err != nil {
 		t.Fatalf("Save() error: %+v", err)
 	}
 	return data, handle
 }
 
 func remove(t testing.TB, be restic.Backend, h restic.Handle) {
-	if err := be.Remove(h); err != nil {
+	if err := be.Remove(context.TODO(), h); err != nil {
 		t.Fatalf("Remove() returned error: %v", err)
 	}
 }
@@ -40,7 +41,7 @@ func (s *Suite) BenchmarkLoadFile(t *testing.B) {
 	t.ResetTimer()
 
 	for i := 0; i < t.N; i++ {
-		rd, err := be.Load(handle, 0, 0)
+		rd, err := be.Load(context.TODO(), handle, 0, 0)
 		if err != nil {
 			t.Fatal(err)
 		}
@@ -82,7 +83,7 @@ func (s *Suite) BenchmarkLoadPartialFile(t *testing.B) {
 	t.ResetTimer()
 
 	for i := 0; i < t.N; i++ {
-		rd, err := be.Load(handle, testLength, 0)
+		rd, err := be.Load(context.TODO(), handle, testLength, 0)
 		if err != nil {
 			t.Fatal(err)
 		}
@@ -126,7 +127,7 @@ func (s *Suite) BenchmarkLoadPartialFileOffset(t *testing.B) {
 	t.ResetTimer()
 
 	for i := 0; i < t.N; i++ {
-		rd, err := be.Load(handle, testLength, int64(testOffset))
+		rd, err := be.Load(context.TODO(), handle, testLength, int64(testOffset))
 		if err != nil {
 			t.Fatal(err)
 		}
@@ -171,11 +172,11 @@ func (s *Suite) BenchmarkSave(t *testing.B) {
 			t.Fatal(err)
 		}
 
-		if err := be.Save(handle, rd); err != nil {
+		if err := be.Save(context.TODO(), handle, rd); err != nil {
 			t.Fatal(err)
 		}
 
-		if err := be.Remove(handle); err != nil {
+		if err := be.Remove(context.TODO(), handle); err != nil {
 			t.Fatal(err)
 		}
 	}

src/restic/backend/test/suite.go

@@ -6,10 +6,12 @@ import (
 	"restic/test"
 	"strings"
 	"testing"
+	"time"
 )
 
 // Suite implements a test suite for restic backends.
 type Suite struct {
+	// Config should be used to configure the backend.
 	Config interface{}
 
 	// NewConfig returns a config for a new temporary backend that will be used in tests.
@@ -26,6 +28,11 @@ type Suite struct {
 
 	// MinimalData instructs the tests to not use excessive data.
 	MinimalData bool
+
+	// WaitForDelayedRemoval is set to a non-zero value to instruct the test
+	// suite to wait for this amount of time until a file that was removed
+	// really disappeared.
+	WaitForDelayedRemoval time.Duration
 }
 
 // RunTests executes all defined tests as subtests of t.

src/restic/backend/test/tests.go

@@ -2,6 +2,7 @@ package test
 
 import (
 	"bytes"
+	"context"
 	"fmt"
 	"io"
 	"io/ioutil"
@@ -34,7 +35,7 @@ func (s *Suite) TestCreateWithConfig(t *testing.T) {
 
 	// remove a config if present
 	cfgHandle := restic.Handle{Type: restic.ConfigFile}
-	cfgPresent, err := b.Test(cfgHandle)
+	cfgPresent, err := b.Test(context.TODO(), cfgHandle)
 	if err != nil {
 		t.Fatalf("unable to test for config: %+v", err)
 	}
@@ -53,7 +54,7 @@ func (s *Suite) TestCreateWithConfig(t *testing.T) {
 	}
 
 	// remove config
-	err = b.Remove(restic.Handle{Type: restic.ConfigFile, Name: ""})
+	err = b.Remove(context.TODO(), restic.Handle{Type: restic.ConfigFile, Name: ""})
 	if err != nil {
 		t.Fatalf("unexpected error removing config: %+v", err)
 	}
@@ -78,12 +79,12 @@ func (s *Suite) TestConfig(t *testing.T) {
 	var testString = "Config"
 
 	// create config and read it back
-	_, err := backend.LoadAll(b, restic.Handle{Type: restic.ConfigFile})
+	_, err := backend.LoadAll(context.TODO(), b, restic.Handle{Type: restic.ConfigFile})
 	if err == nil {
 		t.Fatalf("did not get expected error for non-existing config")
 	}
 
-	err = b.Save(restic.Handle{Type: restic.ConfigFile}, strings.NewReader(testString))
+	err = b.Save(context.TODO(), restic.Handle{Type: restic.ConfigFile}, strings.NewReader(testString))
 	if err != nil {
 		t.Fatalf("Save() error: %+v", err)
 	}
@@ -92,7 +93,7 @@ func (s *Suite) TestConfig(t *testing.T) {
 	// same config
 	for _, name := range []string{"", "foo", "bar", "0000000000000000000000000000000000000000000000000000000000000000"} {
 		h := restic.Handle{Type: restic.ConfigFile, Name: name}
-		buf, err := backend.LoadAll(b, h)
+		buf, err := backend.LoadAll(context.TODO(), b, h)
 		if err != nil {
 			t.Fatalf("unable to read config with name %q: %+v", name, err)
 		}
@@ -113,12 +114,12 @@ func (s *Suite) TestLoad(t *testing.T) {
 	b := s.open(t)
 	defer s.close(t, b)
 
-	rd, err := b.Load(restic.Handle{}, 0, 0)
+	rd, err := b.Load(context.TODO(), restic.Handle{}, 0, 0)
 	if err == nil {
 		t.Fatalf("Load() did not return an error for invalid handle")
 	}
 	if rd != nil {
-		rd.Close()
+		_ = rd.Close()
 	}
 
 	err = testLoad(b, restic.Handle{Type: restic.DataFile, Name: "foobar"}, 0, 0)
@@ -132,14 +133,14 @@ func (s *Suite) TestLoad(t *testing.T) {
 	id := restic.Hash(data)
 
 	handle := restic.Handle{Type: restic.DataFile, Name: id.String()}
-	err = b.Save(handle, bytes.NewReader(data))
+	err = b.Save(context.TODO(), handle, bytes.NewReader(data))
 	if err != nil {
 		t.Fatalf("Save() error: %+v", err)
 	}
 
 	t.Logf("saved %d bytes as %v", length, handle)
 
-	rd, err = b.Load(handle, 100, -1)
+	rd, err = b.Load(context.TODO(), handle, 100, -1)
 	if err == nil {
 		t.Fatalf("Load() returned no error for negative offset!")
 	}
@@ -174,7 +175,7 @@ func (s *Suite) TestLoad(t *testing.T) {
 			d = d[:l]
 		}
 
-		rd, err := b.Load(handle, getlen, int64(o))
+		rd, err := b.Load(context.TODO(), handle, getlen, int64(o))
 		if err != nil {
 			t.Logf("Load, l %v, o %v, len(d) %v, getlen %v", l, o, len(d), getlen)
 			t.Errorf("Load(%d, %d) returned unexpected error: %+v", l, o, err)
@@ -235,13 +236,13 @@ func (s *Suite) TestLoad(t *testing.T) {
 		}
 	}
 
-	test.OK(t, b.Remove(handle))
+	test.OK(t, b.Remove(context.TODO(), handle))
 }
 
 type errorCloser struct {
 	io.Reader
-	size int64
-	t    testing.TB
+	l int
+	t testing.TB
 }
 
 func (ec errorCloser) Close() error {
@@ -249,8 +250,8 @@ func (ec errorCloser) Close() error {
 	return errors.New("forbidden method close was called")
 }
 
-func (ec errorCloser) Size() int64 {
-	return ec.size
+func (ec errorCloser) Len() int {
+	return ec.l
 }
 
 // TestSave tests saving data in the backend.
@@ -276,10 +277,10 @@ func (s *Suite) TestSave(t *testing.T) {
 			Type: restic.DataFile,
 			Name: fmt.Sprintf("%s-%d", id, i),
 		}
-		err := b.Save(h, bytes.NewReader(data))
+		err := b.Save(context.TODO(), h, bytes.NewReader(data))
 		test.OK(t, err)
 
-		buf, err := backend.LoadAll(b, h)
+		buf, err := backend.LoadAll(context.TODO(), b, h)
 		test.OK(t, err)
 		if len(buf) != len(data) {
 			t.Fatalf("number of bytes does not match, want %v, got %v", len(data), len(buf))
@@ -289,14 +290,14 @@ func (s *Suite) TestSave(t *testing.T) {
 			t.Fatalf("data not equal")
 		}
 
-		fi, err := b.Stat(h)
+		fi, err := b.Stat(context.TODO(), h)
 		test.OK(t, err)
 
 		if fi.Size != int64(len(data)) {
 			t.Fatalf("Stat() returned different size, want %q, got %d", len(data), fi.Size)
 		}
 
-		err = b.Remove(h)
+		err = b.Remove(context.TODO(), h)
 		if err != nil {
 			t.Fatalf("error removing item: %+v", err)
 		}
@@ -324,12 +325,12 @@ func (s *Suite) TestSave(t *testing.T) {
 
 	// wrap the tempfile in an errorCloser, so we can detect if the backend
 	// closes the reader
-	err = b.Save(h, errorCloser{t: t, size: int64(length), Reader: tmpfile})
+	err = b.Save(context.TODO(), h, errorCloser{t: t, l: length, Reader: tmpfile})
 	if err != nil {
 		t.Fatal(err)
 	}
 
-	err = b.Remove(h)
+	err = delayedRemove(t, b, s.WaitForDelayedRemoval, h)
 	if err != nil {
 		t.Fatalf("error removing item: %+v", err)
 	}
@@ -339,7 +340,7 @@ func (s *Suite) TestSave(t *testing.T) {
 		t.Fatal(err)
 	}
 
-	err = b.Save(h, tmpfile)
+	err = b.Save(context.TODO(), h, tmpfile)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -348,7 +349,7 @@ func (s *Suite) TestSave(t *testing.T) {
 		t.Fatal(err)
 	}
 
-	err = b.Remove(h)
+	err = b.Remove(context.TODO(), h)
 	if err != nil {
 		t.Fatalf("error removing item: %+v", err)
 	}
@@ -377,13 +378,13 @@ func (s *Suite) TestSaveFilenames(t *testing.T) {
 
 	for i, test := range filenameTests {
 		h := restic.Handle{Name: test.name, Type: restic.DataFile}
-		err := b.Save(h, strings.NewReader(test.data))
+		err := b.Save(context.TODO(), h, strings.NewReader(test.data))
 		if err != nil {
 			t.Errorf("test %d failed: Save() returned %+v", i, err)
 			continue
 		}
 
-		buf, err := backend.LoadAll(b, h)
+		buf, err := backend.LoadAll(context.TODO(), b, h)
 		if err != nil {
 			t.Errorf("test %d failed: Load() returned %+v", i, err)
 			continue
@@ -393,7 +394,7 @@ func (s *Suite) TestSaveFilenames(t *testing.T) {
 			t.Errorf("test %d: returned wrong bytes", i)
 		}
 
-		err = b.Remove(h)
+		err = b.Remove(context.TODO(), h)
 		if err != nil {
 			t.Errorf("test %d failed: Remove() returned %+v", i, err)
 			continue
@@ -414,14 +415,14 @@ var testStrings = []struct {
 func store(t testing.TB, b restic.Backend, tpe restic.FileType, data []byte) restic.Handle {
 	id := restic.Hash(data)
 	h := restic.Handle{Name: id.String(), Type: tpe}
-	err := b.Save(h, bytes.NewReader(data))
+	err := b.Save(context.TODO(), h, bytes.NewReader(data))
 	test.OK(t, err)
 	return h
 }
 
 // testLoad loads a blob (but discards its contents).
 func testLoad(b restic.Backend, h restic.Handle, length int, offset int64) error {
-	rd, err := b.Load(h, 0, 0)
+	rd, err := b.Load(context.TODO(), h, 0, 0)
 	if err != nil {
 		return err
 	}
@@ -434,6 +435,60 @@ func testLoad(b restic.Backend, h restic.Handle, length int, offset int64) error
 	return err
 }
 
+func delayedRemove(t testing.TB, be restic.Backend, maxwait time.Duration, handles ...restic.Handle) error {
+	// Some backend (swift, I'm looking at you) may implement delayed
+	// removal of data. Let's wait a bit if this happens.
+
+	for _, h := range handles {
+		err := be.Remove(context.TODO(), h)
+		if err != nil {
+			return err
+		}
+	}
+
+	for _, h := range handles {
+		start := time.Now()
+		attempt := 0
+		var found bool
+		var err error
+		for time.Since(start) <= maxwait {
+			found, err = be.Test(context.TODO(), h)
+			if err != nil {
+				return err
+			}
+
+			if !found {
+				break
+			}
+
+			time.Sleep(2 * time.Second)
+			attempt++
+		}
+
+		if found {
+			t.Fatalf("removed blob %v still present after %v (%d attempts)", h, time.Since(start), attempt)
+		}
+	}
+
+	return nil
+}
+
+func delayedList(t testing.TB, b restic.Backend, tpe restic.FileType, max int, maxwait time.Duration) restic.IDs {
+	list := restic.NewIDSet()
+	start := time.Now()
+	for i := 0; i < max; i++ {
+		for s := range b.List(context.TODO(), tpe) {
+			id := restic.TestParseID(s)
+			list.Insert(id)
+		}
+		if len(list) < max && time.Since(start) < maxwait {
+			time.Sleep(500 * time.Millisecond)
+		}
+	}
+
+	return list.List()
+}
+
 // TestBackend tests all functions of the backend.
 func (s *Suite) TestBackend(t *testing.T) {
 	b := s.open(t)
@@ -450,12 +505,12 @@ func (s *Suite) TestBackend(t *testing.T) {
 
 			// test if blob is already in repository
 			h := restic.Handle{Type: tpe, Name: id.String()}
-			ret, err := b.Test(h)
+			ret, err := b.Test(context.TODO(), h)
 			test.OK(t, err)
 			test.Assert(t, !ret, "blob was found to exist before creating")
 
 			// try to stat a not existing blob
-			_, err = b.Stat(h)
+			_, err = b.Stat(context.TODO(), h)
 			test.Assert(t, err != nil, "blob data could be extracted before creation")
 
 			// try to read not existing blob
@@ -463,7 +518,7 @@ func (s *Suite) TestBackend(t *testing.T) {
 			test.Assert(t, err != nil, "blob could be read before creation")
 
 			// try to get string out, should fail
-			ret, err = b.Test(h)
+			ret, err = b.Test(context.TODO(), h)
 			test.OK(t, err)
 			test.Assert(t, !ret, "id %q was found (but should not have)", ts.id)
 		}
@@ -474,7 +529,7 @@ func (s *Suite) TestBackend(t *testing.T) {
 
 			// test Load()
 			h := restic.Handle{Type: tpe, Name: ts.id}
-			buf, err := backend.LoadAll(b, h)
+			buf, err := backend.LoadAll(context.TODO(), b, h)
 			test.OK(t, err)
 			test.Equals(t, ts.data, string(buf))
 
@@ -484,7 +539,7 @@ func (s *Suite) TestBackend(t *testing.T) {
 			length := end - start
 
 			buf2 := make([]byte, length)
-			rd, err := b.Load(h, len(buf2), int64(start))
+			rd, err := b.Load(context.TODO(), h, len(buf2), int64(start))
 			test.OK(t, err)
 			n, err := io.ReadFull(rd, buf2)
 			test.OK(t, err)
@@ -504,20 +559,20 @@ func (s *Suite) TestBackend(t *testing.T) {
 
 		// create blob
 		h := restic.Handle{Type: tpe, Name: ts.id}
-		err := b.Save(h, strings.NewReader(ts.data))
+		err := b.Save(context.TODO(), h, strings.NewReader(ts.data))
 		test.Assert(t, err != nil, "expected error for %v, got %v", h, err)
 
 		// remove and recreate
-		err = b.Remove(h)
+		err = delayedRemove(t, b, s.WaitForDelayedRemoval, h)
 		test.OK(t, err)
 
 		// test that the blob is gone
-		ok, err := b.Test(h)
+		ok, err := b.Test(context.TODO(), h)
 		test.OK(t, err)
 		test.Assert(t, !ok, "removed blob still present")
 
 		// create blob
-		err = b.Save(h, strings.NewReader(ts.data))
+		err = b.Save(context.TODO(), h, strings.NewReader(ts.data))
 		test.OK(t, err)
 
 		// list items
@@ -529,12 +584,7 @@ func (s *Suite) TestBackend(t *testing.T) {
 			IDs = append(IDs, id)
 		}
 
-		list := restic.IDs{}
-
-		for s := range b.List(tpe, nil) {
-			list = append(list, restic.TestParseID(s))
-		}
-
+		list := delayedList(t, b, tpe, len(IDs), s.WaitForDelayedRemoval)
 		if len(IDs) != len(list) {
 			t.Fatalf("wrong number of IDs returned: want %d, got %d", len(IDs), len(list))
 		}
@@ -548,22 +598,21 @@ func (s *Suite) TestBackend(t *testing.T) {
 
 		// remove content if requested
 		if test.TestCleanupTempDirs {
+			var handles []restic.Handle
 			for _, ts := range testStrings {
 				id, err := restic.ParseID(ts.id)
 				test.OK(t, err)
 
 				h := restic.Handle{Type: tpe, Name: id.String()}
 
-				found, err := b.Test(h)
+				found, err := b.Test(context.TODO(), h)
 				test.OK(t, err)
 				test.Assert(t, found, fmt.Sprintf("id %q not found", id))
 
-				test.OK(t, b.Remove(h))
-
-				found, err = b.Test(h)
-				test.OK(t, err)
-				test.Assert(t, !found, fmt.Sprintf("id %q not found after removal", id))
+				handles = append(handles, h)
 			}
+
+			test.OK(t, delayedRemove(t, b, s.WaitForDelayedRemoval, handles...))
 		}
 	}
 }
@@ -582,7 +631,7 @@ func (s *Suite) TestDelete(t *testing.T) {
 		return
 	}
 
-	err := be.Delete()
+	err := be.Delete(context.TODO())
 	if err != nil {
 		t.Fatalf("error deleting backend: %+v", err)
 	}

src/restic/backend/test/tests_test.go

@@ -1,6 +1,7 @@
 package test_test
 
 import (
+	"context"
 	"restic"
 	"restic/errors"
 	"testing"
@@ -26,7 +27,7 @@ func newTestSuite(t testing.TB) *test.Suite {
 		Create: func(cfg interface{}) (restic.Backend, error) {
 			c := cfg.(*memConfig)
 			if c.be != nil {
-				ok, err := c.be.Test(restic.Handle{Type: restic.ConfigFile})
+				ok, err := c.be.Test(context.TODO(), restic.Handle{Type: restic.ConfigFile})
 				if err != nil {
 					return nil, err
 				}

src/restic/backend/utils.go

@@ -1,14 +1,15 @@
 package backend
 
 import (
+	"context"
 	"io"
 	"io/ioutil"
 	"restic"
 )
 
 // LoadAll reads all data stored in the backend for the handle.
-func LoadAll(be restic.Backend, h restic.Handle) (buf []byte, err error) {
-	rd, err := be.Load(h, 0, 0)
+func LoadAll(ctx context.Context, be restic.Backend, h restic.Handle) (buf []byte, err error) {
+	rd, err := be.Load(ctx, h, 0, 0)
 	if err != nil {
 		return nil, err
 	}
@@ -28,16 +29,6 @@ func LoadAll(be restic.Backend, h restic.Handle) (buf []byte, err error) {
 	return ioutil.ReadAll(rd)
 }
 
-// Closer wraps an io.Reader and adds a Close() method that does nothing.
-type Closer struct {
-	io.Reader
-}
-
-// Close is a no-op.
-func (c Closer) Close() error {
-	return nil
-}
-
 // LimitedReadCloser wraps io.LimitedReader and exposes the Close() method.
 type LimitedReadCloser struct {
 	io.ReadCloser

src/restic/backend/utils_test.go

@@ -2,6 +2,7 @@ package backend_test
 
 import (
 	"bytes"
+	"context"
 	"math/rand"
 	"restic"
 	"testing"
@@ -21,10 +22,10 @@ func TestLoadAll(t *testing.T) {
 		data := Random(23+i, rand.Intn(MiB)+500*KiB)
 
 		id := restic.Hash(data)
-		err := b.Save(restic.Handle{Name: id.String(), Type: restic.DataFile}, bytes.NewReader(data))
+		err := b.Save(context.TODO(), restic.Handle{Name: id.String(), Type: restic.DataFile}, bytes.NewReader(data))
 		OK(t, err)
 
-		buf, err := backend.LoadAll(b, restic.Handle{Type: restic.DataFile, Name: id.String()})
+		buf, err := backend.LoadAll(context.TODO(), b, restic.Handle{Type: restic.DataFile, Name: id.String()})
 		OK(t, err)
 
 		if len(buf) != len(data) {
@@ -46,10 +47,10 @@ func TestLoadSmallBuffer(t *testing.T) {
 		data := Random(23+i, rand.Intn(MiB)+500*KiB)
 
 		id := restic.Hash(data)
-		err := b.Save(restic.Handle{Name: id.String(), Type: restic.DataFile}, bytes.NewReader(data))
+		err := b.Save(context.TODO(), restic.Handle{Name: id.String(), Type: restic.DataFile}, bytes.NewReader(data))
 		OK(t, err)
 
-		buf, err := backend.LoadAll(b, restic.Handle{Type: restic.DataFile, Name: id.String()})
+		buf, err := backend.LoadAll(context.TODO(), b, restic.Handle{Type: restic.DataFile, Name: id.String()})
 		OK(t, err)
 
 		if len(buf) != len(data) {
@@ -71,10 +72,10 @@ func TestLoadLargeBuffer(t *testing.T) {
 		data := Random(23+i, rand.Intn(MiB)+500*KiB)
 
 		id := restic.Hash(data)
-		err := b.Save(restic.Handle{Name: id.String(), Type: restic.DataFile}, bytes.NewReader(data))
+		err := b.Save(context.TODO(), restic.Handle{Name: id.String(), Type: restic.DataFile}, bytes.NewReader(data))
 		OK(t, err)
 
-		buf, err := backend.LoadAll(b, restic.Handle{Type: restic.DataFile, Name: id.String()})
+		buf, err := backend.LoadAll(context.TODO(), b, restic.Handle{Type: restic.DataFile, Name: id.String()})
 		OK(t, err)
 
 		if len(buf) != len(data) {

src/restic/backend_find.go

@@ -1,6 +1,9 @@
 package restic
 
-import "restic/errors"
+import (
+	"context"
+	"restic/errors"
+)
 
 // ErrNoIDPrefixFound is returned by Find() when no ID for the given prefix
 // could be found.
@@ -14,13 +17,10 @@ var ErrMultipleIDMatches = errors.New("multiple IDs with prefix found")
 // start with prefix. If none is found, nil and ErrNoIDPrefixFound is returned.
 // If more than one is found, nil and ErrMultipleIDMatches is returned.
 func Find(be Lister, t FileType, prefix string) (string, error) {
-	done := make(chan struct{})
-	defer close(done)
-
 	match := ""
 
 	// TODO: optimize by sorting list etc.
-	for name := range be.List(t, done) {
+	for name := range be.List(context.TODO(), t) {
 		if prefix == name[:len(prefix)] {
 			if match == "" {
 				match = name
@@ -42,12 +42,9 @@ const minPrefixLength = 8
 // PrefixLength returns the number of bytes required so that all prefixes of
 // all names of type t are unique.
 func PrefixLength(be Lister, t FileType) (int, error) {
-	done := make(chan struct{})
-	defer close(done)
-
 	// load all IDs of the given type
 	list := make([]string, 0, 100)
-	for name := range be.List(t, done) {
+	for name := range be.List(context.TODO(), t) {
 		list = append(list, name)
 	}
 

src/restic/backend_find_test.go

@@ -1,15 +1,16 @@
 package restic
 
 import (
+	"context"
 	"testing"
 )
 
 type mockBackend struct {
-	list func(FileType, <-chan struct{}) <-chan string
+	list func(context.Context, FileType) <-chan string
 }
 
-func (m mockBackend) List(t FileType, done <-chan struct{}) <-chan string {
-	return m.list(t, done)
+func (m mockBackend) List(ctx context.Context, t FileType) <-chan string {
+	return m.list(ctx, t)
 }
 
 var samples = IDs{
@@ -27,14 +28,14 @@ func TestPrefixLength(t *testing.T) {
 	list := samples
 
 	m := mockBackend{}
-	m.list = func(t FileType, done <-chan struct{}) <-chan string {
+	m.list = func(ctx context.Context, t FileType) <-chan string {
 		ch := make(chan string)
 		go func() {
 			defer close(ch)
 			for _, id := range list {
 				select {
 				case ch <- id.String():
-				case <-done:
+				case <-ctx.Done():
 					return
 				}
 			}

src/restic/buffer_pool.go

@@ -1,21 +0,0 @@
-package restic
-
-import (
-	"sync"
-
-	"github.com/restic/chunker"
-)
-
-var bufPool = sync.Pool{
-	New: func() interface{} {
-		return make([]byte, chunker.MinSize)
-	},
-}
-
-func getBuf() []byte {
-	return bufPool.Get().([]byte)
-}
-
-func freeBuf(data []byte) {
-	bufPool.Put(data)
-}

src/restic/checker/checker.go

@@ -1,6 +1,7 @@
 package checker
 
 import (
+	"context"
 	"crypto/sha256"
 	"fmt"
 	"io"
@@ -12,7 +13,6 @@ import (
 	"restic/hashing"
 
 	"restic"
-	"restic/crypto"
 	"restic/debug"
 	"restic/pack"
 	"restic/repository"
@@ -76,7 +76,7 @@ func (err ErrOldIndexFormat) Error() string {
 }
 
 // LoadIndex loads all index files.
-func (c *Checker) LoadIndex() (hints []error, errs []error) {
+func (c *Checker) LoadIndex(ctx context.Context) (hints []error, errs []error) {
 	debug.Log("Start")
 	type indexRes struct {
 		Index *repository.Index
@@ -86,21 +86,21 @@ func (c *Checker) LoadIndex() (hints []error, errs []error) {
 
 	indexCh := make(chan indexRes)
 
-	worker := func(id restic.ID, done <-chan struct{}) error {
+	worker := func(ctx context.Context, id restic.ID) error {
 		debug.Log("worker got index %v", id)
-		idx, err := repository.LoadIndexWithDecoder(c.repo, id, repository.DecodeIndex)
+		idx, err := repository.LoadIndexWithDecoder(ctx, c.repo, id, repository.DecodeIndex)
 		if errors.Cause(err) == repository.ErrOldIndexFormat {
 			debug.Log("index %v has old format", id.Str())
 			hints = append(hints, ErrOldIndexFormat{id})
 
-			idx, err = repository.LoadIndexWithDecoder(c.repo, id, repository.DecodeOldIndex)
+			idx, err = repository.LoadIndexWithDecoder(ctx, c.repo, id, repository.DecodeOldIndex)
 		}
 
 		err = errors.Wrapf(err, "error loading index %v", id.Str())
 
 		select {
 		case indexCh <- indexRes{Index: idx, ID: id.String(), err: err}:
-		case <-done:
+		case <-ctx.Done():
 		}
 
 		return nil
@@ -109,7 +109,7 @@ func (c *Checker) LoadIndex() (hints []error, errs []error) {
 	go func() {
 		defer close(indexCh)
 		debug.Log("start loading indexes in parallel")
-		err := repository.FilesInParallel(c.repo.Backend(), restic.IndexFile, defaultParallelism,
+		err := repository.FilesInParallel(ctx, c.repo.Backend(), restic.IndexFile, defaultParallelism,
 			repository.ParallelWorkFuncParseID(worker))
 		debug.Log("loading indexes finished, error: %v", err)
 		if err != nil {
@@ -141,7 +141,7 @@ func (c *Checker) LoadIndex() (hints []error, errs []error) {
 
 		debug.Log("process blobs")
 		cnt := 0
-		for blob := range res.Index.Each(done) {
+		for blob := range res.Index.Each(ctx) {
 			c.packs.Insert(blob.PackID)
 			c.blobs.Insert(blob.ID)
 			c.blobRefs.M[blob.ID] = 0
@@ -183,7 +183,7 @@ func (e PackError) Error() string {
 	return "pack " + e.ID.String() + ": " + e.Err.Error()
 }
 
-func packIDTester(repo restic.Repository, inChan <-chan restic.ID, errChan chan<- error, wg *sync.WaitGroup, done <-chan struct{}) {
+func packIDTester(ctx context.Context, repo restic.Repository, inChan <-chan restic.ID, errChan chan<- error, wg *sync.WaitGroup) {
 	debug.Log("worker start")
 	defer debug.Log("worker done")
 
@@ -191,7 +191,7 @@ func packIDTester(repo restic.Repository, inChan <-chan restic.ID, errChan chan<
 
 	for id := range inChan {
 		h := restic.Handle{Type: restic.DataFile, Name: id.String()}
-		ok, err := repo.Backend().Test(h)
+		ok, err := repo.Backend().Test(ctx, h)
 		if err != nil {
 			err = PackError{ID: id, Err: err}
 		} else {
@@ -203,7 +203,7 @@ func packIDTester(repo restic.Repository, inChan <-chan restic.ID, errChan chan<
 		if err != nil {
 			debug.Log("error checking for pack %s: %v", id.Str(), err)
 			select {
-			case <-done:
+			case <-ctx.Done():
 				return
 			case errChan <- err:
 			}
@@ -218,7 +218,7 @@ func packIDTester(repo restic.Repository, inChan <-chan restic.ID, errChan chan<
 // Packs checks that all packs referenced in the index are still available and
 // there are no packs that aren't in an index. errChan is closed after all
 // packs have been checked.
-func (c *Checker) Packs(errChan chan<- error, done <-chan struct{}) {
+func (c *Checker) Packs(ctx context.Context, errChan chan<- error) {
 	defer close(errChan)
 
 	debug.Log("checking for %d packs", len(c.packs))
@@ -229,7 +229,7 @@ func (c *Checker) Packs(errChan chan<- error, done <-chan struct{}) {
 	IDChan := make(chan restic.ID)
 	for i := 0; i < defaultParallelism; i++ {
 		workerWG.Add(1)
-		go packIDTester(c.repo, IDChan, errChan, &workerWG, done)
+		go packIDTester(ctx, c.repo, IDChan, errChan, &workerWG)
 	}
 
 	for id := range c.packs {
@@ -242,12 +242,12 @@ func (c *Checker) Packs(errChan chan<- error, done <-chan struct{}) {
 	workerWG.Wait()
 	debug.Log("workers terminated")
 
-	for id := range c.repo.List(restic.DataFile, done) {
+	for id := range c.repo.List(ctx, restic.DataFile) {
 		debug.Log("check data blob %v", id.Str())
 		if !seenPacks.Has(id) {
 			c.orphanedPacks = append(c.orphanedPacks, id)
 			select {
-			case <-done:
+			case <-ctx.Done():
 				return
 			case errChan <- PackError{ID: id, Orphaned: true, Err: errors.New("not referenced in any index")}:
 			}
@@ -277,8 +277,8 @@ func (e Error) Error() string {
 	return e.Err.Error()
 }
 
-func loadTreeFromSnapshot(repo restic.Repository, id restic.ID) (restic.ID, error) {
-	sn, err := restic.LoadSnapshot(repo, id)
+func loadTreeFromSnapshot(ctx context.Context, repo restic.Repository, id restic.ID) (restic.ID, error) {
+	sn, err := restic.LoadSnapshot(ctx, repo, id)
 	if err != nil {
 		debug.Log("error loading snapshot %v: %v", id.Str(), err)
 		return restic.ID{}, err
@@ -293,7 +293,7 @@ func loadTreeFromSnapshot(repo restic.Repository, id restic.ID) (restic.ID, erro
 }
 
 // loadSnapshotTreeIDs loads all snapshots from backend and returns the tree IDs.
-func loadSnapshotTreeIDs(repo restic.Repository) (restic.IDs, []error) {
+func loadSnapshotTreeIDs(ctx context.Context, repo restic.Repository) (restic.IDs, []error) {
 	var trees struct {
 		IDs restic.IDs
 		sync.Mutex
@@ -304,7 +304,7 @@ func loadSnapshotTreeIDs(repo restic.Repository) (restic.IDs, []error) {
 		sync.Mutex
 	}
 
-	snapshotWorker := func(strID string, done <-chan struct{}) error {
+	snapshotWorker := func(ctx context.Context, strID string) error {
 		id, err := restic.ParseID(strID)
 		if err != nil {
 			return err
@@ -312,7 +312,7 @@ func loadSnapshotTreeIDs(repo restic.Repository) (restic.IDs, []error) {
 
 		debug.Log("load snapshot %v", id.Str())
 
-		treeID, err := loadTreeFromSnapshot(repo, id)
+		treeID, err := loadTreeFromSnapshot(ctx, repo, id)
 		if err != nil {
 			errs.Lock()
 			errs.errs = append(errs.errs, err)
@@ -328,7 +328,7 @@ func loadSnapshotTreeIDs(repo restic.Repository) (restic.IDs, []error) {
 		return nil
 	}
 
-	err := repository.FilesInParallel(repo.Backend(), restic.SnapshotFile, defaultParallelism, snapshotWorker)
+	err := repository.FilesInParallel(ctx, repo.Backend(), restic.SnapshotFile, defaultParallelism, snapshotWorker)
 	if err != nil {
 		errs.errs = append(errs.errs, err)
 	}
@@ -353,9 +353,9 @@ type treeJob struct {
 }
 
 // loadTreeWorker loads trees from repo and sends them to out.
-func loadTreeWorker(repo restic.Repository,
+func loadTreeWorker(ctx context.Context, repo restic.Repository,
 	in <-chan restic.ID, out chan<- treeJob,
-	done <-chan struct{}, wg *sync.WaitGroup) {
+	wg *sync.WaitGroup) {
 
 	defer func() {
 		debug.Log("exiting")
@@ -371,7 +371,7 @@ func loadTreeWorker(repo restic.Repository,
 	outCh = nil
 	for {
 		select {
-		case <-done:
+		case <-ctx.Done():
 			return
 
 		case treeID, ok := <-inCh:
@@ -380,7 +380,7 @@ func loadTreeWorker(repo restic.Repository,
 			}
 			debug.Log("load tree %v", treeID.Str())
 
-			tree, err := repo.LoadTree(treeID)
+			tree, err := repo.LoadTree(ctx, treeID)
 			debug.Log("load tree %v (%v) returned err: %v", tree, treeID.Str(), err)
 			job = treeJob{ID: treeID, error: err, Tree: tree}
 			outCh = out
@@ -395,7 +395,7 @@ func loadTreeWorker(repo restic.Repository,
 }
 
 // checkTreeWorker checks the trees received and sends out errors to errChan.
-func (c *Checker) checkTreeWorker(in <-chan treeJob, out chan<- error, done <-chan struct{}, wg *sync.WaitGroup) {
+func (c *Checker) checkTreeWorker(ctx context.Context, in <-chan treeJob, out chan<- error, wg *sync.WaitGroup) {
 	defer func() {
 		debug.Log("exiting")
 		wg.Done()
@@ -410,7 +410,7 @@ func (c *Checker) checkTreeWorker(in <-chan treeJob, out chan<- error, done <-ch
 	outCh = nil
 	for {
 		select {
-		case <-done:
+		case <-ctx.Done():
 			debug.Log("done channel closed, exiting")
 			return
 
@@ -458,7 +458,7 @@ func (c *Checker) checkTreeWorker(in <-chan treeJob, out chan<- error, done <-ch
 	}
 }
 
-func filterTrees(backlog restic.IDs, loaderChan chan<- restic.ID, in <-chan treeJob, out chan<- treeJob, done <-chan struct{}) {
+func filterTrees(ctx context.Context, backlog restic.IDs, loaderChan chan<- restic.ID, in <-chan treeJob, out chan<- treeJob) {
 	defer func() {
 		debug.Log("closing output channels")
 		close(loaderChan)
@@ -489,7 +489,7 @@ func filterTrees(backlog restic.IDs, loaderChan chan<- restic.ID, in <-chan tree
 		}
 
 		select {
-		case <-done:
+		case <-ctx.Done():
 			return
 
 		case loadCh <- nextTreeID:
@@ -549,15 +549,15 @@ func filterTrees(backlog restic.IDs, loaderChan chan<- restic.ID, in <-chan tree
 // Structure checks that for all snapshots all referenced data blobs and
 // subtrees are available in the index. errChan is closed after all trees have
 // been traversed.
-func (c *Checker) Structure(errChan chan<- error, done <-chan struct{}) {
+func (c *Checker) Structure(ctx context.Context, errChan chan<- error) {
 	defer close(errChan)
 
-	trees, errs := loadSnapshotTreeIDs(c.repo)
+	trees, errs := loadSnapshotTreeIDs(ctx, c.repo)
 	debug.Log("need to check %d trees from snapshots, %d errs returned", len(trees), len(errs))
 
 	for _, err := range errs {
 		select {
-		case <-done:
+		case <-ctx.Done():
 			return
 		case errChan <- err:
 		}
@@ -570,11 +570,11 @@ func (c *Checker) Structure(errChan chan<- error, done <-chan struct{}) {
 	var wg sync.WaitGroup
 	for i := 0; i < defaultParallelism; i++ {
 		wg.Add(2)
-		go loadTreeWorker(c.repo, treeIDChan, treeJobChan1, done, &wg)
-		go c.checkTreeWorker(treeJobChan2, errChan, done, &wg)
+		go loadTreeWorker(ctx, c.repo, treeIDChan, treeJobChan1, &wg)
+		go c.checkTreeWorker(ctx, treeJobChan2, errChan, &wg)
 	}
 
-	filterTrees(trees, treeIDChan, treeJobChan1, treeJobChan2, done)
+	filterTrees(ctx, trees, treeIDChan, treeJobChan1, treeJobChan2)
 
 	wg.Wait()
 }
@@ -659,11 +659,11 @@ func (c *Checker) CountPacks() uint64 {
 }
 
 // checkPack reads a pack and checks the integrity of all blobs.
-func checkPack(r restic.Repository, id restic.ID) error {
+func checkPack(ctx context.Context, r restic.Repository, id restic.ID) error {
 	debug.Log("checking pack %v", id.Str())
 	h := restic.Handle{Type: restic.DataFile, Name: id.String()}
 
-	rd, err := r.Backend().Load(h, 0, 0)
+	rd, err := r.Backend().Load(ctx, h, 0, 0)
 	if err != nil {
 		return err
 	}
@@ -724,7 +724,7 @@ func checkPack(r restic.Repository, id restic.ID) error {
 			continue
 		}
 
-		n, err := crypto.Decrypt(r.Key(), buf, buf)
+		n, err := r.Key().Decrypt(buf, buf)
 		if err != nil {
 			debug.Log("  error decrypting blob %v: %v", blob.ID.Str(), err)
 			errs = append(errs, errors.Errorf("blob %v: %v", i, err))
@@ -748,7 +748,7 @@ func checkPack(r restic.Repository, id restic.ID) error {
 }
 
 // ReadData loads all data from the repository and checks the integrity.
-func (c *Checker) ReadData(p *restic.Progress, errChan chan<- error, done <-chan struct{}) {
+func (c *Checker) ReadData(ctx context.Context, p *restic.Progress, errChan chan<- error) {
 	defer close(errChan)
 
 	p.Start()
@@ -761,7 +761,7 @@ func (c *Checker) ReadData(p *restic.Progress, errChan chan<- error, done <-chan
 			var ok bool
 
 			select {
-			case <-done:
+			case <-ctx.Done():
 				return
 			case id, ok = <-in:
 				if !ok {
@@ -769,21 +769,21 @@ func (c *Checker) ReadData(p *restic.Progress, errChan chan<- error, done <-chan
 				}
 			}
 
-			err := checkPack(c.repo, id)
+			err := checkPack(ctx, c.repo, id)
 			p.Report(restic.Stat{Blobs: 1})
 			if err == nil {
 				continue
 			}
 
 			select {
-			case <-done:
+			case <-ctx.Done():
 				return
 			case errChan <- err:
 			}
 		}
 	}
 
-	ch := c.repo.List(restic.DataFile, done)
+	ch := c.repo.List(ctx, restic.DataFile)
 
 	var wg sync.WaitGroup
 	for i := 0; i < defaultParallelism; i++ {

src/restic/checker/checker_test.go

@@ -1,6 +1,7 @@
 package checker_test
 
 import (
+	"context"
 	"io"
 	"math/rand"
 	"path/filepath"
@@ -16,13 +17,13 @@ import (
 
 var checkerTestData = filepath.Join("testdata", "checker-test-repo.tar.gz")
 
-func collectErrors(f func(chan<- error, <-chan struct{})) (errs []error) {
-	done := make(chan struct{})
-	defer close(done)
+func collectErrors(ctx context.Context, f func(context.Context, chan<- error)) (errs []error) {
+	ctx, cancel := context.WithCancel(ctx)
+	defer cancel()
 
 	errChan := make(chan error)
 
-	go f(errChan, done)
+	go f(ctx, errChan)
 
 	for err := range errChan {
 		errs = append(errs, err)
@@ -32,17 +33,18 @@ func collectErrors(f func(chan<- error, <-chan struct{})) (errs []error) {
 }
 
 func checkPacks(chkr *checker.Checker) []error {
-	return collectErrors(chkr.Packs)
+	return collectErrors(context.TODO(), chkr.Packs)
 }
 
 func checkStruct(chkr *checker.Checker) []error {
-	return collectErrors(chkr.Structure)
+	return collectErrors(context.TODO(), chkr.Structure)
 }
 
 func checkData(chkr *checker.Checker) []error {
 	return collectErrors(
-		func(errCh chan<- error, done <-chan struct{}) {
-			chkr.ReadData(nil, errCh, done)
+		context.TODO(),
+		func(ctx context.Context, errCh chan<- error) {
+			chkr.ReadData(ctx, nil, errCh)
 		},
 	)
 }
@@ -54,7 +56,7 @@ func TestCheckRepo(t *testing.T) {
 	repo := repository.TestOpenLocal(t, repodir)
 
 	chkr := checker.New(repo)
-	hints, errs := chkr.LoadIndex()
+	hints, errs := chkr.LoadIndex(context.TODO())
 	if len(errs) > 0 {
 		t.Fatalf("expected no errors, got %v: %v", len(errs), errs)
 	}
@@ -77,10 +79,10 @@ func TestMissingPack(t *testing.T) {
 		Type: restic.DataFile,
 		Name: "657f7fb64f6a854fff6fe9279998ee09034901eded4e6db9bcee0e59745bbce6",
 	}
-	test.OK(t, repo.Backend().Remove(packHandle))
+	test.OK(t, repo.Backend().Remove(context.TODO(), packHandle))
 
 	chkr := checker.New(repo)
-	hints, errs := chkr.LoadIndex()
+	hints, errs := chkr.LoadIndex(context.TODO())
 	if len(errs) > 0 {
 		t.Fatalf("expected no errors, got %v: %v", len(errs), errs)
 	}
@@ -113,10 +115,10 @@ func TestUnreferencedPack(t *testing.T) {
 		Type: restic.IndexFile,
 		Name: "3f1abfcb79c6f7d0a3be517d2c83c8562fba64ef2c8e9a3544b4edaf8b5e3b44",
 	}
-	test.OK(t, repo.Backend().Remove(indexHandle))
+	test.OK(t, repo.Backend().Remove(context.TODO(), indexHandle))
 
 	chkr := checker.New(repo)
-	hints, errs := chkr.LoadIndex()
+	hints, errs := chkr.LoadIndex(context.TODO())
 	if len(errs) > 0 {
 		t.Fatalf("expected no errors, got %v: %v", len(errs), errs)
 	}
@@ -147,7 +149,7 @@ func TestUnreferencedBlobs(t *testing.T) {
 		Type: restic.SnapshotFile,
 		Name: "51d249d28815200d59e4be7b3f21a157b864dc343353df9d8e498220c2499b02",
 	}
-	test.OK(t, repo.Backend().Remove(snapshotHandle))
+	test.OK(t, repo.Backend().Remove(context.TODO(), snapshotHandle))
 
 	unusedBlobsBySnapshot := restic.IDs{
 		restic.TestParseID("58c748bbe2929fdf30c73262bd8313fe828f8925b05d1d4a87fe109082acb849"),
@@ -161,7 +163,7 @@ func TestUnreferencedBlobs(t *testing.T) {
 	sort.Sort(unusedBlobsBySnapshot)
 
 	chkr := checker.New(repo)
-	hints, errs := chkr.LoadIndex()
+	hints, errs := chkr.LoadIndex(context.TODO())
 	if len(errs) > 0 {
 		t.Fatalf("expected no errors, got %v: %v", len(errs), errs)
 	}
@@ -192,7 +194,7 @@ func TestModifiedIndex(t *testing.T) {
 		Type: restic.IndexFile,
 		Name: "90f838b4ac28735fda8644fe6a08dbc742e57aaf81b30977b4fefa357010eafd",
 	}
-	f, err := repo.Backend().Load(h, 0, 0)
+	f, err := repo.Backend().Load(context.TODO(), h, 0, 0)
 	test.OK(t, err)
 
 	// save the index again with a modified name so that the hash doesn't match
@@ -201,13 +203,13 @@ func TestModifiedIndex(t *testing.T) {
 		Type: restic.IndexFile,
 		Name: "80f838b4ac28735fda8644fe6a08dbc742e57aaf81b30977b4fefa357010eafd",
 	}
-	err = repo.Backend().Save(h2, f)
+	err = repo.Backend().Save(context.TODO(), h2, f)
 	test.OK(t, err)
 
 	test.OK(t, f.Close())
 
 	chkr := checker.New(repo)
-	hints, errs := chkr.LoadIndex()
+	hints, errs := chkr.LoadIndex(context.TODO())
 	if len(errs) == 0 {
 		t.Fatalf("expected errors not found")
 	}
@@ -230,7 +232,7 @@ func TestDuplicatePacksInIndex(t *testing.T) {
 	repo := repository.TestOpenLocal(t, repodir)
 
 	chkr := checker.New(repo)
-	hints, errs := chkr.LoadIndex()
+	hints, errs := chkr.LoadIndex(context.TODO())
 	if len(hints) == 0 {
 		t.Fatalf("did not get expected checker hints for duplicate packs in indexes")
 	}
@@ -259,8 +261,8 @@ type errorBackend struct {
 	ProduceErrors bool
 }
 
-func (b errorBackend) Load(h restic.Handle, length int, offset int64) (io.ReadCloser, error) {
-	rd, err := b.Backend.Load(h, length, offset)
+func (b errorBackend) Load(ctx context.Context, h restic.Handle, length int, offset int64) (io.ReadCloser, error) {
+	rd, err := b.Backend.Load(ctx, h, length, offset)
 	if err != nil {
 		return rd, err
 	}
@@ -303,17 +305,17 @@ func TestCheckerModifiedData(t *testing.T) {
 	defer cleanup()
 
 	arch := archiver.New(repo)
-	_, id, err := arch.Snapshot(nil, []string{"."}, nil, "localhost", nil)
+	_, id, err := arch.Snapshot(context.TODO(), nil, []string{"."}, nil, "localhost", nil)
 	test.OK(t, err)
 	t.Logf("archived as %v", id.Str())
 
 	beError := &errorBackend{Backend: repo.Backend()}
 	checkRepo := repository.New(beError)
-	test.OK(t, checkRepo.SearchKey(test.TestPassword, 5))
+	test.OK(t, checkRepo.SearchKey(context.TODO(), test.TestPassword, 5))
 
 	chkr := checker.New(checkRepo)
 
-	hints, errs := chkr.LoadIndex()
+	hints, errs := chkr.LoadIndex(context.TODO())
 	if len(errs) > 0 {
 		t.Fatalf("expected no errors, got %v: %v", len(errs), errs)
 	}
@@ -349,7 +351,7 @@ func BenchmarkChecker(t *testing.B) {
 	repo := repository.TestOpenLocal(t, repodir)
 
 	chkr := checker.New(repo)
-	hints, errs := chkr.LoadIndex()
+	hints, errs := chkr.LoadIndex(context.TODO())
 	if len(errs) > 0 {
 		t.Fatalf("expected no errors, got %v: %v", len(errs), errs)
 	}

src/restic/checker/testing.go

@@ -1,6 +1,7 @@
 package checker
 
 import (
+	"context"
 	"restic"
 	"testing"
 )
@@ -9,7 +10,7 @@ import (
 func TestCheckRepo(t testing.TB, repo restic.Repository) {
 	chkr := New(repo)
 
-	hints, errs := chkr.LoadIndex()
+	hints, errs := chkr.LoadIndex(context.TODO())
 	if len(errs) != 0 {
 		t.Fatalf("errors loading index: %v", errs)
 	}
@@ -18,12 +19,9 @@ func TestCheckRepo(t testing.TB, repo restic.Repository) {
 		t.Fatalf("errors loading index: %v", hints)
 	}
 
-	done := make(chan struct{})
-	defer close(done)
-
 	// packs
 	errChan := make(chan error)
-	go chkr.Packs(errChan, done)
+	go chkr.Packs(context.TODO(), errChan)
 
 	for err := range errChan {
 		t.Error(err)
@@ -31,7 +29,7 @@ func TestCheckRepo(t testing.TB, repo restic.Repository) {
 
 	// structure
 	errChan = make(chan error)
-	go chkr.Structure(errChan, done)
+	go chkr.Structure(context.TODO(), errChan)
 
 	for err := range errChan {
 		t.Error(err)
@@ -45,7 +43,7 @@ func TestCheckRepo(t testing.TB, repo restic.Repository) {
 
 	// read data
 	errChan = make(chan error)
-	go chkr.ReadData(nil, errChan, done)
+	go chkr.ReadData(context.TODO(), nil, errChan)
 
 	for err := range errChan {
 		t.Error(err)

src/restic/config.go

@@ -1,6 +1,7 @@
 package restic
 
 import (
+	"context"
 	"testing"
 
 	"restic/errors"
@@ -23,7 +24,7 @@ const RepoVersion = 1
 
 // JSONUnpackedLoader loads unpacked JSON.
 type JSONUnpackedLoader interface {
-	LoadJSONUnpacked(FileType, ID, interface{}) error
+	LoadJSONUnpacked(context.Context, FileType, ID, interface{}) error
 }
 
 // CreateConfig creates a config file with a randomly selected polynomial and
@@ -57,12 +58,12 @@ func TestCreateConfig(t testing.TB, pol chunker.Pol) (cfg Config) {
 }
 
 // LoadConfig returns loads, checks and returns the config for a repository.
-func LoadConfig(r JSONUnpackedLoader) (Config, error) {
+func LoadConfig(ctx context.Context, r JSONUnpackedLoader) (Config, error) {
 	var (
 		cfg Config
 	)
 
-	err := r.LoadJSONUnpacked(ConfigFile, ID{}, &cfg)
+	err := r.LoadJSONUnpacked(ctx, ConfigFile, ID{}, &cfg)
 	if err != nil {
 		return Config{}, err
 	}

src/restic/config_test.go

@@ -1,6 +1,7 @@
 package restic_test
 
 import (
+	"context"
 	"restic"
 	"testing"
 
@@ -13,10 +14,10 @@ func (s saver) SaveJSONUnpacked(t restic.FileType, arg interface{}) (restic.ID,
 	return s(t, arg)
 }
 
-type loader func(restic.FileType, restic.ID, interface{}) error
+type loader func(context.Context, restic.FileType, restic.ID, interface{}) error
 
-func (l loader) LoadJSONUnpacked(t restic.FileType, id restic.ID, arg interface{}) error {
-	return l(t, id, arg)
+func (l loader) LoadJSONUnpacked(ctx context.Context, t restic.FileType, id restic.ID, arg interface{}) error {
+	return l(ctx, t, id, arg)
 }
 
 func TestConfig(t *testing.T) {
@@ -36,7 +37,7 @@ func TestConfig(t *testing.T) {
 
 	_, err = saver(save).SaveJSONUnpacked(restic.ConfigFile, cfg1)
 
-	load := func(tpe restic.FileType, id restic.ID, arg interface{}) error {
+	load := func(ctx context.Context, tpe restic.FileType, id restic.ID, arg interface{}) error {
 		Assert(t, tpe == restic.ConfigFile,
 			"wrong backend type: got %v, wanted %v",
 			tpe, restic.ConfigFile)
@@ -46,7 +47,7 @@ func TestConfig(t *testing.T) {
 		return nil
 	}
 
-	cfg2, err := restic.LoadConfig(loader(load))
+	cfg2, err := restic.LoadConfig(context.TODO(), loader(load))
 	OK(t, err)
 
 	Assert(t, cfg1 == cfg2,

src/restic/crypto/crypto.go

@@ -19,7 +19,9 @@ const (
 	macKeySize  = macKeySizeK + macKeySizeR // for Poly1305-AES128
 	ivSize      = aes.BlockSize
 
-	macSize   = poly1305.TagSize
+	macSize = poly1305.TagSize
+
+	// Extension is the number of bytes a plaintext is enlarged by encrypting it.
 	Extension = ivSize + macSize
 )
 
@@ -32,11 +34,14 @@ var (
 // encrypted and authenticated as a JSON data structure in the Data field of the Key
 // structure.
 type Key struct {
-	MAC     MACKey        `json:"mac"`
-	Encrypt EncryptionKey `json:"encrypt"`
+	MACKey        `json:"mac"`
+	EncryptionKey `json:"encrypt"`
 }
 
+// EncryptionKey is key used for encryption
 type EncryptionKey [32]byte
+
+// MACKey is used to sign (authenticate) data.
 type MACKey struct {
 	K [16]byte // for AES-128
 	R [16]byte // for Poly1305
@@ -123,22 +128,22 @@ func poly1305Verify(msg []byte, nonce []byte, key *MACKey, mac []byte) bool {
 func NewRandomKey() *Key {
 	k := &Key{}
 
-	n, err := rand.Read(k.Encrypt[:])
+	n, err := rand.Read(k.EncryptionKey[:])
 	if n != aesKeySize || err != nil {
 		panic("unable to read enough random bytes for encryption key")
 	}
 
-	n, err = rand.Read(k.MAC.K[:])
+	n, err = rand.Read(k.MACKey.K[:])
 	if n != macKeySizeK || err != nil {
 		panic("unable to read enough random bytes for MAC encryption key")
 	}
 
-	n, err = rand.Read(k.MAC.R[:])
+	n, err = rand.Read(k.MACKey.R[:])
 	if n != macKeySizeR || err != nil {
 		panic("unable to read enough random bytes for MAC key")
 	}
 
-	maskKey(&k.MAC)
+	maskKey(&k.MACKey)
 	return k
 }
 
@@ -156,10 +161,12 @@ type jsonMACKey struct {
 	R []byte `json:"r"`
 }
 
+// MarshalJSON converts the MACKey to JSON.
 func (m *MACKey) MarshalJSON() ([]byte, error) {
 	return json.Marshal(jsonMACKey{K: m.K[:], R: m.R[:]})
 }
 
+// UnmarshalJSON fills the key m with data from the JSON representation.
 func (m *MACKey) UnmarshalJSON(data []byte) error {
 	j := jsonMACKey{}
 	err := json.Unmarshal(data, &j)
@@ -194,10 +201,12 @@ func (m *MACKey) Valid() bool {
 	return false
 }
 
+// MarshalJSON converts the EncryptionKey to JSON.
 func (k *EncryptionKey) MarshalJSON() ([]byte, error) {
 	return json.Marshal(k[:])
 }
 
+// UnmarshalJSON fills the key k with data from the JSON representation.
 func (k *EncryptionKey) UnmarshalJSON(data []byte) error {
 	d := make([]byte, aesKeySize)
 	err := json.Unmarshal(data, &d)
@@ -228,8 +237,8 @@ var ErrInvalidCiphertext = errors.New("invalid ciphertext, same slice used for p
 // MAC. Encrypt returns the new ciphertext slice, which is extended when
 // necessary. ciphertext and plaintext may not point to (exactly) the same
 // slice or non-intersecting slices.
-func Encrypt(ks *Key, ciphertext []byte, plaintext []byte) ([]byte, error) {
-	if !ks.Valid() {
+func (k *Key) Encrypt(ciphertext []byte, plaintext []byte) ([]byte, error) {
+	if !k.Valid() {
 		return nil, errors.New("invalid key")
 	}
 
@@ -248,7 +257,7 @@ func Encrypt(ks *Key, ciphertext []byte, plaintext []byte) ([]byte, error) {
 	}
 
 	iv := newIV()
-	c, err := aes.NewCipher(ks.Encrypt[:])
+	c, err := aes.NewCipher(k.EncryptionKey[:])
 	if err != nil {
 		panic(fmt.Sprintf("unable to create cipher: %v", err))
 	}
@@ -261,7 +270,7 @@ func Encrypt(ks *Key, ciphertext []byte, plaintext []byte) ([]byte, error) {
 	// truncate to only cover iv and actual ciphertext
 	ciphertext = ciphertext[:ivSize+len(plaintext)]
 
-	mac := poly1305MAC(ciphertext[ivSize:], ciphertext[:ivSize], &ks.MAC)
+	mac := poly1305MAC(ciphertext[ivSize:], ciphertext[:ivSize], &k.MACKey)
 	ciphertext = append(ciphertext, mac...)
 
 	return ciphertext, nil
@@ -270,8 +279,8 @@ func Encrypt(ks *Key, ciphertext []byte, plaintext []byte) ([]byte, error) {
 // Decrypt verifies and decrypts the ciphertext. Ciphertext must be in the form
 // IV || Ciphertext || MAC. plaintext and ciphertext may point to (exactly) the
 // same slice.
-func Decrypt(ks *Key, plaintext []byte, ciphertextWithMac []byte) (int, error) {
-	if !ks.Valid() {
+func (k *Key) Decrypt(plaintext []byte, ciphertextWithMac []byte) (int, error) {
+	if !k.Valid() {
 		return 0, errors.New("invalid key")
 	}
 
@@ -291,7 +300,7 @@ func Decrypt(ks *Key, plaintext []byte, ciphertextWithMac []byte) (int, error) {
 	ciphertextWithIV, mac := ciphertextWithMac[:l], ciphertextWithMac[l:]
 
 	// verify mac
-	if !poly1305Verify(ciphertextWithIV[ivSize:], ciphertextWithIV[:ivSize], &ks.MAC, mac) {
+	if !poly1305Verify(ciphertextWithIV[ivSize:], ciphertextWithIV[:ivSize], &k.MACKey, mac) {
 		return 0, ErrUnauthenticated
 	}
 
@@ -303,7 +312,7 @@ func Decrypt(ks *Key, plaintext []byte, ciphertextWithMac []byte) (int, error) {
 	}
 
 	// decrypt data
-	c, err := aes.NewCipher(ks.Encrypt[:])
+	c, err := aes.NewCipher(k.EncryptionKey[:])
 	if err != nil {
 		panic(fmt.Sprintf("unable to create cipher: %v", err))
 	}
@@ -318,5 +327,5 @@ func Decrypt(ks *Key, plaintext []byte, ciphertextWithMac []byte) (int, error) {
 
 // Valid tests if the key is valid.
 func (k *Key) Valid() bool {
-	return k.Encrypt.Valid() && k.MAC.Valid()
+	return k.EncryptionKey.Valid() && k.MACKey.Valid()
 }

src/restic/crypto/crypto_int_test.go

@@ -90,18 +90,18 @@ func TestCrypto(t *testing.T) {
 	for _, tv := range testValues {
 		// test encryption
 		k := &Key{
-			Encrypt: tv.ekey,
-			MAC:     tv.skey,
+			EncryptionKey: tv.ekey,
+			MACKey:        tv.skey,
 		}
 
-		msg, err := Encrypt(k, msg, tv.plaintext)
+		msg, err := k.Encrypt(msg, tv.plaintext)
 		if err != nil {
 			t.Fatal(err)
 		}
 
 		// decrypt message
 		buf := make([]byte, len(tv.plaintext))
-		n, err := Decrypt(k, buf, msg)
+		n, err := k.Decrypt(buf, msg)
 		if err != nil {
 			t.Fatal(err)
 		}
@@ -110,7 +110,7 @@ func TestCrypto(t *testing.T) {
 		// change mac, this must fail
 		msg[len(msg)-8] ^= 0x23
 
-		if _, err = Decrypt(k, buf, msg); err != ErrUnauthenticated {
+		if _, err = k.Decrypt(buf, msg); err != ErrUnauthenticated {
 			t.Fatal("wrong MAC value not detected")
 		}
 
@@ -120,13 +120,13 @@ func TestCrypto(t *testing.T) {
 		// tamper with message, this must fail
 		msg[16+5] ^= 0x85
 
-		if _, err = Decrypt(k, buf, msg); err != ErrUnauthenticated {
+		if _, err = k.Decrypt(buf, msg); err != ErrUnauthenticated {
 			t.Fatal("tampered message not detected")
 		}
 
 		// test decryption
 		p := make([]byte, len(tv.ciphertext))
-		n, err = Decrypt(k, p, tv.ciphertext)
+		n, err = k.Decrypt(p, tv.ciphertext)
 		if err != nil {
 			t.Fatal(err)
 		}

src/restic/crypto/crypto_test.go

@@ -26,14 +26,14 @@ func TestEncryptDecrypt(t *testing.T) {
 		data := Random(42, size)
 		buf := make([]byte, size+crypto.Extension)
 
-		ciphertext, err := crypto.Encrypt(k, buf, data)
+		ciphertext, err := k.Encrypt(buf, data)
 		OK(t, err)
 		Assert(t, len(ciphertext) == len(data)+crypto.Extension,
 			"ciphertext length does not match: want %d, got %d",
 			len(data)+crypto.Extension, len(ciphertext))
 
 		plaintext := make([]byte, len(ciphertext))
-		n, err := crypto.Decrypt(k, plaintext, ciphertext)
+		n, err := k.Decrypt(plaintext, ciphertext)
 		OK(t, err)
 		plaintext = plaintext[:n]
 		Assert(t, len(plaintext) == len(data),
@@ -53,7 +53,7 @@ func TestSmallBuffer(t *testing.T) {
 	OK(t, err)
 
 	ciphertext := make([]byte, size/2)
-	ciphertext, err = crypto.Encrypt(k, ciphertext, data)
+	ciphertext, err = k.Encrypt(ciphertext, data)
 	// this must extend the slice
 	Assert(t, cap(ciphertext) > size/2,
 		"expected extended slice, but capacity is only %d bytes",
@@ -61,7 +61,7 @@ func TestSmallBuffer(t *testing.T) {
 
 	// check for the correct plaintext
 	plaintext := make([]byte, len(ciphertext))
-	n, err := crypto.Decrypt(k, plaintext, ciphertext)
+	n, err := k.Decrypt(plaintext, ciphertext)
 	OK(t, err)
 	plaintext = plaintext[:n]
 	Assert(t, bytes.Equal(plaintext, data),
@@ -78,11 +78,11 @@ func TestSameBuffer(t *testing.T) {
 
 	ciphertext := make([]byte, 0, size+crypto.Extension)
 
-	ciphertext, err = crypto.Encrypt(k, ciphertext, data)
+	ciphertext, err = k.Encrypt(ciphertext, data)
 	OK(t, err)
 
 	// use the same buffer for decryption
-	n, err := crypto.Decrypt(k, ciphertext, ciphertext)
+	n, err := k.Decrypt(ciphertext, ciphertext)
 	OK(t, err)
 	ciphertext = ciphertext[:n]
 	Assert(t, bytes.Equal(ciphertext, data),
@@ -94,7 +94,7 @@ func TestCornerCases(t *testing.T) {
 
 	// nil plaintext should encrypt to the empty string
 	// nil ciphertext should allocate a new slice for the ciphertext
-	c, err := crypto.Encrypt(k, nil, nil)
+	c, err := k.Encrypt(nil, nil)
 	OK(t, err)
 
 	Assert(t, len(c) == crypto.Extension,
@@ -102,12 +102,12 @@ func TestCornerCases(t *testing.T) {
 		len(c))
 
 	// this should decrypt to nil
-	n, err := crypto.Decrypt(k, nil, c)
+	n, err := k.Decrypt(nil, c)
 	OK(t, err)
 	Equals(t, 0, n)
 
 	// test encryption for same slice, this should return an error
-	_, err = crypto.Encrypt(k, c, c)
+	_, err = k.Encrypt(c, c)
 	Equals(t, crypto.ErrInvalidCiphertext, err)
 }
 
@@ -123,10 +123,10 @@ func TestLargeEncrypt(t *testing.T) {
 		_, err := io.ReadFull(rand.Reader, data)
 		OK(t, err)
 
-		ciphertext, err := crypto.Encrypt(k, make([]byte, size+crypto.Extension), data)
+		ciphertext, err := k.Encrypt(make([]byte, size+crypto.Extension), data)
 		OK(t, err)
 
-		plaintext, err := crypto.Decrypt(k, []byte{}, ciphertext)
+		plaintext, err := k.Decrypt([]byte{}, ciphertext)
 		OK(t, err)
 
 		Equals(t, plaintext, data)
@@ -144,7 +144,7 @@ func BenchmarkEncrypt(b *testing.B) {
 	b.SetBytes(int64(size))
 
 	for i := 0; i < b.N; i++ {
-		_, err := crypto.Encrypt(k, buf, data)
+		_, err := k.Encrypt(buf, data)
 		OK(b, err)
 	}
 }
@@ -158,14 +158,14 @@ func BenchmarkDecrypt(b *testing.B) {
 	plaintext := make([]byte, size)
 	ciphertext := make([]byte, size+crypto.Extension)
 
-	ciphertext, err := crypto.Encrypt(k, ciphertext, data)
+	ciphertext, err := k.Encrypt(ciphertext, data)
 	OK(b, err)
 
 	b.ResetTimer()
 	b.SetBytes(int64(size))
 
 	for i := 0; i < b.N; i++ {
-		_, err = crypto.Decrypt(k, plaintext, ciphertext)
+		_, err = k.Decrypt(plaintext, ciphertext)
 		OK(b, err)
 	}
 }

src/restic/crypto/kdf.go

@@ -81,10 +81,10 @@ func KDF(p KDFParams, salt []byte, password string) (*Key, error) {
 	}
 
 	// first 32 byte of scrypt output is the encryption key
-	copy(derKeys.Encrypt[:], scryptKeys[:aesKeySize])
+	copy(derKeys.EncryptionKey[:], scryptKeys[:aesKeySize])
 
 	// next 32 byte of scrypt output is the mac key, in the form k||r
-	macKeyFromSlice(&derKeys.MAC, scryptKeys[aesKeySize:])
+	macKeyFromSlice(&derKeys.MACKey, scryptKeys[aesKeySize:])
 
 	return derKeys, nil
 }

src/restic/debug/round_tripper_debug.go

@@ -52,7 +52,9 @@ func (rd *eofDetectReader) Close() error {
 
 func (tr eofDetectRoundTripper) RoundTrip(req *http.Request) (res *http.Response, err error) {
 	res, err = tr.RoundTripper.RoundTrip(req)
-	res.Body = &eofDetectReader{rd: res.Body}
+	if res != nil && res.Body != nil {
+		res.Body = &eofDetectReader{rd: res.Body}
+	}
 	return res, err
 }
 

src/restic/find.go

@@ -1,12 +1,14 @@
 package restic
 
+import "context"
+
 // FindUsedBlobs traverses the tree ID and adds all seen blobs (trees and data
 // blobs) to the set blobs. The tree blobs in the `seen` BlobSet will not be visited
 // again.
-func FindUsedBlobs(repo Repository, treeID ID, blobs BlobSet, seen BlobSet) error {
+func FindUsedBlobs(ctx context.Context, repo Repository, treeID ID, blobs BlobSet, seen BlobSet) error {
 	blobs.Insert(BlobHandle{ID: treeID, Type: TreeBlob})
 
-	tree, err := repo.LoadTree(treeID)
+	tree, err := repo.LoadTree(ctx, treeID)
 	if err != nil {
 		return err
 	}
@@ -26,7 +28,7 @@ func FindUsedBlobs(repo Repository, treeID ID, blobs BlobSet, seen BlobSet) erro
 
 			seen.Insert(h)
 
-			err := FindUsedBlobs(repo, subtreeID, blobs, seen)
+			err := FindUsedBlobs(ctx, repo, subtreeID, blobs, seen)
 			if err != nil {
 				return err
 			}

src/restic/find_test.go

@@ -2,6 +2,7 @@ package restic_test
 
 import (
 	"bufio"
+	"context"
 	"encoding/json"
 	"flag"
 	"fmt"
@@ -92,7 +93,7 @@ func TestFindUsedBlobs(t *testing.T) {
 
 	for i, sn := range snapshots {
 		usedBlobs := restic.NewBlobSet()
-		err := restic.FindUsedBlobs(repo, *sn.Tree, usedBlobs, restic.NewBlobSet())
+		err := restic.FindUsedBlobs(context.TODO(), repo, *sn.Tree, usedBlobs, restic.NewBlobSet())
 		if err != nil {
 			t.Errorf("FindUsedBlobs returned error: %v", err)
 			continue
@@ -128,7 +129,7 @@ func BenchmarkFindUsedBlobs(b *testing.B) {
 	for i := 0; i < b.N; i++ {
 		seen := restic.NewBlobSet()
 		blobs := restic.NewBlobSet()
-		err := restic.FindUsedBlobs(repo, *sn.Tree, blobs, seen)
+		err := restic.FindUsedBlobs(context.TODO(), repo, *sn.Tree, blobs, seen)
 		if err != nil {
 			b.Error(err)
 		}

src/restic/fs/file.go

@@ -19,11 +19,6 @@ type File interface {
 	Stat() (os.FileInfo, error)
 }
 
-// Chmod changes the mode of the named file to mode.
-func Chmod(name string, mode os.FileMode) error {
-	return os.Chmod(fixpath(name), mode)
-}
-
 // Mkdir creates a new directory with the specified name and permission bits.
 // If there is an error, it will be of type *PathError.
 func Mkdir(name string, perm os.FileMode) error {

src/restic/fs/file_unix.go

@@ -5,6 +5,7 @@ package fs
 import (
 	"io/ioutil"
 	"os"
+	"syscall"
 )
 
 // fixpath returns an absolute path on windows, so restic can open long file
@@ -35,3 +36,23 @@ func TempFile(dir, prefix string) (f *os.File, err error) {
 
 	return f, nil
 }
+
+// isNotSuported returns true if the error is caused by an unsupported file system feature.
+func isNotSupported(err error) bool {
+	if perr, ok := err.(*os.PathError); ok && perr.Err == syscall.ENOTSUP {
+		return true
+	}
+	return false
+}
+
+// Chmod changes the mode of the named file to mode.
+func Chmod(name string, mode os.FileMode) error {
+	err := os.Chmod(fixpath(name), mode)
+
+	// ignore the error if the FS does not support setting this mode (e.g. CIFS with gvfs on Linux)
+	if err != nil && isNotSupported(err) {
+		return nil
+	}
+
+	return err
+}

src/restic/fs/file_windows.go

@@ -91,3 +91,8 @@ func MkdirAll(path string, perm os.FileMode) error {
 func TempFile(dir, prefix string) (f *os.File, err error) {
 	return ioutil.TempFile(dir, prefix)
 }
+
+// Chmod changes the mode of the named file to mode.
+func Chmod(name string, mode os.FileMode) error {
+	return os.Chmod(fixpath(name), mode)
+}