amorfo77/restic
1
0
Fork 0
mirror of https://github.com/restic/restic.git synced 2026-05-26 20:15:35 +00:00
Code Issues Projects Releases Wiki Activity

Move things around for gb

Browse Source 
This moves all restic source files to src/, and all vendored
dependencies to vendor/src.
This commit is contained in:
Alexander Neumann
2016-02-14 15:13:44 +01:00
parent 273d028a82
commit b63399d606
510 changed files with 2 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/github.com/restic/restic/.hound.yml
+2
View File
@@ -0,0 +1,2 @@
go:
enabled: true
src/github.com/restic/restic/.travis.yml
+42
View File
@@ -0,0 +1,42 @@
language: go
sudo: false
go:
- 1.3.3
- 1.4.3
- 1.5.3
- 1.6rc2
os:
- linux
- osx
matrix:
exclude:
- os: osx
go: 1.3.3
- os: osx
go: 1.4.3
- os: osx
go: 1.6rc2
notifications:
irc:
channels:
- "chat.freenode.net#restic"
on_success: change
on_failure: change
skip_join: true
install:
- go version
- export GOBIN="$GOPATH/bin"
- export PATH="$PATH:$GOBIN"
- export GOPATH="$GOPATH:${TRAVIS_BUILD_DIR}/Godeps/_workspace"
- go env
script:
- go run run_integration_tests.go
after_success:
- goveralls -coverprofile=all.cov -service=travis-ci -repotoken "$COVERALLS_TOKEN" || true
src/github.com/restic/restic/CONTRIBUTING.md
+179
View File
@@ -0,0 +1,179 @@
This document describes the way you can contribute to the restic project.
Ways to Help Out
================
Thank you for your contribution!
There are several ways you can help us out. First of all code contributions and
bug fixes are most welcome. However even "minor" details as fixing spelling
errors, improving documentation or pointing out usability issues are a great
help also.
The restic project uses the GitHub infrastructure (see the
[project page](https://github.com/restic/restic)) for all related discussions
as well as the `#restic` channel on `irc.freenode.net`.
If you want to find an area that currently needs improving have a look at the
open issues listed at the
[issues page](https://github.com/restic/restic/issues). This is also the place
for discussing enhancement to the restic tools.
If you are unsure what to do, please have a look at the issues, especially
those tagged
[minor complexity](https://github.com/restic/restic/labels/minor%20complexity).
Reporting Bugs
==============
You've found a bug? Thanks for letting us know so we can fix it! It is a good
idea to describe in detail how to reproduce the bug (when you know how), what
environment was used and so on. Please tell us at least the following things:
* What's the version of restic you used? Please include the output of
`restic version` in your bug report.
* What commands did you execute to get to where the bug occurred?
* What did you expect?
* What happened instead?
* Are you aware of a way to reproduce the bug?
Remember, the easier it is for us to reproduce the bug, the earlier it will be
corrected!
In addition, you can compile restic with debug support by running
`go run build.go -tags debug` and instructing it to create a debug log by
setting the environment variable `DEBUG_LOG` to a file, e.g. like this:
$ export DEBUG_LOG=/tmp/restic-debug.log
$ restic backup ~/work
Please be aware that the debug log file will contain potentially sensitive
things like file and directory names, so please either redact it before
uploading it somewhere or post only the parts that are really relevant.
Development Environment
=======================
For development, it is recommended to check out the restic repository within a
`GOPATH`, an introductory text is
["How to Write Go Code"](https://golang.org/doc/code.html). It is recommended
to have a working directory, we're using `~/work/restic` in the following. This
directory mainly contains the directory `src`, where the source code is stored.
First, create the necessary directory structure and clone the restic repository
to the correct location:
$ mkdir --parents ~/work/restic/src/github.com/restic
$ cd ~/work/restic/src/github.com/restic
$ git clone https://github.com/restic/restic
$ cd restic
Now we're in the main directory of the restic repository. The last step is to
set the environment variable `$GOPATH` to the correct value:
$ export GOPATH=~/work/restic:~/work/restic/src/github.com/restic/restic/Godeps/_workspace
The following commands can be used to run all the tests:
$ go test ./...
ok github.com/restic/restic 8.174s
[...]
The restic binary can be built from the directory `cmd/restic` this way:
$ cd cmd/restic
$ go build
$ ./restic version
restic compiled manually on go1.4.2
if you want to run your tests on Linux, OpenBSD or FreeBSD, you can use
[vagrant](https://www.vagrantup.com/) with the proveded `Vagrantfile` to
quickly set up VMs and run the tests, e.g.:
$ vagrant up freebsd
[...]
$ vagrant ssh freebsd -c 'cd restic/restic; go test -v ./...'
[...]
Providing Patches
=================
You have fixed an annoying bug or have added a new feature? Very cool! Let's
get it into the project! The workflow we're using is also described on the
[GitHub Flow](https://guides.github.com/introduction/flow/) website, it boils
down to the following steps:
1. First we would kindly ask you to fork our project on GitHub if you haven't
done so already.
2. Clone the repository locally and create a new branch. If you are working on
the code itself, please set up the development environment as described in
the previous section and instead of cloning add your fork on GitHub as a
remote to the clone of the restic repository.
3. Then commit your changes as fine grained as possible, as smaller patches,
that handle one and only one issue are easier to discuss and merge.
4. Push the new branch with your changes to your fork of the repository.
5. Create a pull request by visiting the GitHub website, it will guide you
through the process.
6. You will receive comments on your code and the feature or bug that they
address. Maybe you need to rework some minor things, in this case push new
commits to the branch you created for the pull request, they will be
automatically added to the pull request.
7. Once your code looks good, we'll merge it. Thanks a low for your
contribution!
Please provide the patches for each bug or feature in a separate branch and
open up a pull request for each.
The restic project uses the `gofmt` tool for Go source indentation, so please
run
gofmt -w **/*.go
in the project root directory before committing. Installing the script
`fmt-check` from https://github.com/edsrzf/gofmt-git-hook locally as a
pre-commit hook checks formatting before committing automatically, just copy
this script to `.git/hooks/pre-commit`.
For each pull request, several different systems run the integration tests on
Linux, OS X and Windows. We won't merge any code that does not pass all tests
for all systems, so when a tests fails, try to find out what's wrong and fix
it. If you need help on this, please leave a comment in the pull request, and
we'll be glad to assist. Having a PR with failing integration tests is nothing
to be ashamed of. In contrast, that happens regularly for all of us. That's
what the tests are there for.
Git Commits
-----------
I would be good if you could follow the same general style regarding Git
commits as the rest of the project, this makes reviewing code, browsing the
history and triaging bugs much easier.
Git commit messages have a very terse summary in the first line of the commit
message, followed by an empty line, followed by a more verbose description or a
List of changed things. For examples, please refer to the excellent [How to
Write a Git Commit Message](http://chris.beams.io/posts/git-commit/).
If you change/add multiple different things that aren't related at all, try to
make several smaller commits. This is much easier to review. Using `git add -p`
allows staging and committing only some changes.
Code Review
===========
The restic project encourages actively reviewing the code, as it will store
your precious data, so it's common practice to receive comments on provided
patches.
If you are reviewing other contributor's code please consider the following
when reviewing:
* Be nice. Please make the review comment as constructive as possible so all
participants will learn something from your review.
As a contributor you might be asked to rewrite portions of your code to make it
fit better into the upstream sources.
src/github.com/restic/restic/Dockerfile
+58
View File
@@ -0,0 +1,58 @@
# This Dockerfiles configures a container that is similar to the Travis CI
# environment and can be used to run tests locally.
#
# build the image:
# docker build -t restic/test .
#
# run tests:
# docker run --rm -v $PWD:/home/travis/gopath/src/github.com/restic/restic restic/test go run run_integration_tests.go
#
# run interactively with:
# docker run --interactive --tty --rm -v $PWD:/home/travis/gopath/src/github.com/restic/restic restic/test /bin/bash
#
# run a tests:
# docker run --rm -v $PWD:/home/travis/gopath/src/github.com/restic/restic restic/test go test -v ./backend
FROM ubuntu:14.04
ARG GOVERSION=1.5.3
ARG GOARCH=amd64
# install dependencies
RUN apt-get update
RUN apt-get install -y --no-install-recommends ca-certificates wget git build-essential openssh-server
# add and configure user
ENV HOME /home/travis
RUN useradd -m -d $HOME -s /bin/bash travis
# run everything below as user travis
USER travis
WORKDIR $HOME
# download and install Go
RUN wget -q -O /tmp/go.tar.gz https://storage.googleapis.com/golang/go${GOVERSION}.linux-${GOARCH}.tar.gz
RUN tar xf /tmp/go.tar.gz && rm -f /tmp/go.tar.gz
ENV GOROOT $HOME/go
ENV PATH $PATH:$GOROOT/bin
ENV GOPATH $HOME/gopath
ENV PATH $PATH:$GOPATH/bin
RUN mkdir -p $GOPATH/src/github.com/restic/restic
# pre-install tools, this speeds up running the tests itself
RUN go get github.com/tools/godep
RUN go get golang.org/x/tools/cmd/cover
RUN go get github.com/mattn/goveralls
RUN go get github.com/mitchellh/gox
RUN go get github.com/pierrre/gotestcover
RUN mkdir $HOME/bin \
&& wget -q -O $HOME/bin/minio https://dl.minio.io/server/minio/release/linux-${GOARCH}/minio \
&& chmod +x $HOME/bin/minio
# set TRAVIS_BUILD_DIR for integration script
ENV TRAVIS_BUILD_DIR $GOPATH/src/github.com/restic/restic
ENV GOPATH $GOPATH:${TRAVIS_BUILD_DIR}/Godeps/_workspace
WORKDIR $TRAVIS_BUILD_DIR
src/github.com/restic/restic/LICENSE
+23
View File
@@ -0,0 +1,23 @@
Copyright (c) 2014, Alexander Neumann <alexander@bumpern.de>
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
src/github.com/restic/restic/Makefile
+12
View File
@@ -0,0 +1,12 @@
.PHONY: all clean test
all: restic
restic: $(SOURCE)
go run build.go
clean:
rm -rf restic
test: $(SOURCE)
go test ./...
src/github.com/restic/restic/README.md
+116
View File
@@ -0,0 +1,116 @@
[![Build Status](https://travis-ci.org/restic/restic.svg?branch=master)](https://travis-ci.org/restic/restic)
[![Build status](https://ci.appveyor.com/api/projects/status/nuy4lfbgfbytw92q/branch/master?svg=true)](https://ci.appveyor.com/project/fd0/restic/branch/master)
[![Report Card](http://goreportcard.com/badge/restic/restic)](http://goreportcard.com/report/restic/restic)
[![Coverage Status](https://coveralls.io/repos/restic/restic/badge.svg)](https://coveralls.io/r/restic/restic)
Restic Design Principles
========================
Restic is a program that does backups right and was designed with the following
principles in mind:
* Easy: Doing backups should be a frictionless process, otherwise you might be
tempted to skip it. Restic should be easy to configure and use, so that, in
the event of a data loss, you can just restore it. Likewise,
restoring data should not be complicated.
* Fast: Backing up your data with restic should only be limited by your
network or hard disk bandwidth so that you can backup your files every day.
Nobody does backups if it takes too much time. Restoring backups should only
transfer data that is needed for the files that are to be restored, so that
this process is also fast.
* Verifiable: Much more important than backup is restore, so restic enables
you to easily verify that all data can be restored.
* Secure: Restic uses cryptography to guarantee confidentiality and integrity
of your data. The location the backup data is stored is assumed not to be a
trusted environment (e.g. a shared space where others like system
administrators are able to access your backups). Restic is built to secure
your data against such attackers.
* Efficient: With the growth of data, additional snapshots should only take
the storage of the actual increment. Even more, duplicate data should be
de-duplicated before it is actually written to the storage back end to save
precious backup space.
Build restic
============
Install Go/Golang (at least version 1.3), then run `go run build.go`,
afterwards you'll find the binary in the current directory:
$ go run build.go
$ ./restic --help
Usage:
restic [OPTIONS] <command>
[...]
More documentation can be found on the [website](https://restic.github.io),
especially in the [user manual](https://restic.github.io/manual).
At the moment, the only tested compiler for restic is the official Go compiler.
Building restic with gccgo may work, but is not supported.
Compatibility
=============
Backward compatibility for backups is important so that our users are always
able to restore saved data. Therefore restic follows [Semantic
Versioning](http://semver.org) to clearly define which versions are compatible.
The repository and data structures contained therein are considered the "Public
API" in the sense of Semantic Versioning. This goes for all released versions
of restic, this may not be the case for the master branch.
We guarantee backward compatibility of all repositories within one major version;
as long as we do not increment the major version, data can be read and restored.
We strive to be fully backward compatible to all prior versions.
Contribute and Documentation
============================
Contributions are welcome! More information can be found in
[`CONTRIBUTING.md`](CONTRIBUTING.md). A document describing the design of
restic and the data structures stored on the back end is contained in
[`doc/Design.md`](doc/Design.md).
The development environment is described in [`CONTRIBUTING.md`](CONTRIBUTING.md).
Contact
=======
If you discover a bug, find something surprising or if you would like to
discuss or ask something, please [open a github issue](https://github.com/restic/restic/issues/new).
If you would like to chat about restic, there is also the IRC channel #restic
on irc.freenode.net.
**Important**: If you discover something that you believe to be a possible critical
security problem, please do *not* open a GitHub issue but send an email directly to
alexander@bumpern.de. If possible, please encrypt your email using the following PGP key
([0x91A6868BD3F7A907](https://pgp.mit.edu/pks/lookup?op=get&search=0xCF8F18F2844575973F79D4E191A6868BD3F7A907)):
```
pub 4096R/91A6868BD3F7A907 2014-11-01
Key fingerprint = CF8F 18F2 8445 7597 3F79 D4E1 91A6 868B D3F7 A907
uid Alexander Neumann <alexander@bumpern.de>
uid Alexander Neumann <alexander@debian.org>
sub 4096R/D5FC2ACF4043FDF1 2014-11-01
```
Talks
=====
The following talks will be or have been given about restic:
* 2016-01-31: Lightning Talk at the Go Devroom at FOSDEM 2016, Brussels, Belgium
* 2016-01-29: [restic - Backups mal richtig](https://media.ccc.de/v/c4.openchaos.2016.01.restic): Public lecture in German at [CCC Cologne e.V.](https://koeln.ccc.de) in Cologne, Germany
* 2015-08-23: [A Solution to the Backup Inconvenience](https://programm.froscon.de/2015/events/1515.html): Lecture at [FROSCON 2015](https://www.froscon.de) in Bonn, Germany
* 2015-02-01: [Lightning Talk at FOSDEM 2015](https://www.youtube.com/watch?v=oM-MfeflUZ8&t=11m40s): A short introduction (with slightly outdated command line)
* 2015-01-27: [Talk about restic at CCC Aachen](https://videoag.fsmpi.rwth-aachen.de/?view=player&lectureid=4442#content) (in German)
License
=======
Restic is licensed under "BSD 2-Clause License". You can find the complete text
in the file `LICENSE`.
src/github.com/restic/restic/VERSION
+1
View File
@@ -0,0 +1 @@
0.1.0
src/github.com/restic/restic/Vagrantfile
Vendored
+122
View File
@@ -0,0 +1,122 @@
# -*- mode: ruby -*-
# vi: set ft=ruby :
GO_VERSION = '1.4.2'
def packages_freebsd
return <<-EOF
pkg install -y git
pkg install -y curl
echo 'fuse_load="YES"' >> /boot/loader.conf
echo 'vfs.usermount=1' >> /etc/sysctl.conf
kldload fuse
sysctl vfs.usermount=1
pw groupmod operator -M vagrant
EOF
end
def packages_openbsd
return <<-EOF
. ~/.profile
pkg_add git curl bash gtar--
ln -sf /usr/local/bin/gtar /usr/local/bin/tar
EOF
end
def packages_linux
return <<-EOF
apt-get update
apt-get install -y git curl
EOF
end
def packages_darwin
return <<-EOF
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew install caskroom/cask/brew-cask
brew cask install osxfuse
EOF
end
def install_gimme
return <<-EOF
rm -rf /opt/gimme
mkdir -p /opt/gimme || true
git clone https://github.com/meatballhat/gimme /opt/gimme
perl -p -i -e 's,/bin/bash,/usr/bin/env bash,' /opt/gimme/gimme
ln -sf /opt/gimme/gimme /usr/bin/gimme
EOF
end
def prepare_user(boxname)
return <<-EOF
mkdir -p ~/go/src
export PATH=/usr/local/bin:$PATH
gimme #{GO_VERSION} >> ~/.profile
echo export 'GOPATH=/vagrant/go' >> ~/.profile
echo export 'CDPATH=.:$GOPATH/src/github.com' >> ~/.profile
echo export 'PATH=$GOPATH/bin:/usr/local/bin:$PATH' >> ~/.profile
. ~/.profile
go get golang.org/x/tools/cmd/cover
go get github.com/tools/godep
echo
echo "Run:"
echo " vagrant rsync #{boxname}"
echo " vagrant ssh #{boxname} -c 'cd project/path; godep go test ./...'"
EOF
end
def fix_perms
return <<-EOF
chown -R vagrant /vagrant/go
EOF
end
# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure(2) do |config|
# use rsync to copy content to the folder
config.vm.synced_folder ".", "/vagrant/go/src/github.com/restic/restic", :type => "rsync"
config.vm.synced_folder ".", "/vagrant", disabled: true
# fix permissions on synced folder
config.vm.provision "fix perms", :type => :shell, :inline => fix_perms
config.vm.define "linux" do |b|
b.vm.box = "ubuntu/trusty64"
b.vm.provision "packages linux", :type => :shell, :inline => packages_linux
b.vm.provision "install gimme", :type => :shell, :inline => install_gimme
b.vm.provision "prepare user", :type => :shell, :privileged => false, :inline => prepare_user("linux")
end
config.vm.define "freebsd" do |b|
b.vm.box = "geoffgarside/freebsd-10.1"
b.vm.provision "packages freebsd", :type => :shell, :inline => packages_freebsd
b.vm.provision "install gimme", :type => :shell, :inline => install_gimme
b.vm.provision "prepare user", :type => :shell, :privileged => false, :inline => prepare_user("freebsd")
end
config.vm.define "openbsd" do |b|
b.vm.box = "tmatilai/openbsd-5.6"
b.vm.provision "packages openbsd", :type => :shell, :inline => packages_openbsd
b.vm.provision "install gimme", :type => :shell, :inline => install_gimme
b.vm.provision "prepare user", :type => :shell, :privileged => false, :inline => prepare_user("openbsd")
end
config.vm.define "darwin" do |b|
#b.vm.box = "jhcook/osx-yosemite-10.10"
b.vm.box = "jhcook/yosemite-clitools"
b.vm.provision "packages darwin", :type => :shell, :privileged => false, :inline => packages_darwin
b.vm.provision "install gimme", :type => :shell, :inline => install_gimme
b.vm.provision "prepare user", :type => :shell, :privileged => false, :inline => prepare_user("darwin")
end
end
src/github.com/restic/restic/appveyor.yml
+25
View File
@@ -0,0 +1,25 @@
clone_folder: c:\gopath\src\github.com\restic\restic
environment:
GOPATH: c:\gopath;c:\gopath\src\github.com\restic\restic\Godeps\_workspace
init:
- ps: >-
$app = Get-WmiObject -Class Win32_Product -Filter "Vendor = 'http://golang.org'"
if ($app) {
$app.Uninstall()
}
install:
- rmdir c:\go /s /q
- appveyor DownloadFile https://storage.googleapis.com/golang/go1.5.3.windows-amd64.msi
- msiexec /i go1.5.3.windows-amd64.msi /q
- go version
- go env
- appveyor DownloadFile http://sourceforge.netcologne.de/project/gnuwin32/tar/1.13-1/tar-1.13-1-bin.zip -FileName tar.zip
- 7z x tar.zip bin/tar.exe
- set PATH=bin/;%PATH%
build_script:
- go run run_integration_tests.go
src/github.com/restic/restic/archiver.go
+816
View File
@@ -0,0 +1,816 @@
package restic
import (
"crypto/sha256"
"encoding/json"
"fmt"
"io"
"os"
"path/filepath"
"sort"
"sync"
"time"
"github.com/restic/chunker"
"github.com/restic/restic/backend"
"github.com/restic/restic/debug"
"github.com/restic/restic/pack"
"github.com/restic/restic/pipe"
"github.com/restic/restic/repository"
"github.com/juju/errors"
)
const (
maxConcurrentBlobs = 32
maxConcurrency = 10
)
var archiverAbortOnAllErrors = func(str string, fi os.FileInfo, err error) error { return err }
var archiverAllowAllFiles = func(string, os.FileInfo) bool { return true }
// Archiver is used to backup a set of directories.
type Archiver struct {
repo *repository.Repository
knownBlobs struct {
backend.IDSet
sync.Mutex
}
blobToken chan struct{}
Error func(dir string, fi os.FileInfo, err error) error
SelectFilter pipe.SelectFunc
Excludes []string
}
// NewArchiver returns a new archiver.
func NewArchiver(repo *repository.Repository) *Archiver {
arch := &Archiver{
repo: repo,
blobToken: make(chan struct{}, maxConcurrentBlobs),
knownBlobs: struct {
backend.IDSet
sync.Mutex
}{
IDSet: backend.NewIDSet(),
},
}
for i := 0; i < maxConcurrentBlobs; i++ {
arch.blobToken <- struct{}{}
}
arch.Error = archiverAbortOnAllErrors
arch.SelectFilter = archiverAllowAllFiles
return arch
}
// isKnownBlob returns true iff the blob is not yet in the list of known blobs.
// When the blob is not known, false is returned and the blob is added to the
// list. This means that the caller false is returned to is responsible to save
// the blob to the backend.
func (arch *Archiver) isKnownBlob(id backend.ID) bool {
arch.knownBlobs.Lock()
defer arch.knownBlobs.Unlock()
if arch.knownBlobs.Has(id) {
return true
}
arch.knownBlobs.Insert(id)
_, err := arch.repo.Index().Lookup(id)
if err == nil {
return true
}
return false
}
// Save stores a blob read from rd in the repository.
func (arch *Archiver) Save(t pack.BlobType, id backend.ID, length uint, rd io.Reader) error {
debug.Log("Archiver.Save", "Save(%v, %v)\n", t, id.Str())
if arch.isKnownBlob(id) {
debug.Log("Archiver.Save", "blob %v is known\n", id.Str())
return nil
}
err := arch.repo.SaveFrom(t, &id, length, rd)
if err != nil {
debug.Log("Archiver.Save", "Save(%v, %v): error %v\n", t, id.Str(), err)
return err
}
debug.Log("Archiver.Save", "Save(%v, %v): new blob\n", t, id.Str())
return nil
}
// SaveTreeJSON stores a tree in the repository.
func (arch *Archiver) SaveTreeJSON(item interface{}) (backend.ID, error) {
data, err := json.Marshal(item)
if err != nil {
return backend.ID{}, err
}
data = append(data, '\n')
// check if tree has been saved before
id := backend.Hash(data)
if arch.isKnownBlob(id) {
return id, nil
}
return arch.repo.SaveJSON(pack.Tree, item)
}
func (arch *Archiver) reloadFileIfChanged(node *Node, file *os.File) (*Node, error) {
fi, err := file.Stat()
if err != nil {
return nil, err
}
if fi.ModTime() == node.ModTime {
return node, nil
}
err = arch.Error(node.path, fi, errors.New("file has changed"))
if err != nil {
return nil, err
}
node, err = NodeFromFileInfo(node.path, fi)
if err != nil {
debug.Log("Archiver.SaveFile", "NodeFromFileInfo returned error for %v: %v", node.path, err)
return nil, err
}
return node, nil
}
type saveResult struct {
id backend.ID
bytes uint64
}
func (arch *Archiver) saveChunk(chunk *chunker.Chunk, p *Progress, token struct{}, file *os.File, resultChannel chan<- saveResult) {
hash := chunk.Digest
id := backend.ID{}
copy(id[:], hash)
err := arch.Save(pack.Data, id, chunk.Length, chunk.Reader(file))
// TODO handle error
if err != nil {
panic(err)
}
p.Report(Stat{Bytes: uint64(chunk.Length)})
arch.blobToken <- token
resultChannel <- saveResult{id: id, bytes: uint64(chunk.Length)}
}
func waitForResults(resultChannels [](<-chan saveResult)) ([]saveResult, error) {
results := []saveResult{}
for _, ch := range resultChannels {
results = append(results, <-ch)
}
if len(results) != len(resultChannels) {
return nil, fmt.Errorf("chunker returned %v chunks, but only %v blobs saved", len(resultChannels), len(results))
}
return results, nil
}
func updateNodeContent(node *Node, results []saveResult) error {
debug.Log("Archiver.Save", "checking size for file %s", node.path)
var bytes uint64
node.Content = make([]backend.ID, len(results))
for i, b := range results {
node.Content[i] = b.id
bytes += b.bytes
debug.Log("Archiver.Save", " adding blob %s, %d bytes", b.id.Str(), b.bytes)
}
if bytes != node.Size {
return fmt.Errorf("errors saving node %q: saved %d bytes, wanted %d bytes", node.path, bytes, node.Size)
}
debug.Log("Archiver.SaveFile", "SaveFile(%q): %v blobs\n", node.path, len(results))
return nil
}
// SaveFile stores the content of the file on the backend as a Blob by calling
// Save for each chunk.
func (arch *Archiver) SaveFile(p *Progress, node *Node) error {
file, err := node.OpenForReading()
defer file.Close()
if err != nil {
return err
}
node, err = arch.reloadFileIfChanged(node, file)
if err != nil {
return err
}
chnker := chunker.New(file, arch.repo.Config.ChunkerPolynomial, sha256.New())
resultChannels := [](<-chan saveResult){}
for {
chunk, err := chnker.Next()
if err == io.EOF {
break
}
if err != nil {
return errors.Annotate(err, "SaveFile() chunker.Next()")
}
resCh := make(chan saveResult, 1)
go arch.saveChunk(chunk, p, <-arch.blobToken, file, resCh)
resultChannels = append(resultChannels, resCh)
}
results, err := waitForResults(resultChannels)
if err != nil {
return err
}
err = updateNodeContent(node, results)
return err
}
func (arch *Archiver) fileWorker(wg *sync.WaitGroup, p *Progress, done <-chan struct{}, entCh <-chan pipe.Entry) {
defer func() {
debug.Log("Archiver.fileWorker", "done")
wg.Done()
}()
for {
select {
case e, ok := <-entCh:
if !ok {
// channel is closed
return
}
debug.Log("Archiver.fileWorker", "got job %v", e)
// check for errors
if e.Error() != nil {
debug.Log("Archiver.fileWorker", "job %v has errors: %v", e.Path(), e.Error())
// TODO: integrate error reporting
fmt.Fprintf(os.Stderr, "error for %v: %v\n", e.Path(), e.Error())
// ignore this file
e.Result() <- nil
p.Report(Stat{Errors: 1})
continue
}
node, err := NodeFromFileInfo(e.Fullpath(), e.Info())
if err != nil {
// TODO: integrate error reporting
debug.Log("Archiver.fileWorker", "NodeFromFileInfo returned error for %v: %v", node.path, err)
e.Result() <- nil
p.Report(Stat{Errors: 1})
continue
}
// try to use old node, if present
if e.Node != nil {
debug.Log("Archiver.fileWorker", " %v use old data", e.Path())
oldNode := e.Node.(*Node)
// check if all content is still available in the repository
contentMissing := false
for _, blob := range oldNode.blobs {
if ok, err := arch.repo.Backend().Test(backend.Data, blob.Storage.String()); !ok || err != nil {
debug.Log("Archiver.fileWorker", " %v not using old data, %v (%v) is missing", e.Path(), blob.ID.Str(), blob.Storage.Str())
contentMissing = true
break
}
}
if !contentMissing {
node.Content = oldNode.Content
node.blobs = oldNode.blobs
debug.Log("Archiver.fileWorker", " %v content is complete", e.Path())
}
} else {
debug.Log("Archiver.fileWorker", " %v no old data", e.Path())
}
// otherwise read file normally
if node.Type == "file" && len(node.Content) == 0 {
debug.Log("Archiver.fileWorker", " read and save %v, content: %v", e.Path(), node.Content)
err = arch.SaveFile(p, node)
if err != nil {
// TODO: integrate error reporting
fmt.Fprintf(os.Stderr, "error for %v: %v\n", node.path, err)
// ignore this file
e.Result() <- nil
p.Report(Stat{Errors: 1})
continue
}
} else {
// report old data size
p.Report(Stat{Bytes: node.Size})
}
debug.Log("Archiver.fileWorker", " processed %v, %d/%d blobs", e.Path(), len(node.Content), len(node.blobs))
e.Result() <- node
p.Report(Stat{Files: 1})
case <-done:
// pipeline was cancelled
return
}
}
}
func (arch *Archiver) dirWorker(wg *sync.WaitGroup, p *Progress, done <-chan struct{}, dirCh <-chan pipe.Dir) {
debug.Log("Archiver.dirWorker", "start")
defer func() {
debug.Log("Archiver.dirWorker", "done")
wg.Done()
}()
for {
select {
case dir, ok := <-dirCh:
if !ok {
// channel is closed
return
}
debug.Log("Archiver.dirWorker", "save dir %v (%d entries), error %v\n", dir.Path(), len(dir.Entries), dir.Error())
// ignore dir nodes with errors
if dir.Error() != nil {
fmt.Fprintf(os.Stderr, "error walking dir %v: %v\n", dir.Path(), dir.Error())
dir.Result() <- nil
p.Report(Stat{Errors: 1})
continue
}
tree := NewTree()
// wait for all content
for _, ch := range dir.Entries {
debug.Log("Archiver.dirWorker", "receiving result from %v", ch)
res := <-ch
// if we get a nil pointer here, an error has happened while
// processing this entry. Ignore it for now.
if res == nil {
debug.Log("Archiver.dirWorker", "got nil result?")
continue
}
// else insert node
node := res.(*Node)
tree.Insert(node)
if node.Type == "dir" {
debug.Log("Archiver.dirWorker", "got tree node for %s: %v", node.path, node.Subtree)
if node.Subtree.IsNull() {
panic("invalid null subtree ID")
}
}
}
node := &Node{}
if dir.Path() != "" && dir.Info() != nil {
n, err := NodeFromFileInfo(dir.Path(), dir.Info())
if err != nil {
n.Error = err.Error()
dir.Result() <- n
continue
}
node = n
}
if err := dir.Error(); err != nil {
node.Error = err.Error()
}
id, err := arch.SaveTreeJSON(tree)
if err != nil {
panic(err)
}
debug.Log("Archiver.dirWorker", "save tree for %s: %v", dir.Path(), id.Str())
if id.IsNull() {
panic("invalid null subtree ID return from SaveTreeJSON()")
}
node.Subtree = &id
debug.Log("Archiver.dirWorker", "sending result to %v", dir.Result())
dir.Result() <- node
if dir.Path() != "" {
p.Report(Stat{Dirs: 1})
}
case <-done:
// pipeline was cancelled
return
}
}
}
type archivePipe struct {
Old <-chan WalkTreeJob
New <-chan pipe.Job
}
func copyJobs(done <-chan struct{}, in <-chan pipe.Job, out chan<- pipe.Job) {
var (
// disable sending on the outCh until we received a job
outCh chan<- pipe.Job
// enable receiving from in
inCh = in
job pipe.Job
ok bool
)
for {
select {
case <-done:
return
case job, ok = <-inCh:
if !ok {
// input channel closed, we're done
debug.Log("copyJobs", "input channel closed, we're done")
return
}
inCh = nil
outCh = out
case outCh <- job:
outCh = nil
inCh = in
}
}
}
type archiveJob struct {
hasOld bool
old WalkTreeJob
new pipe.Job
}
func (a *archivePipe) compare(done <-chan struct{}, out chan<- pipe.Job) {
defer func() {
close(out)
debug.Log("ArchivePipe.compare", "done")
}()
debug.Log("ArchivePipe.compare", "start")
var (
loadOld, loadNew bool = true, true
ok bool
oldJob WalkTreeJob
newJob pipe.Job
)
for {
if loadOld {
oldJob, ok = <-a.Old
// if the old channel is closed, just pass through the new jobs
if !ok {
debug.Log("ArchivePipe.compare", "old channel is closed, copy from new channel")
// handle remaining newJob
if !loadNew {
out <- archiveJob{new: newJob}.Copy()
}
copyJobs(done, a.New, out)
return
}
loadOld = false
}
if loadNew {
newJob, ok = <-a.New
// if the new channel is closed, there are no more files in the current snapshot, return
if !ok {
debug.Log("ArchivePipe.compare", "new channel is closed, we're done")
return
}
loadNew = false
}
debug.Log("ArchivePipe.compare", "old job: %v", oldJob.Path)
debug.Log("ArchivePipe.compare", "new job: %v", newJob.Path())
// at this point we have received an old job as well as a new job, compare paths
file1 := oldJob.Path
file2 := newJob.Path()
dir1 := filepath.Dir(file1)
dir2 := filepath.Dir(file2)
if file1 == file2 {
debug.Log("ArchivePipe.compare", " same filename %q", file1)
// send job
out <- archiveJob{hasOld: true, old: oldJob, new: newJob}.Copy()
loadOld = true
loadNew = true
continue
} else if dir1 < dir2 {
debug.Log("ArchivePipe.compare", " %q < %q, file %q added", dir1, dir2, file2)
// file is new, send new job and load new
loadNew = true
out <- archiveJob{new: newJob}.Copy()
continue
} else if dir1 == dir2 {
if file1 < file2 {
debug.Log("ArchivePipe.compare", " %q < %q, file %q removed", file1, file2, file1)
// file has been removed, load new old
loadOld = true
continue
} else {
debug.Log("ArchivePipe.compare", " %q > %q, file %q added", file1, file2, file2)
// file is new, send new job and load new
loadNew = true
out <- archiveJob{new: newJob}.Copy()
continue
}
}
debug.Log("ArchivePipe.compare", " %q > %q, file %q removed", file1, file2, file1)
// file has been removed, throw away old job and load new
loadOld = true
}
}
func (j archiveJob) Copy() pipe.Job {
if !j.hasOld {
return j.new
}
// handle files
if isRegularFile(j.new.Info()) {
debug.Log("archiveJob.Copy", " job %v is file", j.new.Path())
// if type has changed, return new job directly
if j.old.Node == nil {
return j.new
}
// if file is newer, return the new job
if j.old.Node.isNewer(j.new.Fullpath(), j.new.Info()) {
debug.Log("archiveJob.Copy", " job %v is newer", j.new.Path())
return j.new
}
debug.Log("archiveJob.Copy", " job %v add old data", j.new.Path())
// otherwise annotate job with old data
e := j.new.(pipe.Entry)
e.Node = j.old.Node
return e
}
// dirs and other types are just returned
return j.new
}
const saveIndexTime = 30 * time.Second
// saveIndexes regularly queries the master index for full indexes and saves them.
func (arch *Archiver) saveIndexes(wg *sync.WaitGroup, done <-chan struct{}) {
defer wg.Done()
ticker := time.NewTicker(saveIndexTime)
defer ticker.Stop()
for {
select {
case <-done:
return
case <-ticker.C:
debug.Log("Archiver.saveIndexes", "saving full indexes")
err := arch.repo.SaveFullIndex()
if err != nil {
debug.Log("Archiver.saveIndexes", "save indexes returned an error: %v", err)
fmt.Fprintf(os.Stderr, "error saving preliminary index: %v\n", err)
}
}
}
}
// unique returns a slice that only contains unique strings.
func unique(items []string) []string {
seen := make(map[string]struct{})
for _, item := range items {
seen[item] = struct{}{}
}
items = items[:0]
for item := range seen {
items = append(items, item)
}
return items
}
// Snapshot creates a snapshot of the given paths. If parentID is set, this is
// used to compare the files to the ones archived at the time this snapshot was
// taken.
func (arch *Archiver) Snapshot(p *Progress, paths []string, parentID *backend.ID) (*Snapshot, backend.ID, error) {
paths = unique(paths)
sort.Strings(paths)
debug.Log("Archiver.Snapshot", "start for %v", paths)
debug.RunHook("Archiver.Snapshot", nil)
// signal the whole pipeline to stop
done := make(chan struct{})
var err error
p.Start()
defer p.Done()
// create new snapshot
sn, err := NewSnapshot(paths)
if err != nil {
return nil, backend.ID{}, err
}
sn.Excludes = arch.Excludes
jobs := archivePipe{}
// use parent snapshot (if some was given)
if parentID != nil {
sn.Parent = parentID
// load parent snapshot
parent, err := LoadSnapshot(arch.repo, *parentID)
if err != nil {
return nil, backend.ID{}, err
}
// start walker on old tree
ch := make(chan WalkTreeJob)
go WalkTree(arch.repo, *parent.Tree, done, ch)
jobs.Old = ch
} else {
// use closed channel
ch := make(chan WalkTreeJob)
close(ch)
jobs.Old = ch
}
// start walker
pipeCh := make(chan pipe.Job)
resCh := make(chan pipe.Result, 1)
go func() {
pipe.Walk(paths, arch.SelectFilter, done, pipeCh, resCh)
debug.Log("Archiver.Snapshot", "pipe.Walk done")
}()
jobs.New = pipeCh
ch := make(chan pipe.Job)
go jobs.compare(done, ch)
var wg sync.WaitGroup
entCh := make(chan pipe.Entry)
dirCh := make(chan pipe.Dir)
// split
wg.Add(1)
go func() {
pipe.Split(ch, dirCh, entCh)
debug.Log("Archiver.Snapshot", "split done")
close(dirCh)
close(entCh)
wg.Done()
}()
// run workers
for i := 0; i < maxConcurrency; i++ {
wg.Add(2)
go arch.fileWorker(&wg, p, done, entCh)
go arch.dirWorker(&wg, p, done, dirCh)
}
// run index saver
var wgIndexSaver sync.WaitGroup
stopIndexSaver := make(chan struct{})
wgIndexSaver.Add(1)
go arch.saveIndexes(&wgIndexSaver, stopIndexSaver)
// wait for all workers to terminate
debug.Log("Archiver.Snapshot", "wait for workers")
wg.Wait()
// stop index saver
close(stopIndexSaver)
wgIndexSaver.Wait()
debug.Log("Archiver.Snapshot", "workers terminated")
// receive the top-level tree
root := (<-resCh).(*Node)
debug.Log("Archiver.Snapshot", "root node received: %v", root.Subtree.Str())
sn.Tree = root.Subtree
// save snapshot
id, err := arch.repo.SaveJSONUnpacked(backend.Snapshot, sn)
if err != nil {
return nil, backend.ID{}, err
}
// store ID in snapshot struct
sn.id = &id
debug.Log("Archiver.Snapshot", "saved snapshot %v", id.Str())
// flush repository
err = arch.repo.Flush()
if err != nil {
return nil, backend.ID{}, err
}
// save index
err = arch.repo.SaveIndex()
if err != nil {
debug.Log("Archiver.Snapshot", "error saving index: %v", err)
return nil, backend.ID{}, err
}
debug.Log("Archiver.Snapshot", "saved indexes")
return sn, id, nil
}
func isRegularFile(fi os.FileInfo) bool {
if fi == nil {
return false
}
return fi.Mode()&(os.ModeType|os.ModeCharDevice) == 0
}
// Scan traverses the dirs to collect Stat information while emitting progress
// information with p.
func Scan(dirs []string, filter pipe.SelectFunc, p *Progress) (Stat, error) {
p.Start()
defer p.Done()
var stat Stat
for _, dir := range dirs {
debug.Log("Scan", "Start for %v", dir)
err := filepath.Walk(dir, func(str string, fi os.FileInfo, err error) error {
// TODO: integrate error reporting
if err != nil {
fmt.Fprintf(os.Stderr, "error for %v: %v\n", str, err)
return nil
}
if fi == nil {
fmt.Fprintf(os.Stderr, "error for %v: FileInfo is nil\n", str)
return nil
}
if !filter(str, fi) {
debug.Log("Scan.Walk", "path %v excluded", str)
if fi.IsDir() {
return filepath.SkipDir
}
return nil
}
s := Stat{}
if fi.IsDir() {
s.Dirs++
} else {
s.Files++
if isRegularFile(fi) {
s.Bytes += uint64(fi.Size())
}
}
p.Report(s)
stat.Add(s)
// TODO: handle error?
return nil
})
debug.Log("Scan", "Done for %v, err: %v", dir, err)
if err != nil {
return Stat{}, err
}
}
return stat, nil
}
src/github.com/restic/restic/archiver_duplication_test.go
+150
View File
@@ -0,0 +1,150 @@
package restic_test
import (
"bytes"
"crypto/rand"
"errors"
"io"
mrand "math/rand"
"sync"
"testing"
"time"
"github.com/restic/restic"
"github.com/restic/restic/backend"
"github.com/restic/restic/pack"
"github.com/restic/restic/repository"
)
const parallelSaves = 50
const testSaveIndexTime = 100 * time.Millisecond
const testTimeout = 2 * time.Second
var DupID backend.ID
func randomID() backend.ID {
if mrand.Float32() < 0.5 {
return DupID
}
id := backend.ID{}
_, err := io.ReadFull(rand.Reader, id[:])
if err != nil {
panic(err)
}
return id
}
// forgetfulBackend returns a backend that forgets everything.
func forgetfulBackend() backend.Backend {
be := &backend.MockBackend{}
be.TestFn = func(t backend.Type, name string) (bool, error) {
return false, nil
}
be.LoadFn = func(h backend.Handle, p []byte, off int64) (int, error) {
return 0, errors.New("not found")
}
be.SaveFn = func(h backend.Handle, p []byte) error {
return nil
}
be.StatFn = func(h backend.Handle) (backend.BlobInfo, error) {
return backend.BlobInfo{}, errors.New("not found")
}
be.RemoveFn = func(t backend.Type, name string) error {
return nil
}
be.ListFn = func(t backend.Type, done <-chan struct{}) <-chan string {
ch := make(chan string)
close(ch)
return ch
}
be.DeleteFn = func() error {
return nil
}
return be
}
func testArchiverDuplication(t *testing.T) {
_, err := io.ReadFull(rand.Reader, DupID[:])
if err != nil {
t.Fatal(err)
}
repo := repository.New(forgetfulBackend())
err = repo.Init("foo")
if err != nil {
t.Fatal(err)
}
arch := restic.NewArchiver(repo)
wg := &sync.WaitGroup{}
done := make(chan struct{})
for i := 0; i < parallelSaves; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for {
select {
case <-done:
return
default:
}
id := randomID()
if repo.Index().Has(id) {
continue
}
buf := make([]byte, 50)
err := arch.Save(pack.Data, id, uint(len(buf)), bytes.NewReader(buf))
if err != nil {
t.Fatal(err)
}
}
}()
}
saveIndex := func() {
defer wg.Done()
ticker := time.NewTicker(testSaveIndexTime)
defer ticker.Stop()
for {
select {
case <-done:
return
case <-ticker.C:
err := repo.SaveFullIndex()
if err != nil {
t.Fatal(err)
}
}
}
}
wg.Add(1)
go saveIndex()
<-time.After(testTimeout)
close(done)
wg.Wait()
}
func TestArchiverDuplication(t *testing.T) {
for i := 0; i < 5; i++ {
testArchiverDuplication(t)
}
}
src/github.com/restic/restic/archiver_int_test.go
+143
View File
@@ -0,0 +1,143 @@
package restic
import (
"os"
"testing"
"github.com/restic/restic/pipe"
)
var treeJobs = []string{
"foo/baz/subdir",
"foo/baz",
"foo",
"quu/bar/file1",
"quu/bar/file2",
"quu/foo/file1",
"quu/foo/file2",
"quu/foo/file3",
"quu/foo",
"quu/fooz",
"quu",
"yy/a",
"yy/b",
"yy",
}
var pipeJobs = []string{
"foo/baz/subdir",
"foo/baz/subdir2", // subdir2 added
"foo/baz",
"foo",
"quu/bar/.file1.swp", // file with . added
"quu/bar/file1",
"quu/bar/file2",
"quu/foo/file1", // file2 removed
"quu/foo/file3",
"quu/foo",
"quu",
"quv/file1", // files added and removed
"quv/file2",
"quv",
"yy",
"zz/file1", // files removed and added at the end
"zz/file2",
"zz",
}
var resultJobs = []struct {
path string
action string
}{
{"foo/baz/subdir", "same, not a file"},
{"foo/baz/subdir2", "new, no old job"},
{"foo/baz", "same, not a file"},
{"foo", "same, not a file"},
{"quu/bar/.file1.swp", "new, no old job"},
{"quu/bar/file1", "same, not a file"},
{"quu/bar/file2", "same, not a file"},
{"quu/foo/file1", "same, not a file"},
{"quu/foo/file3", "same, not a file"},
{"quu/foo", "same, not a file"},
{"quu", "same, not a file"},
{"quv/file1", "new, no old job"},
{"quv/file2", "new, no old job"},
{"quv", "new, no old job"},
{"yy", "same, not a file"},
{"zz/file1", "testPipeJob"},
{"zz/file2", "testPipeJob"},
{"zz", "testPipeJob"},
}
type testPipeJob struct {
path string
err error
fi os.FileInfo
res chan<- pipe.Result
}
func (j testPipeJob) Path() string { return j.path }
func (j testPipeJob) Fullpath() string { return j.path }
func (j testPipeJob) Error() error { return j.err }
func (j testPipeJob) Info() os.FileInfo { return j.fi }
func (j testPipeJob) Result() chan<- pipe.Result { return j.res }
func testTreeWalker(done <-chan struct{}, out chan<- WalkTreeJob) {
for _, e := range treeJobs {
select {
case <-done:
return
case out <- WalkTreeJob{Path: e}:
}
}
close(out)
}
func testPipeWalker(done <-chan struct{}, out chan<- pipe.Job) {
for _, e := range pipeJobs {
select {
case <-done:
return
case out <- testPipeJob{path: e}:
}
}
close(out)
}
func TestArchivePipe(t *testing.T) {
done := make(chan struct{})
treeCh := make(chan WalkTreeJob)
pipeCh := make(chan pipe.Job)
go testTreeWalker(done, treeCh)
go testPipeWalker(done, pipeCh)
p := archivePipe{Old: treeCh, New: pipeCh}
ch := make(chan pipe.Job)
go p.compare(done, ch)
i := 0
for job := range ch {
if job.Path() != resultJobs[i].path {
t.Fatalf("wrong job received: wanted %v, got %v", resultJobs[i], job)
}
// switch j := job.(type) {
// case archivePipeJob:
// if j.action != resultJobs[i].action {
// t.Fatalf("wrong action for %v detected: wanted %q, got %q", job.Path(), resultJobs[i].action, j.action)
// }
// case testPipeJob:
// if resultJobs[i].action != "testPipeJob" {
// t.Fatalf("unexpected testPipeJob, expected %q: %v", resultJobs[i].action, j)
// }
// }
i++
}
}
src/github.com/restic/restic/archiver_test.go
+342
View File
@@ -0,0 +1,342 @@
package restic_test
import (
"bytes"
"crypto/sha256"
"io"
"testing"
"time"
"github.com/restic/chunker"
"github.com/restic/restic"
"github.com/restic/restic/backend"
"github.com/restic/restic/checker"
"github.com/restic/restic/crypto"
"github.com/restic/restic/pack"
"github.com/restic/restic/repository"
. "github.com/restic/restic/test"
)
var testPol = chunker.Pol(0x3DA3358B4DC173)
type Rdr interface {
io.ReadSeeker
io.ReaderAt
}
type chunkedData struct {
buf []byte
chunks []*chunker.Chunk
}
func benchmarkChunkEncrypt(b testing.TB, buf, buf2 []byte, rd Rdr, key *crypto.Key) {
rd.Seek(0, 0)
ch := chunker.New(rd, testPol, sha256.New())
for {
chunk, err := ch.Next()
if err == io.EOF {
break
}
OK(b, err)
// reduce length of buf
buf = buf[:chunk.Length]
n, err := io.ReadFull(chunk.Reader(rd), buf)
OK(b, err)
Assert(b, uint(n) == chunk.Length, "invalid length: got %d, expected %d", n, chunk.Length)
_, err = crypto.Encrypt(key, buf2, buf)
OK(b, err)
}
}
func BenchmarkChunkEncrypt(b *testing.B) {
repo := SetupRepo()
defer TeardownRepo(repo)
data := Random(23, 10<<20) // 10MiB
rd := bytes.NewReader(data)
buf := make([]byte, chunker.MaxSize)
buf2 := make([]byte, chunker.MaxSize)
b.ResetTimer()
b.SetBytes(int64(len(data)))
for i := 0; i < b.N; i++ {
benchmarkChunkEncrypt(b, buf, buf2, rd, repo.Key())
}
}
func benchmarkChunkEncryptP(b *testing.PB, buf []byte, rd Rdr, key *crypto.Key) {
ch := chunker.New(rd, testPol, sha256.New())
for {
chunk, err := ch.Next()
if err == io.EOF {
break
}
// reduce length of chunkBuf
buf = buf[:chunk.Length]
io.ReadFull(chunk.Reader(rd), buf)
crypto.Encrypt(key, buf, buf)
}
}
func BenchmarkChunkEncryptParallel(b *testing.B) {
repo := SetupRepo()
defer TeardownRepo(repo)
data := Random(23, 10<<20) // 10MiB
buf := make([]byte, chunker.MaxSize)
b.ResetTimer()
b.SetBytes(int64(len(data)))
b.RunParallel(func(pb *testing.PB) {
for pb.Next() {
rd := bytes.NewReader(data)
benchmarkChunkEncryptP(pb, buf, rd, repo.Key())
}
})
}
func archiveDirectory(b testing.TB) {
repo := SetupRepo()
defer TeardownRepo(repo)
arch := restic.NewArchiver(repo)
_, id, err := arch.Snapshot(nil, []string{BenchArchiveDirectory}, nil)
OK(b, err)
b.Logf("snapshot archived as %v", id)
}
func TestArchiveDirectory(t *testing.T) {
if BenchArchiveDirectory == "" {
t.Skip("benchdir not set, skipping TestArchiveDirectory")
}
archiveDirectory(t)
}
func BenchmarkArchiveDirectory(b *testing.B) {
if BenchArchiveDirectory == "" {
b.Skip("benchdir not set, skipping BenchmarkArchiveDirectory")
}
for i := 0; i < b.N; i++ {
archiveDirectory(b)
}
}
func archiveWithDedup(t testing.TB) {
repo := SetupRepo()
defer TeardownRepo(repo)
if BenchArchiveDirectory == "" {
t.Skip("benchdir not set, skipping TestArchiverDedup")
}
var cnt struct {
before, after, after2 struct {
packs, dataBlobs, treeBlobs uint
}
}
// archive a few files
sn := SnapshotDir(t, repo, BenchArchiveDirectory, nil)
t.Logf("archived snapshot %v", sn.ID().Str())
// get archive stats
cnt.before.packs = repo.Count(backend.Data)
cnt.before.dataBlobs = repo.Index().Count(pack.Data)
cnt.before.treeBlobs = repo.Index().Count(pack.Tree)
t.Logf("packs %v, data blobs %v, tree blobs %v",
cnt.before.packs, cnt.before.dataBlobs, cnt.before.treeBlobs)
// archive the same files again, without parent snapshot
sn2 := SnapshotDir(t, repo, BenchArchiveDirectory, nil)
t.Logf("archived snapshot %v", sn2.ID().Str())
// get archive stats again
cnt.after.packs = repo.Count(backend.Data)
cnt.after.dataBlobs = repo.Index().Count(pack.Data)
cnt.after.treeBlobs = repo.Index().Count(pack.Tree)
t.Logf("packs %v, data blobs %v, tree blobs %v",
cnt.after.packs, cnt.after.dataBlobs, cnt.after.treeBlobs)
// if there are more data blobs, something is wrong
if cnt.after.dataBlobs > cnt.before.dataBlobs {
t.Fatalf("TestArchiverDedup: too many data blobs in repository: before %d, after %d",
cnt.before.dataBlobs, cnt.after.dataBlobs)
}
// archive the same files again, with a parent snapshot
sn3 := SnapshotDir(t, repo, BenchArchiveDirectory, sn2.ID())
t.Logf("archived snapshot %v, parent %v", sn3.ID().Str(), sn2.ID().Str())
// get archive stats again
cnt.after2.packs = repo.Count(backend.Data)
cnt.after2.dataBlobs = repo.Index().Count(pack.Data)
cnt.after2.treeBlobs = repo.Index().Count(pack.Tree)
t.Logf("packs %v, data blobs %v, tree blobs %v",
cnt.after2.packs, cnt.after2.dataBlobs, cnt.after2.treeBlobs)
// if there are more data blobs, something is wrong
if cnt.after2.dataBlobs > cnt.before.dataBlobs {
t.Fatalf("TestArchiverDedup: too many data blobs in repository: before %d, after %d",
cnt.before.dataBlobs, cnt.after2.dataBlobs)
}
}
func TestArchiveDedup(t *testing.T) {
archiveWithDedup(t)
}
func BenchmarkLoadTree(t *testing.B) {
repo := SetupRepo()
defer TeardownRepo(repo)
if BenchArchiveDirectory == "" {
t.Skip("benchdir not set, skipping TestArchiverDedup")
}
// archive a few files
arch := restic.NewArchiver(repo)
sn, _, err := arch.Snapshot(nil, []string{BenchArchiveDirectory}, nil)
OK(t, err)
t.Logf("archived snapshot %v", sn.ID())
list := make([]backend.ID, 0, 10)
done := make(chan struct{})
for _, idx := range repo.Index().All() {
for blob := range idx.Each(done) {
if blob.Type != pack.Tree {
continue
}
list = append(list, blob.ID)
if len(list) == cap(list) {
close(done)
break
}
}
}
// start benchmark
t.ResetTimer()
for i := 0; i < t.N; i++ {
for _, id := range list {
_, err := restic.LoadTree(repo, id)
OK(t, err)
}
}
}
// Saves several identical chunks concurrently and later checks that there are no
// unreferenced packs in the repository. See also #292 and #358.
func TestParallelSaveWithDuplication(t *testing.T) {
for seed := 0; seed < 10; seed++ {
testParallelSaveWithDuplication(t, seed)
}
}
func testParallelSaveWithDuplication(t *testing.T, seed int) {
repo := SetupRepo()
defer TeardownRepo(repo)
dataSizeMb := 128
duplication := 7
arch := restic.NewArchiver(repo)
data, chunks := getRandomData(seed, dataSizeMb*1024*1024)
reader := bytes.NewReader(data)
errChannels := [](<-chan error){}
// interweaved processing of subsequent chunks
maxParallel := 2*duplication - 1
barrier := make(chan struct{}, maxParallel)
for _, c := range chunks {
for dupIdx := 0; dupIdx < duplication; dupIdx++ {
errChan := make(chan error)
errChannels = append(errChannels, errChan)
go func(reader *bytes.Reader, c *chunker.Chunk, errChan chan<- error) {
barrier <- struct{}{}
hash := c.Digest
id := backend.ID{}
copy(id[:], hash)
time.Sleep(time.Duration(hash[0]))
err := arch.Save(pack.Data, id, c.Length, c.Reader(reader))
<-barrier
errChan <- err
}(reader, c, errChan)
}
}
for _, errChan := range errChannels {
OK(t, <-errChan)
}
OK(t, repo.Flush())
OK(t, repo.SaveIndex())
chkr := createAndInitChecker(t, repo)
assertNoUnreferencedPacks(t, chkr)
}
func getRandomData(seed int, size int) ([]byte, []*chunker.Chunk) {
buf := Random(seed, size)
chunks := []*chunker.Chunk{}
chunker := chunker.New(bytes.NewReader(buf), testPol, sha256.New())
for {
c, err := chunker.Next()
if err == io.EOF {
break
}
chunks = append(chunks, c)
}
return buf, chunks
}
func createAndInitChecker(t *testing.T, repo *repository.Repository) *checker.Checker {
chkr := checker.New(repo)
hints, errs := chkr.LoadIndex()
if len(errs) > 0 {
t.Fatalf("expected no errors, got %v: %v", len(errs), errs)
}
if len(hints) > 0 {
t.Errorf("expected no hints, got %v: %v", len(hints), hints)
}
return chkr
}
func assertNoUnreferencedPacks(t *testing.T, chkr *checker.Checker) {
done := make(chan struct{})
defer close(done)
errChan := make(chan error)
go chkr.Packs(errChan, done)
for err := range errChan {
OK(t, err)
}
}
src/github.com/restic/restic/backend/doc.go
+5
View File
@@ -0,0 +1,5 @@
// Package backend provides local and remote storage for restic repositories.
// All backends need to implement the Backend interface. There is a
// MockBackend, which can be used for mocking in tests, and a MemBackend, which
// stores all data in a hash internally.
package backend
src/github.com/restic/restic/backend/generic.go
+70
View File
@@ -0,0 +1,70 @@
package backend
import "errors"
// ErrNoIDPrefixFound is returned by Find() when no ID for the given prefix
// could be found.
var ErrNoIDPrefixFound = errors.New("no ID found")
// ErrMultipleIDMatches is returned by Find() when multiple IDs with the given
// prefix are found.
var ErrMultipleIDMatches = errors.New("multiple IDs with prefix found")
// Find loads the list of all blobs of type t and searches for names which
// start with prefix. If none is found, nil and ErrNoIDPrefixFound is returned.
// If more than one is found, nil and ErrMultipleIDMatches is returned.
func Find(be Lister, t Type, prefix string) (string, error) {
done := make(chan struct{})
defer close(done)
match := ""
// TODO: optimize by sorting list etc.
for name := range be.List(t, done) {
if prefix == name[:len(prefix)] {
if match == "" {
match = name
} else {
return "", ErrMultipleIDMatches
}
}
}
if match != "" {
return match, nil
}
return "", ErrNoIDPrefixFound
}
const minPrefixLength = 8
// PrefixLength returns the number of bytes required so that all prefixes of
// all names of type t are unique.
func PrefixLength(be Lister, t Type) (int, error) {
done := make(chan struct{})
defer close(done)
// load all IDs of the given type
list := make([]string, 0, 100)
for name := range be.List(t, done) {
list = append(list, name)
}
// select prefixes of length l, test if the last one is the same as the current one
outer:
for l := minPrefixLength; l < IDSize; l++ {
var last string
for _, name := range list {
if last == name[:l] {
continue outer
}
last = name[:l]
}
return l, nil
}
return IDSize, nil
}
src/github.com/restic/restic/backend/generic_test.go
+61
View File
@@ -0,0 +1,61 @@
package backend_test
import (
"testing"
"github.com/restic/restic/backend"
. "github.com/restic/restic/test"
)
type mockBackend struct {
list func(backend.Type, <-chan struct{}) <-chan string
}
func (m mockBackend) List(t backend.Type, done <-chan struct{}) <-chan string {
return m.list(t, done)
}
var samples = backend.IDs{
ParseID("20bdc1402a6fc9b633aaffffffffffffffffffffffffffffffffffffffffffff"),
ParseID("20bdc1402a6fc9b633ccd578c4a92d0f4ef1a457fa2e16c596bc73fb409d6cc0"),
ParseID("20bdc1402a6fc9b633ffffffffffffffffffffffffffffffffffffffffffffff"),
ParseID("20ff988befa5fc40350f00d531a767606efefe242c837aaccb80673f286be53d"),
ParseID("326cb59dfe802304f96ee9b5b9af93bdee73a30f53981e5ec579aedb6f1d0f07"),
ParseID("86b60b9594d1d429c4aa98fa9562082cabf53b98c7dc083abe5dae31074dd15a"),
ParseID("96c8dbe225079e624b5ce509f5bd817d1453cd0a85d30d536d01b64a8669aeae"),
ParseID("fa31d65b87affcd167b119e9d3d2a27b8236ca4836cb077ed3e96fcbe209b792"),
}
func TestPrefixLength(t *testing.T) {
list := samples
m := mockBackend{}
m.list = func(t backend.Type, done <-chan struct{}) <-chan string {
ch := make(chan string)
go func() {
defer close(ch)
for _, id := range list {
select {
case ch <- id.String():
case <-done:
return
}
}
}()
return ch
}
l, err := backend.PrefixLength(m, backend.Snapshot)
OK(t, err)
Equals(t, 19, l)
list = samples[:3]
l, err = backend.PrefixLength(m, backend.Snapshot)
OK(t, err)
Equals(t, 19, l)
list = samples[3:]
l, err = backend.PrefixLength(m, backend.Snapshot)
OK(t, err)
Equals(t, 8, l)
}
src/github.com/restic/restic/backend/handle.go
+48
View File
@@ -0,0 +1,48 @@
package backend
import (
"errors"
"fmt"
)
// Handle is used to store and access data in a backend.
type Handle struct {
Type Type
Name string
}
func (h Handle) String() string {
name := h.Name
if len(name) > 10 {
name = name[:10]
}
return fmt.Sprintf("<%s/%s>", h.Type, name)
}
// Valid returns an error if h is not valid.
func (h Handle) Valid() error {
if h.Type == "" {
return errors.New("type is empty")
}
switch h.Type {
case Data:
case Key:
case Lock:
case Snapshot:
case Index:
case Config:
default:
return fmt.Errorf("invalid Type %q", h.Type)
}
if h.Type == Config {
return nil
}
if h.Name == "" {
return errors.New("invalid Name")
}
return nil
}
src/github.com/restic/restic/backend/handle_test.go
+28
View File
@@ -0,0 +1,28 @@
package backend
import "testing"
var handleTests = []struct {
h Handle
valid bool
}{
{Handle{Name: "foo"}, false},
{Handle{Type: "foobar"}, false},
{Handle{Type: Config, Name: ""}, true},
{Handle{Type: Data, Name: ""}, false},
{Handle{Type: "", Name: "x"}, false},
{Handle{Type: Lock, Name: "010203040506"}, true},
}
func TestHandleValid(t *testing.T) {
for i, test := range handleTests {
err := test.h.Valid()
if err != nil && test.valid {
t.Errorf("test %v failed: error returned for valid handle: %v", i, err)
}
if !test.valid && err == nil {
t.Errorf("test %v failed: expected error for invalid handle not found", i)
}
}
}
src/github.com/restic/restic/backend/id.go
+108
View File
@@ -0,0 +1,108 @@
package backend
import (
"bytes"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
)
// Hash returns the ID for data.
func Hash(data []byte) ID {
return sha256.Sum256(data)
}
// IDSize contains the size of an ID, in bytes.
const IDSize = sha256.Size
// ID references content within a repository.
type ID [IDSize]byte
// ParseID converts the given string to an ID.
func ParseID(s string) (ID, error) {
b, err := hex.DecodeString(s)
if err != nil {
return ID{}, err
}
if len(b) != IDSize {
return ID{}, errors.New("invalid length for hash")
}
id := ID{}
copy(id[:], b)
return id, nil
}
func (id ID) String() string {
return hex.EncodeToString(id[:])
}
const shortStr = 4
// Str returns the shortened string version of id.
func (id *ID) Str() string {
if id == nil {
return "[nil]"
}
if id.IsNull() {
return "[null]"
}
return hex.EncodeToString(id[:shortStr])
}
// IsNull returns true iff id only consists of null bytes.
func (id ID) IsNull() bool {
var nullID ID
return id == nullID
}
// Equal compares an ID to another other.
func (id ID) Equal(other ID) bool {
return id == other
}
// EqualString compares this ID to another one, given as a string.
func (id ID) EqualString(other string) (bool, error) {
s, err := hex.DecodeString(other)
if err != nil {
return false, err
}
id2 := ID{}
copy(id2[:], s)
return id == id2, nil
}
// Compare compares this ID to another one, returning -1, 0, or 1.
func (id ID) Compare(other ID) int {
return bytes.Compare(other[:], id[:])
}
// MarshalJSON returns the JSON encoding of id.
func (id ID) MarshalJSON() ([]byte, error) {
return json.Marshal(id.String())
}
// UnmarshalJSON parses the JSON-encoded data and stores the result in id.
func (id *ID) UnmarshalJSON(b []byte) error {
var s string
err := json.Unmarshal(b, &s)
if err != nil {
return err
}
_, err = hex.Decode(id[:], []byte(s))
if err != nil {
return err
}
return nil
}
src/github.com/restic/restic/backend/id_int_test.go
+16
View File
@@ -0,0 +1,16 @@
package backend
import "testing"
func TestIDMethods(t *testing.T) {
var id ID
if id.Str() != "[null]" {
t.Errorf("ID.Str() returned wrong value, want %v, got %v", "[null]", id.Str())
}
var pid *ID
if pid.Str() != "[nil]" {
t.Errorf("ID.Str() returned wrong value, want %v, got %v", "[nil]", pid.Str())
}
}
src/github.com/restic/restic/backend/id_test.go
+43
View File
@@ -0,0 +1,43 @@
package backend_test
import (
"testing"
"github.com/restic/restic/backend"
. "github.com/restic/restic/test"
)
var TestStrings = []struct {
id string
data string
}{
{"c3ab8ff13720e8ad9047dd39466b3c8974e592c2fa383d4a3960714caef0c4f2", "foobar"},
{"248d6a61d20638b8e5c026930c3e6039a33ce45964ff2167f6ecedd419db06c1", "abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq"},
{"cc5d46bdb4991c6eae3eb739c9c8a7a46fe9654fab79c47b4fe48383b5b25e1c", "foo/bar"},
{"4e54d2c721cbdb730f01b10b62dec622962b36966ec685880effa63d71c808f2", "foo/../../baz"},
}
func TestID(t *testing.T) {
for _, test := range TestStrings {
id, err := backend.ParseID(test.id)
OK(t, err)
id2, err := backend.ParseID(test.id)
OK(t, err)
Assert(t, id.Equal(id2), "ID.Equal() does not work as expected")
ret, err := id.EqualString(test.id)
OK(t, err)
Assert(t, ret, "ID.EqualString() returned wrong value")
// test json marshalling
buf, err := id.MarshalJSON()
OK(t, err)
Equals(t, "\""+test.id+"\"", string(buf))
var id3 backend.ID
err = id3.UnmarshalJSON(buf)
OK(t, err)
Equals(t, id, id3)
}
}
src/github.com/restic/restic/backend/ids.go
+69
View File
@@ -0,0 +1,69 @@
package backend
import (
"encoding/hex"
"fmt"
)
// IDs is an ordered list of IDs that implements sort.Interface.
type IDs []ID
func (ids IDs) Len() int {
return len(ids)
}
func (ids IDs) Less(i, j int) bool {
if len(ids[i]) < len(ids[j]) {
return true
}
for k, b := range ids[i] {
if b == ids[j][k] {
continue
}
if b < ids[j][k] {
return true
}
return false
}
return false
}
func (ids IDs) Swap(i, j int) {
ids[i], ids[j] = ids[j], ids[i]
}
// Uniq returns list without duplicate IDs. The returned list retains the order
// of the original list so that the order of the first occurrence of each ID
// stays the same.
func (ids IDs) Uniq() (list IDs) {
seen := NewIDSet()
for _, id := range ids {
if seen.Has(id) {
continue
}
list = append(list, id)
seen.Insert(id)
}
return list
}
type shortID ID
func (id shortID) String() string {
return hex.EncodeToString(id[:shortStr])
}
func (ids IDs) String() string {
elements := make([]shortID, 0, len(ids))
for _, id := range ids {
elements = append(elements, shortID(id))
}
return fmt.Sprintf("%v", elements)
}
src/github.com/restic/restic/backend/ids_test.go
+58
View File
@@ -0,0 +1,58 @@
package backend_test
import (
"reflect"
"testing"
"github.com/restic/restic/backend"
. "github.com/restic/restic/test"
)
var uniqTests = []struct {
before, after backend.IDs
}{
{
backend.IDs{
ParseID("7bb086db0d06285d831485da8031281e28336a56baa313539eaea1c73a2a1a40"),
ParseID("1285b30394f3b74693cc29a758d9624996ae643157776fce8154aabd2f01515f"),
ParseID("7bb086db0d06285d831485da8031281e28336a56baa313539eaea1c73a2a1a40"),
},
backend.IDs{
ParseID("7bb086db0d06285d831485da8031281e28336a56baa313539eaea1c73a2a1a40"),
ParseID("1285b30394f3b74693cc29a758d9624996ae643157776fce8154aabd2f01515f"),
},
},
{
backend.IDs{
ParseID("1285b30394f3b74693cc29a758d9624996ae643157776fce8154aabd2f01515f"),
ParseID("7bb086db0d06285d831485da8031281e28336a56baa313539eaea1c73a2a1a40"),
ParseID("7bb086db0d06285d831485da8031281e28336a56baa313539eaea1c73a2a1a40"),
},
backend.IDs{
ParseID("1285b30394f3b74693cc29a758d9624996ae643157776fce8154aabd2f01515f"),
ParseID("7bb086db0d06285d831485da8031281e28336a56baa313539eaea1c73a2a1a40"),
},
},
{
backend.IDs{
ParseID("1285b30394f3b74693cc29a758d9624996ae643157776fce8154aabd2f01515f"),
ParseID("f658198b405d7e80db5ace1980d125c8da62f636b586c46bf81dfb856a49d0c8"),
ParseID("7bb086db0d06285d831485da8031281e28336a56baa313539eaea1c73a2a1a40"),
ParseID("7bb086db0d06285d831485da8031281e28336a56baa313539eaea1c73a2a1a40"),
},
backend.IDs{
ParseID("1285b30394f3b74693cc29a758d9624996ae643157776fce8154aabd2f01515f"),
ParseID("f658198b405d7e80db5ace1980d125c8da62f636b586c46bf81dfb856a49d0c8"),
ParseID("7bb086db0d06285d831485da8031281e28336a56baa313539eaea1c73a2a1a40"),
},
},
}
func TestUniqIDs(t *testing.T) {
for i, test := range uniqTests {
uniq := test.before.Uniq()
if !reflect.DeepEqual(uniq, test.after) {
t.Errorf("uniqIDs() test %v failed\n wanted: %v\n got: %v", i, test.after, uniq)
}
}
}
src/github.com/restic/restic/backend/idset.go
+74
View File
@@ -0,0 +1,74 @@
package backend
import "sort"
// IDSet is a set of IDs.
type IDSet map[ID]struct{}
// NewIDSet returns a new IDSet, populated with ids.
func NewIDSet(ids ...ID) IDSet {
m := make(IDSet)
for _, id := range ids {
m[id] = struct{}{}
}
return m
}
// Has returns true iff id is contained in the set.
func (s IDSet) Has(id ID) bool {
_, ok := s[id]
return ok
}
// Insert adds id to the set.
func (s IDSet) Insert(id ID) {
s[id] = struct{}{}
}
// Delete removes id from the set.
func (s IDSet) Delete(id ID) {
delete(s, id)
}
// List returns a slice of all IDs in the set.
func (s IDSet) List() IDs {
list := make(IDs, 0, len(s))
for id := range s {
list = append(list, id)
}
sort.Sort(list)
return list
}
// Equals returns true iff s equals other.
func (s IDSet) Equals(other IDSet) bool {
if len(s) != len(other) {
return false
}
for id := range s {
if _, ok := other[id]; !ok {
return false
}
}
for id := range other {
if _, ok := s[id]; !ok {
return false
}
}
return true
}
func (s IDSet) String() string {
str := s.List().String()
if len(str) < 2 {
return "{}"
}
return "{" + str[1:len(str)-1] + "}"
}
src/github.com/restic/restic/backend/idset_test.go
+35
View File
@@ -0,0 +1,35 @@
package backend_test
import (
"testing"
"github.com/restic/restic/backend"
. "github.com/restic/restic/test"
)
var idsetTests = []struct {
id backend.ID
seen bool
}{
{ParseID("7bb086db0d06285d831485da8031281e28336a56baa313539eaea1c73a2a1a40"), false},
{ParseID("1285b30394f3b74693cc29a758d9624996ae643157776fce8154aabd2f01515f"), false},
{ParseID("7bb086db0d06285d831485da8031281e28336a56baa313539eaea1c73a2a1a40"), true},
{ParseID("7bb086db0d06285d831485da8031281e28336a56baa313539eaea1c73a2a1a40"), true},
{ParseID("1285b30394f3b74693cc29a758d9624996ae643157776fce8154aabd2f01515f"), true},
{ParseID("f658198b405d7e80db5ace1980d125c8da62f636b586c46bf81dfb856a49d0c8"), false},
{ParseID("7bb086db0d06285d831485da8031281e28336a56baa313539eaea1c73a2a1a40"), true},
{ParseID("1285b30394f3b74693cc29a758d9624996ae643157776fce8154aabd2f01515f"), true},
{ParseID("f658198b405d7e80db5ace1980d125c8da62f636b586c46bf81dfb856a49d0c8"), true},
{ParseID("7bb086db0d06285d831485da8031281e28336a56baa313539eaea1c73a2a1a40"), true},
}
func TestIDSet(t *testing.T) {
set := backend.NewIDSet()
for i, test := range idsetTests {
seen := set.Has(test.id)
if seen != test.seen {
t.Errorf("IDSet test %v failed: wanted %v, got %v", i, test.seen, seen)
}
set.Insert(test.id)
}
}
src/github.com/restic/restic/backend/interface.go
+61
View File
@@ -0,0 +1,61 @@
package backend
// Type is the type of a Blob.
type Type string
// These are the different data types a backend can store.
const (
Data Type = "data"
Key = "key"
Lock = "lock"
Snapshot = "snapshot"
Index = "index"
Config = "config"
)
// Backend is used to store and access data.
type Backend interface {
// Location returns a string that describes the type and location of the
// repository.
Location() string
// Test a boolean value whether a Blob with the name and type exists.
Test(t Type, name string) (bool, error)
// Remove removes a Blob with type t and name.
Remove(t Type, name string) error
// Close the backend
Close() error
Lister
// Load returns the data stored in the backend for h at the given offset
// and saves it in p. Load has the same semantics as io.ReaderAt.
Load(h Handle, p []byte, off int64) (int, error)
// Save stores the data in the backend under the given handle.
Save(h Handle, p []byte) error
// Stat returns information about the blob identified by h.
Stat(h Handle) (BlobInfo, error)
}
// Lister implements listing data items stored in a backend.
type Lister interface {
// List returns a channel that yields all names of blobs of type t in an
// arbitrary order. A goroutine is started for this. If the channel done is
// closed, sending stops.
List(t Type, done <-chan struct{}) <-chan string
}
// Deleter are backends that allow to self-delete all content stored in them.
type Deleter interface {
// Delete the complete repository.
Delete() error
}
// BlobInfo is returned by Stat() and contains information about a stored blob.
type BlobInfo struct {
Size int64
}
src/github.com/restic/restic/backend/local/backend_test.go
+87
View File
@@ -0,0 +1,87 @@
// DO NOT EDIT, AUTOMATICALLY GENERATED
package local_test
import (
"testing"
"github.com/restic/restic/backend/test"
)
var SkipMessage string
func TestLocalBackendCreate(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestCreate(t)
}
func TestLocalBackendOpen(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestOpen(t)
}
func TestLocalBackendCreateWithConfig(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestCreateWithConfig(t)
}
func TestLocalBackendLocation(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestLocation(t)
}
func TestLocalBackendConfig(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestConfig(t)
}
func TestLocalBackendLoad(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestLoad(t)
}
func TestLocalBackendSave(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestSave(t)
}
func TestLocalBackendSaveFilenames(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestSaveFilenames(t)
}
func TestLocalBackendBackend(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestBackend(t)
}
func TestLocalBackendDelete(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestDelete(t)
}
func TestLocalBackendCleanup(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestCleanup(t)
}
src/github.com/restic/restic/backend/local/config.go
+15
View File
@@ -0,0 +1,15 @@
package local
import (
"errors"
"strings"
)
// ParseConfig parses a local backend config.
func ParseConfig(cfg string) (interface{}, error) {
if !strings.HasPrefix(cfg, "local:") {
return nil, errors.New(`invalid format, prefix "local" not found`)
}
return cfg[6:], nil
}
src/github.com/restic/restic/backend/local/doc.go
+2
View File
@@ -0,0 +1,2 @@
// Package local implements repository storage in a local directory.
package local
src/github.com/restic/restic/backend/local/local.go
+288
View File
@@ -0,0 +1,288 @@
package local
import (
"errors"
"fmt"
"io"
"io/ioutil"
"os"
"path/filepath"
"github.com/restic/restic/backend"
"github.com/restic/restic/debug"
)
// Local is a backend in a local directory.
type Local struct {
p string
}
func paths(dir string) []string {
return []string{
dir,
filepath.Join(dir, backend.Paths.Data),
filepath.Join(dir, backend.Paths.Snapshots),
filepath.Join(dir, backend.Paths.Index),
filepath.Join(dir, backend.Paths.Locks),
filepath.Join(dir, backend.Paths.Keys),
filepath.Join(dir, backend.Paths.Temp),
}
}
// Open opens the local backend as specified by config.
func Open(dir string) (*Local, error) {
// test if all necessary dirs are there
for _, d := range paths(dir) {
if _, err := os.Stat(d); err != nil {
return nil, fmt.Errorf("%s does not exist", d)
}
}
return &Local{p: dir}, nil
}
// Create creates all the necessary files and directories for a new local
// backend at dir. Afterwards a new config blob should be created.
func Create(dir string) (*Local, error) {
// test if config file already exists
_, err := os.Lstat(filepath.Join(dir, backend.Paths.Config))
if err == nil {
return nil, errors.New("config file already exists")
}
// create paths for data, refs and temp
for _, d := range paths(dir) {
err := os.MkdirAll(d, backend.Modes.Dir)
if err != nil {
return nil, err
}
}
// open backend
return Open(dir)
}
// Location returns this backend's location (the directory name).
func (b *Local) Location() string {
return b.p
}
// Construct path for given Type and name.
func filename(base string, t backend.Type, name string) string {
if t == backend.Config {
return filepath.Join(base, "config")
}
return filepath.Join(dirname(base, t, name), name)
}
// Construct directory for given Type.
func dirname(base string, t backend.Type, name string) string {
var n string
switch t {
case backend.Data:
n = backend.Paths.Data
if len(name) > 2 {
n = filepath.Join(n, name[:2])
}
case backend.Snapshot:
n = backend.Paths.Snapshots
case backend.Index:
n = backend.Paths.Index
case backend.Lock:
n = backend.Paths.Locks
case backend.Key:
n = backend.Paths.Keys
}
return filepath.Join(base, n)
}
// Load returns the data stored in the backend for h at the given offset
// and saves it in p. Load has the same semantics as io.ReaderAt.
func (b *Local) Load(h backend.Handle, p []byte, off int64) (n int, err error) {
if err := h.Valid(); err != nil {
return 0, err
}
f, err := os.Open(filename(b.p, h.Type, h.Name))
if err != nil {
return 0, err
}
defer func() {
e := f.Close()
if err == nil && e != nil {
err = e
}
}()
if off > 0 {
_, err = f.Seek(off, 0)
if err != nil {
return 0, err
}
}
return io.ReadFull(f, p)
}
// writeToTempfile saves p into a tempfile in tempdir.
func writeToTempfile(tempdir string, p []byte) (filename string, err error) {
tmpfile, err := ioutil.TempFile(tempdir, "temp-")
if err != nil {
return "", err
}
n, err := tmpfile.Write(p)
if err != nil {
return "", err
}
if n != len(p) {
return "", errors.New("not all bytes writen")
}
if err = tmpfile.Sync(); err != nil {
return "", err
}
err = tmpfile.Close()
if err != nil {
return "", err
}
return tmpfile.Name(), nil
}
// Save stores data in the backend at the handle.
func (b *Local) Save(h backend.Handle, p []byte) (err error) {
if err := h.Valid(); err != nil {
return err
}
tmpfile, err := writeToTempfile(filepath.Join(b.p, backend.Paths.Temp), p)
debug.Log("local.Save", "saved %v (%d bytes) to %v", h, len(p), tmpfile)
filename := filename(b.p, h.Type, h.Name)
// test if new path already exists
if _, err := os.Stat(filename); err == nil {
return fmt.Errorf("Rename(): file %v already exists", filename)
}
// create directories if necessary, ignore errors
if h.Type == backend.Data {
err = os.MkdirAll(filepath.Dir(filename), backend.Modes.Dir)
if err != nil {
return err
}
}
err = os.Rename(tmpfile, filename)
debug.Log("local.Save", "save %v: rename %v -> %v: %v",
h, filepath.Base(tmpfile), filepath.Base(filename), err)
if err != nil {
return err
}
// set mode to read-only
fi, err := os.Stat(filename)
if err != nil {
return err
}
return setNewFileMode(filename, fi)
}
// Stat returns information about a blob.
func (b *Local) Stat(h backend.Handle) (backend.BlobInfo, error) {
if err := h.Valid(); err != nil {
return backend.BlobInfo{}, err
}
fi, err := os.Stat(filename(b.p, h.Type, h.Name))
if err != nil {
return backend.BlobInfo{}, err
}
return backend.BlobInfo{Size: fi.Size()}, nil
}
// Test returns true if a blob of the given type and name exists in the backend.
func (b *Local) Test(t backend.Type, name string) (bool, error) {
_, err := os.Stat(filename(b.p, t, name))
if err != nil {
if os.IsNotExist(err) {
return false, nil
}
return false, err
}
return true, nil
}
// Remove removes the blob with the given name and type.
func (b *Local) Remove(t backend.Type, name string) error {
fn := filename(b.p, t, name)
// reset read-only flag
err := os.Chmod(fn, 0666)
if err != nil {
return err
}
return os.Remove(fn)
}
// List returns a channel that yields all names of blobs of type t. A
// goroutine is started for this. If the channel done is closed, sending
// stops.
func (b *Local) List(t backend.Type, done <-chan struct{}) <-chan string {
var pattern string
if t == backend.Data {
pattern = filepath.Join(dirname(b.p, t, ""), "*", "*")
} else {
pattern = filepath.Join(dirname(b.p, t, ""), "*")
}
ch := make(chan string)
matches, err := filepath.Glob(pattern)
if err != nil {
close(ch)
return ch
}
for i := range matches {
matches[i] = filepath.Base(matches[i])
}
go func() {
defer close(ch)
for _, m := range matches {
if m == "" {
continue
}
select {
case ch <- m:
case <-done:
return
}
}
}()
return ch
}
// Delete removes the repository and all files.
func (b *Local) Delete() error {
return os.RemoveAll(b.p)
}
// Close closes all open files.
func (b *Local) Close() error {
// this does not need to do anything, all open files are closed within the
// same function.
return nil
}
src/github.com/restic/restic/backend/local/local_test.go
+59
View File
@@ -0,0 +1,59 @@
package local_test
import (
"fmt"
"io/ioutil"
"os"
"github.com/restic/restic/backend"
"github.com/restic/restic/backend/local"
"github.com/restic/restic/backend/test"
)
var tempBackendDir string
//go:generate go run ../test/generate_backend_tests.go
func createTempdir() error {
if tempBackendDir != "" {
return nil
}
tempdir, err := ioutil.TempDir("", "restic-local-test-")
if err != nil {
return err
}
fmt.Printf("created new test backend at %v\n", tempdir)
tempBackendDir = tempdir
return nil
}
func init() {
test.CreateFn = func() (backend.Backend, error) {
err := createTempdir()
if err != nil {
return nil, err
}
return local.Create(tempBackendDir)
}
test.OpenFn = func() (backend.Backend, error) {
err := createTempdir()
if err != nil {
return nil, err
}
return local.Open(tempBackendDir)
}
test.CleanupFn = func() error {
if tempBackendDir == "" {
return nil
}
fmt.Printf("removing test backend at %v\n", tempBackendDir)
err := os.RemoveAll(tempBackendDir)
tempBackendDir = ""
return err
}
}
src/github.com/restic/restic/backend/local/local_unix.go
+12
View File
@@ -0,0 +1,12 @@
// +build !windows
package local
import (
"os"
)
// set file to readonly
func setNewFileMode(f string, fi os.FileInfo) error {
return os.Chmod(f, fi.Mode()&os.FileMode(^uint32(0222)))
}
src/github.com/restic/restic/backend/local/local_windows.go
+12
View File
@@ -0,0 +1,12 @@
package local
import (
"os"
)
// We don't modify read-only on windows,
// since it will make us unable to delete the file,
// and this isn't common practice on this platform.
func setNewFileMode(f string, fi os.FileInfo) error {
return nil
}
src/github.com/restic/restic/backend/mem/backend_test.go
+87
View File
@@ -0,0 +1,87 @@
// DO NOT EDIT, AUTOMATICALLY GENERATED
package mem_test
import (
"testing"
"github.com/restic/restic/backend/test"
)
var SkipMessage string
func TestMemBackendCreate(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestCreate(t)
}
func TestMemBackendOpen(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestOpen(t)
}
func TestMemBackendCreateWithConfig(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestCreateWithConfig(t)
}
func TestMemBackendLocation(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestLocation(t)
}
func TestMemBackendConfig(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestConfig(t)
}
func TestMemBackendLoad(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestLoad(t)
}
func TestMemBackendSave(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestSave(t)
}
func TestMemBackendSaveFilenames(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestSaveFilenames(t)
}
func TestMemBackendBackend(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestBackend(t)
}
func TestMemBackendDelete(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestDelete(t)
}
func TestMemBackendCleanup(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestCleanup(t)
}
src/github.com/restic/restic/backend/mem/mem_backend.go
+223
View File
@@ -0,0 +1,223 @@
package mem
import (
"errors"
"io"
"sync"
"github.com/restic/restic/backend"
"github.com/restic/restic/debug"
)
type entry struct {
Type backend.Type
Name string
}
type memMap map[entry][]byte
// MemoryBackend is a mock backend that uses a map for storing all data in
// memory. This should only be used for tests.
type MemoryBackend struct {
data memMap
m sync.Mutex
backend.MockBackend
}
// New returns a new backend that saves all data in a map in memory.
func New() *MemoryBackend {
be := &MemoryBackend{
data: make(memMap),
}
be.MockBackend.TestFn = func(t backend.Type, name string) (bool, error) {
return memTest(be, t, name)
}
be.MockBackend.LoadFn = func(h backend.Handle, p []byte, off int64) (int, error) {
return memLoad(be, h, p, off)
}
be.MockBackend.SaveFn = func(h backend.Handle, p []byte) error {
return memSave(be, h, p)
}
be.MockBackend.StatFn = func(h backend.Handle) (backend.BlobInfo, error) {
return memStat(be, h)
}
be.MockBackend.RemoveFn = func(t backend.Type, name string) error {
return memRemove(be, t, name)
}
be.MockBackend.ListFn = func(t backend.Type, done <-chan struct{}) <-chan string {
return memList(be, t, done)
}
be.MockBackend.DeleteFn = func() error {
be.m.Lock()
defer be.m.Unlock()
be.data = make(memMap)
return nil
}
be.MockBackend.LocationFn = func() string {
return "Memory Backend"
}
debug.Log("MemoryBackend.New", "created new memory backend")
return be
}
func (be *MemoryBackend) insert(t backend.Type, name string, data []byte) error {
be.m.Lock()
defer be.m.Unlock()
if _, ok := be.data[entry{t, name}]; ok {
return errors.New("already present")
}
be.data[entry{t, name}] = data
return nil
}
func memTest(be *MemoryBackend, t backend.Type, name string) (bool, error) {
be.m.Lock()
defer be.m.Unlock()
debug.Log("MemoryBackend.Test", "test %v %v", t, name)
if _, ok := be.data[entry{t, name}]; ok {
return true, nil
}
return false, nil
}
func memLoad(be *MemoryBackend, h backend.Handle, p []byte, off int64) (int, error) {
if err := h.Valid(); err != nil {
return 0, err
}
be.m.Lock()
defer be.m.Unlock()
if h.Type == backend.Config {
h.Name = ""
}
debug.Log("MemoryBackend.Load", "get %v offset %v len %v", h, off, len(p))
if _, ok := be.data[entry{h.Type, h.Name}]; !ok {
return 0, errors.New("no such data")
}
buf := be.data[entry{h.Type, h.Name}]
if off > int64(len(buf)) {
return 0, errors.New("offset beyond end of file")
}
buf = buf[off:]
n := copy(p, buf)
if len(p) > len(buf) {
return n, io.ErrUnexpectedEOF
}
return n, nil
}
func memSave(be *MemoryBackend, h backend.Handle, p []byte) error {
if err := h.Valid(); err != nil {
return err
}
be.m.Lock()
defer be.m.Unlock()
if h.Type == backend.Config {
h.Name = ""
}
if _, ok := be.data[entry{h.Type, h.Name}]; ok {
return errors.New("file already exists")
}
debug.Log("MemoryBackend.Save", "save %v bytes at %v", len(p), h)
buf := make([]byte, len(p))
copy(buf, p)
be.data[entry{h.Type, h.Name}] = buf
return nil
}
func memStat(be *MemoryBackend, h backend.Handle) (backend.BlobInfo, error) {
be.m.Lock()
defer be.m.Unlock()
if err := h.Valid(); err != nil {
return backend.BlobInfo{}, err
}
if h.Type == backend.Config {
h.Name = ""
}
debug.Log("MemoryBackend.Stat", "stat %v", h)
e, ok := be.data[entry{h.Type, h.Name}]
if !ok {
return backend.BlobInfo{}, errors.New("no such data")
}
return backend.BlobInfo{Size: int64(len(e))}, nil
}
func memRemove(be *MemoryBackend, t backend.Type, name string) error {
be.m.Lock()
defer be.m.Unlock()
debug.Log("MemoryBackend.Remove", "get %v %v", t, name)
if _, ok := be.data[entry{t, name}]; !ok {
return errors.New("no such data")
}
delete(be.data, entry{t, name})
return nil
}
func memList(be *MemoryBackend, t backend.Type, done <-chan struct{}) <-chan string {
be.m.Lock()
defer be.m.Unlock()
ch := make(chan string)
var ids []string
for entry := range be.data {
if entry.Type != t {
continue
}
ids = append(ids, entry.Name)
}
debug.Log("MemoryBackend.List", "list %v: %v", t, ids)
go func() {
defer close(ch)
for _, id := range ids {
select {
case ch <- id:
case <-done:
return
}
}
}()
return ch
}
src/github.com/restic/restic/backend/mem/mem_backend_test.go
+38
View File
@@ -0,0 +1,38 @@
package mem_test
import (
"errors"
"github.com/restic/restic/backend"
"github.com/restic/restic/backend/mem"
"github.com/restic/restic/backend/test"
)
var be backend.Backend
//go:generate go run ../test/generate_backend_tests.go
func init() {
test.CreateFn = func() (backend.Backend, error) {
if be != nil {
return nil, errors.New("temporary memory backend dir already exists")
}
be = mem.New()
return be, nil
}
test.OpenFn = func() (backend.Backend, error) {
if be == nil {
return nil, errors.New("repository not initialized")
}
return be, nil
}
test.CleanupFn = func() error {
be = nil
return nil
}
}
src/github.com/restic/restic/backend/mock_backend.go
+103
View File
@@ -0,0 +1,103 @@
package backend
import "errors"
// MockBackend implements a backend whose functions can be specified. This
// should only be used for tests.
type MockBackend struct {
CloseFn func() error
LoadFn func(h Handle, p []byte, off int64) (int, error)
SaveFn func(h Handle, p []byte) error
StatFn func(h Handle) (BlobInfo, error)
ListFn func(Type, <-chan struct{}) <-chan string
RemoveFn func(Type, string) error
TestFn func(Type, string) (bool, error)
DeleteFn func() error
LocationFn func() string
}
// Close the backend.
func (m *MockBackend) Close() error {
if m.CloseFn == nil {
return nil
}
return m.CloseFn()
}
// Location returns a location string.
func (m *MockBackend) Location() string {
if m.LocationFn == nil {
return ""
}
return m.LocationFn()
}
// Load loads data from the backend.
func (m *MockBackend) Load(h Handle, p []byte, off int64) (int, error) {
if m.LoadFn == nil {
return 0, errors.New("not implemented")
}
return m.LoadFn(h, p, off)
}
// Save data in the backend.
func (m *MockBackend) Save(h Handle, p []byte) error {
if m.SaveFn == nil {
return errors.New("not implemented")
}
return m.SaveFn(h, p)
}
// Stat an object in the backend.
func (m *MockBackend) Stat(h Handle) (BlobInfo, error) {
if m.StatFn == nil {
return BlobInfo{}, errors.New("not implemented")
}
return m.StatFn(h)
}
// List items of type t.
func (m *MockBackend) List(t Type, done <-chan struct{}) <-chan string {
if m.ListFn == nil {
ch := make(chan string)
close(ch)
return ch
}
return m.ListFn(t, done)
}
// Remove data from the backend.
func (m *MockBackend) Remove(t Type, name string) error {
if m.RemoveFn == nil {
return errors.New("not implemented")
}
return m.RemoveFn(t, name)
}
// Test for the existence of a specific item.
func (m *MockBackend) Test(t Type, name string) (bool, error) {
if m.TestFn == nil {
return false, errors.New("not implemented")
}
return m.TestFn(t, name)
}
// Delete all data.
func (m *MockBackend) Delete() error {
if m.DeleteFn == nil {
return errors.New("not implemented")
}
return m.DeleteFn()
}
// Make sure that MockBackend implements the backend interface.
var _ Backend = &MockBackend{}
src/github.com/restic/restic/backend/paths.go
+26
View File
@@ -0,0 +1,26 @@
package backend
import "os"
// Paths contains the default paths for file-based backends (e.g. local).
var Paths = struct {
Data string
Snapshots string
Index string
Locks string
Keys string
Temp string
Config string
}{
"data",
"snapshots",
"index",
"locks",
"keys",
"tmp",
"config",
}
// Modes holds the default modes for directories and files for file-based
// backends.
var Modes = struct{ Dir, File os.FileMode }{0700, 0600}
src/github.com/restic/restic/backend/s3/backend_test.go
+87
View File
@@ -0,0 +1,87 @@
// DO NOT EDIT, AUTOMATICALLY GENERATED
package s3_test
import (
"testing"
"github.com/restic/restic/backend/test"
)
var SkipMessage string
func TestS3BackendCreate(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestCreate(t)
}
func TestS3BackendOpen(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestOpen(t)
}
func TestS3BackendCreateWithConfig(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestCreateWithConfig(t)
}
func TestS3BackendLocation(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestLocation(t)
}
func TestS3BackendConfig(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestConfig(t)
}
func TestS3BackendLoad(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestLoad(t)
}
func TestS3BackendSave(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestSave(t)
}
func TestS3BackendSaveFilenames(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestSaveFilenames(t)
}
func TestS3BackendBackend(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestBackend(t)
}
func TestS3BackendDelete(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestDelete(t)
}
func TestS3BackendCleanup(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestCleanup(t)
}
src/github.com/restic/restic/backend/s3/config.go
+72
View File
@@ -0,0 +1,72 @@
package s3
import (
"errors"
"net/url"
"path"
"strings"
)
// Config contains all configuration necessary to connect to an s3 compatible
// server.
type Config struct {
Endpoint string
UseHTTP bool
KeyID, Secret string
Bucket string
Prefix string
}
const defaultPrefix = "restic"
// ParseConfig parses the string s and extracts the s3 config. The two
// supported configuration formats are s3://host/bucketname/prefix and
// s3:host:bucketname/prefix. The host can also be a valid s3 region
// name. If no prefix is given the prefix "restic" will be used.
func ParseConfig(s string) (interface{}, error) {
switch {
case strings.HasPrefix(s, "s3:http"):
// assume that a URL has been specified, parse it and
// use the host as the endpoint and the path as the
// bucket name and prefix
url, err := url.Parse(s[3:])
if err != nil {
return nil, err
}
if url.Path == "" {
return nil, errors.New("s3: bucket name not found")
}
path := strings.SplitN(url.Path[1:], "/", 2)
return createConfig(url.Host, path, url.Scheme == "http")
case strings.HasPrefix(s, "s3://"):
s = s[5:]
case strings.HasPrefix(s, "s3:"):
s = s[3:]
default:
return nil, errors.New("s3: invalid format")
}
// use the first entry of the path as the endpoint and the
// remainder as bucket name and prefix
path := strings.SplitN(s, "/", 3)
return createConfig(path[0], path[1:], false)
}
func createConfig(endpoint string, p []string, useHTTP bool) (interface{}, error) {
var prefix string
switch {
case len(p) < 1:
return nil, errors.New("s3: invalid format, host/region or bucket name not found")
case len(p) == 1 || p[1] == "":
prefix = defaultPrefix
default:
prefix = path.Clean(p[1])
}
return Config{
Endpoint: endpoint,
UseHTTP: useHTTP,
Bucket: p[0],
Prefix: prefix,
}, nil
}
src/github.com/restic/restic/backend/s3/config_test.go
+99
View File
@@ -0,0 +1,99 @@
package s3
import "testing"
var configTests = []struct {
s string
cfg Config
}{
{"s3://eu-central-1/bucketname", Config{
Endpoint: "eu-central-1",
Bucket: "bucketname",
Prefix: "restic",
}},
{"s3://eu-central-1/bucketname/", Config{
Endpoint: "eu-central-1",
Bucket: "bucketname",
Prefix: "restic",
}},
{"s3://eu-central-1/bucketname/prefix/directory", Config{
Endpoint: "eu-central-1",
Bucket: "bucketname",
Prefix: "prefix/directory",
}},
{"s3://eu-central-1/bucketname/prefix/directory/", Config{
Endpoint: "eu-central-1",
Bucket: "bucketname",
Prefix: "prefix/directory",
}},
{"s3:eu-central-1/foobar", Config{
Endpoint: "eu-central-1",
Bucket: "foobar",
Prefix: "restic",
}},
{"s3:eu-central-1/foobar/", Config{
Endpoint: "eu-central-1",
Bucket: "foobar",
Prefix: "restic",
}},
{"s3:eu-central-1/foobar/prefix/directory", Config{
Endpoint: "eu-central-1",
Bucket: "foobar",
Prefix: "prefix/directory",
}},
{"s3:eu-central-1/foobar/prefix/directory/", Config{
Endpoint: "eu-central-1",
Bucket: "foobar",
Prefix: "prefix/directory",
}},
{"s3:https://hostname:9999/foobar", Config{
Endpoint: "hostname:9999",
Bucket: "foobar",
Prefix: "restic",
}},
{"s3:https://hostname:9999/foobar/", Config{
Endpoint: "hostname:9999",
Bucket: "foobar",
Prefix: "restic",
}},
{"s3:http://hostname:9999/foobar", Config{
Endpoint: "hostname:9999",
Bucket: "foobar",
Prefix: "restic",
UseHTTP: true,
}},
{"s3:http://hostname:9999/foobar/", Config{
Endpoint: "hostname:9999",
Bucket: "foobar",
Prefix: "restic",
UseHTTP: true,
}},
{"s3:http://hostname:9999/bucket/prefix/directory", Config{
Endpoint: "hostname:9999",
Bucket: "bucket",
Prefix: "prefix/directory",
UseHTTP: true,
}},
{"s3:http://hostname:9999/bucket/prefix/directory/", Config{
Endpoint: "hostname:9999",
Bucket: "bucket",
Prefix: "prefix/directory",
UseHTTP: true,
}},
}
func TestParseConfig(t *testing.T) {
for i, test := range configTests {
cfg, err := ParseConfig(test.s)
if err != nil {
t.Errorf("test %d:%s failed: %v", i, test.s, err)
continue
}
if cfg != test.cfg {
t.Errorf("test %d:\ninput:\n %s\n wrong config, want:\n %v\ngot:\n %v",
i, test.s, test.cfg, cfg)
continue
}
}
}
src/github.com/restic/restic/backend/s3/s3.go
+237
View File
@@ -0,0 +1,237 @@
package s3
import (
"bytes"
"errors"
"io"
"strings"
"github.com/minio/minio-go"
"github.com/restic/restic/backend"
"github.com/restic/restic/debug"
)
const connLimit = 10
// s3 is a backend which stores the data on an S3 endpoint.
type s3 struct {
client minio.CloudStorageClient
connChan chan struct{}
bucketname string
prefix string
}
// Open opens the S3 backend at bucket and region. The bucket is created if it
// does not exist yet.
func Open(cfg Config) (backend.Backend, error) {
debug.Log("s3.Open", "open, config %#v", cfg)
client, err := minio.New(cfg.Endpoint, cfg.KeyID, cfg.Secret, cfg.UseHTTP)
if err != nil {
return nil, err
}
be := &s3{client: client, bucketname: cfg.Bucket, prefix: cfg.Prefix}
be.createConnections()
if err := client.BucketExists(cfg.Bucket); err != nil {
debug.Log("s3.Open", "BucketExists(%v) returned err %v, trying to create the bucket", cfg.Bucket, err)
// create new bucket with default ACL in default region
err = client.MakeBucket(cfg.Bucket, "", "")
if err != nil {
return nil, err
}
}
return be, nil
}
func (be *s3) s3path(t backend.Type, name string) string {
var path string
if be.prefix != "" {
path = be.prefix + "/"
}
path += string(t)
if t == backend.Config {
return path
}
return path + "/" + name
}
func (be *s3) createConnections() {
be.connChan = make(chan struct{}, connLimit)
for i := 0; i < connLimit; i++ {
be.connChan <- struct{}{}
}
}
// Location returns this backend's location (the bucket name).
func (be *s3) Location() string {
return be.bucketname
}
// Load returns the data stored in the backend for h at the given offset
// and saves it in p. Load has the same semantics as io.ReaderAt.
func (be s3) Load(h backend.Handle, p []byte, off int64) (int, error) {
debug.Log("s3.Load", "%v, offset %v, len %v", h, off, len(p))
path := be.s3path(h.Type, h.Name)
obj, err := be.client.GetObject(be.bucketname, path)
if err != nil {
debug.Log("s3.GetReader", " err %v", err)
return 0, err
}
if off > 0 {
_, err = obj.Seek(off, 0)
if err != nil {
return 0, err
}
}
<-be.connChan
defer func() {
be.connChan <- struct{}{}
}()
return io.ReadFull(obj, p)
}
// Save stores data in the backend at the handle.
func (be s3) Save(h backend.Handle, p []byte) (err error) {
if err := h.Valid(); err != nil {
return err
}
debug.Log("s3.Save", "%v bytes at %d", len(p), h)
path := be.s3path(h.Type, h.Name)
// Check key does not already exist
_, err = be.client.StatObject(be.bucketname, path)
if err == nil {
debug.Log("s3.blob.Finalize()", "%v already exists", h)
return errors.New("key already exists")
}
<-be.connChan
defer func() {
be.connChan <- struct{}{}
}()
debug.Log("s3.Save", "PutObject(%v, %v, %v, %v)",
be.bucketname, path, int64(len(p)), "binary/octet-stream")
n, err := be.client.PutObject(be.bucketname, path, bytes.NewReader(p), "binary/octet-stream")
debug.Log("s3.Save", "%v -> %v bytes, err %#v", path, n, err)
return err
}
// Stat returns information about a blob.
func (be s3) Stat(h backend.Handle) (backend.BlobInfo, error) {
debug.Log("s3.Stat", "%v")
path := be.s3path(h.Type, h.Name)
obj, err := be.client.GetObject(be.bucketname, path)
if err != nil {
debug.Log("s3.Stat", "GetObject() err %v", err)
return backend.BlobInfo{}, err
}
fi, err := obj.Stat()
if err != nil {
debug.Log("s3.Stat", "Stat() err %v", err)
return backend.BlobInfo{}, err
}
return backend.BlobInfo{Size: fi.Size}, nil
}
// Test returns true if a blob of the given type and name exists in the backend.
func (be *s3) Test(t backend.Type, name string) (bool, error) {
found := false
path := be.s3path(t, name)
_, err := be.client.StatObject(be.bucketname, path)
if err == nil {
found = true
}
// If error, then not found
return found, nil
}
// Remove removes the blob with the given name and type.
func (be *s3) Remove(t backend.Type, name string) error {
path := be.s3path(t, name)
err := be.client.RemoveObject(be.bucketname, path)
debug.Log("s3.Remove", "%v %v -> err %v", t, name, err)
return err
}
// List returns a channel that yields all names of blobs of type t. A
// goroutine is started for this. If the channel done is closed, sending
// stops.
func (be *s3) List(t backend.Type, done <-chan struct{}) <-chan string {
debug.Log("s3.List", "listing %v", t)
ch := make(chan string)
prefix := be.s3path(t, "")
listresp := be.client.ListObjects(be.bucketname, prefix, true, done)
go func() {
defer close(ch)
for obj := range listresp {
m := strings.TrimPrefix(obj.Key, prefix)
if m == "" {
continue
}
select {
case ch <- m:
case <-done:
return
}
}
}()
return ch
}
// Remove keys for a specified backend type.
func (be *s3) removeKeys(t backend.Type) error {
done := make(chan struct{})
defer close(done)
for key := range be.List(backend.Data, done) {
err := be.Remove(backend.Data, key)
if err != nil {
return err
}
}
return nil
}
// Delete removes all restic keys in the bucket. It will not remove the bucket itself.
func (be *s3) Delete() error {
alltypes := []backend.Type{
backend.Data,
backend.Key,
backend.Lock,
backend.Snapshot,
backend.Index}
for _, t := range alltypes {
err := be.removeKeys(t)
if err != nil {
return nil
}
}
return be.Remove(backend.Config, "")
}
// Close does nothing
func (be *s3) Close() error { return nil }
src/github.com/restic/restic/backend/s3/s3_test.go
+72
View File
@@ -0,0 +1,72 @@
package s3_test
import (
"errors"
"fmt"
"net/url"
"os"
"github.com/restic/restic/backend"
"github.com/restic/restic/backend/s3"
"github.com/restic/restic/backend/test"
. "github.com/restic/restic/test"
)
//go:generate go run ../test/generate_backend_tests.go
func init() {
if TestS3Server == "" {
SkipMessage = "s3 test server not available"
return
}
url, err := url.Parse(TestS3Server)
if err != nil {
fmt.Fprintf(os.Stderr, "invalid url: %v\n", err)
return
}
cfg := s3.Config{
Endpoint: url.Host,
Bucket: "restictestbucket",
KeyID: os.Getenv("AWS_ACCESS_KEY_ID"),
Secret: os.Getenv("AWS_SECRET_ACCESS_KEY"),
}
if url.Scheme == "http" {
cfg.UseHTTP = true
}
test.CreateFn = func() (backend.Backend, error) {
be, err := s3.Open(cfg)
if err != nil {
return nil, err
}
exists, err := be.Test(backend.Config, "")
if err != nil {
return nil, err
}
if exists {
return nil, errors.New("config already exists")
}
return be, nil
}
test.OpenFn = func() (backend.Backend, error) {
return s3.Open(cfg)
}
// test.CleanupFn = func() error {
// if tempBackendDir == "" {
// return nil
// }
// fmt.Printf("removing test backend at %v\n", tempBackendDir)
// err := os.RemoveAll(tempBackendDir)
// tempBackendDir = ""
// return err
// }
}
src/github.com/restic/restic/backend/sftp/backend_test.go
+87
View File
@@ -0,0 +1,87 @@
// DO NOT EDIT, AUTOMATICALLY GENERATED
package sftp_test
import (
"testing"
"github.com/restic/restic/backend/test"
)
var SkipMessage string
func TestSftpBackendCreate(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestCreate(t)
}
func TestSftpBackendOpen(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestOpen(t)
}
func TestSftpBackendCreateWithConfig(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestCreateWithConfig(t)
}
func TestSftpBackendLocation(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestLocation(t)
}
func TestSftpBackendConfig(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestConfig(t)
}
func TestSftpBackendLoad(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestLoad(t)
}
func TestSftpBackendSave(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestSave(t)
}
func TestSftpBackendSaveFilenames(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestSaveFilenames(t)
}
func TestSftpBackendBackend(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestBackend(t)
}
func TestSftpBackendDelete(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestDelete(t)
}
func TestSftpBackendCleanup(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestCleanup(t)
}
src/github.com/restic/restic/backend/sftp/config.go
+68
View File
@@ -0,0 +1,68 @@
package sftp
import (
"errors"
"net/url"
"strings"
)
// Config collects all information required to connect to an sftp server.
type Config struct {
User, Host, Dir string
}
// ParseConfig extracts all information for the sftp connection from the string s.
func ParseConfig(s string) (interface{}, error) {
if strings.HasPrefix(s, "sftp://") {
return parseFormat1(s)
}
// otherwise parse in the sftp:user@host:path format, which means we'll get
// "user@host:path" in s
return parseFormat2(s)
}
// parseFormat1 parses the first format, starting with a slash, so the user
// either specified "sftp://host/path", so we'll get everything after the first
// colon character
func parseFormat1(s string) (Config, error) {
url, err := url.Parse(s)
if err != nil {
return Config{}, err
}
cfg := Config{
Host: url.Host,
Dir: url.Path[1:],
}
if url.User != nil {
cfg.User = url.User.Username()
}
return cfg, nil
}
// parseFormat2 parses the second format, sftp:user@host:path
func parseFormat2(s string) (cfg Config, err error) {
// split user/host and path at the second colon
data := strings.SplitN(s, ":", 3)
if len(data) < 3 {
return Config{}, errors.New("sftp: invalid format, hostname or path not found")
}
if data[0] != "sftp" {
return Config{}, errors.New(`invalid format, does not start with "sftp:"`)
}
userhost := data[1]
cfg.Dir = data[2]
data = strings.SplitN(userhost, "@", 2)
if len(data) == 2 {
cfg.User = data[0]
cfg.Host = data[1]
} else {
cfg.Host = userhost
}
return cfg, nil
}
src/github.com/restic/restic/backend/sftp/config_test.go
+60
View File
@@ -0,0 +1,60 @@
package sftp
import "testing"
var configTests = []struct {
s string
cfg Config
}{
// first form, user specified sftp://user@host/dir
{
"sftp://user@host/dir/subdir",
Config{User: "user", Host: "host", Dir: "dir/subdir"},
},
{
"sftp://host/dir/subdir",
Config{Host: "host", Dir: "dir/subdir"},
},
{
"sftp://host//dir/subdir",
Config{Host: "host", Dir: "/dir/subdir"},
},
{
"sftp://host:10022//dir/subdir",
Config{Host: "host:10022", Dir: "/dir/subdir"},
},
{
"sftp://user@host:10022//dir/subdir",
Config{User: "user", Host: "host:10022", Dir: "/dir/subdir"},
},
// second form, user specified sftp:user@host:/dir
{
"sftp:foo@bar:/baz/quux",
Config{User: "foo", Host: "bar", Dir: "/baz/quux"},
},
{
"sftp:bar:../baz/quux",
Config{Host: "bar", Dir: "../baz/quux"},
},
{
"sftp:fux@bar:baz/qu:ux",
Config{User: "fux", Host: "bar", Dir: "baz/qu:ux"},
},
}
func TestParseConfig(t *testing.T) {
for i, test := range configTests {
cfg, err := ParseConfig(test.s)
if err != nil {
t.Errorf("test %d failed: %v", i, err)
continue
}
if cfg != test.cfg {
t.Errorf("test %d: wrong config, want:\n %v\ngot:\n %v",
i, test.cfg, cfg)
continue
}
}
}
src/github.com/restic/restic/backend/sftp/doc.go
+3
View File
@@ -0,0 +1,3 @@
// Package sftp implements repository storage in a directory on a remote server
// via the sftp protocol.
package sftp
src/github.com/restic/restic/backend/sftp/sftp.go
+465
View File
@@ -0,0 +1,465 @@
package sftp
import (
"crypto/rand"
"encoding/hex"
"fmt"
"io"
"log"
"os"
"os/exec"
"path/filepath"
"strings"
"github.com/juju/errors"
"github.com/pkg/sftp"
"github.com/restic/restic/backend"
"github.com/restic/restic/debug"
)
const (
tempfileRandomSuffixLength = 10
)
// SFTP is a backend in a directory accessed via SFTP.
type SFTP struct {
c *sftp.Client
p string
cmd *exec.Cmd
}
func startClient(program string, args ...string) (*SFTP, error) {
// Connect to a remote host and request the sftp subsystem via the 'ssh'
// command. This assumes that passwordless login is correctly configured.
cmd := exec.Command(program, args...)
// send errors from ssh to stderr
cmd.Stderr = os.Stderr
// ignore signals sent to the parent (e.g. SIGINT)
cmd.SysProcAttr = ignoreSigIntProcAttr()
// get stdin and stdout
wr, err := cmd.StdinPipe()
if err != nil {
log.Fatal(err)
}
rd, err := cmd.StdoutPipe()
if err != nil {
log.Fatal(err)
}
// start the process
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
// open the SFTP session
client, err := sftp.NewClientPipe(rd, wr)
if err != nil {
log.Fatal(err)
}
return &SFTP{c: client, cmd: cmd}, nil
}
func paths(dir string) []string {
return []string{
dir,
Join(dir, backend.Paths.Data),
Join(dir, backend.Paths.Snapshots),
Join(dir, backend.Paths.Index),
Join(dir, backend.Paths.Locks),
Join(dir, backend.Paths.Keys),
Join(dir, backend.Paths.Temp),
}
}
// Open opens an sftp backend. When the command is started via
// exec.Command, it is expected to speak sftp on stdin/stdout. The backend
// is expected at the given path.
func Open(dir string, program string, args ...string) (*SFTP, error) {
sftp, err := startClient(program, args...)
if err != nil {
return nil, err
}
// test if all necessary dirs and files are there
for _, d := range paths(dir) {
if _, err := sftp.c.Lstat(d); err != nil {
return nil, fmt.Errorf("%s does not exist", d)
}
}
sftp.p = dir
return sftp, nil
}
func buildSSHCommand(cfg Config) []string {
hostport := strings.Split(cfg.Host, ":")
args := []string{hostport[0]}
if len(hostport) > 1 {
args = append(args, "-p", hostport[1])
}
if cfg.User != "" {
args = append(args, "-l")
args = append(args, cfg.User)
}
args = append(args, "-s")
args = append(args, "sftp")
return args
}
// OpenWithConfig opens an sftp backend as described by the config by running
// "ssh" with the appropiate arguments.
func OpenWithConfig(cfg Config) (*SFTP, error) {
return Open(cfg.Dir, "ssh", buildSSHCommand(cfg)...)
}
// Create creates all the necessary files and directories for a new sftp
// backend at dir. Afterwards a new config blob should be created.
func Create(dir string, program string, args ...string) (*SFTP, error) {
sftp, err := startClient(program, args...)
if err != nil {
return nil, err
}
// test if config file already exists
_, err = sftp.c.Lstat(Join(dir, backend.Paths.Config))
if err == nil {
return nil, errors.New("config file already exists")
}
// create paths for data, refs and temp blobs
for _, d := range paths(dir) {
err = sftp.mkdirAll(d, backend.Modes.Dir)
if err != nil {
return nil, err
}
}
err = sftp.c.Close()
if err != nil {
return nil, err
}
err = sftp.cmd.Wait()
if err != nil {
return nil, err
}
// open backend
return Open(dir, program, args...)
}
// CreateWithConfig creates an sftp backend as described by the config by running
// "ssh" with the appropiate arguments.
func CreateWithConfig(cfg Config) (*SFTP, error) {
return Create(cfg.Dir, "ssh", buildSSHCommand(cfg)...)
}
// Location returns this backend's location (the directory name).
func (r *SFTP) Location() string {
return r.p
}
// Return temp directory in correct directory for this backend.
func (r *SFTP) tempFile() (string, *sftp.File, error) {
// choose random suffix
buf := make([]byte, tempfileRandomSuffixLength)
_, err := io.ReadFull(rand.Reader, buf)
if err != nil {
return "", nil, errors.Annotatef(err,
"unable to read %d random bytes for tempfile name",
tempfileRandomSuffixLength)
}
// construct tempfile name
name := Join(r.p, backend.Paths.Temp, "temp-"+hex.EncodeToString(buf))
// create file in temp dir
f, err := r.c.Create(name)
if err != nil {
return "", nil, errors.Annotatef(err, "creating tempfile %q failed", name)
}
return name, f, nil
}
func (r *SFTP) mkdirAll(dir string, mode os.FileMode) error {
// check if directory already exists
fi, err := r.c.Lstat(dir)
if err == nil {
if fi.IsDir() {
return nil
}
return fmt.Errorf("mkdirAll(%s): entry exists but is not a directory", dir)
}
// create parent directories
errMkdirAll := r.mkdirAll(filepath.Dir(dir), backend.Modes.Dir)
// create directory
errMkdir := r.c.Mkdir(dir)
// test if directory was created successfully
fi, err = r.c.Lstat(dir)
if err != nil {
// return previous errors
return fmt.Errorf("mkdirAll(%s): unable to create directories: %v, %v", dir, errMkdirAll, errMkdir)
}
if !fi.IsDir() {
return fmt.Errorf("mkdirAll(%s): entry exists but is not a directory", dir)
}
// set mode
return r.c.Chmod(dir, mode)
}
// Rename temp file to final name according to type and name.
func (r *SFTP) renameFile(oldname string, t backend.Type, name string) error {
filename := r.filename(t, name)
// create directories if necessary
if t == backend.Data {
err := r.mkdirAll(filepath.Dir(filename), backend.Modes.Dir)
if err != nil {
return err
}
}
// test if new file exists
if _, err := r.c.Lstat(filename); err == nil {
return fmt.Errorf("Close(): file %v already exists", filename)
}
err := r.c.Rename(oldname, filename)
if err != nil {
return err
}
// set mode to read-only
fi, err := r.c.Lstat(filename)
if err != nil {
return err
}
return r.c.Chmod(filename, fi.Mode()&os.FileMode(^uint32(0222)))
}
// Join joins the given paths and cleans them afterwards.
func Join(parts ...string) string {
return filepath.Clean(strings.Join(parts, "/"))
}
// Construct path for given backend.Type and name.
func (r *SFTP) filename(t backend.Type, name string) string {
if t == backend.Config {
return Join(r.p, "config")
}
return Join(r.dirname(t, name), name)
}
// Construct directory for given backend.Type.
func (r *SFTP) dirname(t backend.Type, name string) string {
var n string
switch t {
case backend.Data:
n = backend.Paths.Data
if len(name) > 2 {
n = Join(n, name[:2])
}
case backend.Snapshot:
n = backend.Paths.Snapshots
case backend.Index:
n = backend.Paths.Index
case backend.Lock:
n = backend.Paths.Locks
case backend.Key:
n = backend.Paths.Keys
}
return Join(r.p, n)
}
// Load returns the data stored in the backend for h at the given offset
// and saves it in p. Load has the same semantics as io.ReaderAt.
func (r *SFTP) Load(h backend.Handle, p []byte, off int64) (n int, err error) {
if err := h.Valid(); err != nil {
return 0, err
}
f, err := r.c.Open(r.filename(h.Type, h.Name))
if err != nil {
return 0, err
}
defer func() {
e := f.Close()
if err == nil && e != nil {
err = e
}
}()
if off > 0 {
_, err = f.Seek(off, 0)
if err != nil {
return 0, err
}
}
return io.ReadFull(f, p)
}
// Save stores data in the backend at the handle.
func (r *SFTP) Save(h backend.Handle, p []byte) (err error) {
if err := h.Valid(); err != nil {
return err
}
filename, tmpfile, err := r.tempFile()
debug.Log("sftp.Save", "save %v (%d bytes) to %v", h, len(p), filename)
n, err := tmpfile.Write(p)
if err != nil {
return err
}
if n != len(p) {
return errors.New("not all bytes writen")
}
err = tmpfile.Close()
if err != nil {
return err
}
err = r.renameFile(filename, h.Type, h.Name)
debug.Log("sftp.Save", "save %v: rename %v: %v",
h, filepath.Base(filename), err)
if err != nil {
return fmt.Errorf("sftp: renameFile: %v", err)
}
return nil
}
// Stat returns information about a blob.
func (r *SFTP) Stat(h backend.Handle) (backend.BlobInfo, error) {
if err := h.Valid(); err != nil {
return backend.BlobInfo{}, err
}
fi, err := r.c.Lstat(r.filename(h.Type, h.Name))
if err != nil {
return backend.BlobInfo{}, err
}
return backend.BlobInfo{Size: fi.Size()}, nil
}
// Test returns true if a blob of the given type and name exists in the backend.
func (r *SFTP) Test(t backend.Type, name string) (bool, error) {
_, err := r.c.Lstat(r.filename(t, name))
if os.IsNotExist(err) {
return false, nil
}
if err != nil {
return false, err
}
return true, nil
}
// Remove removes the content stored at name.
func (r *SFTP) Remove(t backend.Type, name string) error {
return r.c.Remove(r.filename(t, name))
}
// List returns a channel that yields all names of blobs of type t. A
// goroutine is started for this. If the channel done is closed, sending
// stops.
func (r *SFTP) List(t backend.Type, done <-chan struct{}) <-chan string {
ch := make(chan string)
go func() {
defer close(ch)
if t == backend.Data {
// read first level
basedir := r.dirname(t, "")
list1, err := r.c.ReadDir(basedir)
if err != nil {
return
}
dirs := make([]string, 0, len(list1))
for _, d := range list1 {
dirs = append(dirs, d.Name())
}
// read files
for _, dir := range dirs {
entries, err := r.c.ReadDir(Join(basedir, dir))
if err != nil {
continue
}
items := make([]string, 0, len(entries))
for _, entry := range entries {
items = append(items, entry.Name())
}
for _, file := range items {
select {
case ch <- file:
case <-done:
return
}
}
}
} else {
entries, err := r.c.ReadDir(r.dirname(t, ""))
if err != nil {
return
}
items := make([]string, 0, len(entries))
for _, entry := range entries {
items = append(items, entry.Name())
}
for _, file := range items {
select {
case ch <- file:
case <-done:
return
}
}
}
}()
return ch
}
// Close closes the sftp connection and terminates the underlying command.
func (r *SFTP) Close() error {
if r == nil {
return nil
}
err := r.c.Close()
debug.Log("sftp.Close", "Close returned error %v", err)
if err := r.cmd.Process.Kill(); err != nil {
return err
}
return r.cmd.Wait()
}
src/github.com/restic/restic/backend/sftp/sftp_backend_test.go
+80
View File
@@ -0,0 +1,80 @@
package sftp_test
import (
"fmt"
"io/ioutil"
"os"
"path/filepath"
"strings"
"github.com/restic/restic/backend"
"github.com/restic/restic/backend/sftp"
"github.com/restic/restic/backend/test"
. "github.com/restic/restic/test"
)
var tempBackendDir string
//go:generate go run ../test/generate_backend_tests.go
func createTempdir() error {
if tempBackendDir != "" {
return nil
}
tempdir, err := ioutil.TempDir("", "restic-local-test-")
if err != nil {
return err
}
fmt.Printf("created new test backend at %v\n", tempdir)
tempBackendDir = tempdir
return nil
}
func init() {
sftpserver := ""
for _, dir := range strings.Split(TestSFTPPath, ":") {
testpath := filepath.Join(dir, "sftp-server")
_, err := os.Stat(testpath)
if !os.IsNotExist(err) {
sftpserver = testpath
break
}
}
if sftpserver == "" {
SkipMessage = "sftp server binary not found, skipping tests"
return
}
test.CreateFn = func() (backend.Backend, error) {
err := createTempdir()
if err != nil {
return nil, err
}
return sftp.Create(tempBackendDir, sftpserver)
}
test.OpenFn = func() (backend.Backend, error) {
err := createTempdir()
if err != nil {
return nil, err
}
return sftp.Open(tempBackendDir, sftpserver)
}
test.CleanupFn = func() error {
if tempBackendDir == "" {
return nil
}
fmt.Printf("removing test backend at %v\n", tempBackendDir)
err := os.RemoveAll(tempBackendDir)
tempBackendDir = ""
return err
}
}
src/github.com/restic/restic/backend/sftp/sftp_unix.go
+13
View File
@@ -0,0 +1,13 @@
// +build !windows
package sftp
import (
"syscall"
)
// ignoreSigIntProcAttr returns a syscall.SysProcAttr that
// disables SIGINT on parent.
func ignoreSigIntProcAttr() *syscall.SysProcAttr {
return &syscall.SysProcAttr{Setsid: true}
}
src/github.com/restic/restic/backend/sftp/sftp_windows.go
+11
View File
@@ -0,0 +1,11 @@
package sftp
import (
"syscall"
)
// ignoreSigIntProcAttr returns a default syscall.SysProcAttr
// on Windows.
func ignoreSigIntProcAttr() *syscall.SysProcAttr {
return &syscall.SysProcAttr{}
}
src/github.com/restic/restic/backend/test/backend_test.go
+87
View File
@@ -0,0 +1,87 @@
// DO NOT EDIT, AUTOMATICALLY GENERATED
package test_test
import (
"testing"
"github.com/restic/restic/backend/test"
)
var SkipMessage string
func TestTestBackendCreate(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestCreate(t)
}
func TestTestBackendOpen(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestOpen(t)
}
func TestTestBackendCreateWithConfig(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestCreateWithConfig(t)
}
func TestTestBackendLocation(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestLocation(t)
}
func TestTestBackendConfig(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestConfig(t)
}
func TestTestBackendLoad(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestLoad(t)
}
func TestTestBackendSave(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestSave(t)
}
func TestTestBackendSaveFilenames(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestSaveFilenames(t)
}
func TestTestBackendBackend(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestBackend(t)
}
func TestTestBackendDelete(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestDelete(t)
}
func TestTestBackendCleanup(t *testing.T) {
if SkipMessage != "" {
t.Skip(SkipMessage)
}
test.TestCleanup(t)
}
src/github.com/restic/restic/backend/test/generate_backend_tests.go
+140
View File
@@ -0,0 +1,140 @@
// +build ignore
package main
import (
"bufio"
"flag"
"fmt"
"io"
"log"
"os"
"os/exec"
"path/filepath"
"regexp"
"text/template"
"unicode"
"unicode/utf8"
)
var data struct {
Package string
PackagePrefix string
Funcs []string
}
var testTemplate = `
// DO NOT EDIT, AUTOMATICALLY GENERATED
package {{ .Package }}
import (
"testing"
"github.com/restic/restic/backend/test"
)
var SkipMessage string
{{ $prefix := .PackagePrefix }}
{{ range $f := .Funcs }}
func Test{{ $prefix }}{{ $f }}(t *testing.T){
if SkipMessage != "" { t.Skip(SkipMessage) }
test.Test{{ $f }}(t)
}
{{ end }}
`
var testFile = flag.String("testfile", "../test/tests.go", "file to search test functions in")
var outputFile = flag.String("output", "backend_test.go", "output file to write generated code to")
var packageName = flag.String("package", "", "the package name to use")
var prefix = flag.String("prefix", "", "test function prefix")
var quiet = flag.Bool("quiet", false, "be quiet")
func errx(err error) {
if err == nil {
return
}
fmt.Fprintf(os.Stderr, "error: %v\n", err)
os.Exit(1)
}
var funcRegex = regexp.MustCompile(`^func\s+Test(.+)\s*\(`)
func findTestFunctions() (funcs []string) {
f, err := os.Open(*testFile)
errx(err)
sc := bufio.NewScanner(f)
for sc.Scan() {
match := funcRegex.FindStringSubmatch(sc.Text())
if len(match) > 0 {
funcs = append(funcs, match[1])
}
}
if err := sc.Err(); err != nil {
log.Fatalf("Error scanning file: %v", err)
}
errx(f.Close())
return funcs
}
func generateOutput(wr io.Writer, data interface{}) {
t := template.Must(template.New("backendtest").Parse(testTemplate))
cmd := exec.Command("gofmt")
cmd.Stdout = wr
in, err := cmd.StdinPipe()
errx(err)
errx(cmd.Start())
errx(t.Execute(in, data))
errx(in.Close())
errx(cmd.Wait())
}
func packageTestFunctionPrefix(pkg string) string {
if pkg == "" {
return ""
}
r, n := utf8.DecodeRuneInString(pkg)
return string(unicode.ToUpper(r)) + pkg[n:]
}
func init() {
flag.Parse()
}
func main() {
dir, err := os.Getwd()
if err != nil {
fmt.Fprintf(os.Stderr, "Getwd() %v\n", err)
os.Exit(1)
}
pkg := *packageName
if pkg == "" {
pkg = filepath.Base(dir)
}
f, err := os.Create(*outputFile)
errx(err)
data.Package = pkg + "_test"
if *prefix != "" {
data.PackagePrefix = *prefix
} else {
data.PackagePrefix = packageTestFunctionPrefix(pkg) + "Backend"
}
data.Funcs = findTestFunctions()
generateOutput(f, data)
errx(f.Close())
if !*quiet {
fmt.Printf("wrote backend tests for package %v to %v\n", data.Package, *outputFile)
}
}
src/github.com/restic/restic/backend/test/tests.go
+525
View File
@@ -0,0 +1,525 @@
package test
import (
"bytes"
"fmt"
"io"
"io/ioutil"
"math/rand"
"reflect"
"sort"
"testing"
"github.com/restic/restic/backend"
. "github.com/restic/restic/test"
)
// CreateFn is a function that creates a temporary repository for the tests.
var CreateFn func() (backend.Backend, error)
// OpenFn is a function that opens a previously created temporary repository.
var OpenFn func() (backend.Backend, error)
// CleanupFn removes temporary files and directories created during the tests.
var CleanupFn func() error
var but backend.Backend // backendUnderTest
var butInitialized bool
func open(t testing.TB) backend.Backend {
if OpenFn == nil {
t.Fatal("OpenFn not set")
}
if CreateFn == nil {
t.Fatalf("CreateFn not set")
}
if !butInitialized {
be, err := CreateFn()
if err != nil {
t.Fatalf("Create returned unexpected error: %v", err)
}
but = be
butInitialized = true
}
if but == nil {
var err error
but, err = OpenFn()
if err != nil {
t.Fatalf("Open returned unexpected error: %v", err)
}
}
return but
}
func close(t testing.TB) {
if but == nil {
t.Fatalf("trying to close non-existing backend")
}
err := but.Close()
if err != nil {
t.Fatalf("Close returned unexpected error: %v", err)
}
but = nil
}
// TestCreate creates a backend.
func TestCreate(t testing.TB) {
if CreateFn == nil {
t.Fatalf("CreateFn not set!")
}
be, err := CreateFn()
if err != nil {
fmt.Printf("foo\n")
t.Fatalf("Create returned error: %v", err)
}
butInitialized = true
err = be.Close()
if err != nil {
t.Fatalf("Close returned error: %v", err)
}
}
// TestOpen opens a previously created backend.
func TestOpen(t testing.TB) {
if OpenFn == nil {
t.Fatalf("OpenFn not set!")
}
be, err := OpenFn()
if err != nil {
t.Fatalf("Open returned error: %v", err)
}
err = be.Close()
if err != nil {
t.Fatalf("Close returned error: %v", err)
}
}
// TestCreateWithConfig tests that creating a backend in a location which already
// has a config file fails.
func TestCreateWithConfig(t testing.TB) {
if CreateFn == nil {
t.Fatalf("CreateFn not set")
}
b := open(t)
defer close(t)
// save a config
store(t, b, backend.Config, []byte("test config"))
// now create the backend again, this must fail
_, err := CreateFn()
if err == nil {
t.Fatalf("expected error not found for creating a backend with an existing config file")
}
// remove config
err = b.Remove(backend.Config, "")
if err != nil {
t.Fatalf("unexpected error removing config: %v", err)
}
}
// TestLocation tests that a location string is returned.
func TestLocation(t testing.TB) {
b := open(t)
defer close(t)
l := b.Location()
if l == "" {
t.Fatalf("invalid location string %q", l)
}
}
// TestConfig saves and loads a config from the backend.
func TestConfig(t testing.TB) {
b := open(t)
defer close(t)
var testString = "Config"
// create config and read it back
_, err := backend.LoadAll(b, backend.Handle{Type: backend.Config}, nil)
if err == nil {
t.Fatalf("did not get expected error for non-existing config")
}
err = b.Save(backend.Handle{Type: backend.Config}, []byte(testString))
if err != nil {
t.Fatalf("Save() error: %v", err)
}
// try accessing the config with different names, should all return the
// same config
for _, name := range []string{"", "foo", "bar", "0000000000000000000000000000000000000000000000000000000000000000"} {
buf, err := backend.LoadAll(b, backend.Handle{Type: backend.Config}, nil)
if err != nil {
t.Fatalf("unable to read config with name %q: %v", name, err)
}
if string(buf) != testString {
t.Fatalf("wrong data returned, want %q, got %q", testString, string(buf))
}
}
}
// TestLoad tests the backend's Load function.
func TestLoad(t testing.TB) {
b := open(t)
defer close(t)
_, err := b.Load(backend.Handle{}, nil, 0)
if err == nil {
t.Fatalf("Load() did not return an error for invalid handle")
}
_, err = b.Load(backend.Handle{Type: backend.Data, Name: "foobar"}, nil, 0)
if err == nil {
t.Fatalf("Load() did not return an error for non-existing blob")
}
length := rand.Intn(1<<24) + 2000
data := Random(23, length)
id := backend.Hash(data)
handle := backend.Handle{Type: backend.Data, Name: id.String()}
err = b.Save(handle, data)
if err != nil {
t.Fatalf("Save() error: %v", err)
}
for i := 0; i < 50; i++ {
l := rand.Intn(length + 2000)
o := rand.Intn(length + 2000)
d := data
if o < len(d) {
d = d[o:]
} else {
o = len(d)
d = d[:0]
}
if l > 0 && l < len(d) {
d = d[:l]
}
buf := make([]byte, l)
n, err := b.Load(handle, buf, int64(o))
// if we requested data beyond the end of the file, ignore
// ErrUnexpectedEOF error
if l > len(d) && err == io.ErrUnexpectedEOF {
err = nil
buf = buf[:len(d)]
}
if err != nil {
t.Errorf("Load(%d, %d): unexpected error: %v", len(buf), int64(o), err)
continue
}
if n != len(buf) {
t.Errorf("Load(%d, %d): wrong length returned, want %d, got %d",
len(buf), int64(o), len(buf), n)
continue
}
buf = buf[:n]
if !bytes.Equal(buf, d) {
t.Errorf("Load(%d, %d) returned wrong bytes", len(buf), int64(o))
continue
}
}
// load with a too-large buffer, this should return io.ErrUnexpectedEOF
buf := make([]byte, length+100)
n, err := b.Load(handle, buf, 0)
if n != length {
t.Errorf("wrong length for larger buffer returned, want %d, got %d", length, n)
}
if err != io.ErrUnexpectedEOF {
t.Errorf("wrong error returned for larger buffer: want io.ErrUnexpectedEOF, got %#v", err)
}
OK(t, b.Remove(backend.Data, id.String()))
}
// TestSave tests saving data in the backend.
func TestSave(t testing.TB) {
b := open(t)
defer close(t)
var id backend.ID
for i := 0; i < 10; i++ {
length := rand.Intn(1<<23) + 200000
data := Random(23, length)
// use the first 32 byte as the ID
copy(id[:], data)
h := backend.Handle{
Type: backend.Data,
Name: fmt.Sprintf("%s-%d", id, i),
}
err := b.Save(h, data)
OK(t, err)
buf, err := backend.LoadAll(b, h, nil)
OK(t, err)
if len(buf) != len(data) {
t.Fatalf("number of bytes does not match, want %v, got %v", len(data), len(buf))
}
if !bytes.Equal(buf, data) {
t.Fatalf("data not equal")
}
fi, err := b.Stat(h)
OK(t, err)
if fi.Size != int64(len(data)) {
t.Fatalf("Stat() returned different size, want %q, got %d", len(data), fi.Size)
}
err = b.Remove(h.Type, h.Name)
if err != nil {
t.Fatalf("error removing item: %v", err)
}
}
}
var filenameTests = []struct {
name string
data string
}{
{"1dfc6bc0f06cb255889e9ea7860a5753e8eb9665c9a96627971171b444e3113e", "x"},
{"foobar", "foobar"},
{
"1dfc6bc0f06cb255889e9ea7860a5753e8eb9665c9a96627971171b444e3113e4bf8f2d9144cc5420a80f04a4880ad6155fc58903a4fb6457c476c43541dcaa6-5",
"foobar content of data blob",
},
}
// TestSaveFilenames tests saving data with various file names in the backend.
func TestSaveFilenames(t testing.TB) {
b := open(t)
defer close(t)
for i, test := range filenameTests {
h := backend.Handle{Name: test.name, Type: backend.Data}
err := b.Save(h, []byte(test.data))
if err != nil {
t.Errorf("test %d failed: Save() returned %v", i, err)
continue
}
buf, err := backend.LoadAll(b, h, nil)
if err != nil {
t.Errorf("test %d failed: Load() returned %v", i, err)
continue
}
if !bytes.Equal(buf, []byte(test.data)) {
t.Errorf("test %d: returned wrong bytes", i)
}
err = b.Remove(h.Type, h.Name)
if err != nil {
t.Errorf("test %d failed: Remove() returned %v", i, err)
continue
}
}
}
var testStrings = []struct {
id string
data string
}{
{"c3ab8ff13720e8ad9047dd39466b3c8974e592c2fa383d4a3960714caef0c4f2", "foobar"},
{"248d6a61d20638b8e5c026930c3e6039a33ce45964ff2167f6ecedd419db06c1", "abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq"},
{"cc5d46bdb4991c6eae3eb739c9c8a7a46fe9654fab79c47b4fe48383b5b25e1c", "foo/bar"},
{"4e54d2c721cbdb730f01b10b62dec622962b36966ec685880effa63d71c808f2", "foo/../../baz"},
}
func store(t testing.TB, b backend.Backend, tpe backend.Type, data []byte) {
id := backend.Hash(data)
err := b.Save(backend.Handle{Name: id.String(), Type: tpe}, data)
OK(t, err)
}
func read(t testing.TB, rd io.Reader, expectedData []byte) {
buf, err := ioutil.ReadAll(rd)
OK(t, err)
if expectedData != nil {
Equals(t, expectedData, buf)
}
}
// TestBackend tests all functions of the backend.
func TestBackend(t testing.TB) {
b := open(t)
defer close(t)
for _, tpe := range []backend.Type{
backend.Data, backend.Key, backend.Lock,
backend.Snapshot, backend.Index,
} {
// detect non-existing files
for _, test := range testStrings {
id, err := backend.ParseID(test.id)
OK(t, err)
// test if blob is already in repository
ret, err := b.Test(tpe, id.String())
OK(t, err)
Assert(t, !ret, "blob was found to exist before creating")
// try to stat a not existing blob
h := backend.Handle{Type: tpe, Name: id.String()}
_, err = b.Stat(h)
Assert(t, err != nil, "blob data could be extracted before creation")
// try to read not existing blob
_, err = b.Load(h, nil, 0)
Assert(t, err != nil, "blob reader could be obtained before creation")
// try to get string out, should fail
ret, err = b.Test(tpe, id.String())
OK(t, err)
Assert(t, !ret, "id %q was found (but should not have)", test.id)
}
// add files
for _, test := range testStrings {
store(t, b, tpe, []byte(test.data))
// test Load()
h := backend.Handle{Type: tpe, Name: test.id}
buf, err := backend.LoadAll(b, h, nil)
OK(t, err)
Equals(t, test.data, string(buf))
// try to read it out with an offset and a length
start := 1
end := len(test.data) - 2
length := end - start
buf2 := make([]byte, length)
n, err := b.Load(h, buf2, int64(start))
OK(t, err)
Equals(t, length, n)
Equals(t, test.data[start:end], string(buf2))
}
// test adding the first file again
test := testStrings[0]
// create blob
err := b.Save(backend.Handle{Type: tpe, Name: test.id}, []byte(test.data))
Assert(t, err != nil, "expected error, got %v", err)
// remove and recreate
err = b.Remove(tpe, test.id)
OK(t, err)
// test that the blob is gone
ok, err := b.Test(tpe, test.id)
OK(t, err)
Assert(t, ok == false, "removed blob still present")
// create blob
err = b.Save(backend.Handle{Type: tpe, Name: test.id}, []byte(test.data))
OK(t, err)
// list items
IDs := backend.IDs{}
for _, test := range testStrings {
id, err := backend.ParseID(test.id)
OK(t, err)
IDs = append(IDs, id)
}
list := backend.IDs{}
for s := range b.List(tpe, nil) {
list = append(list, ParseID(s))
}
if len(IDs) != len(list) {
t.Fatalf("wrong number of IDs returned: want %d, got %d", len(IDs), len(list))
}
sort.Sort(IDs)
sort.Sort(list)
if !reflect.DeepEqual(IDs, list) {
t.Fatalf("lists aren't equal, want:\n %v\n got:\n%v\n", IDs, list)
}
// remove content if requested
if TestCleanupTempDirs {
for _, test := range testStrings {
id, err := backend.ParseID(test.id)
OK(t, err)
found, err := b.Test(tpe, id.String())
OK(t, err)
OK(t, b.Remove(tpe, id.String()))
found, err = b.Test(tpe, id.String())
OK(t, err)
Assert(t, !found, fmt.Sprintf("id %q not found after removal", id))
}
}
}
}
// TestDelete tests the Delete function.
func TestDelete(t testing.TB) {
b := open(t)
defer close(t)
be, ok := b.(backend.Deleter)
if !ok {
return
}
err := be.Delete()
if err != nil {
t.Fatalf("error deleting backend: %v", err)
}
}
// TestCleanup runs the cleanup function after all tests are run.
func TestCleanup(t testing.TB) {
if CleanupFn == nil {
t.Log("CleanupFn function not set")
return
}
if !TestCleanupTempDirs {
t.Logf("not cleaning up backend")
return
}
err := CleanupFn()
if err != nil {
t.Fatalf("Cleanup returned error: %v", err)
}
}
src/github.com/restic/restic/backend/test/tests_test.go
+38
View File
@@ -0,0 +1,38 @@
package test_test
import (
"errors"
"github.com/restic/restic/backend"
"github.com/restic/restic/backend/mem"
"github.com/restic/restic/backend/test"
)
var be backend.Backend
//go:generate go run ../test/generate_backend_tests.go
func init() {
test.CreateFn = func() (backend.Backend, error) {
if be != nil {
return nil, errors.New("temporary memory backend dir already exists")
}
be = mem.New()
return be, nil
}
test.OpenFn = func() (backend.Backend, error) {
if be == nil {
return nil, errors.New("repository not initialized")
}
return be, nil
}
test.CleanupFn = func() error {
be = nil
return nil
}
}
src/github.com/restic/restic/backend/utils.go
+25
View File
@@ -0,0 +1,25 @@
package backend
import "io"
// LoadAll reads all data stored in the backend for the handle. The buffer buf
// is resized to accomodate all data in the blob. Errors returned by be.Load()
// are passed on, except io.ErrUnexpectedEOF is silenced and nil returned
// instead, since it means this function is working properly.
func LoadAll(be Backend, h Handle, buf []byte) ([]byte, error) {
fi, err := be.Stat(h)
if err != nil {
return nil, err
}
if fi.Size > int64(len(buf)) {
buf = make([]byte, int(fi.Size))
}
n, err := be.Load(h, buf, 0)
if err == io.ErrUnexpectedEOF {
err = nil
}
buf = buf[:n]
return buf, err
}
src/github.com/restic/restic/backend/utils_test.go
+91
View File
@@ -0,0 +1,91 @@
package backend_test
import (
"bytes"
"math/rand"
"testing"
"github.com/restic/restic/backend"
"github.com/restic/restic/backend/mem"
. "github.com/restic/restic/test"
)
const KiB = 1 << 10
const MiB = 1 << 20
func TestLoadAll(t *testing.T) {
b := mem.New()
for i := 0; i < 20; i++ {
data := Random(23+i, rand.Intn(MiB)+500*KiB)
id := backend.Hash(data)
err := b.Save(backend.Handle{Name: id.String(), Type: backend.Data}, data)
OK(t, err)
buf, err := backend.LoadAll(b, backend.Handle{Type: backend.Data, Name: id.String()}, nil)
OK(t, err)
if len(buf) != len(data) {
t.Errorf("length of returned buffer does not match, want %d, got %d", len(data), len(buf))
continue
}
if !bytes.Equal(buf, data) {
t.Errorf("wrong data returned")
continue
}
}
}
func TestLoadSmallBuffer(t *testing.T) {
b := mem.New()
for i := 0; i < 20; i++ {
data := Random(23+i, rand.Intn(MiB)+500*KiB)
id := backend.Hash(data)
err := b.Save(backend.Handle{Name: id.String(), Type: backend.Data}, data)
OK(t, err)
buf := make([]byte, len(data)-23)
buf, err = backend.LoadAll(b, backend.Handle{Type: backend.Data, Name: id.String()}, buf)
OK(t, err)
if len(buf) != len(data) {
t.Errorf("length of returned buffer does not match, want %d, got %d", len(data), len(buf))
continue
}
if !bytes.Equal(buf, data) {
t.Errorf("wrong data returned")
continue
}
}
}
func TestLoadLargeBuffer(t *testing.T) {
b := mem.New()
for i := 0; i < 20; i++ {
data := Random(23+i, rand.Intn(MiB)+500*KiB)
id := backend.Hash(data)
err := b.Save(backend.Handle{Name: id.String(), Type: backend.Data}, data)
OK(t, err)
buf := make([]byte, len(data)+100)
buf, err = backend.LoadAll(b, backend.Handle{Type: backend.Data, Name: id.String()}, buf)
OK(t, err)
if len(buf) != len(data) {
t.Errorf("length of returned buffer does not match, want %d, got %d", len(data), len(buf))
continue
}
if !bytes.Equal(buf, data) {
t.Errorf("wrong data returned")
continue
}
}
}
src/github.com/restic/restic/build.go
+377
View File
@@ -0,0 +1,377 @@
// +build ignore
package main
import (
"fmt"
"io"
"io/ioutil"
"os"
"os/exec"
"path"
"path/filepath"
"runtime"
"strings"
"time"
)
var (
verbose bool
keepGopath bool
runTests bool
)
const timeFormat = "2006-01-02 15:04:05"
// specialDir returns true if the file begins with a special character ('.' or '_').
func specialDir(name string) bool {
if name == "." {
return false
}
base := filepath.Base(name)
return base[0] == '_' || base[0] == '.'
}
// excludePath returns true if the file should not be copied to the new GOPATH.
func excludePath(name string) bool {
ext := path.Ext(name)
if ext == ".go" || ext == ".s" {
return false
}
parentDir := filepath.Base(filepath.Dir(name))
if parentDir == "testdata" {
return false
}
return true
}
// updateGopath builds a valid GOPATH at dst, with all Go files in src/ copied
// to dst/prefix/, so calling
//
// updateGopath("/tmp/gopath", "/home/u/restic", "github.com/restic/restic")
//
// with "/home/u/restic" containing the file "foo.go" yields the following tree
// at "/tmp/gopath":
//
// /tmp/gopath
// └── src
// └── github.com
// └── restic
// └── restic
// └── foo.go
func updateGopath(dst, src, prefix string) error {
return filepath.Walk(src, func(name string, fi os.FileInfo, err error) error {
if specialDir(name) {
if fi.IsDir() {
return filepath.SkipDir
}
return nil
}
if fi.IsDir() {
return nil
}
if excludePath(name) {
return nil
}
intermediatePath, err := filepath.Rel(src, name)
if err != nil {
return err
}
fileSrc := filepath.Join(src, intermediatePath)
fileDst := filepath.Join(dst, "src", prefix, intermediatePath)
return copyFile(fileDst, fileSrc)
})
}
// copyFile creates dst from src, preserving file attributes and timestamps.
func copyFile(dst, src string) error {
fi, err := os.Stat(src)
if err != nil {
return err
}
fsrc, err := os.Open(src)
if err != nil {
return err
}
if err = os.MkdirAll(filepath.Dir(dst), 0755); err != nil {
fmt.Printf("MkdirAll(%v)\n", filepath.Dir(dst))
return err
}
fdst, err := os.Create(dst)
if err != nil {
return err
}
if _, err = io.Copy(fdst, fsrc); err != nil {
return err
}
if err == nil {
err = fsrc.Close()
}
if err == nil {
err = fdst.Close()
}
if err == nil {
err = os.Chmod(dst, fi.Mode())
}
if err == nil {
err = os.Chtimes(dst, fi.ModTime(), fi.ModTime())
}
return nil
}
// die prints the message with fmt.Fprintf() to stderr and exits with an error
// code.
func die(message string, args ...interface{}) {
fmt.Fprintf(os.Stderr, message, args...)
os.Exit(1)
}
func showUsage(output io.Writer) {
fmt.Fprintf(output, "USAGE: go run build.go OPTIONS\n")
fmt.Fprintf(output, "\n")
fmt.Fprintf(output, "OPTIONS:\n")
fmt.Fprintf(output, " -v --verbose output more messages\n")
fmt.Fprintf(output, " -t --tags specify additional build tags\n")
fmt.Fprintf(output, " -k --keep-gopath do not remove the GOPATH after build\n")
fmt.Fprintf(output, " -T --test run tests\n")
}
func verbosePrintf(message string, args ...interface{}) {
if !verbose {
return
}
fmt.Printf("build: "+message, args...)
}
// cleanEnv returns a clean environment with GOPATH and GOBIN removed (if
// present).
func cleanEnv() (env []string) {
for _, v := range os.Environ() {
if strings.HasPrefix(v, "GOPATH=") || strings.HasPrefix(v, "GOBIN=") {
continue
}
env = append(env, v)
}
return env
}
// build runs "go build args..." with GOPATH set to gopath.
func build(gopath string, args ...string) error {
args = append([]string{"build"}, args...)
cmd := exec.Command("go", args...)
cmd.Env = append(cleanEnv(), "GOPATH="+gopath)
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
verbosePrintf("go %s\n", args)
return cmd.Run()
}
// test runs "go test args..." with GOPATH set to gopath.
func test(gopath string, args ...string) error {
args = append([]string{"test"}, args...)
cmd := exec.Command("go", args...)
cmd.Env = append(cleanEnv(), "GOPATH="+gopath)
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
verbosePrintf("go %s\n", args)
return cmd.Run()
}
// getVersion returns the version string from the file VERSION in the current
// directory.
func getVersionFromFile() string {
buf, err := ioutil.ReadFile("VERSION")
if err != nil {
verbosePrintf("error reading file VERSION: %v\n", err)
return ""
}
return strings.TrimSpace(string(buf))
}
// getVersion returns a version string which is a combination of the contents
// of the file VERSION in the current directory and the version from git (if
// available).
func getVersion() string {
versionFile := getVersionFromFile()
versionGit := getVersionFromGit()
verbosePrintf("version from file 'VERSION' is %q, version from git %q\n",
versionFile, versionGit)
switch {
case versionFile == "":
return versionGit
case versionGit == "":
return versionFile
}
return fmt.Sprintf("%s (%s)", versionFile, versionGit)
}
// getVersionFromGit returns a version string that identifies the currently
// checked out git commit.
func getVersionFromGit() string {
cmd := exec.Command("git", "describe",
"--long", "--tags", "--dirty", "--always")
out, err := cmd.Output()
if err != nil {
verbosePrintf("git describe returned error: %v\n", err)
return ""
}
version := strings.TrimSpace(string(out))
verbosePrintf("git version is %s\n", version)
return version
}
// Constants represents a set of constants that are set in the final binary to
// the given value via compiler flags.
type Constants map[string]string
// LDFlags returns the string that can be passed to go build's `-ldflags`.
func (cs Constants) LDFlags() string {
l := make([]string, 0, len(cs))
v := runtime.Version()
if strings.HasPrefix(v, "go1.5") || strings.HasPrefix(v, "go1.6") {
for k, v := range cs {
l = append(l, fmt.Sprintf(`-X "%s=%s"`, k, v))
}
} else {
for k, v := range cs {
l = append(l, fmt.Sprintf(`-X %q %q`, k, v))
}
}
return strings.Join(l, " ")
}
func main() {
buildTags := []string{}
skipNext := false
params := os.Args[1:]
for i, arg := range params {
if skipNext {
skipNext = false
continue
}
switch arg {
case "-v", "--verbose":
verbose = true
case "-k", "--keep-gopath":
keepGopath = true
case "-t", "-tags", "--tags":
skipNext = true
buildTags = strings.Split(params[i+1], " ")
case "-T", "--test":
runTests = true
case "-h":
showUsage(os.Stdout)
default:
fmt.Fprintf(os.Stderr, "Error: unknown option %q\n\n", arg)
showUsage(os.Stderr)
os.Exit(1)
}
}
if len(buildTags) == 0 {
verbosePrintf("adding build-tag release\n")
buildTags = []string{"release"}
}
for i := range buildTags {
buildTags[i] = strings.TrimSpace(buildTags[i])
}
verbosePrintf("build tags: %s\n", buildTags)
root, err := os.Getwd()
if err != nil {
die("Getwd(): %v\n", err)
}
gopath, err := ioutil.TempDir("", "restic-build-")
if err != nil {
die("TempDir(): %v\n", err)
}
verbosePrintf("create GOPATH at %v\n", gopath)
if err = updateGopath(gopath, root, "github.com/restic/restic"); err != nil {
die("copying files from %v to %v failed: %v\n", root, gopath, err)
}
vendor := filepath.Join(root, "Godeps", "_workspace", "src")
if err = updateGopath(gopath, vendor, ""); err != nil {
die("copying files from %v to %v failed: %v\n", root, gopath, err)
}
defer func() {
if !keepGopath {
verbosePrintf("remove %v\n", gopath)
if err = os.RemoveAll(gopath); err != nil {
die("remove GOPATH at %s failed: %v\n", err)
}
} else {
verbosePrintf("leaving temporary GOPATH at %v\n", gopath)
}
}()
output := "restic"
if runtime.GOOS == "windows" {
output = "restic.exe"
}
version := getVersion()
compileTime := time.Now().Format(timeFormat)
constants := Constants{`main.compiledAt`: compileTime}
if version != "" {
constants["main.version"] = version
}
ldflags := "-s " + constants.LDFlags()
verbosePrintf("ldflags: %s\n", ldflags)
args := []string{
"-tags", strings.Join(buildTags, " "),
"-ldflags", ldflags,
"-o", output, "github.com/restic/restic/cmd/restic",
}
err = build(gopath, args...)
if err != nil {
die("build failed: %v\n", err)
}
if runTests {
verbosePrintf("running tests\n")
err = test(gopath, "github.com/restic/restic/...")
if err != nil {
die("running tests failed: %v\n", err)
}
}
}
src/github.com/restic/restic/cache.go
+289
View File
@@ -0,0 +1,289 @@
package restic
import (
"errors"
"fmt"
"io"
"os"
"path/filepath"
"runtime"
"strings"
"github.com/restic/restic/backend"
"github.com/restic/restic/debug"
"github.com/restic/restic/repository"
)
// Cache is used to locally cache items from a repository.
type Cache struct {
base string
}
// NewCache returns a new cache at cacheDir. If it is the empty string, the
// default cache location is chosen.
func NewCache(repo *repository.Repository, cacheDir string) (*Cache, error) {
var err error
if cacheDir == "" {
cacheDir, err = getCacheDir()
if err != nil {
return nil, err
}
}
basedir := filepath.Join(cacheDir, repo.Config.ID)
debug.Log("Cache.New", "opened cache at %v", basedir)
return &Cache{base: basedir}, nil
}
// Has checks if the local cache has the id.
func (c *Cache) Has(t backend.Type, subtype string, id backend.ID) (bool, error) {
filename, err := c.filename(t, subtype, id)
if err != nil {
return false, err
}
fd, err := os.Open(filename)
defer fd.Close()
if err != nil {
if os.IsNotExist(err) {
debug.Log("Cache.Has", "test for file %v: not cached", filename)
return false, nil
}
debug.Log("Cache.Has", "test for file %v: error %v", filename, err)
return false, err
}
debug.Log("Cache.Has", "test for file %v: is cached", filename)
return true, nil
}
// Store returns an io.WriteCloser that is used to save new information to the
// cache. The returned io.WriteCloser must be closed by the caller after all
// data has been written.
func (c *Cache) Store(t backend.Type, subtype string, id backend.ID) (io.WriteCloser, error) {
filename, err := c.filename(t, subtype, id)
if err != nil {
return nil, err
}
dirname := filepath.Dir(filename)
err = os.MkdirAll(dirname, 0700)
if err != nil {
return nil, err
}
file, err := os.Create(filename)
if err != nil {
debug.Log("Cache.Store", "error creating file %v: %v", filename, err)
return nil, err
}
debug.Log("Cache.Store", "created file %v", filename)
return file, nil
}
// Load returns information from the cache. The returned io.ReadCloser must be
// closed by the caller.
func (c *Cache) Load(t backend.Type, subtype string, id backend.ID) (io.ReadCloser, error) {
filename, err := c.filename(t, subtype, id)
if err != nil {
return nil, err
}
return os.Open(filename)
}
func (c *Cache) purge(t backend.Type, subtype string, id backend.ID) error {
filename, err := c.filename(t, subtype, id)
if err != nil {
return err
}
err = os.Remove(filename)
debug.Log("Cache.purge", "Remove file %v: %v", filename, err)
if err != nil && os.IsNotExist(err) {
return nil
}
return err
}
// Clear removes information from the cache that isn't present in the repository any more.
func (c *Cache) Clear(repo *repository.Repository) error {
list, err := c.list(backend.Snapshot)
if err != nil {
return err
}
for _, entry := range list {
debug.Log("Cache.Clear", "found entry %v", entry)
if ok, err := repo.Backend().Test(backend.Snapshot, entry.ID.String()); !ok || err != nil {
debug.Log("Cache.Clear", "snapshot %v doesn't exist any more, removing %v", entry.ID, entry)
err = c.purge(backend.Snapshot, entry.Subtype, entry.ID)
if err != nil {
return err
}
}
}
return nil
}
type cacheEntry struct {
ID backend.ID
Subtype string
}
func (c cacheEntry) String() string {
if c.Subtype != "" {
return c.ID.Str() + "." + c.Subtype
}
return c.ID.Str()
}
func (c *Cache) list(t backend.Type) ([]cacheEntry, error) {
var dir string
switch t {
case backend.Snapshot:
dir = filepath.Join(c.base, "snapshots")
default:
return nil, fmt.Errorf("cache not supported for type %v", t)
}
fd, err := os.Open(dir)
if err != nil {
if os.IsNotExist(err) {
return []cacheEntry{}, nil
}
return nil, err
}
defer fd.Close()
fis, err := fd.Readdir(-1)
if err != nil {
return nil, err
}
entries := make([]cacheEntry, 0, len(fis))
for _, fi := range fis {
parts := strings.SplitN(fi.Name(), ".", 2)
id, err := backend.ParseID(parts[0])
// ignore invalid cache entries for now
if err != nil {
debug.Log("Cache.List", "unable to parse name %v as id: %v", parts[0], err)
continue
}
e := cacheEntry{ID: id}
if len(parts) == 2 {
e.Subtype = parts[1]
}
entries = append(entries, e)
}
return entries, nil
}
func (c *Cache) filename(t backend.Type, subtype string, id backend.ID) (string, error) {
filename := id.String()
if subtype != "" {
filename += "." + subtype
}
switch t {
case backend.Snapshot:
return filepath.Join(c.base, "snapshots", filename), nil
}
return "", fmt.Errorf("cache not supported for type %v", t)
}
func getCacheDir() (string, error) {
if dir := os.Getenv("RESTIC_CACHE"); dir != "" {
return dir, nil
}
if runtime.GOOS == "windows" {
return getWindowsCacheDir()
}
return getXDGCacheDir()
}
// getWindowsCacheDir will return %APPDATA%\restic or create
// a folder in the temporary folder called "restic".
func getWindowsCacheDir() (string, error) {
cachedir := os.Getenv("APPDATA")
if cachedir == "" {
cachedir = os.TempDir()
}
cachedir = filepath.Join(cachedir, "restic")
fi, err := os.Stat(cachedir)
if os.IsNotExist(err) {
err = os.MkdirAll(cachedir, 0700)
if err != nil {
return "", err
}
return cachedir, nil
}
if err != nil {
return "", err
}
if !fi.IsDir() {
return "", fmt.Errorf("cache dir %v is not a directory", cachedir)
}
return cachedir, nil
}
// getXDGCacheDir returns the cache directory according to XDG basedir spec, see
// http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html
func getXDGCacheDir() (string, error) {
xdgcache := os.Getenv("XDG_CACHE_HOME")
home := os.Getenv("HOME")
if xdgcache == "" && home == "" {
return "", errors.New("unable to locate cache directory (XDG_CACHE_HOME and HOME unset)")
}
cachedir := ""
if xdgcache != "" {
cachedir = filepath.Join(xdgcache, "restic")
} else if home != "" {
cachedir = filepath.Join(home, ".cache", "restic")
}
fi, err := os.Stat(cachedir)
if os.IsNotExist(err) {
err = os.MkdirAll(cachedir, 0700)
if err != nil {
return "", err
}
fi, err = os.Stat(cachedir)
debug.Log("getCacheDir", "create cache dir %v", cachedir)
}
if err != nil {
return "", err
}
if !fi.IsDir() {
return "", fmt.Errorf("cache dir %v is not a directory", cachedir)
}
return cachedir, nil
}
src/github.com/restic/restic/cache_test.go
+23
View File
@@ -0,0 +1,23 @@
package restic_test
import (
"testing"
"github.com/restic/restic"
. "github.com/restic/restic/test"
)
func TestCache(t *testing.T) {
repo := SetupRepo()
defer TeardownRepo(repo)
_, err := restic.NewCache(repo, "")
OK(t, err)
arch := restic.NewArchiver(repo)
// archive some files, this should automatically cache all blobs from the snapshot
_, _, err = arch.Snapshot(nil, []string{BenchArchiveDirectory}, nil)
// TODO: test caching index
}
src/github.com/restic/restic/checker/checker.go
+738
View File
@@ -0,0 +1,738 @@
package checker
import (
"bytes"
"errors"
"fmt"
"sync"
"github.com/restic/restic"
"github.com/restic/restic/backend"
"github.com/restic/restic/crypto"
"github.com/restic/restic/debug"
"github.com/restic/restic/pack"
"github.com/restic/restic/repository"
)
// Checker runs various checks on a repository. It is advisable to create an
// exclusive Lock in the repository before running any checks.
//
// A Checker only tests for internal errors within the data structures of the
// repository (e.g. missing blobs), and needs a valid Repository to work on.
type Checker struct {
packs map[backend.ID]struct{}
blobs map[backend.ID]struct{}
blobRefs struct {
sync.Mutex
M map[backend.ID]uint
}
indexes map[backend.ID]*repository.Index
orphanedPacks backend.IDs
masterIndex *repository.MasterIndex
repo *repository.Repository
}
// New returns a new checker which runs on repo.
func New(repo *repository.Repository) *Checker {
c := &Checker{
packs: make(map[backend.ID]struct{}),
blobs: make(map[backend.ID]struct{}),
masterIndex: repository.NewMasterIndex(),
indexes: make(map[backend.ID]*repository.Index),
repo: repo,
}
c.blobRefs.M = make(map[backend.ID]uint)
return c
}
const defaultParallelism = 40
// ErrDuplicatePacks is returned when a pack is found in more than one index.
type ErrDuplicatePacks struct {
PackID backend.ID
Indexes backend.IDSet
}
func (e ErrDuplicatePacks) Error() string {
return fmt.Sprintf("pack %v contained in several indexes: %v", e.PackID.Str(), e.Indexes)
}
// ErrOldIndexFormat is returned when an index with the old format is
// found.
type ErrOldIndexFormat struct {
backend.ID
}
func (err ErrOldIndexFormat) Error() string {
return fmt.Sprintf("index %v has old format", err.ID.Str())
}
// LoadIndex loads all index files.
func (c *Checker) LoadIndex() (hints []error, errs []error) {
debug.Log("LoadIndex", "Start")
type indexRes struct {
Index *repository.Index
ID string
}
indexCh := make(chan indexRes)
worker := func(id backend.ID, done <-chan struct{}) error {
debug.Log("LoadIndex", "worker got index %v", id)
idx, err := repository.LoadIndexWithDecoder(c.repo, id.String(), repository.DecodeIndex)
if err == repository.ErrOldIndexFormat {
debug.Log("LoadIndex", "index %v has old format", id.Str())
hints = append(hints, ErrOldIndexFormat{id})
idx, err = repository.LoadIndexWithDecoder(c.repo, id.String(), repository.DecodeOldIndex)
}
if err != nil {
return err
}
select {
case indexCh <- indexRes{Index: idx, ID: id.String()}:
case <-done:
}
return nil
}
var perr error
go func() {
defer close(indexCh)
debug.Log("LoadIndex", "start loading indexes in parallel")
perr = repository.FilesInParallel(c.repo.Backend(), backend.Index, defaultParallelism,
repository.ParallelWorkFuncParseID(worker))
debug.Log("LoadIndex", "loading indexes finished, error: %v", perr)
}()
done := make(chan struct{})
defer close(done)
if perr != nil {
errs = append(errs, perr)
return hints, errs
}
packToIndex := make(map[backend.ID]backend.IDSet)
for res := range indexCh {
debug.Log("LoadIndex", "process index %v", res.ID)
idxID, err := backend.ParseID(res.ID)
if err != nil {
errs = append(errs, fmt.Errorf("unable to parse as index ID: %v", res.ID))
continue
}
c.indexes[idxID] = res.Index
c.masterIndex.Insert(res.Index)
debug.Log("LoadIndex", "process blobs")
cnt := 0
for blob := range res.Index.Each(done) {
c.packs[blob.PackID] = struct{}{}
c.blobs[blob.ID] = struct{}{}
c.blobRefs.M[blob.ID] = 0
cnt++
if _, ok := packToIndex[blob.PackID]; !ok {
packToIndex[blob.PackID] = backend.NewIDSet()
}
packToIndex[blob.PackID].Insert(idxID)
}
debug.Log("LoadIndex", "%d blobs processed", cnt)
}
debug.Log("LoadIndex", "done, error %v", perr)
debug.Log("LoadIndex", "checking for duplicate packs")
for packID := range c.packs {
debug.Log("LoadIndex", " check pack %v: contained in %d indexes", packID.Str(), len(packToIndex[packID]))
if len(packToIndex[packID]) > 1 {
hints = append(hints, ErrDuplicatePacks{
PackID: packID,
Indexes: packToIndex[packID],
})
}
}
c.repo.SetIndex(c.masterIndex)
return hints, errs
}
// PackError describes an error with a specific pack.
type PackError struct {
ID backend.ID
Orphaned bool
Err error
}
func (e PackError) Error() string {
return "pack " + e.ID.String() + ": " + e.Err.Error()
}
func packIDTester(repo *repository.Repository, inChan <-chan backend.ID, errChan chan<- error, wg *sync.WaitGroup, done <-chan struct{}) {
debug.Log("Checker.testPackID", "worker start")
defer debug.Log("Checker.testPackID", "worker done")
defer wg.Done()
for id := range inChan {
ok, err := repo.Backend().Test(backend.Data, id.String())
if err != nil {
err = PackError{ID: id, Err: err}
} else {
if !ok {
err = PackError{ID: id, Err: errors.New("does not exist")}
}
}
if err != nil {
debug.Log("Checker.testPackID", "error checking for pack %s: %v", id.Str(), err)
select {
case <-done:
return
case errChan <- err:
}
continue
}
debug.Log("Checker.testPackID", "pack %s exists", id.Str())
}
}
// Packs checks that all packs referenced in the index are still available and
// there are no packs that aren't in an index. errChan is closed after all
// packs have been checked.
func (c *Checker) Packs(errChan chan<- error, done <-chan struct{}) {
defer close(errChan)
debug.Log("Checker.Packs", "checking for %d packs", len(c.packs))
seenPacks := make(map[backend.ID]struct{})
var workerWG sync.WaitGroup
IDChan := make(chan backend.ID)
for i := 0; i < defaultParallelism; i++ {
workerWG.Add(1)
go packIDTester(c.repo, IDChan, errChan, &workerWG, done)
}
for id := range c.packs {
seenPacks[id] = struct{}{}
IDChan <- id
}
close(IDChan)
debug.Log("Checker.Packs", "waiting for %d workers to terminate", defaultParallelism)
workerWG.Wait()
debug.Log("Checker.Packs", "workers terminated")
for id := range c.repo.List(backend.Data, done) {
debug.Log("Checker.Packs", "check data blob %v", id.Str())
if _, ok := seenPacks[id]; !ok {
c.orphanedPacks = append(c.orphanedPacks, id)
select {
case <-done:
return
case errChan <- PackError{ID: id, Orphaned: true, Err: errors.New("not referenced in any index")}:
}
}
}
}
// Error is an error that occurred while checking a repository.
type Error struct {
TreeID *backend.ID
BlobID *backend.ID
Err error
}
func (e Error) Error() string {
if e.BlobID != nil && e.TreeID != nil {
msg := "tree " + e.TreeID.Str()
msg += ", blob " + e.BlobID.Str()
msg += ": " + e.Err.Error()
return msg
}
if e.TreeID != nil {
return "tree " + e.TreeID.Str() + ": " + e.Err.Error()
}
return e.Err.Error()
}
func loadTreeFromSnapshot(repo *repository.Repository, id backend.ID) (backend.ID, error) {
sn, err := restic.LoadSnapshot(repo, id)
if err != nil {
debug.Log("Checker.loadTreeFromSnapshot", "error loading snapshot %v: %v", id.Str(), err)
return backend.ID{}, err
}
if sn.Tree == nil {
debug.Log("Checker.loadTreeFromSnapshot", "snapshot %v has no tree", id.Str())
return backend.ID{}, fmt.Errorf("snapshot %v has no tree", id)
}
return *sn.Tree, nil
}
// loadSnapshotTreeIDs loads all snapshots from backend and returns the tree IDs.
func loadSnapshotTreeIDs(repo *repository.Repository) (backend.IDs, []error) {
var trees struct {
IDs backend.IDs
sync.Mutex
}
var errs struct {
errs []error
sync.Mutex
}
snapshotWorker := func(strID string, done <-chan struct{}) error {
id, err := backend.ParseID(strID)
if err != nil {
return err
}
debug.Log("Checker.Snaphots", "load snapshot %v", id.Str())
treeID, err := loadTreeFromSnapshot(repo, id)
if err != nil {
errs.Lock()
errs.errs = append(errs.errs, err)
errs.Unlock()
return nil
}
debug.Log("Checker.Snaphots", "snapshot %v has tree %v", id.Str(), treeID.Str())
trees.Lock()
trees.IDs = append(trees.IDs, treeID)
trees.Unlock()
return nil
}
err := repository.FilesInParallel(repo.Backend(), backend.Snapshot, defaultParallelism, snapshotWorker)
if err != nil {
errs.errs = append(errs.errs, err)
}
return trees.IDs, errs.errs
}
// TreeError collects several errors that occurred while processing a tree.
type TreeError struct {
ID backend.ID
Errors []error
}
func (e TreeError) Error() string {
return fmt.Sprintf("tree %v: %v", e.ID.Str(), e.Errors)
}
type treeJob struct {
backend.ID
error
*restic.Tree
}
// loadTreeWorker loads trees from repo and sends them to out.
func loadTreeWorker(repo *repository.Repository,
in <-chan backend.ID, out chan<- treeJob,
done <-chan struct{}, wg *sync.WaitGroup) {
defer func() {
debug.Log("checker.loadTreeWorker", "exiting")
wg.Done()
}()
var (
inCh = in
outCh = out
job treeJob
)
outCh = nil
for {
select {
case <-done:
return
case treeID, ok := <-inCh:
if !ok {
return
}
debug.Log("checker.loadTreeWorker", "load tree %v", treeID.Str())
tree, err := restic.LoadTree(repo, treeID)
debug.Log("checker.loadTreeWorker", "load tree %v (%v) returned err: %v", tree, treeID.Str(), err)
job = treeJob{ID: treeID, error: err, Tree: tree}
outCh = out
inCh = nil
case outCh <- job:
debug.Log("checker.loadTreeWorker", "sent tree %v", job.ID.Str())
outCh = nil
inCh = in
}
}
}
// checkTreeWorker checks the trees received and sends out errors to errChan.
func (c *Checker) checkTreeWorker(in <-chan treeJob, out chan<- error, done <-chan struct{}, wg *sync.WaitGroup) {
defer func() {
debug.Log("checker.checkTreeWorker", "exiting")
wg.Done()
}()
var (
inCh = in
outCh = out
treeError TreeError
)
outCh = nil
for {
select {
case <-done:
debug.Log("checker.checkTreeWorker", "done channel closed, exiting")
return
case job, ok := <-inCh:
if !ok {
debug.Log("checker.checkTreeWorker", "input channel closed, exiting")
return
}
id := job.ID
alreadyChecked := false
c.blobRefs.Lock()
if c.blobRefs.M[id] > 0 {
alreadyChecked = true
}
c.blobRefs.M[id]++
debug.Log("checker.checkTreeWorker", "tree %v refcount %d", job.ID.Str(), c.blobRefs.M[id])
c.blobRefs.Unlock()
if alreadyChecked {
continue
}
debug.Log("checker.checkTreeWorker", "check tree %v (tree %v, err %v)", job.ID.Str(), job.Tree, job.error)
var errs []error
if job.error != nil {
errs = append(errs, job.error)
} else {
errs = c.checkTree(job.ID, job.Tree)
}
if len(errs) > 0 {
debug.Log("checker.checkTreeWorker", "checked tree %v: %v errors", job.ID.Str(), len(errs))
treeError = TreeError{ID: job.ID, Errors: errs}
outCh = out
inCh = nil
}
case outCh <- treeError:
debug.Log("checker.checkTreeWorker", "tree %v: sent %d errors", treeError.ID, len(treeError.Errors))
outCh = nil
inCh = in
}
}
}
func filterTrees(backlog backend.IDs, loaderChan chan<- backend.ID, in <-chan treeJob, out chan<- treeJob, done <-chan struct{}) {
defer func() {
debug.Log("checker.filterTrees", "closing output channels")
close(loaderChan)
close(out)
}()
var (
inCh = in
outCh = out
loadCh = loaderChan
job treeJob
nextTreeID backend.ID
outstandingLoadTreeJobs = 0
)
outCh = nil
loadCh = nil
for {
if loadCh == nil && len(backlog) > 0 {
loadCh = loaderChan
nextTreeID, backlog = backlog[0], backlog[1:]
}
if loadCh == nil && outCh == nil && outstandingLoadTreeJobs == 0 {
debug.Log("checker.filterTrees", "backlog is empty, all channels nil, exiting")
return
}
select {
case <-done:
return
case loadCh <- nextTreeID:
outstandingLoadTreeJobs++
loadCh = nil
case j, ok := <-inCh:
if !ok {
debug.Log("checker.filterTrees", "input channel closed")
inCh = nil
in = nil
continue
}
outstandingLoadTreeJobs--
debug.Log("checker.filterTrees", "input job tree %v", j.ID.Str())
var err error
if j.error != nil {
debug.Log("checker.filterTrees", "received job with error: %v (tree %v, ID %v)", j.error, j.Tree, j.ID.Str())
} else if j.Tree == nil {
debug.Log("checker.filterTrees", "received job with nil tree pointer: %v (ID %v)", j.error, j.ID.Str())
err = errors.New("tree is nil and error is nil")
} else {
debug.Log("checker.filterTrees", "subtrees for tree %v: %v", j.ID.Str(), j.Tree.Subtrees())
for _, id := range j.Tree.Subtrees() {
if id.IsNull() {
// We do not need to raise this error here, it is
// checked when the tree is checked. Just make sure
// that we do not add any null IDs to the backlog.
debug.Log("checker.filterTrees", "tree %v has nil subtree", j.ID.Str())
continue
}
backlog = append(backlog, id)
}
}
if err != nil {
// send a new job with the new error instead of the old one
j = treeJob{ID: j.ID, error: err}
}
job = j
outCh = out
inCh = nil
case outCh <- job:
outCh = nil
inCh = in
}
}
}
// Structure checks that for all snapshots all referenced data blobs and
// subtrees are available in the index. errChan is closed after all trees have
// been traversed.
func (c *Checker) Structure(errChan chan<- error, done <-chan struct{}) {
defer close(errChan)
trees, errs := loadSnapshotTreeIDs(c.repo)
debug.Log("checker.Structure", "need to check %d trees from snapshots, %d errs returned", len(trees), len(errs))
for _, err := range errs {
select {
case <-done:
return
case errChan <- err:
}
}
treeIDChan := make(chan backend.ID)
treeJobChan1 := make(chan treeJob)
treeJobChan2 := make(chan treeJob)
var wg sync.WaitGroup
for i := 0; i < defaultParallelism; i++ {
wg.Add(2)
go loadTreeWorker(c.repo, treeIDChan, treeJobChan1, done, &wg)
go c.checkTreeWorker(treeJobChan2, errChan, done, &wg)
}
filterTrees(trees, treeIDChan, treeJobChan1, treeJobChan2, done)
wg.Wait()
}
func (c *Checker) checkTree(id backend.ID, tree *restic.Tree) (errs []error) {
debug.Log("Checker.checkTree", "checking tree %v", id.Str())
var blobs []backend.ID
for _, node := range tree.Nodes {
switch node.Type {
case "file":
for b, blobID := range node.Content {
if blobID.IsNull() {
errs = append(errs, Error{TreeID: &id, Err: fmt.Errorf("file %q blob %d has null ID", node.Name, b)})
continue
}
blobs = append(blobs, blobID)
}
case "dir":
if node.Subtree == nil {
errs = append(errs, Error{TreeID: &id, Err: fmt.Errorf("dir node %q has no subtree", node.Name)})
continue
}
if node.Subtree.IsNull() {
errs = append(errs, Error{TreeID: &id, Err: fmt.Errorf("dir node %q subtree id is null", node.Name)})
continue
}
}
}
for _, blobID := range blobs {
c.blobRefs.Lock()
c.blobRefs.M[blobID]++
debug.Log("Checker.checkTree", "blob %v refcount %d", blobID.Str(), c.blobRefs.M[blobID])
c.blobRefs.Unlock()
if _, ok := c.blobs[blobID]; !ok {
debug.Log("Checker.trees", "tree %v references blob %v which isn't contained in index", id.Str(), blobID.Str())
errs = append(errs, Error{TreeID: &id, BlobID: &blobID, Err: errors.New("not found in index")})
}
}
return errs
}
// UnusedBlobs returns all blobs that have never been referenced.
func (c *Checker) UnusedBlobs() (blobs backend.IDs) {
c.blobRefs.Lock()
defer c.blobRefs.Unlock()
debug.Log("Checker.UnusedBlobs", "checking %d blobs", len(c.blobs))
for id := range c.blobs {
if c.blobRefs.M[id] == 0 {
debug.Log("Checker.UnusedBlobs", "blob %v not referenced", id.Str())
blobs = append(blobs, id)
}
}
return blobs
}
// OrphanedPacks returns a slice of unused packs (only available after Packs() was run).
func (c *Checker) OrphanedPacks() backend.IDs {
return c.orphanedPacks
}
// CountPacks returns the number of packs in the repository.
func (c *Checker) CountPacks() uint64 {
return uint64(len(c.packs))
}
// checkPack reads a pack and checks the integrity of all blobs.
func checkPack(r *repository.Repository, id backend.ID) error {
debug.Log("Checker.checkPack", "checking pack %v", id.Str())
h := backend.Handle{Type: backend.Data, Name: id.String()}
buf, err := backend.LoadAll(r.Backend(), h, nil)
if err != nil {
return err
}
hash := backend.Hash(buf)
if !hash.Equal(id) {
debug.Log("Checker.checkPack", "Pack ID does not match, want %v, got %v", id.Str(), hash.Str())
return fmt.Errorf("Pack ID does not match, want %v, got %v", id.Str(), hash.Str())
}
unpacker, err := pack.NewUnpacker(r.Key(), bytes.NewReader(buf))
if err != nil {
return err
}
var errs []error
for i, blob := range unpacker.Entries {
debug.Log("Checker.checkPack", " check blob %d: %v", i, blob.ID.Str())
plainBuf := make([]byte, blob.Length)
plainBuf, err = crypto.Decrypt(r.Key(), plainBuf, buf[blob.Offset:blob.Offset+blob.Length])
if err != nil {
debug.Log("Checker.checkPack", " error decrypting blob %v: %v", blob.ID.Str(), err)
errs = append(errs, fmt.Errorf("blob %v: %v", i, err))
continue
}
hash := backend.Hash(plainBuf)
if !hash.Equal(blob.ID) {
debug.Log("Checker.checkPack", " Blob ID does not match, want %v, got %v", blob.ID.Str(), hash.Str())
errs = append(errs, fmt.Errorf("Blob ID does not match, want %v, got %v", blob.ID.Str(), hash.Str()))
continue
}
}
if len(errs) > 0 {
return fmt.Errorf("pack %v contains %v errors: %v", id.Str(), len(errs), errs)
}
return nil
}
// ReadData loads all data from the repository and checks the integrity.
func (c *Checker) ReadData(p *restic.Progress, errChan chan<- error, done <-chan struct{}) {
defer close(errChan)
p.Start()
defer p.Done()
worker := func(wg *sync.WaitGroup, in <-chan backend.ID) {
defer wg.Done()
for {
var id backend.ID
var ok bool
select {
case <-done:
return
case id, ok = <-in:
if !ok {
return
}
}
err := checkPack(c.repo, id)
p.Report(restic.Stat{Blobs: 1})
if err == nil {
continue
}
select {
case <-done:
return
case errChan <- err:
}
}
}
ch := c.repo.List(backend.Data, done)
var wg sync.WaitGroup
for i := 0; i < defaultParallelism; i++ {
wg.Add(1)
go worker(&wg, ch)
}
wg.Wait()
}
src/github.com/restic/restic/checker/checker_test.go
+283
View File
@@ -0,0 +1,283 @@
package checker_test
import (
"fmt"
"math/rand"
"path/filepath"
"sort"
"testing"
"github.com/restic/restic"
"github.com/restic/restic/backend"
"github.com/restic/restic/backend/mem"
"github.com/restic/restic/checker"
"github.com/restic/restic/repository"
. "github.com/restic/restic/test"
)
var checkerTestData = filepath.Join("testdata", "checker-test-repo.tar.gz")
func list(repo *repository.Repository, t backend.Type) (IDs []string) {
done := make(chan struct{})
defer close(done)
for id := range repo.List(t, done) {
IDs = append(IDs, id.String())
}
return IDs
}
func collectErrors(f func(chan<- error, <-chan struct{})) (errs []error) {
done := make(chan struct{})
defer close(done)
errChan := make(chan error)
go f(errChan, done)
for err := range errChan {
errs = append(errs, err)
}
return errs
}
func checkPacks(chkr *checker.Checker) []error {
return collectErrors(chkr.Packs)
}
func checkStruct(chkr *checker.Checker) []error {
return collectErrors(chkr.Structure)
}
func checkData(chkr *checker.Checker) []error {
return collectErrors(
func(errCh chan<- error, done <-chan struct{}) {
chkr.ReadData(nil, errCh, done)
},
)
}
func TestCheckRepo(t *testing.T) {
WithTestEnvironment(t, checkerTestData, func(repodir string) {
repo := OpenLocalRepo(t, repodir)
chkr := checker.New(repo)
hints, errs := chkr.LoadIndex()
if len(errs) > 0 {
t.Fatalf("expected no errors, got %v: %v", len(errs), errs)
}
if len(hints) > 0 {
t.Errorf("expected no hints, got %v: %v", len(hints), hints)
}
OKs(t, checkPacks(chkr))
OKs(t, checkStruct(chkr))
})
}
func TestMissingPack(t *testing.T) {
WithTestEnvironment(t, checkerTestData, func(repodir string) {
repo := OpenLocalRepo(t, repodir)
packID := "657f7fb64f6a854fff6fe9279998ee09034901eded4e6db9bcee0e59745bbce6"
OK(t, repo.Backend().Remove(backend.Data, packID))
chkr := checker.New(repo)
hints, errs := chkr.LoadIndex()
if len(errs) > 0 {
t.Fatalf("expected no errors, got %v: %v", len(errs), errs)
}
if len(hints) > 0 {
t.Errorf("expected no hints, got %v: %v", len(hints), hints)
}
errs = checkPacks(chkr)
Assert(t, len(errs) == 1,
"expected exactly one error, got %v", len(errs))
if err, ok := errs[0].(checker.PackError); ok {
Equals(t, packID, err.ID.String())
} else {
t.Errorf("expected error returned by checker.Packs() to be PackError, got %v", err)
}
})
}
func TestUnreferencedPack(t *testing.T) {
WithTestEnvironment(t, checkerTestData, func(repodir string) {
repo := OpenLocalRepo(t, repodir)
// index 3f1a only references pack 60e0
indexID := "3f1abfcb79c6f7d0a3be517d2c83c8562fba64ef2c8e9a3544b4edaf8b5e3b44"
packID := "60e0438dcb978ec6860cc1f8c43da648170ee9129af8f650f876bad19f8f788e"
OK(t, repo.Backend().Remove(backend.Index, indexID))
chkr := checker.New(repo)
hints, errs := chkr.LoadIndex()
if len(errs) > 0 {
t.Fatalf("expected no errors, got %v: %v", len(errs), errs)
}
if len(hints) > 0 {
t.Errorf("expected no hints, got %v: %v", len(hints), hints)
}
errs = checkPacks(chkr)
Assert(t, len(errs) == 1,
"expected exactly one error, got %v", len(errs))
if err, ok := errs[0].(checker.PackError); ok {
Equals(t, packID, err.ID.String())
} else {
t.Errorf("expected error returned by checker.Packs() to be PackError, got %v", err)
}
})
}
func TestUnreferencedBlobs(t *testing.T) {
WithTestEnvironment(t, checkerTestData, func(repodir string) {
repo := OpenLocalRepo(t, repodir)
snID := "51d249d28815200d59e4be7b3f21a157b864dc343353df9d8e498220c2499b02"
OK(t, repo.Backend().Remove(backend.Snapshot, snID))
unusedBlobsBySnapshot := backend.IDs{
ParseID("58c748bbe2929fdf30c73262bd8313fe828f8925b05d1d4a87fe109082acb849"),
ParseID("988a272ab9768182abfd1fe7d7a7b68967825f0b861d3b36156795832c772235"),
ParseID("c01952de4d91da1b1b80bc6e06eaa4ec21523f4853b69dc8231708b9b7ec62d8"),
ParseID("bec3a53d7dc737f9a9bee68b107ec9e8ad722019f649b34d474b9982c3a3fec7"),
ParseID("2a6f01e5e92d8343c4c6b78b51c5a4dc9c39d42c04e26088c7614b13d8d0559d"),
ParseID("18b51b327df9391732ba7aaf841a4885f350d8a557b2da8352c9acf8898e3f10"),
}
sort.Sort(unusedBlobsBySnapshot)
chkr := checker.New(repo)
hints, errs := chkr.LoadIndex()
if len(errs) > 0 {
t.Fatalf("expected no errors, got %v: %v", len(errs), errs)
}
if len(hints) > 0 {
t.Errorf("expected no hints, got %v: %v", len(hints), hints)
}
OKs(t, checkPacks(chkr))
OKs(t, checkStruct(chkr))
blobs := chkr.UnusedBlobs()
sort.Sort(blobs)
Equals(t, unusedBlobsBySnapshot, blobs)
})
}
var checkerDuplicateIndexTestData = filepath.Join("testdata", "duplicate-packs-in-index-test-repo.tar.gz")
func TestDuplicatePacksInIndex(t *testing.T) {
WithTestEnvironment(t, checkerDuplicateIndexTestData, func(repodir string) {
repo := OpenLocalRepo(t, repodir)
chkr := checker.New(repo)
hints, errs := chkr.LoadIndex()
if len(hints) == 0 {
t.Fatalf("did not get expected checker hints for duplicate packs in indexes")
}
found := false
for _, hint := range hints {
if _, ok := hint.(checker.ErrDuplicatePacks); ok {
found = true
} else {
t.Errorf("got unexpected hint: %v", hint)
}
}
if !found {
t.Fatalf("did not find hint ErrDuplicatePacks")
}
if len(errs) > 0 {
t.Errorf("expected no errors, got %v: %v", len(errs), errs)
}
})
}
// errorBackend randomly modifies data after reading.
type errorBackend struct {
backend.Backend
ProduceErrors bool
}
func (b errorBackend) Load(h backend.Handle, p []byte, off int64) (int, error) {
fmt.Printf("load %v\n", h)
n, err := b.Backend.Load(h, p, off)
if b.ProduceErrors {
induceError(p)
}
return n, err
}
// induceError flips a bit in the slice.
func induceError(data []byte) {
if rand.Float32() < 0.2 {
return
}
pos := rand.Intn(len(data))
data[pos] ^= 1
}
func TestCheckerModifiedData(t *testing.T) {
be := mem.New()
repo := repository.New(be)
OK(t, repo.Init(TestPassword))
arch := restic.NewArchiver(repo)
_, id, err := arch.Snapshot(nil, []string{"."}, nil)
OK(t, err)
t.Logf("archived as %v", id.Str())
beError := &errorBackend{Backend: be}
checkRepo := repository.New(beError)
OK(t, checkRepo.SearchKey(TestPassword))
chkr := checker.New(checkRepo)
hints, errs := chkr.LoadIndex()
if len(errs) > 0 {
t.Fatalf("expected no errors, got %v: %v", len(errs), errs)
}
if len(hints) > 0 {
t.Errorf("expected no hints, got %v: %v", len(hints), hints)
}
beError.ProduceErrors = true
errFound := false
for _, err := range checkPacks(chkr) {
t.Logf("pack error: %v", err)
}
for _, err := range checkStruct(chkr) {
t.Logf("struct error: %v", err)
}
for _, err := range checkData(chkr) {
t.Logf("struct error: %v", err)
errFound = true
}
if !errFound {
t.Fatal("no error found, checker is broken")
}
}
src/github.com/restic/restic/checker/repacker.go
+163
View File
@@ -0,0 +1,163 @@
package checker
import (
"errors"
"github.com/restic/restic/backend"
"github.com/restic/restic/debug"
"github.com/restic/restic/repository"
)
// Repacker extracts still used blobs from packs with unused blobs and creates
// new packs.
type Repacker struct {
unusedBlobs backend.IDSet
repo *repository.Repository
}
// NewRepacker returns a new repacker that (when Repack() in run) cleans up the
// repository and creates new packs and indexs so that all blobs in unusedBlobs
// aren't used any more.
func NewRepacker(repo *repository.Repository, unusedBlobs backend.IDSet) *Repacker {
return &Repacker{
repo: repo,
unusedBlobs: unusedBlobs,
}
}
// Repack runs the process of finding still used blobs in packs with unused
// blobs, extracts them and creates new packs with just the still-in-use blobs.
func (r *Repacker) Repack() error {
debug.Log("Repacker.Repack", "searching packs for %v", r.unusedBlobs)
unneededPacks, err := FindPacksForBlobs(r.repo, r.unusedBlobs)
if err != nil {
return err
}
debug.Log("Repacker.Repack", "found packs: %v", unneededPacks)
blobs, err := FindBlobsForPacks(r.repo, unneededPacks)
if err != nil {
return err
}
debug.Log("Repacker.Repack", "found blobs: %v", blobs)
for id := range r.unusedBlobs {
debug.Log("Repacker.Repack", "remove unused blob %v", id.Str())
blobs.Delete(id)
}
debug.Log("Repacker.Repack", "need to repack blobs: %v", blobs)
err = RepackBlobs(r.repo, r.repo, blobs)
if err != nil {
return err
}
debug.Log("Repacker.Repack", "remove unneeded packs: %v", unneededPacks)
for packID := range unneededPacks {
err = r.repo.Backend().Remove(backend.Data, packID.String())
if err != nil {
return err
}
}
debug.Log("Repacker.Repack", "rebuild index, unneeded packs: %v", unneededPacks)
idx, err := r.repo.Index().RebuildIndex(unneededPacks)
newIndexID, err := repository.SaveIndex(r.repo, idx)
debug.Log("Repacker.Repack", "saved new index at %v, err %v", newIndexID.Str(), err)
if err != nil {
return err
}
debug.Log("Repacker.Repack", "remove old indexes: %v", idx.Supersedes())
for _, id := range idx.Supersedes() {
err = r.repo.Backend().Remove(backend.Index, id.String())
if err != nil {
debug.Log("Repacker.Repack", "error removing index %v: %v", id.Str(), err)
return err
}
debug.Log("Repacker.Repack", "removed index %v", id.Str())
}
return nil
}
// FindPacksForBlobs returns the set of packs that contain the blobs.
func FindPacksForBlobs(repo *repository.Repository, blobs backend.IDSet) (backend.IDSet, error) {
packs := backend.NewIDSet()
idx := repo.Index()
for id := range blobs {
blob, err := idx.Lookup(id)
if err != nil {
return nil, err
}
packs.Insert(blob.PackID)
}
return packs, nil
}
// FindBlobsForPacks returns the set of blobs contained in a pack of packs.
func FindBlobsForPacks(repo *repository.Repository, packs backend.IDSet) (backend.IDSet, error) {
blobs := backend.NewIDSet()
for packID := range packs {
for _, packedBlob := range repo.Index().ListPack(packID) {
blobs.Insert(packedBlob.ID)
}
}
return blobs, nil
}
// repackBlob loads a single blob from src and saves it in dst.
func repackBlob(src, dst *repository.Repository, id backend.ID) error {
blob, err := src.Index().Lookup(id)
if err != nil {
return err
}
debug.Log("RepackBlobs", "repacking blob %v, len %v", id.Str(), blob.PlaintextLength())
buf := make([]byte, 0, blob.PlaintextLength())
buf, err = src.LoadBlob(blob.Type, id, buf)
if err != nil {
return err
}
if uint(len(buf)) != blob.PlaintextLength() {
debug.Log("RepackBlobs", "repack blob %v: len(buf) isn't equal to length: %v = %v", id.Str(), len(buf), blob.PlaintextLength())
return errors.New("LoadBlob returned wrong data, len() doesn't match")
}
_, err = dst.SaveAndEncrypt(blob.Type, buf, &id)
if err != nil {
return err
}
return nil
}
// RepackBlobs reads all blobs in blobIDs from src and saves them into new pack
// files in dst. Source and destination repo may be the same.
func RepackBlobs(src, dst *repository.Repository, blobIDs backend.IDSet) (err error) {
for id := range blobIDs {
err = repackBlob(src, dst, id)
if err != nil {
return err
}
}
err = dst.Flush()
if err != nil {
return err
}
return nil
}
src/github.com/restic/restic/checker/repacker_test.go
+127
View File
@@ -0,0 +1,127 @@
package checker_test
import (
"testing"
"github.com/restic/restic/backend"
"github.com/restic/restic/checker"
. "github.com/restic/restic/test"
)
var findPackTests = []struct {
blobIDs backend.IDSet
packIDs backend.IDSet
}{
{
backend.IDSet{
ParseID("534f211b4fc2cf5b362a24e8eba22db5372a75b7e974603ff9263f5a471760f4"): struct{}{},
ParseID("51aa04744b518c6a85b4e7643cfa99d58789c2a6ca2a3fda831fa3032f28535c"): struct{}{},
ParseID("454515bca5f4f60349a527bd814cc2681bc3625716460cc6310771c966d8a3bf"): struct{}{},
ParseID("c01952de4d91da1b1b80bc6e06eaa4ec21523f4853b69dc8231708b9b7ec62d8"): struct{}{},
},
backend.IDSet{
ParseID("19a731a515618ec8b75fc0ff3b887d8feb83aef1001c9899f6702761142ed068"): struct{}{},
ParseID("657f7fb64f6a854fff6fe9279998ee09034901eded4e6db9bcee0e59745bbce6"): struct{}{},
},
},
}
var findBlobTests = []struct {
packIDs backend.IDSet
blobIDs backend.IDSet
}{
{
backend.IDSet{
ParseID("60e0438dcb978ec6860cc1f8c43da648170ee9129af8f650f876bad19f8f788e"): struct{}{},
},
backend.IDSet{
ParseID("356493f0b00a614d36c698591bbb2b1d801932d85328c1f508019550034549fc"): struct{}{},
ParseID("b8a6bcdddef5c0f542b4648b2ef79bc0ed4377d4109755d2fb78aff11e042663"): struct{}{},
ParseID("5714f7274a8aa69b1692916739dc3835d09aac5395946b8ec4f58e563947199a"): struct{}{},
ParseID("b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c"): struct{}{},
ParseID("08d0444e9987fa6e35ce4232b2b71473e1a8f66b2f9664cc44dc57aad3c5a63a"): struct{}{},
},
},
{
backend.IDSet{
ParseID("60e0438dcb978ec6860cc1f8c43da648170ee9129af8f650f876bad19f8f788e"): struct{}{},
ParseID("ff7e12cd66d896b08490e787d1915c641e678d7e6b4a00e60db5d13054f4def4"): struct{}{},
},
backend.IDSet{
ParseID("356493f0b00a614d36c698591bbb2b1d801932d85328c1f508019550034549fc"): struct{}{},
ParseID("b8a6bcdddef5c0f542b4648b2ef79bc0ed4377d4109755d2fb78aff11e042663"): struct{}{},
ParseID("5714f7274a8aa69b1692916739dc3835d09aac5395946b8ec4f58e563947199a"): struct{}{},
ParseID("b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c"): struct{}{},
ParseID("08d0444e9987fa6e35ce4232b2b71473e1a8f66b2f9664cc44dc57aad3c5a63a"): struct{}{},
ParseID("aa79d596dbd4c863e5400deaca869830888fe1ce9f51b4a983f532c77f16a596"): struct{}{},
ParseID("b2396c92781307111accf2ebb1cd62b58134b744d90cb6f153ca456a98dc3e76"): struct{}{},
ParseID("5249af22d3b2acd6da8048ac37b2a87fa346fabde55ed23bb866f7618843c9fe"): struct{}{},
ParseID("f41c2089a9d58a4b0bf39369fa37588e6578c928aea8e90a4490a6315b9905c1"): struct{}{},
},
},
}
func TestRepackerFindPacks(t *testing.T) {
WithTestEnvironment(t, checkerTestData, func(repodir string) {
repo := OpenLocalRepo(t, repodir)
OK(t, repo.LoadIndex())
for _, test := range findPackTests {
packIDs, err := checker.FindPacksForBlobs(repo, test.blobIDs)
OK(t, err)
Equals(t, test.packIDs, packIDs)
}
for _, test := range findBlobTests {
blobs, err := checker.FindBlobsForPacks(repo, test.packIDs)
OK(t, err)
Assert(t, test.blobIDs.Equals(blobs),
"list of blobs for packs %v does not match, expected:\n %v\ngot:\n %v",
test.packIDs, test.blobIDs, blobs)
}
})
}
func TestRepacker(t *testing.T) {
WithTestEnvironment(t, checkerTestData, func(repodir string) {
repo := OpenLocalRepo(t, repodir)
OK(t, repo.LoadIndex())
repo.Backend().Remove(backend.Snapshot, "c2b53c5e6a16db92fbb9aa08bd2794c58b379d8724d661ee30d20898bdfdff22")
unusedBlobs := backend.IDSet{
ParseID("5714f7274a8aa69b1692916739dc3835d09aac5395946b8ec4f58e563947199a"): struct{}{},
ParseID("08d0444e9987fa6e35ce4232b2b71473e1a8f66b2f9664cc44dc57aad3c5a63a"): struct{}{},
ParseID("356493f0b00a614d36c698591bbb2b1d801932d85328c1f508019550034549fc"): struct{}{},
ParseID("b8a6bcdddef5c0f542b4648b2ef79bc0ed4377d4109755d2fb78aff11e042663"): struct{}{},
}
chkr := checker.New(repo)
_, errs := chkr.LoadIndex()
OKs(t, errs)
errs = checkStruct(chkr)
OKs(t, errs)
list := backend.NewIDSet(chkr.UnusedBlobs()...)
if !unusedBlobs.Equals(list) {
t.Fatalf("expected unused blobs:\n %v\ngot:\n %v", unusedBlobs, list)
}
repacker := checker.NewRepacker(repo, unusedBlobs)
OK(t, repacker.Repack())
chkr = checker.New(repo)
_, errs = chkr.LoadIndex()
OKs(t, errs)
OKs(t, checkPacks(chkr))
OKs(t, checkStruct(chkr))
blobs := chkr.UnusedBlobs()
Assert(t, len(blobs) == 0,
"expected zero unused blobs, got %v", blobs)
})
}
src/github.com/restic/restic/checker/testdata/checker-test-repo.tar.gz
Vendored
BIN
View File
Binary file not shown.
src/github.com/restic/restic/checker/testdata/duplicate-packs-in-index-test-repo.tar.gz
Vendored
BIN
View File
Binary file not shown.
src/github.com/restic/restic/cmd/restic/.gitignore
+1
View File
@@ -0,0 +1 @@
config.mk
src/github.com/restic/restic/cmd/restic/cleanup.go
+65
View File
@@ -0,0 +1,65 @@
package main
import (
"fmt"
"os"
"os/signal"
"sync"
"syscall"
"github.com/restic/restic/debug"
)
var cleanupHandlers struct {
sync.Mutex
list []func() error
done bool
}
var stderr = os.Stderr
func init() {
c := make(chan os.Signal)
signal.Notify(c, syscall.SIGINT)
go CleanupHandler(c)
}
// AddCleanupHandler adds the function f to the list of cleanup handlers so
// that it is executed when all the cleanup handlers are run, e.g. when SIGINT
// is received.
func AddCleanupHandler(f func() error) {
cleanupHandlers.Lock()
defer cleanupHandlers.Unlock()
cleanupHandlers.list = append(cleanupHandlers.list, f)
}
// RunCleanupHandlers runs all registered cleanup handlers
func RunCleanupHandlers() {
cleanupHandlers.Lock()
defer cleanupHandlers.Unlock()
if cleanupHandlers.done {
return
}
cleanupHandlers.done = true
for _, f := range cleanupHandlers.list {
err := f()
if err != nil {
fmt.Fprintf(stderr, "error in cleanup handler: %v\n", err)
}
}
}
// CleanupHandler handles the SIGINT signal.
func CleanupHandler(c <-chan os.Signal) {
for s := range c {
debug.Log("CleanupHandler", "signal %v received, cleaning up", s)
fmt.Println("\x1b[2KInterrupt received, cleaning up")
RunCleanupHandlers()
fmt.Println("exiting")
os.Exit(0)
}
}
src/github.com/restic/restic/cmd/restic/cmd_backup.go
+338
View File
@@ -0,0 +1,338 @@
package main
import (
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"time"
"github.com/restic/restic"
"github.com/restic/restic/backend"
"github.com/restic/restic/debug"
"github.com/restic/restic/filter"
"github.com/restic/restic/repository"
"golang.org/x/crypto/ssh/terminal"
)
type CmdBackup struct {
Parent string `short:"p" long:"parent" description:"use this parent snapshot (default: last snapshot in repo that has the same target)"`
Force bool `short:"f" long:"force" description:"Force re-reading the target. Overrides the \"parent\" flag"`
Excludes []string `short:"e" long:"exclude" description:"Exclude a pattern (can be specified multiple times)"`
global *GlobalOptions
}
func init() {
_, err := parser.AddCommand("backup",
"save file/directory",
"The backup command creates a snapshot of a file or directory",
&CmdBackup{global: &globalOpts})
if err != nil {
panic(err)
}
}
func formatBytes(c uint64) string {
b := float64(c)
switch {
case c > 1<<40:
return fmt.Sprintf("%.3f TiB", b/(1<<40))
case c > 1<<30:
return fmt.Sprintf("%.3f GiB", b/(1<<30))
case c > 1<<20:
return fmt.Sprintf("%.3f MiB", b/(1<<20))
case c > 1<<10:
return fmt.Sprintf("%.3f KiB", b/(1<<10))
default:
return fmt.Sprintf("%dB", c)
}
}
func formatSeconds(sec uint64) string {
hours := sec / 3600
sec -= hours * 3600
min := sec / 60
sec -= min * 60
if hours > 0 {
return fmt.Sprintf("%d:%02d:%02d", hours, min, sec)
}
return fmt.Sprintf("%d:%02d", min, sec)
}
func formatPercent(numerator uint64, denominator uint64) string {
if denominator == 0 {
return ""
}
percent := 100.0 * float64(numerator) / float64(denominator)
if percent > 100 {
percent = 100
}
return fmt.Sprintf("%3.2f%%", percent)
}
func formatRate(bytes uint64, duration time.Duration) string {
sec := float64(duration) / float64(time.Second)
rate := float64(bytes) / sec / (1 << 20)
return fmt.Sprintf("%.2fMiB/s", rate)
}
func formatDuration(d time.Duration) string {
sec := uint64(d / time.Second)
return formatSeconds(sec)
}
func printTree2(indent int, t *restic.Tree) {
for _, node := range t.Nodes {
if node.Tree() != nil {
fmt.Printf("%s%s/\n", strings.Repeat(" ", indent), node.Name)
printTree2(indent+1, node.Tree())
} else {
fmt.Printf("%s%s\n", strings.Repeat(" ", indent), node.Name)
}
}
}
func (cmd CmdBackup) Usage() string {
return "DIR/FILE [DIR/FILE] [...]"
}
func (cmd CmdBackup) newScanProgress() *restic.Progress {
if !cmd.global.ShowProgress() {
return nil
}
p := restic.NewProgress(time.Second)
p.OnUpdate = func(s restic.Stat, d time.Duration, ticker bool) {
fmt.Printf("\x1b[2K[%s] %d directories, %d files, %s\r", formatDuration(d), s.Dirs, s.Files, formatBytes(s.Bytes))
}
p.OnDone = func(s restic.Stat, d time.Duration, ticker bool) {
fmt.Printf("\x1b[2Kscanned %d directories, %d files in %s\n", s.Dirs, s.Files, formatDuration(d))
}
return p
}
func (cmd CmdBackup) newArchiveProgress(todo restic.Stat) *restic.Progress {
if !cmd.global.ShowProgress() {
return nil
}
archiveProgress := restic.NewProgress(time.Second)
var bps, eta uint64
itemsTodo := todo.Files + todo.Dirs
archiveProgress.OnUpdate = func(s restic.Stat, d time.Duration, ticker bool) {
sec := uint64(d / time.Second)
if todo.Bytes > 0 && sec > 0 && ticker {
bps = s.Bytes / sec
if s.Bytes >= todo.Bytes {
eta = 0
} else if bps > 0 {
eta = (todo.Bytes - s.Bytes) / bps
}
}
itemsDone := s.Files + s.Dirs
status1 := fmt.Sprintf("[%s] %s %s/s %s / %s %d / %d items %d errors ",
formatDuration(d),
formatPercent(s.Bytes, todo.Bytes),
formatBytes(bps),
formatBytes(s.Bytes), formatBytes(todo.Bytes),
itemsDone, itemsTodo,
s.Errors)
status2 := fmt.Sprintf("ETA %s ", formatSeconds(eta))
w, _, err := terminal.GetSize(int(os.Stdout.Fd()))
if err == nil {
maxlen := w - len(status2)
if maxlen < 4 {
status1 = ""
} else if len(status1) > maxlen {
status1 = status1[:maxlen-4]
status1 += "... "
}
}
fmt.Printf("\x1b[2K%s%s\r", status1, status2)
}
archiveProgress.OnDone = func(s restic.Stat, d time.Duration, ticker bool) {
fmt.Printf("\nduration: %s, %s\n", formatDuration(d), formatRate(todo.Bytes, d))
}
return archiveProgress
}
func samePaths(expected, actual []string) bool {
if expected == nil || actual == nil {
return true
}
if len(expected) != len(actual) {
return false
}
for i := range expected {
if expected[i] != actual[i] {
return false
}
}
return true
}
var errNoSnapshotFound = errors.New("no snapshot found")
func findLatestSnapshot(repo *repository.Repository, targets []string) (backend.ID, error) {
var (
latest time.Time
latestID backend.ID
found bool
)
for snapshotID := range repo.List(backend.Snapshot, make(chan struct{})) {
snapshot, err := restic.LoadSnapshot(repo, snapshotID)
if err != nil {
return backend.ID{}, fmt.Errorf("Error listing snapshot: %v", err)
}
if snapshot.Time.After(latest) && samePaths(snapshot.Paths, targets) {
latest = snapshot.Time
latestID = snapshotID
found = true
}
}
if !found {
return backend.ID{}, errNoSnapshotFound
}
return latestID, nil
}
// filterExisting returns a slice of all existing items, or an error if no
// items exist at all.
func filterExisting(items []string) (result []string, err error) {
for _, item := range items {
_, err := os.Lstat(item)
if err != nil && os.IsNotExist(err) {
continue
}
result = append(result, item)
}
if len(result) == 0 {
return nil, errors.New("all target directories/files do not exist")
}
return
}
func (cmd CmdBackup) Execute(args []string) error {
if len(args) == 0 {
return fmt.Errorf("wrong number of parameters, Usage: %s", cmd.Usage())
}
target := make([]string, 0, len(args))
for _, d := range args {
if a, err := filepath.Abs(d); err == nil {
d = a
}
target = append(target, d)
}
target, err := filterExisting(target)
if err != nil {
return err
}
repo, err := cmd.global.OpenRepository()
if err != nil {
return err
}
lock, err := lockRepo(repo)
defer unlockRepo(lock)
if err != nil {
return err
}
err = repo.LoadIndex()
if err != nil {
return err
}
var parentSnapshotID *backend.ID
// Force using a parent
if !cmd.Force && cmd.Parent != "" {
id, err := restic.FindSnapshot(repo, cmd.Parent)
if err != nil {
return fmt.Errorf("invalid id %q: %v", cmd.Parent, err)
}
parentSnapshotID = &id
}
// Find last snapshot to set it as parent, if not already set
if !cmd.Force && parentSnapshotID == nil {
id, err := findLatestSnapshot(repo, target)
if err == nil {
parentSnapshotID = &id
} else if err != errNoSnapshotFound {
return err
}
}
if parentSnapshotID != nil {
cmd.global.Verbosef("using parent snapshot %v\n", parentSnapshotID.Str())
}
cmd.global.Verbosef("scan %v\n", target)
selectFilter := func(item string, fi os.FileInfo) bool {
matched, err := filter.List(cmd.Excludes, item)
if err != nil {
cmd.global.Warnf("error for exclude pattern: %v", err)
}
if matched {
debug.Log("backup.Execute", "path %q excluded by a filter", item)
}
return !matched
}
stat, err := restic.Scan(target, selectFilter, cmd.newScanProgress())
if err != nil {
return err
}
arch := restic.NewArchiver(repo)
arch.Excludes = cmd.Excludes
arch.SelectFilter = selectFilter
arch.Error = func(dir string, fi os.FileInfo, err error) error {
// TODO: make ignoring errors configurable
cmd.global.Warnf("\x1b[2K\rerror for %s: %v\n", dir, err)
return nil
}
_, id, err := arch.Snapshot(cmd.newArchiveProgress(stat), target, parentSnapshotID)
if err != nil {
return err
}
cmd.global.Verbosef("snapshot %s saved\n", id.Str())
return nil
}
src/github.com/restic/restic/cmd/restic/cmd_cache.go
+56
View File
@@ -0,0 +1,56 @@
package main
import (
"fmt"
"github.com/restic/restic"
)
type CmdCache struct {
global *GlobalOptions
}
func init() {
_, err := parser.AddCommand("cache",
"manage cache",
"The cache command creates and manages the local cache",
&CmdCache{global: &globalOpts})
if err != nil {
panic(err)
}
}
func (cmd CmdCache) Usage() string {
return "[update|clear]"
}
func (cmd CmdCache) Execute(args []string) error {
// if len(args) == 0 || len(args) > 2 {
// return fmt.Errorf("wrong number of parameters, Usage: %s", cmd.Usage())
// }
repo, err := cmd.global.OpenRepository()
if err != nil {
return err
}
lock, err := lockRepo(repo)
defer unlockRepo(lock)
if err != nil {
return err
}
cache, err := restic.NewCache(repo, cmd.global.CacheDir)
if err != nil {
return err
}
fmt.Printf("clear cache for old snapshots\n")
err = cache.Clear(repo)
if err != nil {
return err
}
fmt.Printf("done\n")
return nil
}
src/github.com/restic/restic/cmd/restic/cmd_cat.go
+199
View File
@@ -0,0 +1,199 @@
package main
import (
"encoding/json"
"errors"
"fmt"
"os"
"github.com/restic/restic"
"github.com/restic/restic/backend"
"github.com/restic/restic/debug"
"github.com/restic/restic/pack"
"github.com/restic/restic/repository"
)
type CmdCat struct {
global *GlobalOptions
}
func init() {
_, err := parser.AddCommand("cat",
"dump something",
"The cat command dumps data structures or data from a repository",
&CmdCat{global: &globalOpts})
if err != nil {
panic(err)
}
}
func (cmd CmdCat) Usage() string {
return "[pack|blob|tree|snapshot|key|masterkey|config|lock] ID"
}
func (cmd CmdCat) Execute(args []string) error {
if len(args) < 1 || (args[0] != "masterkey" && args[0] != "config" && len(args) != 2) {
return fmt.Errorf("type or ID not specified, Usage: %s", cmd.Usage())
}
repo, err := cmd.global.OpenRepository()
if err != nil {
return err
}
lock, err := lockRepo(repo)
defer unlockRepo(lock)
if err != nil {
return err
}
tpe := args[0]
var id backend.ID
if tpe != "masterkey" && tpe != "config" {
id, err = backend.ParseID(args[1])
if err != nil {
if tpe != "snapshot" {
return err
}
// find snapshot id with prefix
id, err = restic.FindSnapshot(repo, args[1])
if err != nil {
return err
}
}
}
// handle all types that don't need an index
switch tpe {
case "config":
buf, err := json.MarshalIndent(repo.Config, "", " ")
if err != nil {
return err
}
fmt.Println(string(buf))
return nil
case "index":
buf, err := repo.LoadAndDecrypt(backend.Index, id)
if err != nil {
return err
}
_, err = os.Stdout.Write(append(buf, '\n'))
return err
case "snapshot":
sn := &restic.Snapshot{}
err = repo.LoadJSONUnpacked(backend.Snapshot, id, sn)
if err != nil {
return err
}
buf, err := json.MarshalIndent(&sn, "", " ")
if err != nil {
return err
}
fmt.Println(string(buf))
return nil
case "key":
h := backend.Handle{Type: backend.Key, Name: id.String()}
buf, err := backend.LoadAll(repo.Backend(), h, nil)
if err != nil {
return err
}
key := &repository.Key{}
err = json.Unmarshal(buf, key)
if err != nil {
return err
}
buf, err = json.MarshalIndent(&key, "", " ")
if err != nil {
return err
}
fmt.Println(string(buf))
return nil
case "masterkey":
buf, err := json.MarshalIndent(repo.Key(), "", " ")
if err != nil {
return err
}
fmt.Println(string(buf))
return nil
case "lock":
lock, err := restic.LoadLock(repo, id)
if err != nil {
return err
}
buf, err := json.MarshalIndent(&lock, "", " ")
if err != nil {
return err
}
fmt.Println(string(buf))
return nil
}
// load index, handle all the other types
err = repo.LoadIndex()
if err != nil {
return err
}
switch tpe {
case "pack":
h := backend.Handle{Type: backend.Data, Name: id.String()}
buf, err := backend.LoadAll(repo.Backend(), h, nil)
if err != nil {
return err
}
_, err = os.Stdout.Write(buf)
return err
case "blob":
blob, err := repo.Index().Lookup(id)
if err != nil {
return err
}
buf := make([]byte, blob.Length)
data, err := repo.LoadBlob(blob.Type, id, buf)
if err != nil {
return err
}
_, err = os.Stdout.Write(data)
return err
case "tree":
debug.Log("cat", "cat tree %v", id.Str())
tree := restic.NewTree()
err = repo.LoadJSONPack(pack.Tree, id, tree)
if err != nil {
debug.Log("cat", "unable to load tree %v: %v", id.Str(), err)
return err
}
buf, err := json.MarshalIndent(&tree, "", " ")
if err != nil {
debug.Log("cat", "error json.MarshalIndent(): %v", err)
return err
}
_, err = os.Stdout.Write(append(buf, '\n'))
return nil
default:
return errors.New("invalid type")
}
}
src/github.com/restic/restic/cmd/restic/cmd_check.go
+165
View File
@@ -0,0 +1,165 @@
package main
import (
"errors"
"fmt"
"os"
"time"
"golang.org/x/crypto/ssh/terminal"
"github.com/restic/restic"
"github.com/restic/restic/checker"
)
type CmdCheck struct {
ReadData bool `long:"read-data" default:"false" description:"Read data blobs"`
CheckUnused bool `long:"check-unused" default:"false" description:"Check for unused blobs"`
global *GlobalOptions
}
func init() {
_, err := parser.AddCommand("check",
"check the repository",
"The check command check the integrity and consistency of the repository",
&CmdCheck{global: &globalOpts})
if err != nil {
panic(err)
}
}
func (cmd CmdCheck) Usage() string {
return "[check-options]"
}
func (cmd CmdCheck) newReadProgress(todo restic.Stat) *restic.Progress {
if !cmd.global.ShowProgress() {
return nil
}
readProgress := restic.NewProgress(time.Second)
readProgress.OnUpdate = func(s restic.Stat, d time.Duration, ticker bool) {
status := fmt.Sprintf("[%s] %s %d / %d items",
formatDuration(d),
formatPercent(s.Blobs, todo.Blobs),
s.Blobs, todo.Blobs)
w, _, err := terminal.GetSize(int(os.Stdout.Fd()))
if err == nil {
if len(status) > w {
max := w - len(status) - 4
status = status[:max] + "... "
}
}
fmt.Printf("\x1b[2K%s\r", status)
}
readProgress.OnDone = func(s restic.Stat, d time.Duration, ticker bool) {
fmt.Printf("\nduration: %s\n", formatDuration(d))
}
return readProgress
}
func (cmd CmdCheck) Execute(args []string) error {
if len(args) != 0 {
return errors.New("check has no arguments")
}
repo, err := cmd.global.OpenRepository()
if err != nil {
return err
}
if !cmd.global.NoLock {
cmd.global.Verbosef("Create exclusive lock for repository\n")
lock, err := lockRepoExclusive(repo)
defer unlockRepo(lock)
if err != nil {
return err
}
}
chkr := checker.New(repo)
cmd.global.Verbosef("Load indexes\n")
hints, errs := chkr.LoadIndex()
dupFound := false
for _, hint := range hints {
cmd.global.Printf("%v\n", hint)
if _, ok := hint.(checker.ErrDuplicatePacks); ok {
dupFound = true
}
}
if dupFound {
cmd.global.Printf("\nrun `restic rebuild-index' to correct this\n")
}
if len(errs) > 0 {
for _, err := range errs {
cmd.global.Warnf("error: %v\n", err)
}
return fmt.Errorf("LoadIndex returned errors")
}
done := make(chan struct{})
defer close(done)
errorsFound := false
errChan := make(chan error)
cmd.global.Verbosef("Check all packs\n")
go chkr.Packs(errChan, done)
for err := range errChan {
errorsFound = true
fmt.Fprintf(os.Stderr, "%v\n", err)
}
cmd.global.Verbosef("Check snapshots, trees and blobs\n")
errChan = make(chan error)
go chkr.Structure(errChan, done)
for err := range errChan {
errorsFound = true
if e, ok := err.(checker.TreeError); ok {
fmt.Fprintf(os.Stderr, "error for tree %v:\n", e.ID.Str())
for _, treeErr := range e.Errors {
fmt.Fprintf(os.Stderr, " %v\n", treeErr)
}
} else {
fmt.Fprintf(os.Stderr, "error: %v\n", err)
}
}
if cmd.CheckUnused {
for _, id := range chkr.UnusedBlobs() {
cmd.global.Verbosef("unused blob %v\n", id.Str())
errorsFound = true
}
}
if cmd.ReadData {
cmd.global.Verbosef("Read all data\n")
p := cmd.newReadProgress(restic.Stat{Blobs: chkr.CountPacks()})
errChan := make(chan error)
go chkr.ReadData(p, errChan, done)
for err := range errChan {
errorsFound = true
fmt.Fprintf(os.Stderr, "%v\n", err)
}
}
if errorsFound {
return errors.New("repository contains errors")
}
return nil
}
src/github.com/restic/restic/cmd/restic/cmd_dump.go
+177
View File
@@ -0,0 +1,177 @@
// +build debug
package main
import (
"encoding/json"
"fmt"
"io"
"os"
"github.com/juju/errors"
"github.com/restic/restic"
"github.com/restic/restic/backend"
"github.com/restic/restic/pack"
"github.com/restic/restic/repository"
)
type CmdDump struct {
global *GlobalOptions
repo *repository.Repository
}
func init() {
_, err := parser.AddCommand("dump",
"dump data structures",
"The dump command dumps data structures from a repository as JSON documents",
&CmdDump{global: &globalOpts})
if err != nil {
panic(err)
}
}
func (cmd CmdDump) Usage() string {
return "[indexes|snapshots|trees|all]"
}
func prettyPrintJSON(wr io.Writer, item interface{}) error {
buf, err := json.MarshalIndent(item, "", " ")
if err != nil {
return err
}
_, err = wr.Write(append(buf, '\n'))
return err
}
func printSnapshots(repo *repository.Repository, wr io.Writer) error {
done := make(chan struct{})
defer close(done)
for id := range repo.List(backend.Snapshot, done) {
snapshot, err := restic.LoadSnapshot(repo, id)
if err != nil {
fmt.Fprintf(os.Stderr, "LoadSnapshot(%v): %v", id.Str(), err)
continue
}
fmt.Fprintf(wr, "snapshot_id: %v\n", id)
err = prettyPrintJSON(wr, snapshot)
if err != nil {
return err
}
}
return nil
}
func printTrees(repo *repository.Repository, wr io.Writer) error {
done := make(chan struct{})
defer close(done)
trees := []backend.ID{}
for _, idx := range repo.Index().All() {
for blob := range idx.Each(nil) {
if blob.Type != pack.Tree {
continue
}
trees = append(trees, blob.ID)
}
}
for _, id := range trees {
tree, err := restic.LoadTree(repo, id)
if err != nil {
fmt.Fprintf(os.Stderr, "LoadTree(%v): %v", id.Str(), err)
continue
}
fmt.Fprintf(wr, "tree_id: %v\n", id)
prettyPrintJSON(wr, tree)
}
return nil
}
func (cmd CmdDump) DumpIndexes() error {
done := make(chan struct{})
defer close(done)
for id := range cmd.repo.List(backend.Index, done) {
fmt.Printf("index_id: %v\n", id)
idx, err := repository.LoadIndex(cmd.repo, id.String())
if err != nil {
return err
}
err = idx.Dump(os.Stdout)
if err != nil {
return err
}
}
return nil
}
func (cmd CmdDump) Execute(args []string) error {
if len(args) != 1 {
return fmt.Errorf("type not specified, Usage: %s", cmd.Usage())
}
repo, err := cmd.global.OpenRepository()
if err != nil {
return err
}
cmd.repo = repo
lock, err := lockRepo(repo)
defer unlockRepo(lock)
if err != nil {
return err
}
err = repo.LoadIndex()
if err != nil {
return err
}
tpe := args[0]
switch tpe {
case "indexes":
return cmd.DumpIndexes()
case "snapshots":
return printSnapshots(repo, os.Stdout)
case "trees":
return printTrees(repo, os.Stdout)
case "all":
fmt.Printf("snapshots:\n")
err := printSnapshots(repo, os.Stdout)
if err != nil {
return err
}
fmt.Printf("\ntrees:\n")
err = printTrees(repo, os.Stdout)
if err != nil {
return err
}
fmt.Printf("\nindexes:\n")
err = cmd.DumpIndexes()
if err != nil {
return err
}
return nil
default:
return errors.Errorf("no such type %q", tpe)
}
}
src/github.com/restic/restic/cmd/restic/cmd_find.go
+197
View File
@@ -0,0 +1,197 @@
package main
import (
"fmt"
"path/filepath"
"time"
"github.com/restic/restic"
"github.com/restic/restic/backend"
"github.com/restic/restic/debug"
"github.com/restic/restic/repository"
)
type findResult struct {
node *restic.Node
path string
}
type CmdFind struct {
Oldest string `short:"o" long:"oldest" description:"Oldest modification date/time"`
Newest string `short:"n" long:"newest" description:"Newest modification date/time"`
Snapshot string `short:"s" long:"snapshot" description:"Snapshot ID to search in"`
oldest, newest time.Time
pattern string
global *GlobalOptions
}
var timeFormats = []string{
"2006-01-02",
"2006-01-02 15:04",
"2006-01-02 15:04:05",
"2006-01-02 15:04:05 -0700",
"2006-01-02 15:04:05 MST",
"02.01.2006",
"02.01.2006 15:04",
"02.01.2006 15:04:05",
"02.01.2006 15:04:05 -0700",
"02.01.2006 15:04:05 MST",
"Mon Jan 2 15:04:05 -0700 MST 2006",
}
func init() {
_, err := parser.AddCommand("find",
"find a file/directory",
"The find command searches for files or directories in snapshots",
&CmdFind{global: &globalOpts})
if err != nil {
panic(err)
}
}
func parseTime(str string) (time.Time, error) {
for _, fmt := range timeFormats {
if t, err := time.ParseInLocation(fmt, str, time.Local); err == nil {
return t, nil
}
}
return time.Time{}, fmt.Errorf("unable to parse time: %q", str)
}
func (c CmdFind) findInTree(repo *repository.Repository, id backend.ID, path string) ([]findResult, error) {
debug.Log("restic.find", "checking tree %v\n", id)
tree, err := restic.LoadTree(repo, id)
if err != nil {
return nil, err
}
results := []findResult{}
for _, node := range tree.Nodes {
debug.Log("restic.find", " testing entry %q\n", node.Name)
m, err := filepath.Match(c.pattern, node.Name)
if err != nil {
return nil, err
}
if m {
debug.Log("restic.find", " pattern matches\n")
if !c.oldest.IsZero() && node.ModTime.Before(c.oldest) {
debug.Log("restic.find", " ModTime is older than %s\n", c.oldest)
continue
}
if !c.newest.IsZero() && node.ModTime.After(c.newest) {
debug.Log("restic.find", " ModTime is newer than %s\n", c.newest)
continue
}
results = append(results, findResult{node: node, path: path})
} else {
debug.Log("restic.find", " pattern does not match\n")
}
if node.Type == "dir" {
subdirResults, err := c.findInTree(repo, *node.Subtree, filepath.Join(path, node.Name))
if err != nil {
return nil, err
}
results = append(results, subdirResults...)
}
}
return results, nil
}
func (c CmdFind) findInSnapshot(repo *repository.Repository, id backend.ID) error {
debug.Log("restic.find", "searching in snapshot %s\n for entries within [%s %s]", id.Str(), c.oldest, c.newest)
sn, err := restic.LoadSnapshot(repo, id)
if err != nil {
return err
}
results, err := c.findInTree(repo, *sn.Tree, "")
if err != nil {
return err
}
if len(results) == 0 {
return nil
}
c.global.Verbosef("found %d matching entries in snapshot %s\n", len(results), id)
for _, res := range results {
res.node.Name = filepath.Join(res.path, res.node.Name)
c.global.Printf(" %s\n", res.node)
}
return nil
}
func (CmdFind) Usage() string {
return "[find-OPTIONS] PATTERN"
}
func (c CmdFind) Execute(args []string) error {
if len(args) != 1 {
return fmt.Errorf("wrong number of arguments, Usage: %s", c.Usage())
}
var err error
if c.Oldest != "" {
c.oldest, err = parseTime(c.Oldest)
if err != nil {
return err
}
}
if c.Newest != "" {
c.newest, err = parseTime(c.Newest)
if err != nil {
return err
}
}
repo, err := c.global.OpenRepository()
if err != nil {
return err
}
lock, err := lockRepo(repo)
defer unlockRepo(lock)
if err != nil {
return err
}
err = repo.LoadIndex()
if err != nil {
return err
}
c.pattern = args[0]
if c.Snapshot != "" {
snapshotID, err := restic.FindSnapshot(repo, c.Snapshot)
if err != nil {
return fmt.Errorf("invalid id %q: %v", args[1], err)
}
return c.findInSnapshot(repo, snapshotID)
}
done := make(chan struct{})
defer close(done)
for snapshotID := range repo.List(backend.Snapshot, done) {
err := c.findInSnapshot(repo, snapshotID)
if err != nil {
return err
}
}
return nil
}
src/github.com/restic/restic/cmd/restic/cmd_init.go
+52
View File
@@ -0,0 +1,52 @@
package main
import (
"errors"
"github.com/restic/restic/repository"
)
type CmdInit struct {
global *GlobalOptions
}
func (cmd CmdInit) Execute(args []string) error {
if cmd.global.Repo == "" {
return errors.New("Please specify repository location (-r)")
}
be, err := create(cmd.global.Repo)
if err != nil {
cmd.global.Exitf(1, "creating backend at %s failed: %v\n", cmd.global.Repo, err)
}
if cmd.global.password == "" {
cmd.global.password = cmd.global.ReadPasswordTwice(
"enter password for new backend: ",
"enter password again: ")
}
s := repository.New(be)
err = s.Init(cmd.global.password)
if err != nil {
cmd.global.Exitf(1, "creating key in backend at %s failed: %v\n", cmd.global.Repo, err)
}
cmd.global.Verbosef("created restic backend %v at %s\n", s.Config.ID[:10], cmd.global.Repo)
cmd.global.Verbosef("\n")
cmd.global.Verbosef("Please note that knowledge of your password is required to access\n")
cmd.global.Verbosef("the repository. Losing your password means that your data is\n")
cmd.global.Verbosef("irrecoverably lost.\n")
return nil
}
func init() {
_, err := parser.AddCommand("init",
"create repository",
"The init command creates a new repository",
&CmdInit{global: &globalOpts})
if err != nil {
panic(err)
}
}
src/github.com/restic/restic/cmd/restic/cmd_key.go
+165
View File
@@ -0,0 +1,165 @@
package main
import (
"errors"
"fmt"
"github.com/restic/restic/backend"
"github.com/restic/restic/repository"
)
type CmdKey struct {
global *GlobalOptions
newPassword string
}
func init() {
_, err := parser.AddCommand("key",
"manage keys",
"The key command manages keys (passwords) of a repository",
&CmdKey{global: &globalOpts})
if err != nil {
panic(err)
}
}
func (cmd CmdKey) listKeys(s *repository.Repository) error {
tab := NewTable()
tab.Header = fmt.Sprintf(" %-10s %-10s %-10s %s", "ID", "User", "Host", "Created")
tab.RowFormat = "%s%-10s %-10s %-10s %s"
plen, err := s.PrefixLength(backend.Key)
if err != nil {
return err
}
done := make(chan struct{})
defer close(done)
for id := range s.List(backend.Key, done) {
k, err := repository.LoadKey(s, id.String())
if err != nil {
cmd.global.Warnf("LoadKey() failed: %v\n", err)
continue
}
var current string
if id.String() == s.KeyName() {
current = "*"
} else {
current = " "
}
tab.Rows = append(tab.Rows, []interface{}{current, id.String()[:plen],
k.Username, k.Hostname, k.Created.Format(TimeFormat)})
}
return tab.Write(cmd.global.stdout)
}
func (cmd CmdKey) getNewPassword() string {
if cmd.newPassword != "" {
return cmd.newPassword
}
return cmd.global.ReadPasswordTwice(
"enter password for new key: ",
"enter password again: ")
}
func (cmd CmdKey) addKey(repo *repository.Repository) error {
id, err := repository.AddKey(repo, cmd.getNewPassword(), repo.Key())
if err != nil {
return fmt.Errorf("creating new key failed: %v\n", err)
}
cmd.global.Verbosef("saved new key as %s\n", id)
return nil
}
func (cmd CmdKey) deleteKey(repo *repository.Repository, name string) error {
if name == repo.KeyName() {
return errors.New("refusing to remove key currently used to access repository")
}
err := repo.Backend().Remove(backend.Key, name)
if err != nil {
return err
}
cmd.global.Verbosef("removed key %v\n", name)
return nil
}
func (cmd CmdKey) changePassword(repo *repository.Repository) error {
id, err := repository.AddKey(repo, cmd.getNewPassword(), repo.Key())
if err != nil {
return fmt.Errorf("creating new key failed: %v\n", err)
}
err = repo.Backend().Remove(backend.Key, repo.KeyName())
if err != nil {
return err
}
cmd.global.Verbosef("saved new key as %s\n", id)
return nil
}
func (cmd CmdKey) Usage() string {
return "[list|add|rm|passwd] [ID]"
}
func (cmd CmdKey) Execute(args []string) error {
if len(args) < 1 || (args[0] == "rm" && len(args) != 2) {
return fmt.Errorf("wrong number of arguments, Usage: %s", cmd.Usage())
}
repo, err := cmd.global.OpenRepository()
if err != nil {
return err
}
switch args[0] {
case "list":
lock, err := lockRepo(repo)
defer unlockRepo(lock)
if err != nil {
return err
}
return cmd.listKeys(repo)
case "add":
lock, err := lockRepo(repo)
defer unlockRepo(lock)
if err != nil {
return err
}
return cmd.addKey(repo)
case "rm":
lock, err := lockRepoExclusive(repo)
defer unlockRepo(lock)
if err != nil {
return err
}
id, err := backend.Find(repo.Backend(), backend.Key, args[1])
if err != nil {
return err
}
return cmd.deleteKey(repo, id)
case "passwd":
lock, err := lockRepoExclusive(repo)
defer unlockRepo(lock)
if err != nil {
return err
}
return cmd.changePassword(repo)
}
return nil
}
src/github.com/restic/restic/cmd/restic/cmd_list.go
+80
View File
@@ -0,0 +1,80 @@
package main
import (
"errors"
"fmt"
"github.com/restic/restic/backend"
)
type CmdList struct {
global *GlobalOptions
}
func init() {
_, err := parser.AddCommand("list",
"lists data",
"The list command lists structures or data of a repository",
&CmdList{global: &globalOpts})
if err != nil {
panic(err)
}
}
func (cmd CmdList) Usage() string {
return "[blobs|packs|index|snapshots|keys|locks]"
}
func (cmd CmdList) Execute(args []string) error {
if len(args) != 1 {
return fmt.Errorf("type not specified, Usage: %s", cmd.Usage())
}
repo, err := cmd.global.OpenRepository()
if err != nil {
return err
}
if !cmd.global.NoLock {
lock, err := lockRepo(repo)
defer unlockRepo(lock)
if err != nil {
return err
}
}
var t backend.Type
switch args[0] {
case "blobs":
err = repo.LoadIndex()
if err != nil {
return err
}
for _, idx := range repo.Index().All() {
for blob := range idx.Each(nil) {
cmd.global.Printf("%s\n", blob.ID)
}
}
return nil
case "packs":
t = backend.Data
case "index":
t = backend.Index
case "snapshots":
t = backend.Snapshot
case "keys":
t = backend.Key
case "locks":
t = backend.Lock
default:
return errors.New("invalid type")
}
for id := range repo.List(t, nil) {
cmd.global.Printf("%s\n", id)
}
return nil
}
src/github.com/restic/restic/cmd/restic/cmd_ls.go
+101
View File
@@ -0,0 +1,101 @@
package main
import (
"fmt"
"os"
"path/filepath"
"github.com/restic/restic"
"github.com/restic/restic/backend"
"github.com/restic/restic/repository"
)
type CmdLs struct {
Long bool `short:"l" long:"long" description:"Use a long listing format showing size and mode"`
global *GlobalOptions
}
func init() {
_, err := parser.AddCommand("ls",
"list files",
"The ls command lists all files and directories in a snapshot",
&CmdLs{global: &globalOpts})
if err != nil {
panic(err)
}
}
func (cmd CmdLs) printNode(prefix string, n *restic.Node) string {
if !cmd.Long {
return filepath.Join(prefix, n.Name)
}
switch n.Type {
case "file":
return fmt.Sprintf("%s %5d %5d %6d %s %s",
n.Mode, n.UID, n.GID, n.Size, n.ModTime, filepath.Join(prefix, n.Name))
case "dir":
return fmt.Sprintf("%s %5d %5d %6d %s %s",
n.Mode|os.ModeDir, n.UID, n.GID, n.Size, n.ModTime, filepath.Join(prefix, n.Name))
case "symlink":
return fmt.Sprintf("%s %5d %5d %6d %s %s -> %s",
n.Mode|os.ModeSymlink, n.UID, n.GID, n.Size, n.ModTime, filepath.Join(prefix, n.Name), n.LinkTarget)
default:
return fmt.Sprintf("<Node(%s) %s>", n.Type, n.Name)
}
}
func (cmd CmdLs) printTree(prefix string, repo *repository.Repository, id backend.ID) error {
tree, err := restic.LoadTree(repo, id)
if err != nil {
return err
}
for _, entry := range tree.Nodes {
cmd.global.Printf(cmd.printNode(prefix, entry) + "\n")
if entry.Type == "dir" && entry.Subtree != nil {
err = cmd.printTree(filepath.Join(prefix, entry.Name), repo, *entry.Subtree)
if err != nil {
return err
}
}
}
return nil
}
func (cmd CmdLs) Usage() string {
return "snapshot-ID [DIR]"
}
func (cmd CmdLs) Execute(args []string) error {
if len(args) < 1 || len(args) > 2 {
return fmt.Errorf("wrong number of arguments, Usage: %s", cmd.Usage())
}
repo, err := cmd.global.OpenRepository()
if err != nil {
return err
}
err = repo.LoadIndex()
if err != nil {
return err
}
id, err := restic.FindSnapshot(repo, args[0])
if err != nil {
return err
}
sn, err := restic.LoadSnapshot(repo, id)
if err != nil {
return err
}
cmd.global.Verbosef("snapshot of %v at %s:\n", sn.Paths, sn.Time)
return cmd.printTree("", repo, *sn.Tree)
}
src/github.com/restic/restic/cmd/restic/cmd_mount.go
+107
View File
@@ -0,0 +1,107 @@
// +build !openbsd
// +build !windows
package main
import (
"fmt"
"os"
"github.com/restic/restic/fuse"
systemFuse "bazil.org/fuse"
"bazil.org/fuse/fs"
)
type CmdMount struct {
Root bool `long:"owner-root" description:"use 'root' as the owner of files and dirs" default:"false"`
global *GlobalOptions
ready chan struct{}
done chan struct{}
}
func init() {
_, err := parser.AddCommand("mount",
"mount a repository",
"The mount command mounts a repository read-only to a given directory",
&CmdMount{
global: &globalOpts,
ready: make(chan struct{}, 1),
done: make(chan struct{}),
})
if err != nil {
panic(err)
}
}
func (cmd CmdMount) Usage() string {
return "MOUNTPOINT"
}
func (cmd CmdMount) Execute(args []string) error {
if len(args) == 0 {
return fmt.Errorf("wrong number of parameters, Usage: %s", cmd.Usage())
}
repo, err := cmd.global.OpenRepository()
if err != nil {
return err
}
err = repo.LoadIndex()
if err != nil {
return err
}
mountpoint := args[0]
if _, err := os.Stat(mountpoint); os.IsNotExist(err) {
cmd.global.Verbosef("Mountpoint %s doesn't exist, creating it\n", mountpoint)
err = os.Mkdir(mountpoint, os.ModeDir|0700)
if err != nil {
return err
}
}
c, err := systemFuse.Mount(
mountpoint,
systemFuse.ReadOnly(),
systemFuse.FSName("restic"),
)
if err != nil {
return err
}
root := fs.Tree{}
root.Add("snapshots", fuse.NewSnapshotsDir(repo, cmd.Root))
cmd.global.Printf("Now serving %s at %s\n", repo.Backend().Location(), mountpoint)
cmd.global.Printf("Don't forget to umount after quitting!\n")
AddCleanupHandler(func() error {
return systemFuse.Unmount(mountpoint)
})
cmd.ready <- struct{}{}
errServe := make(chan error)
go func() {
err = fs.Serve(c, &root)
if err != nil {
errServe <- err
}
<-c.Ready
errServe <- c.MountError
}()
select {
case err := <-errServe:
return err
case <-cmd.done:
err := c.Close()
if err != nil {
cmd.global.Printf("Error closing fuse connection: %s\n", err)
}
return systemFuse.Unmount(mountpoint)
}
}
src/github.com/restic/restic/cmd/restic/cmd_optimize.go
+84
View File
@@ -0,0 +1,84 @@
package main
import (
"errors"
"fmt"
"github.com/restic/restic/backend"
"github.com/restic/restic/checker"
)
type CmdOptimize struct {
global *GlobalOptions
}
func init() {
_, err := parser.AddCommand("optimize",
"optimize the repository",
"The optimize command reorganizes the repository and removes uneeded data",
&CmdOptimize{global: &globalOpts})
if err != nil {
panic(err)
}
}
func (cmd CmdOptimize) Usage() string {
return "[optimize-options]"
}
func (cmd CmdOptimize) Execute(args []string) error {
if len(args) != 0 {
return errors.New("optimize has no arguments")
}
repo, err := cmd.global.OpenRepository()
if err != nil {
return err
}
cmd.global.Verbosef("Create exclusive lock for repository\n")
lock, err := lockRepoExclusive(repo)
defer unlockRepo(lock)
if err != nil {
return err
}
chkr := checker.New(repo)
cmd.global.Verbosef("Load indexes\n")
_, errs := chkr.LoadIndex()
if len(errs) > 0 {
for _, err := range errs {
cmd.global.Warnf("error: %v\n", err)
}
return fmt.Errorf("LoadIndex returned errors")
}
done := make(chan struct{})
errChan := make(chan error)
go chkr.Structure(errChan, done)
for err := range errChan {
if e, ok := err.(checker.TreeError); ok {
cmd.global.Warnf("error for tree %v:\n", e.ID.Str())
for _, treeErr := range e.Errors {
cmd.global.Warnf(" %v\n", treeErr)
}
} else {
cmd.global.Warnf("error: %v\n", err)
}
}
unusedBlobs := backend.NewIDSet(chkr.UnusedBlobs()...)
cmd.global.Verbosef("%d unused blobs found, repacking...\n", len(unusedBlobs))
repacker := checker.NewRepacker(repo, unusedBlobs)
err = repacker.Repack()
if err != nil {
return err
}
cmd.global.Verbosef("repacking done\n")
return nil
}
src/github.com/restic/restic/cmd/restic/cmd_rebuild_index.go
+204
View File
@@ -0,0 +1,204 @@
package main
import (
"bytes"
"fmt"
"github.com/restic/restic/backend"
"github.com/restic/restic/debug"
"github.com/restic/restic/pack"
"github.com/restic/restic/repository"
)
type CmdRebuildIndex struct {
global *GlobalOptions
repo *repository.Repository
}
func init() {
_, err := parser.AddCommand("rebuild-index",
"rebuild the index",
"The rebuild-index command builds a new index",
&CmdRebuildIndex{global: &globalOpts})
if err != nil {
panic(err)
}
}
func (cmd CmdRebuildIndex) storeIndex(index *repository.Index) (*repository.Index, error) {
debug.Log("RebuildIndex.RebuildIndex", "saving index")
cmd.global.Printf(" saving new index\n")
id, err := repository.SaveIndex(cmd.repo, index)
if err != nil {
debug.Log("RebuildIndex.RebuildIndex", "error saving index: %v", err)
return nil, err
}
debug.Log("RebuildIndex.RebuildIndex", "index saved as %v", id.Str())
index = repository.NewIndex()
return index, nil
}
func (cmd CmdRebuildIndex) RebuildIndex() error {
debug.Log("RebuildIndex.RebuildIndex", "start")
done := make(chan struct{})
defer close(done)
indexIDs := backend.NewIDSet()
for id := range cmd.repo.List(backend.Index, done) {
indexIDs.Insert(id)
}
cmd.global.Printf("rebuilding index from %d indexes\n", len(indexIDs))
debug.Log("RebuildIndex.RebuildIndex", "found %v indexes", len(indexIDs))
combinedIndex := repository.NewIndex()
packsDone := backend.NewIDSet()
type Blob struct {
id backend.ID
tpe pack.BlobType
}
blobsDone := make(map[Blob]struct{})
i := 0
for indexID := range indexIDs {
cmd.global.Printf(" loading index %v\n", i)
debug.Log("RebuildIndex.RebuildIndex", "load index %v", indexID.Str())
idx, err := repository.LoadIndex(cmd.repo, indexID.String())
if err != nil {
return err
}
debug.Log("RebuildIndex.RebuildIndex", "adding blobs from index %v", indexID.Str())
for packedBlob := range idx.Each(done) {
packsDone.Insert(packedBlob.PackID)
b := Blob{
id: packedBlob.ID,
tpe: packedBlob.Type,
}
if _, ok := blobsDone[b]; ok {
continue
}
blobsDone[b] = struct{}{}
combinedIndex.Store(packedBlob)
}
combinedIndex.AddToSupersedes(indexID)
if repository.IndexFull(combinedIndex) {
combinedIndex, err = cmd.storeIndex(combinedIndex)
if err != nil {
return err
}
}
i++
}
var err error
if combinedIndex.Length() > 0 {
combinedIndex, err = cmd.storeIndex(combinedIndex)
if err != nil {
return err
}
}
cmd.global.Printf("removing %d old indexes\n", len(indexIDs))
for id := range indexIDs {
debug.Log("RebuildIndex.RebuildIndex", "remove index %v", id.Str())
err := cmd.repo.Backend().Remove(backend.Index, id.String())
if err != nil {
debug.Log("RebuildIndex.RebuildIndex", "error removing index %v: %v", id.Str(), err)
return err
}
}
cmd.global.Printf("checking for additional packs\n")
newPacks := 0
var buf []byte
for packID := range cmd.repo.List(backend.Data, done) {
if packsDone.Has(packID) {
continue
}
debug.Log("RebuildIndex.RebuildIndex", "pack %v not indexed", packID.Str())
newPacks++
var err error
h := backend.Handle{Type: backend.Data, Name: packID.String()}
buf, err = backend.LoadAll(cmd.repo.Backend(), h, buf)
if err != nil {
debug.Log("RebuildIndex.RebuildIndex", "error while loading pack %v", packID.Str())
return fmt.Errorf("error while loading pack %v: %v", packID.Str(), err)
}
hash := backend.Hash(buf)
if !hash.Equal(packID) {
debug.Log("RebuildIndex.RebuildIndex", "Pack ID does not match, want %v, got %v", packID.Str(), hash.Str())
return fmt.Errorf("Pack ID does not match, want %v, got %v", packID.Str(), hash.Str())
}
up, err := pack.NewUnpacker(cmd.repo.Key(), bytes.NewReader(buf))
if err != nil {
debug.Log("RebuildIndex.RebuildIndex", "error while unpacking pack %v", packID.Str())
return err
}
for _, blob := range up.Entries {
debug.Log("RebuildIndex.RebuildIndex", "pack %v: blob %v", packID.Str(), blob)
combinedIndex.Store(repository.PackedBlob{
Type: blob.Type,
ID: blob.ID,
PackID: packID,
Offset: blob.Offset,
Length: blob.Length,
})
}
if repository.IndexFull(combinedIndex) {
combinedIndex, err = cmd.storeIndex(combinedIndex)
if err != nil {
return err
}
}
}
if combinedIndex.Length() > 0 {
combinedIndex, err = cmd.storeIndex(combinedIndex)
if err != nil {
return err
}
}
cmd.global.Printf("added %d packs to the index\n", newPacks)
debug.Log("RebuildIndex.RebuildIndex", "done")
return nil
}
func (cmd CmdRebuildIndex) Execute(args []string) error {
repo, err := cmd.global.OpenRepository()
if err != nil {
return err
}
cmd.repo = repo
lock, err := lockRepoExclusive(repo)
defer unlockRepo(lock)
if err != nil {
return err
}
return cmd.RebuildIndex()
}
src/github.com/restic/restic/cmd/restic/cmd_restore.go
+116
View File
@@ -0,0 +1,116 @@
package main
import (
"errors"
"fmt"
"github.com/restic/restic"
"github.com/restic/restic/debug"
"github.com/restic/restic/filter"
)
type CmdRestore struct {
Exclude []string `short:"e" long:"exclude" description:"Exclude a pattern (can be specified multiple times)"`
Include []string `short:"i" long:"include" description:"Include a pattern, exclude everything else (can be specified multiple times)"`
Target string `short:"t" long:"target" description:"Directory to restore to"`
global *GlobalOptions
}
func init() {
_, err := parser.AddCommand("restore",
"restore a snapshot",
"The restore command restores a snapshot to a directory",
&CmdRestore{global: &globalOpts})
if err != nil {
panic(err)
}
}
func (cmd CmdRestore) Usage() string {
return "snapshot-ID"
}
func (cmd CmdRestore) Execute(args []string) error {
if len(args) != 1 {
return fmt.Errorf("wrong number of arguments, Usage: %s", cmd.Usage())
}
if cmd.Target == "" {
return errors.New("please specify a directory to restore to (--target)")
}
if len(cmd.Exclude) > 0 && len(cmd.Include) > 0 {
return errors.New("exclude and include patterns are mutually exclusive")
}
snapshotIDString := args[0]
debug.Log("restore", "restore %v to %v", snapshotIDString, cmd.Target)
repo, err := cmd.global.OpenRepository()
if err != nil {
return err
}
if !cmd.global.NoLock {
lock, err := lockRepo(repo)
defer unlockRepo(lock)
if err != nil {
return err
}
}
err = repo.LoadIndex()
if err != nil {
return err
}
id, err := restic.FindSnapshot(repo, snapshotIDString)
if err != nil {
cmd.global.Exitf(1, "invalid id %q: %v", snapshotIDString, err)
}
res, err := restic.NewRestorer(repo, id)
if err != nil {
cmd.global.Exitf(2, "creating restorer failed: %v\n", err)
}
res.Error = func(dir string, node *restic.Node, err error) error {
cmd.global.Warnf("error for %s: %+v\n", dir, err)
return nil
}
selectExcludeFilter := func(item string, dstpath string, node *restic.Node) bool {
matched, err := filter.List(cmd.Exclude, item)
if err != nil {
cmd.global.Warnf("error for exclude pattern: %v", err)
}
return !matched
}
selectIncludeFilter := func(item string, dstpath string, node *restic.Node) bool {
matched, err := filter.List(cmd.Include, item)
if err != nil {
cmd.global.Warnf("error for include pattern: %v", err)
}
return matched
}
if len(cmd.Exclude) > 0 {
res.SelectFilter = selectExcludeFilter
} else if len(cmd.Include) > 0 {
res.SelectFilter = selectIncludeFilter
}
cmd.global.Verbosef("restoring %s to %s\n", res.Snapshot(), cmd.Target)
err = res.RestoreTo(cmd.Target)
if err != nil {
return err
}
return nil
}
src/github.com/restic/restic/cmd/restic/cmd_snapshots.go
+134
View File
@@ -0,0 +1,134 @@
package main
import (
"encoding/hex"
"fmt"
"io"
"os"
"sort"
"strings"
"github.com/restic/restic"
"github.com/restic/restic/backend"
)
type Table struct {
Header string
Rows [][]interface{}
RowFormat string
}
func NewTable() Table {
return Table{
Rows: [][]interface{}{},
}
}
func (t Table) Write(w io.Writer) error {
_, err := fmt.Fprintln(w, t.Header)
if err != nil {
return err
}
_, err = fmt.Fprintln(w, strings.Repeat("-", 70))
if err != nil {
return err
}
for _, row := range t.Rows {
_, err = fmt.Fprintf(w, t.RowFormat+"\n", row...)
if err != nil {
return err
}
}
return nil
}
const TimeFormat = "2006-01-02 15:04:05"
type CmdSnapshots struct {
global *GlobalOptions
}
func init() {
_, err := parser.AddCommand("snapshots",
"show snapshots",
"The snapshots command lists all snapshots stored in a repository",
&CmdSnapshots{global: &globalOpts})
if err != nil {
panic(err)
}
}
func (cmd CmdSnapshots) Usage() string {
return ""
}
func (cmd CmdSnapshots) Execute(args []string) error {
if len(args) != 0 {
return fmt.Errorf("wrong number of arguments, usage: %s", cmd.Usage())
}
repo, err := cmd.global.OpenRepository()
if err != nil {
return err
}
lock, err := lockRepo(repo)
defer unlockRepo(lock)
if err != nil {
return err
}
tab := NewTable()
tab.Header = fmt.Sprintf("%-8s %-19s %-10s %s", "ID", "Date", "Source", "Directory")
tab.RowFormat = "%-8s %-19s %-10s %s"
done := make(chan struct{})
defer close(done)
list := []*restic.Snapshot{}
for id := range repo.List(backend.Snapshot, done) {
sn, err := restic.LoadSnapshot(repo, id)
if err != nil {
fmt.Fprintf(os.Stderr, "error loading snapshot %s: %v\n", id, err)
continue
}
pos := sort.Search(len(list), func(i int) bool {
return list[i].Time.After(sn.Time)
})
if pos < len(list) {
list = append(list, nil)
copy(list[pos+1:], list[pos:])
list[pos] = sn
} else {
list = append(list, sn)
}
}
plen, err := repo.PrefixLength(backend.Snapshot)
if err != nil {
return err
}
for _, sn := range list {
if len(sn.Paths) == 0 {
continue
}
id := sn.ID()
tab.Rows = append(tab.Rows, []interface{}{hex.EncodeToString(id[:plen/2]), sn.Time.Format(TimeFormat), sn.Hostname, sn.Paths[0]})
if len(sn.Paths) > 1 {
for _, path := range sn.Paths[1:] {
tab.Rows = append(tab.Rows, []interface{}{"", "", "", path})
}
}
}
tab.Write(os.Stdout)
return nil
}
src/github.com/restic/restic/cmd/restic/cmd_unlock.go
+43
View File
@@ -0,0 +1,43 @@
package main
import "github.com/restic/restic"
type CmdUnlock struct {
RemoveAll bool `long:"remove-all" description:"Remove all locks, even stale ones"`
global *GlobalOptions
}
func init() {
_, err := parser.AddCommand("unlock",
"remove locks",
"The unlock command checks for stale locks and removes them",
&CmdUnlock{global: &globalOpts})
if err != nil {
panic(err)
}
}
func (cmd CmdUnlock) Usage() string {
return "[unlock-options]"
}
func (cmd CmdUnlock) Execute(args []string) error {
repo, err := cmd.global.OpenRepository()
if err != nil {
return err
}
fn := restic.RemoveStaleLocks
if cmd.RemoveAll {
fn = restic.RemoveAllLocks
}
err = fn(repo)
if err != nil {
return err
}
cmd.global.Verbosef("successfully removed locks\n")
return nil
}
src/github.com/restic/restic/cmd/restic/cmd_version.go
+25
View File
@@ -0,0 +1,25 @@
package main
import (
"fmt"
"runtime"
)
type CmdVersion struct{}
func init() {
_, err := parser.AddCommand("version",
"display version",
"The version command displays detailed information about the version",
&CmdVersion{})
if err != nil {
panic(err)
}
}
func (cmd CmdVersion) Execute(args []string) error {
fmt.Printf("restic %s\ncompiled at %s with %v\n",
version, compiledAt, runtime.Version())
return nil
}
src/github.com/restic/restic/cmd/restic/doc.go
+2
View File
@@ -0,0 +1,2 @@
// This package contains the code for the restic executable.
package main
src/github.com/restic/restic/cmd/restic/global.go
+287
View File
@@ -0,0 +1,287 @@
package main
import (
"errors"
"fmt"
"io"
"os"
"strings"
"syscall"
"github.com/jessevdk/go-flags"
"github.com/restic/restic/backend"
"github.com/restic/restic/backend/local"
"github.com/restic/restic/backend/s3"
"github.com/restic/restic/backend/sftp"
"github.com/restic/restic/debug"
"github.com/restic/restic/location"
"github.com/restic/restic/repository"
"golang.org/x/crypto/ssh/terminal"
)
var version = "compiled manually"
var compiledAt = "unknown time"
// GlobalOptions holds all those options that can be set for every command.
type GlobalOptions struct {
Repo string `short:"r" long:"repo" description:"Repository directory to backup to/restore from"`
CacheDir string ` long:"cache-dir" description:"Directory to use as a local cache"`
Quiet bool `short:"q" long:"quiet" default:"false" description:"Do not output comprehensive progress report"`
NoLock bool ` long:"no-lock" default:"false" description:"Do not lock the repo, this allows some operations on read-only repos."`
Options []string `short:"o" long:"option" description:"Specify options in the form 'foo.key=value'"`
password string
stdout io.Writer
stderr io.Writer
}
func init() {
restoreTerminal()
}
// checkErrno returns nil when err is set to syscall.Errno(0), since this is no
// error condition.
func checkErrno(err error) error {
e, ok := err.(syscall.Errno)
if !ok {
return err
}
if e == 0 {
return nil
}
return err
}
// restoreTerminal installs a cleanup handler that restores the previous
// terminal state on exit.
func restoreTerminal() {
fd := int(os.Stdout.Fd())
if !terminal.IsTerminal(fd) {
return
}
state, err := terminal.GetState(fd)
if err != nil {
fmt.Fprintf(os.Stderr, "unable to get terminal state: %v\n", err)
return
}
AddCleanupHandler(func() error {
err := checkErrno(terminal.Restore(fd, state))
if err != nil {
fmt.Fprintf(os.Stderr, "unable to get restore terminal state: %#+v\n", err)
}
return err
})
}
var globalOpts = GlobalOptions{stdout: os.Stdout, stderr: os.Stderr}
var parser = flags.NewParser(&globalOpts, flags.HelpFlag|flags.PassDoubleDash)
// Printf writes the message to the configured stdout stream.
func (o GlobalOptions) Printf(format string, args ...interface{}) {
_, err := fmt.Fprintf(o.stdout, format, args...)
if err != nil {
fmt.Fprintf(os.Stderr, "unable to write to stdout: %v\n", err)
os.Exit(100)
}
}
// Verbosef calls Printf to write the message when the verbose flag is set.
func (o GlobalOptions) Verbosef(format string, args ...interface{}) {
if o.Quiet {
return
}
o.Printf(format, args...)
}
// ShowProgress returns true iff the progress status should be written, i.e.
// the quiet flag is not set and the output is a terminal.
func (o GlobalOptions) ShowProgress() bool {
if o.Quiet {
return false
}
if !terminal.IsTerminal(int(os.Stdout.Fd())) {
return false
}
return true
}
// Warnf writes the message to the configured stderr stream.
func (o GlobalOptions) Warnf(format string, args ...interface{}) {
_, err := fmt.Fprintf(o.stderr, format, args...)
if err != nil {
fmt.Fprintf(os.Stderr, "unable to write to stderr: %v\n", err)
os.Exit(100)
}
}
// Exitf uses Warnf to write the message and then calls os.Exit(exitcode).
func (o GlobalOptions) Exitf(exitcode int, format string, args ...interface{}) {
if format[len(format)-1] != '\n' {
format += "\n"
}
o.Warnf(format, args...)
os.Exit(exitcode)
}
// readPassword reads the password from the given reader directly.
func readPassword(in io.Reader) (password string, err error) {
buf := make([]byte, 1000)
n, err := io.ReadFull(in, buf)
buf = buf[:n]
if err != nil && err != io.ErrUnexpectedEOF {
return "", err
}
return strings.TrimRight(string(buf), "\r\n"), nil
}
// readPasswordTerminal reads the password from the given reader which must be a
// tty. Prompt is printed on the writer out before attempting to read the
// password.
func readPasswordTerminal(in *os.File, out io.Writer, prompt string) (password string, err error) {
fmt.Fprint(out, prompt)
buf, err := terminal.ReadPassword(int(in.Fd()))
fmt.Fprintln(out)
if err != nil {
return "", err
}
password = string(buf)
return password, nil
}
// ReadPassword reads the password from stdin.
func (o GlobalOptions) ReadPassword(prompt string) string {
var (
password string
err error
)
if terminal.IsTerminal(int(os.Stdin.Fd())) {
password, err = readPasswordTerminal(os.Stdin, os.Stderr, prompt)
} else {
password, err = readPassword(os.Stdin)
}
if err != nil {
o.Exitf(2, "unable to read password: %v", err)
}
if len(password) == 0 {
o.Exitf(1, "an empty password is not a password")
}
return password
}
// ReadPasswordTwice calls ReadPassword two times and returns an error when the
// passwords don't match.
func (o GlobalOptions) ReadPasswordTwice(prompt1, prompt2 string) string {
pw1 := o.ReadPassword(prompt1)
pw2 := o.ReadPassword(prompt2)
if pw1 != pw2 {
o.Exitf(1, "passwords do not match")
}
return pw1
}
// OpenRepository reads the password and opens the repository.
func (o GlobalOptions) OpenRepository() (*repository.Repository, error) {
if o.Repo == "" {
return nil, errors.New("Please specify repository location (-r)")
}
be, err := open(o.Repo)
if err != nil {
return nil, err
}
s := repository.New(be)
if o.password == "" {
o.password = o.ReadPassword("enter password for repository: ")
}
err = s.SearchKey(o.password)
if err != nil {
return nil, fmt.Errorf("unable to open repo: %v", err)
}
return s, nil
}
// Open the backend specified by a location config.
func open(s string) (backend.Backend, error) {
debug.Log("open", "parsing location %v", s)
loc, err := location.Parse(s)
if err != nil {
return nil, err
}
switch loc.Scheme {
case "local":
debug.Log("open", "opening local repository at %#v", loc.Config)
return local.Open(loc.Config.(string))
case "sftp":
debug.Log("open", "opening sftp repository at %#v", loc.Config)
return sftp.OpenWithConfig(loc.Config.(sftp.Config))
case "s3":
cfg := loc.Config.(s3.Config)
if cfg.KeyID == "" {
cfg.KeyID = os.Getenv("AWS_ACCESS_KEY_ID")
}
if cfg.Secret == "" {
cfg.Secret = os.Getenv("AWS_SECRET_ACCESS_KEY")
}
debug.Log("open", "opening s3 repository at %#v", cfg)
return s3.Open(cfg)
}
debug.Log("open", "invalid repository location: %v", s)
return nil, fmt.Errorf("invalid scheme %q", loc.Scheme)
}
// Create the backend specified by URI.
func create(s string) (backend.Backend, error) {
debug.Log("open", "parsing location %v", s)
loc, err := location.Parse(s)
if err != nil {
return nil, err
}
switch loc.Scheme {
case "local":
debug.Log("open", "create local repository at %#v", loc.Config)
return local.Create(loc.Config.(string))
case "sftp":
debug.Log("open", "create sftp repository at %#v", loc.Config)
return sftp.CreateWithConfig(loc.Config.(sftp.Config))
case "s3":
cfg := loc.Config.(s3.Config)
if cfg.KeyID == "" {
cfg.KeyID = os.Getenv("AWS_ACCESS_KEY_ID")
}
if cfg.Secret == "" {
cfg.Secret = os.Getenv("AWS_SECRET_ACCESS_KEY")
}
debug.Log("open", "create s3 repository at %#v", loc.Config)
return s3.Open(cfg)
}
debug.Log("open", "invalid repository scheme: %v", s)
return nil, fmt.Errorf("invalid scheme %q", loc.Scheme)
}
src/github.com/restic/restic/cmd/restic/integration_fuse_test.go
+162
View File
@@ -0,0 +1,162 @@
// +build !openbsd
// +build !windows
package main
import (
"fmt"
"io/ioutil"
"os"
"path/filepath"
"testing"
"time"
"github.com/restic/restic"
"github.com/restic/restic/backend"
"github.com/restic/restic/repository"
. "github.com/restic/restic/test"
)
const (
mountWait = 20
mountSleep = 100 * time.Millisecond
mountTestSubdir = "snapshots"
)
// waitForMount blocks (max mountWait * mountSleep) until the subdir
// "snapshots" appears in the dir.
func waitForMount(dir string) error {
for i := 0; i < mountWait; i++ {
f, err := os.Open(dir)
if err != nil {
return err
}
names, err := f.Readdirnames(-1)
if err != nil {
return err
}
if err = f.Close(); err != nil {
return err
}
for _, name := range names {
if name == mountTestSubdir {
return nil
}
}
time.Sleep(mountSleep)
}
return fmt.Errorf("subdir %q of dir %s never appeared", mountTestSubdir, dir)
}
func cmdMount(t testing.TB, global GlobalOptions, dir string, ready, done chan struct{}) {
defer func() {
ready <- struct{}{}
}()
cmd := &CmdMount{global: &global, ready: ready, done: done}
OK(t, cmd.Execute([]string{dir}))
if TestCleanupTempDirs {
RemoveAll(t, dir)
}
}
func TestMount(t *testing.T) {
if !RunFuseTest {
t.Skip("Skipping fuse tests")
}
checkSnapshots := func(repo *repository.Repository, mountpoint string, snapshotIDs []backend.ID) {
snapshotsDir, err := os.Open(filepath.Join(mountpoint, "snapshots"))
OK(t, err)
namesInSnapshots, err := snapshotsDir.Readdirnames(-1)
OK(t, err)
Assert(t,
len(namesInSnapshots) == len(snapshotIDs),
"Invalid number of snapshots: expected %d, got %d", len(snapshotIDs), len(namesInSnapshots))
namesMap := make(map[string]bool)
for _, name := range namesInSnapshots {
namesMap[name] = false
}
for _, id := range snapshotIDs {
snapshot, err := restic.LoadSnapshot(repo, id)
OK(t, err)
_, ok := namesMap[snapshot.Time.Format(time.RFC3339)]
Assert(t, ok, "Snapshot %s isn't present in fuse dir", snapshot.Time.Format(time.RFC3339))
namesMap[snapshot.Time.Format(time.RFC3339)] = true
}
for name, present := range namesMap {
Assert(t, present, "Directory %s is present in fuse dir but is not a snapshot", name)
}
OK(t, snapshotsDir.Close())
}
withTestEnvironment(t, func(env *testEnvironment, global GlobalOptions) {
cmdInit(t, global)
repo, err := global.OpenRepository()
OK(t, err)
mountpoint, err := ioutil.TempDir(TestTempDir, "restic-test-mount-")
OK(t, err)
// We remove the mountpoint now to check that cmdMount creates it
RemoveAll(t, mountpoint)
ready := make(chan struct{}, 2)
done := make(chan struct{})
go cmdMount(t, global, mountpoint, ready, done)
<-ready
defer close(done)
OK(t, waitForMount(mountpoint))
mountpointDir, err := os.Open(mountpoint)
OK(t, err)
names, err := mountpointDir.Readdirnames(-1)
OK(t, err)
Assert(t, len(names) == 1 && names[0] == "snapshots", `The fuse virtual directory "snapshots" doesn't exist`)
OK(t, mountpointDir.Close())
checkSnapshots(repo, mountpoint, []backend.ID{})
datafile := filepath.Join("testdata", "backup-data.tar.gz")
fd, err := os.Open(datafile)
if os.IsNotExist(err) {
t.Skipf("unable to find data file %q, skipping", datafile)
return
}
OK(t, err)
OK(t, fd.Close())
SetupTarTestFixture(t, env.testdata, datafile)
// first backup
cmdBackup(t, global, []string{env.testdata}, nil)
snapshotIDs := cmdList(t, global, "snapshots")
Assert(t, len(snapshotIDs) == 1,
"expected one snapshot, got %v", snapshotIDs)
checkSnapshots(repo, mountpoint, snapshotIDs)
// second backup, implicit incremental
cmdBackup(t, global, []string{env.testdata}, nil)
snapshotIDs = cmdList(t, global, "snapshots")
Assert(t, len(snapshotIDs) == 2,
"expected two snapshots, got %v", snapshotIDs)
checkSnapshots(repo, mountpoint, snapshotIDs)
// third backup, explicit incremental
cmdBackup(t, global, []string{env.testdata}, &snapshotIDs[0])
snapshotIDs = cmdList(t, global, "snapshots")
Assert(t, len(snapshotIDs) == 3,
"expected three snapshots, got %v", snapshotIDs)
checkSnapshots(repo, mountpoint, snapshotIDs)
})
}
src/github.com/restic/restic/cmd/restic/integration_helpers_test.go
+228
View File
@@ -0,0 +1,228 @@
package main
import (
"fmt"
"io/ioutil"
"os"
"path/filepath"
"runtime"
"testing"
. "github.com/restic/restic/test"
)
type dirEntry struct {
path string
fi os.FileInfo
}
func walkDir(dir string) <-chan *dirEntry {
ch := make(chan *dirEntry, 100)
go func() {
err := filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
if err != nil {
fmt.Fprintf(os.Stderr, "error: %v\n", err)
return nil
}
name, err := filepath.Rel(dir, path)
if err != nil {
fmt.Fprintf(os.Stderr, "error: %v\n", err)
return nil
}
ch <- &dirEntry{
path: name,
fi: info,
}
return nil
})
if err != nil {
fmt.Fprintf(os.Stderr, "Walk() error: %v\n", err)
}
close(ch)
}()
// first element is root
_ = <-ch
return ch
}
func isSymlink(fi os.FileInfo) bool {
mode := fi.Mode() & (os.ModeType | os.ModeCharDevice)
return mode == os.ModeSymlink
}
func sameModTime(fi1, fi2 os.FileInfo) bool {
switch runtime.GOOS {
case "darwin", "freebsd", "openbsd":
if isSymlink(fi1) && isSymlink(fi2) {
return true
}
}
return fi1.ModTime() == fi2.ModTime()
}
// directoriesEqualContents checks if both directories contain exactly the same
// contents.
func directoriesEqualContents(dir1, dir2 string) bool {
ch1 := walkDir(dir1)
ch2 := walkDir(dir2)
changes := false
var a, b *dirEntry
for {
var ok bool
if ch1 != nil && a == nil {
a, ok = <-ch1
if !ok {
ch1 = nil
}
}
if ch2 != nil && b == nil {
b, ok = <-ch2
if !ok {
ch2 = nil
}
}
if ch1 == nil && ch2 == nil {
break
}
if ch1 == nil {
fmt.Printf("+%v\n", b.path)
changes = true
} else if ch2 == nil {
fmt.Printf("-%v\n", a.path)
changes = true
} else if !a.equals(b) {
if a.path < b.path {
fmt.Printf("-%v\n", a.path)
changes = true
a = nil
continue
} else if a.path > b.path {
fmt.Printf("+%v\n", b.path)
changes = true
b = nil
continue
} else {
fmt.Printf("%%%v\n", a.path)
changes = true
}
}
a, b = nil, nil
}
if changes {
return false
}
return true
}
type dirStat struct {
files, dirs, other uint
size uint64
}
func isFile(fi os.FileInfo) bool {
return fi.Mode()&(os.ModeType|os.ModeCharDevice) == 0
}
// dirStats walks dir and collects stats.
func dirStats(dir string) (stat dirStat) {
for entry := range walkDir(dir) {
if isFile(entry.fi) {
stat.files++
stat.size += uint64(entry.fi.Size())
continue
}
if entry.fi.IsDir() {
stat.dirs++
continue
}
stat.other++
}
return stat
}
type testEnvironment struct {
base, cache, repo, testdata string
}
func configureRestic(t testing.TB, cache, repo string) GlobalOptions {
return GlobalOptions{
CacheDir: cache,
Repo: repo,
Quiet: true,
password: TestPassword,
stdout: os.Stdout,
stderr: os.Stderr,
}
}
func cleanupTempdir(t testing.TB, tempdir string) {
if !TestCleanupTempDirs {
t.Logf("leaving temporary directory %v used for test", tempdir)
return
}
RemoveAll(t, tempdir)
}
// withTestEnvironment creates a test environment and calls f with it. After f has
// returned, the temporary directory is removed.
func withTestEnvironment(t testing.TB, f func(*testEnvironment, GlobalOptions)) {
if !RunIntegrationTest {
t.Skip("integration tests disabled")
}
tempdir, err := ioutil.TempDir(TestTempDir, "restic-test-")
OK(t, err)
env := testEnvironment{
base: tempdir,
cache: filepath.Join(tempdir, "cache"),
repo: filepath.Join(tempdir, "repo"),
testdata: filepath.Join(tempdir, "testdata"),
}
OK(t, os.MkdirAll(env.testdata, 0700))
OK(t, os.MkdirAll(env.cache, 0700))
OK(t, os.MkdirAll(env.repo, 0700))
f(&env, configureRestic(t, env.cache, env.repo))
if !TestCleanupTempDirs {
t.Logf("leaving temporary directory %v used for test", tempdir)
return
}
RemoveAll(t, tempdir)
}
// removeFile resets the read-only flag and then deletes the file.
func removeFile(fn string) error {
err := os.Chmod(fn, 0666)
if err != nil {
return err
}
return os.Remove(fn)
}
src/github.com/restic/restic/cmd/restic/integration_helpers_unix_test.go
+41
View File
@@ -0,0 +1,41 @@
//+build !windows
package main
import (
"fmt"
"os"
"syscall"
)
func (e *dirEntry) equals(other *dirEntry) bool {
if e.path != other.path {
fmt.Fprintf(os.Stderr, "%v: path does not match (%v != %v)\n", e.path, e.path, other.path)
return false
}
if e.fi.Mode() != other.fi.Mode() {
fmt.Fprintf(os.Stderr, "%v: mode does not match (%v != %v)\n", e.path, e.fi.Mode(), other.fi.Mode())
return false
}
if !sameModTime(e.fi, other.fi) {
fmt.Fprintf(os.Stderr, "%v: ModTime does not match (%v != %v)\n", e.path, e.fi.ModTime(), other.fi.ModTime())
return false
}
stat, _ := e.fi.Sys().(*syscall.Stat_t)
stat2, _ := other.fi.Sys().(*syscall.Stat_t)
if stat.Uid != stat2.Uid {
fmt.Fprintf(os.Stderr, "%v: UID does not match (%v != %v)\n", e.path, stat.Uid, stat2.Uid)
return false
}
if stat.Gid != stat2.Gid {
fmt.Fprintf(os.Stderr, "%v: GID does not match (%v != %v)\n", e.path, stat.Gid, stat2.Gid)
return false
}
return true
}
src/github.com/restic/restic/cmd/restic/integration_helpers_windows_test.go
+27
View File
@@ -0,0 +1,27 @@
//+build windows
package main
import (
"fmt"
"os"
)
func (e *dirEntry) equals(other *dirEntry) bool {
if e.path != other.path {
fmt.Fprintf(os.Stderr, "%v: path does not match (%v != %v)\n", e.path, e.path, other.path)
return false
}
if e.fi.Mode() != other.fi.Mode() {
fmt.Fprintf(os.Stderr, "%v: mode does not match (%v != %v)\n", e.path, e.fi.Mode(), other.fi.Mode())
return false
}
if !sameModTime(e.fi, other.fi) {
fmt.Fprintf(os.Stderr, "%v: ModTime does not match (%v != %v)\n", e.path, e.fi.ModTime(), other.fi.ModTime())
return false
}
return true
}
src/github.com/restic/restic/cmd/restic/integration_test.go
+816
View File
@@ -0,0 +1,816 @@
package main
import (
"bufio"
"bytes"
"crypto/rand"
"fmt"
"io"
"io/ioutil"
"os"
"path/filepath"
"regexp"
"strings"
"syscall"
"testing"
"time"
"github.com/restic/restic/backend"
"github.com/restic/restic/debug"
"github.com/restic/restic/filter"
"github.com/restic/restic/repository"
. "github.com/restic/restic/test"
)
func parseIDsFromReader(t testing.TB, rd io.Reader) backend.IDs {
IDs := backend.IDs{}
sc := bufio.NewScanner(rd)
for sc.Scan() {
id, err := backend.ParseID(sc.Text())
if err != nil {
t.Logf("parse id %v: %v", sc.Text(), err)
continue
}
IDs = append(IDs, id)
}
return IDs
}
func cmdInit(t testing.TB, global GlobalOptions) {
cmd := &CmdInit{global: &global}
OK(t, cmd.Execute(nil))
t.Logf("repository initialized at %v", global.Repo)
}
func cmdBackup(t testing.TB, global GlobalOptions, target []string, parentID *backend.ID) {
cmdBackupExcludes(t, global, target, parentID, nil)
}
func cmdBackupExcludes(t testing.TB, global GlobalOptions, target []string, parentID *backend.ID, excludes []string) {
cmd := &CmdBackup{global: &global, Excludes: excludes}
if parentID != nil {
cmd.Parent = parentID.String()
}
t.Logf("backing up %v", target)
OK(t, cmd.Execute(target))
}
func cmdList(t testing.TB, global GlobalOptions, tpe string) backend.IDs {
cmd := &CmdList{global: &global}
return executeAndParseIDs(t, cmd, tpe)
}
func executeAndParseIDs(t testing.TB, cmd *CmdList, args ...string) backend.IDs {
buf := bytes.NewBuffer(nil)
cmd.global.stdout = buf
OK(t, cmd.Execute(args))
return parseIDsFromReader(t, buf)
}
func cmdRestore(t testing.TB, global GlobalOptions, dir string, snapshotID backend.ID) {
cmdRestoreExcludes(t, global, dir, snapshotID, nil)
}
func cmdRestoreExcludes(t testing.TB, global GlobalOptions, dir string, snapshotID backend.ID, excludes []string) {
cmd := &CmdRestore{global: &global, Target: dir, Exclude: excludes}
OK(t, cmd.Execute([]string{snapshotID.String()}))
}
func cmdRestoreIncludes(t testing.TB, global GlobalOptions, dir string, snapshotID backend.ID, includes []string) {
cmd := &CmdRestore{global: &global, Target: dir, Include: includes}
OK(t, cmd.Execute([]string{snapshotID.String()}))
}
func cmdCheck(t testing.TB, global GlobalOptions) {
cmd := &CmdCheck{
global: &global,
ReadData: true,
CheckUnused: true,
}
OK(t, cmd.Execute(nil))
}
func cmdCheckOutput(t testing.TB, global GlobalOptions) string {
buf := bytes.NewBuffer(nil)
global.stdout = buf
cmd := &CmdCheck{global: &global, ReadData: true}
OK(t, cmd.Execute(nil))
return string(buf.Bytes())
}
func cmdRebuildIndex(t testing.TB, global GlobalOptions) {
global.stdout = ioutil.Discard
cmd := &CmdRebuildIndex{global: &global}
OK(t, cmd.Execute(nil))
}
func cmdOptimize(t testing.TB, global GlobalOptions) {
cmd := &CmdOptimize{global: &global}
OK(t, cmd.Execute(nil))
}
func cmdLs(t testing.TB, global GlobalOptions, snapshotID string) []string {
var buf bytes.Buffer
global.stdout = &buf
cmd := &CmdLs{global: &global}
OK(t, cmd.Execute([]string{snapshotID}))
return strings.Split(string(buf.Bytes()), "\n")
}
func cmdFind(t testing.TB, global GlobalOptions, pattern string) []string {
var buf bytes.Buffer
global.stdout = &buf
cmd := &CmdFind{global: &global}
OK(t, cmd.Execute([]string{pattern}))
return strings.Split(string(buf.Bytes()), "\n")
}
func TestBackup(t *testing.T) {
withTestEnvironment(t, func(env *testEnvironment, global GlobalOptions) {
datafile := filepath.Join("testdata", "backup-data.tar.gz")
fd, err := os.Open(datafile)
if os.IsNotExist(err) {
t.Skipf("unable to find data file %q, skipping", datafile)
return
}
OK(t, err)
OK(t, fd.Close())
cmdInit(t, global)
SetupTarTestFixture(t, env.testdata, datafile)
// first backup
cmdBackup(t, global, []string{env.testdata}, nil)
snapshotIDs := cmdList(t, global, "snapshots")
Assert(t, len(snapshotIDs) == 1,
"expected one snapshot, got %v", snapshotIDs)
cmdCheck(t, global)
stat1 := dirStats(env.repo)
// second backup, implicit incremental
cmdBackup(t, global, []string{env.testdata}, nil)
snapshotIDs = cmdList(t, global, "snapshots")
Assert(t, len(snapshotIDs) == 2,
"expected two snapshots, got %v", snapshotIDs)
stat2 := dirStats(env.repo)
if stat2.size > stat1.size+stat1.size/10 {
t.Error("repository size has grown by more than 10 percent")
}
t.Logf("repository grown by %d bytes", stat2.size-stat1.size)
cmdCheck(t, global)
// third backup, explicit incremental
cmdBackup(t, global, []string{env.testdata}, &snapshotIDs[0])
snapshotIDs = cmdList(t, global, "snapshots")
Assert(t, len(snapshotIDs) == 3,
"expected three snapshots, got %v", snapshotIDs)
stat3 := dirStats(env.repo)
if stat3.size > stat1.size+stat1.size/10 {
t.Error("repository size has grown by more than 10 percent")
}
t.Logf("repository grown by %d bytes", stat3.size-stat2.size)
// restore all backups and compare
for i, snapshotID := range snapshotIDs {
restoredir := filepath.Join(env.base, fmt.Sprintf("restore%d", i))
t.Logf("restoring snapshot %v to %v", snapshotID.Str(), restoredir)
cmdRestore(t, global, restoredir, snapshotIDs[0])
Assert(t, directoriesEqualContents(env.testdata, filepath.Join(restoredir, "testdata")),
"directories are not equal")
}
cmdCheck(t, global)
})
}
func TestBackupNonExistingFile(t *testing.T) {
withTestEnvironment(t, func(env *testEnvironment, global GlobalOptions) {
datafile := filepath.Join("testdata", "backup-data.tar.gz")
fd, err := os.Open(datafile)
if os.IsNotExist(err) {
t.Skipf("unable to find data file %q, skipping", datafile)
return
}
OK(t, err)
OK(t, fd.Close())
SetupTarTestFixture(t, env.testdata, datafile)
cmdInit(t, global)
global.stderr = ioutil.Discard
p := filepath.Join(env.testdata, "0", "0")
dirs := []string{
filepath.Join(p, "0"),
filepath.Join(p, "1"),
filepath.Join(p, "nonexisting"),
filepath.Join(p, "5"),
}
cmdBackup(t, global, dirs, nil)
})
}
func TestBackupMissingFile1(t *testing.T) {
withTestEnvironment(t, func(env *testEnvironment, global GlobalOptions) {
datafile := filepath.Join("testdata", "backup-data.tar.gz")
fd, err := os.Open(datafile)
if os.IsNotExist(err) {
t.Skipf("unable to find data file %q, skipping", datafile)
return
}
OK(t, err)
OK(t, fd.Close())
SetupTarTestFixture(t, env.testdata, datafile)
cmdInit(t, global)
global.stderr = ioutil.Discard
ranHook := false
debug.Hook("pipe.walk1", func(context interface{}) {
pathname := context.(string)
if pathname != filepath.Join("testdata", "0", "0", "9") {
return
}
t.Logf("in hook, removing test file testdata/0/0/9/37")
ranHook = true
OK(t, os.Remove(filepath.Join(env.testdata, "0", "0", "9", "37")))
})
cmdBackup(t, global, []string{env.testdata}, nil)
cmdCheck(t, global)
Assert(t, ranHook, "hook did not run")
debug.RemoveHook("pipe.walk1")
})
}
func TestBackupMissingFile2(t *testing.T) {
withTestEnvironment(t, func(env *testEnvironment, global GlobalOptions) {
datafile := filepath.Join("testdata", "backup-data.tar.gz")
fd, err := os.Open(datafile)
if os.IsNotExist(err) {
t.Skipf("unable to find data file %q, skipping", datafile)
return
}
OK(t, err)
OK(t, fd.Close())
SetupTarTestFixture(t, env.testdata, datafile)
cmdInit(t, global)
global.stderr = ioutil.Discard
ranHook := false
debug.Hook("pipe.walk2", func(context interface{}) {
pathname := context.(string)
if pathname != filepath.Join("testdata", "0", "0", "9", "37") {
return
}
t.Logf("in hook, removing test file testdata/0/0/9/37")
ranHook = true
OK(t, os.Remove(filepath.Join(env.testdata, "0", "0", "9", "37")))
})
cmdBackup(t, global, []string{env.testdata}, nil)
cmdCheck(t, global)
Assert(t, ranHook, "hook did not run")
debug.RemoveHook("pipe.walk2")
})
}
func TestBackupDirectoryError(t *testing.T) {
withTestEnvironment(t, func(env *testEnvironment, global GlobalOptions) {
datafile := filepath.Join("testdata", "backup-data.tar.gz")
fd, err := os.Open(datafile)
if os.IsNotExist(err) {
t.Skipf("unable to find data file %q, skipping", datafile)
return
}
OK(t, err)
OK(t, fd.Close())
SetupTarTestFixture(t, env.testdata, datafile)
cmdInit(t, global)
global.stderr = ioutil.Discard
ranHook := false
testdir := filepath.Join(env.testdata, "0", "0", "9")
// install hook that removes the dir right before readdirnames()
debug.Hook("pipe.readdirnames", func(context interface{}) {
path := context.(string)
if path != testdir {
return
}
t.Logf("in hook, removing test file %v", testdir)
ranHook = true
OK(t, os.RemoveAll(testdir))
})
cmdBackup(t, global, []string{filepath.Join(env.testdata, "0", "0")}, nil)
cmdCheck(t, global)
Assert(t, ranHook, "hook did not run")
debug.RemoveHook("pipe.walk2")
snapshots := cmdList(t, global, "snapshots")
Assert(t, len(snapshots) > 0,
"no snapshots found in repo (%v)", datafile)
files := cmdLs(t, global, snapshots[0].String())
Assert(t, len(files) > 1, "snapshot is empty")
})
}
func includes(haystack []string, needle string) bool {
for _, s := range haystack {
if s == needle {
return true
}
}
return false
}
func loadSnapshotMap(t testing.TB, global GlobalOptions) map[string]struct{} {
snapshotIDs := cmdList(t, global, "snapshots")
m := make(map[string]struct{})
for _, id := range snapshotIDs {
m[id.String()] = struct{}{}
}
return m
}
func lastSnapshot(old, new map[string]struct{}) (map[string]struct{}, string) {
for k := range new {
if _, ok := old[k]; !ok {
old[k] = struct{}{}
return old, k
}
}
return old, ""
}
var backupExcludeFilenames = []string{
"testfile1",
"foo.tar.gz",
"private/secret/passwords.txt",
"work/source/test.c",
}
func TestBackupExclude(t *testing.T) {
withTestEnvironment(t, func(env *testEnvironment, global GlobalOptions) {
cmdInit(t, global)
datadir := filepath.Join(env.base, "testdata")
for _, filename := range backupExcludeFilenames {
fp := filepath.Join(datadir, filename)
OK(t, os.MkdirAll(filepath.Dir(fp), 0755))
f, err := os.Create(fp)
OK(t, err)
fmt.Fprintf(f, filename)
OK(t, f.Close())
}
snapshots := make(map[string]struct{})
cmdBackup(t, global, []string{datadir}, nil)
snapshots, snapshotID := lastSnapshot(snapshots, loadSnapshotMap(t, global))
files := cmdLs(t, global, snapshotID)
Assert(t, includes(files, filepath.Join("testdata", "foo.tar.gz")),
"expected file %q in first snapshot, but it's not included", "foo.tar.gz")
cmdBackupExcludes(t, global, []string{datadir}, nil, []string{"*.tar.gz"})
snapshots, snapshotID = lastSnapshot(snapshots, loadSnapshotMap(t, global))
files = cmdLs(t, global, snapshotID)
Assert(t, !includes(files, filepath.Join("testdata", "foo.tar.gz")),
"expected file %q not in first snapshot, but it's included", "foo.tar.gz")
cmdBackupExcludes(t, global, []string{datadir}, nil, []string{"*.tar.gz", "private/secret"})
snapshots, snapshotID = lastSnapshot(snapshots, loadSnapshotMap(t, global))
files = cmdLs(t, global, snapshotID)
Assert(t, !includes(files, filepath.Join("testdata", "foo.tar.gz")),
"expected file %q not in first snapshot, but it's included", "foo.tar.gz")
Assert(t, !includes(files, filepath.Join("testdata", "private", "secret", "passwords.txt")),
"expected file %q not in first snapshot, but it's included", "passwords.txt")
})
}
const (
incrementalFirstWrite = 20 * 1042 * 1024
incrementalSecondWrite = 12 * 1042 * 1024
incrementalThirdWrite = 4 * 1042 * 1024
)
func appendRandomData(filename string, bytes uint) error {
f, err := os.OpenFile(filename, os.O_WRONLY|os.O_CREATE, 0666)
if err != nil {
fmt.Fprint(os.Stderr, err)
return err
}
_, err = f.Seek(0, 2)
if err != nil {
fmt.Fprint(os.Stderr, err)
return err
}
_, err = io.Copy(f, io.LimitReader(rand.Reader, int64(bytes)))
if err != nil {
fmt.Fprint(os.Stderr, err)
return err
}
return f.Close()
}
func TestIncrementalBackup(t *testing.T) {
withTestEnvironment(t, func(env *testEnvironment, global GlobalOptions) {
cmdInit(t, global)
datadir := filepath.Join(env.base, "testdata")
testfile := filepath.Join(datadir, "testfile")
OK(t, appendRandomData(testfile, incrementalFirstWrite))
cmdBackup(t, global, []string{datadir}, nil)
cmdCheck(t, global)
stat1 := dirStats(env.repo)
OK(t, appendRandomData(testfile, incrementalSecondWrite))
cmdBackup(t, global, []string{datadir}, nil)
cmdCheck(t, global)
stat2 := dirStats(env.repo)
if stat2.size-stat1.size > incrementalFirstWrite {
t.Errorf("repository size has grown by more than %d bytes", incrementalFirstWrite)
}
t.Logf("repository grown by %d bytes", stat2.size-stat1.size)
OK(t, appendRandomData(testfile, incrementalThirdWrite))
cmdBackup(t, global, []string{datadir}, nil)
cmdCheck(t, global)
stat3 := dirStats(env.repo)
if stat3.size-stat2.size > incrementalFirstWrite {
t.Errorf("repository size has grown by more than %d bytes", incrementalFirstWrite)
}
t.Logf("repository grown by %d bytes", stat3.size-stat2.size)
})
}
func cmdKey(t testing.TB, global GlobalOptions, args ...string) string {
var buf bytes.Buffer
global.stdout = &buf
cmd := &CmdKey{global: &global}
OK(t, cmd.Execute(args))
return buf.String()
}
func cmdKeyListOtherIDs(t testing.TB, global GlobalOptions) []string {
var buf bytes.Buffer
global.stdout = &buf
cmd := &CmdKey{global: &global}
OK(t, cmd.Execute([]string{"list"}))
scanner := bufio.NewScanner(&buf)
exp := regexp.MustCompile(`^ ([a-f0-9]+) `)
IDs := []string{}
for scanner.Scan() {
if id := exp.FindStringSubmatch(scanner.Text()); id != nil {
IDs = append(IDs, id[1])
}
}
return IDs
}
func cmdKeyAddNewKey(t testing.TB, global GlobalOptions, newPassword string) {
cmd := &CmdKey{global: &global, newPassword: newPassword}
OK(t, cmd.Execute([]string{"add"}))
}
func cmdKeyPasswd(t testing.TB, global GlobalOptions, newPassword string) {
cmd := &CmdKey{global: &global, newPassword: newPassword}
OK(t, cmd.Execute([]string{"passwd"}))
}
func cmdKeyRemove(t testing.TB, global GlobalOptions, IDs []string) {
cmd := &CmdKey{global: &global}
t.Logf("remove %d keys: %q\n", len(IDs), IDs)
for _, id := range IDs {
OK(t, cmd.Execute([]string{"rm", id}))
}
}
func TestKeyAddRemove(t *testing.T) {
passwordList := []string{
"OnnyiasyatvodsEvVodyawit",
"raicneirvOjEfEigonOmLasOd",
}
withTestEnvironment(t, func(env *testEnvironment, global GlobalOptions) {
cmdInit(t, global)
cmdKeyPasswd(t, global, "geheim2")
global.password = "geheim2"
t.Logf("changed password to %q", global.password)
for _, newPassword := range passwordList {
cmdKeyAddNewKey(t, global, newPassword)
t.Logf("added new password %q", newPassword)
global.password = newPassword
cmdKeyRemove(t, global, cmdKeyListOtherIDs(t, global))
}
global.password = passwordList[len(passwordList)-1]
t.Logf("testing access with last password %q\n", global.password)
cmdKey(t, global, "list")
cmdCheck(t, global)
})
}
func testFileSize(filename string, size int64) error {
fi, err := os.Stat(filename)
if err != nil {
return err
}
if fi.Size() != size {
return fmt.Errorf("wrong file size for %v: expected %v, got %v", filename, size, fi.Size())
}
return nil
}
func TestRestoreFilter(t *testing.T) {
testfiles := []struct {
name string
size uint
}{
{"testfile1.c", 100},
{"testfile2.exe", 101},
{"subdir1/subdir2/testfile3.docx", 102},
{"subdir1/subdir2/testfile4.c", 102},
}
withTestEnvironment(t, func(env *testEnvironment, global GlobalOptions) {
cmdInit(t, global)
for _, test := range testfiles {
p := filepath.Join(env.testdata, test.name)
OK(t, os.MkdirAll(filepath.Dir(p), 0755))
OK(t, appendRandomData(p, test.size))
}
cmdBackup(t, global, []string{env.testdata}, nil)
cmdCheck(t, global)
snapshotID := cmdList(t, global, "snapshots")[0]
// no restore filter should restore all files
cmdRestore(t, global, filepath.Join(env.base, "restore0"), snapshotID)
for _, test := range testfiles {
OK(t, testFileSize(filepath.Join(env.base, "restore0", "testdata", test.name), int64(test.size)))
}
for i, pat := range []string{"*.c", "*.exe", "*", "*file3*"} {
base := filepath.Join(env.base, fmt.Sprintf("restore%d", i+1))
cmdRestoreExcludes(t, global, base, snapshotID, []string{pat})
for _, test := range testfiles {
err := testFileSize(filepath.Join(base, "testdata", test.name), int64(test.size))
if ok, _ := filter.Match(pat, filepath.Base(test.name)); !ok {
OK(t, err)
} else {
Assert(t, os.IsNotExist(err),
"expected %v to not exist in restore step %v, but it exists, err %v", test.name, i+1, err)
}
}
}
})
}
func TestRestoreWithPermissionFailure(t *testing.T) {
withTestEnvironment(t, func(env *testEnvironment, global GlobalOptions) {
datafile := filepath.Join("testdata", "repo-restore-permissions-test.tar.gz")
SetupTarTestFixture(t, env.base, datafile)
snapshots := cmdList(t, global, "snapshots")
Assert(t, len(snapshots) > 0,
"no snapshots found in repo (%v)", datafile)
global.stderr = ioutil.Discard
cmdRestore(t, global, filepath.Join(env.base, "restore"), snapshots[0])
// make sure that all files have been restored, regardeless of any
// permission errors
files := cmdLs(t, global, snapshots[0].String())
for _, filename := range files {
fi, err := os.Lstat(filepath.Join(env.base, "restore", filename))
OK(t, err)
Assert(t, !isFile(fi) || fi.Size() > 0,
"file %v restored, but filesize is 0", filename)
}
})
}
func setZeroModTime(filename string) error {
var utimes = []syscall.Timespec{
syscall.NsecToTimespec(0),
syscall.NsecToTimespec(0),
}
return syscall.UtimesNano(filename, utimes)
}
func TestRestoreNoMetadataOnIgnoredIntermediateDirs(t *testing.T) {
withTestEnvironment(t, func(env *testEnvironment, global GlobalOptions) {
cmdInit(t, global)
p := filepath.Join(env.testdata, "subdir1", "subdir2", "subdir3", "file.ext")
OK(t, os.MkdirAll(filepath.Dir(p), 0755))
OK(t, appendRandomData(p, 200))
OK(t, setZeroModTime(filepath.Join(env.testdata, "subdir1", "subdir2")))
cmdBackup(t, global, []string{env.testdata}, nil)
cmdCheck(t, global)
snapshotID := cmdList(t, global, "snapshots")[0]
// restore with filter "*.ext", this should restore "file.ext", but
// since the directories are ignored and only created because of
// "file.ext", no meta data should be restored for them.
cmdRestoreIncludes(t, global, filepath.Join(env.base, "restore0"), snapshotID, []string{"*.ext"})
f1 := filepath.Join(env.base, "restore0", "testdata", "subdir1", "subdir2")
fi, err := os.Stat(f1)
OK(t, err)
Assert(t, fi.ModTime() != time.Unix(0, 0),
"meta data of intermediate directory has been restore although it was ignored")
// restore with filter "*", this should restore meta data on everything.
cmdRestoreIncludes(t, global, filepath.Join(env.base, "restore1"), snapshotID, []string{"*"})
f2 := filepath.Join(env.base, "restore1", "testdata", "subdir1", "subdir2")
fi, err = os.Stat(f2)
OK(t, err)
Assert(t, fi.ModTime() == time.Unix(0, 0),
"meta data of intermediate directory hasn't been restore")
})
}
func TestFind(t *testing.T) {
withTestEnvironment(t, func(env *testEnvironment, global GlobalOptions) {
datafile := filepath.Join("testdata", "backup-data.tar.gz")
cmdInit(t, global)
SetupTarTestFixture(t, env.testdata, datafile)
cmdBackup(t, global, []string{env.testdata}, nil)
cmdCheck(t, global)
results := cmdFind(t, global, "unexistingfile")
Assert(t, len(results) != 0, "unexisting file found in repo (%v)", datafile)
results = cmdFind(t, global, "testfile")
Assert(t, len(results) != 1, "file not found in repo (%v)", datafile)
results = cmdFind(t, global, "test")
Assert(t, len(results) < 2, "less than two file found in repo (%v)", datafile)
})
}
func TestRebuildIndex(t *testing.T) {
withTestEnvironment(t, func(env *testEnvironment, global GlobalOptions) {
datafile := filepath.Join("..", "..", "checker", "testdata", "duplicate-packs-in-index-test-repo.tar.gz")
SetupTarTestFixture(t, env.base, datafile)
out := cmdCheckOutput(t, global)
if !strings.Contains(out, "contained in several indexes") {
t.Fatalf("did not find checker hint for packs in several indexes")
}
if !strings.Contains(out, "restic rebuild-index") {
t.Fatalf("did not find hint for rebuild-index comman")
}
cmdRebuildIndex(t, global)
out = cmdCheckOutput(t, global)
if len(out) != 0 {
t.Fatalf("expected no output from the checker, got: %v", out)
}
})
}
func TestRebuildIndexAlwaysFull(t *testing.T) {
repository.IndexFull = func(*repository.Index) bool { return true }
TestRebuildIndex(t)
}
var optimizeTests = []struct {
testFilename string
snapshots backend.IDSet
}{
{
filepath.Join("..", "..", "checker", "testdata", "checker-test-repo.tar.gz"),
backend.NewIDSet(ParseID("a13c11e582b77a693dd75ab4e3a3ba96538a056594a4b9076e4cacebe6e06d43")),
},
{
filepath.Join("testdata", "old-index-repo.tar.gz"),
nil,
},
{
filepath.Join("testdata", "old-index-repo.tar.gz"),
backend.NewIDSet(
ParseID("f7d83db709977178c9d1a09e4009355e534cde1a135b8186b8b118a3fc4fcd41"),
ParseID("51d249d28815200d59e4be7b3f21a157b864dc343353df9d8e498220c2499b02"),
),
},
}
func TestOptimizeRemoveUnusedBlobs(t *testing.T) {
for i, test := range optimizeTests {
withTestEnvironment(t, func(env *testEnvironment, global GlobalOptions) {
SetupTarTestFixture(t, env.base, test.testFilename)
for id := range test.snapshots {
OK(t, removeFile(filepath.Join(env.repo, "snapshots", id.String())))
}
cmdOptimize(t, global)
output := cmdCheckOutput(t, global)
if len(output) > 0 {
t.Errorf("expected no output for check in test %d, got:\n%v", i, output)
}
})
}
}
func TestCheckRestoreNoLock(t *testing.T) {
withTestEnvironment(t, func(env *testEnvironment, global GlobalOptions) {
datafile := filepath.Join("testdata", "small-repo.tar.gz")
SetupTarTestFixture(t, env.base, datafile)
err := filepath.Walk(env.repo, func(p string, fi os.FileInfo, e error) error {
if e != nil {
return e
}
return os.Chmod(p, fi.Mode() & ^(os.FileMode(0222)))
})
OK(t, err)
global.NoLock = true
cmdCheck(t, global)
snapshotIDs := cmdList(t, global, "snapshots")
if len(snapshotIDs) == 0 {
t.Fatalf("found no snapshots")
}
cmdRestore(t, global, filepath.Join(env.base, "restore"), snapshotIDs[0])
})
}
src/github.com/restic/restic/cmd/restic/lock.go
+125
View File
@@ -0,0 +1,125 @@
package main
import (
"fmt"
"os"
"sync"
"time"
"github.com/restic/restic"
"github.com/restic/restic/debug"
"github.com/restic/restic/repository"
)
var globalLocks struct {
locks []*restic.Lock
cancelRefresh chan struct{}
refreshWG sync.WaitGroup
sync.Mutex
}
func lockRepo(repo *repository.Repository) (*restic.Lock, error) {
return lockRepository(repo, false)
}
func lockRepoExclusive(repo *repository.Repository) (*restic.Lock, error) {
return lockRepository(repo, true)
}
func lockRepository(repo *repository.Repository, exclusive bool) (*restic.Lock, error) {
lockFn := restic.NewLock
if exclusive {
lockFn = restic.NewExclusiveLock
}
lock, err := lockFn(repo)
if err != nil {
return nil, err
}
globalLocks.Lock()
if globalLocks.cancelRefresh == nil {
debug.Log("main.lockRepository", "start goroutine for lock refresh")
globalLocks.cancelRefresh = make(chan struct{})
globalLocks.refreshWG = sync.WaitGroup{}
globalLocks.refreshWG.Add(1)
go refreshLocks(&globalLocks.refreshWG, globalLocks.cancelRefresh)
}
globalLocks.locks = append(globalLocks.locks, lock)
globalLocks.Unlock()
return lock, err
}
var refreshInterval = 5 * time.Minute
func refreshLocks(wg *sync.WaitGroup, done <-chan struct{}) {
debug.Log("main.refreshLocks", "start")
defer func() {
wg.Done()
globalLocks.Lock()
globalLocks.cancelRefresh = nil
globalLocks.Unlock()
}()
ticker := time.NewTicker(refreshInterval)
for {
select {
case <-done:
debug.Log("main.refreshLocks", "terminate")
return
case <-ticker.C:
debug.Log("main.refreshLocks", "refreshing locks")
globalLocks.Lock()
for _, lock := range globalLocks.locks {
err := lock.Refresh()
if err != nil {
fmt.Fprintf(os.Stderr, "unable to refresh lock: %v\n", err)
}
}
globalLocks.Unlock()
}
}
}
func unlockRepo(lock *restic.Lock) error {
globalLocks.Lock()
defer globalLocks.Unlock()
debug.Log("unlockRepo", "unlocking repository")
if err := lock.Unlock(); err != nil {
debug.Log("unlockRepo", "error while unlocking: %v", err)
return err
}
for i := 0; i < len(globalLocks.locks); i++ {
if lock == globalLocks.locks[i] {
globalLocks.locks = append(globalLocks.locks[:i], globalLocks.locks[i+1:]...)
return nil
}
}
return nil
}
func unlockAll() error {
globalLocks.Lock()
defer globalLocks.Unlock()
debug.Log("unlockAll", "unlocking %d locks", len(globalLocks.locks))
for _, lock := range globalLocks.locks {
if err := lock.Unlock(); err != nil {
debug.Log("unlockAll", "error while unlocking: %v", err)
return err
}
debug.Log("unlockAll", "successfully removed lock")
}
return nil
}
func init() {
AddCleanupHandler(unlockAll)
}
src/github.com/restic/restic/cmd/restic/main.go
+45
View File
@@ -0,0 +1,45 @@
package main
import (
"fmt"
"os"
"runtime"
"github.com/jessevdk/go-flags"
"github.com/restic/restic"
"github.com/restic/restic/debug"
)
func init() {
// set GOMAXPROCS to number of CPUs
runtime.GOMAXPROCS(runtime.NumCPU())
}
func main() {
// defer profile.Start(profile.MemProfileRate(100000), profile.ProfilePath(".")).Stop()
// defer profile.Start(profile.CPUProfile, profile.ProfilePath(".")).Stop()
globalOpts.Repo = os.Getenv("RESTIC_REPOSITORY")
globalOpts.password = os.Getenv("RESTIC_PASSWORD")
debug.Log("restic", "main %#v", os.Args)
_, err := parser.Parse()
if e, ok := err.(*flags.Error); ok && e.Type == flags.ErrHelp {
parser.WriteHelp(os.Stdout)
os.Exit(0)
}
if err != nil {
fmt.Fprintf(os.Stderr, "%v\n", err)
}
if restic.IsAlreadyLocked(err) {
fmt.Fprintf(os.Stderr, "\nthe `unlock` command can be used to remove stale locks\n")
}
RunCleanupHandlers()
if err != nil {
os.Exit(1)
}
}
src/github.com/restic/restic/cmd/restic/testdata/backup-data.tar.gz
Vendored
BIN
View File
Binary file not shown.
src/github.com/restic/restic/cmd/restic/testdata/old-index-repo.tar.gz
Vendored
BIN
View File
Binary file not shown.
src/github.com/restic/restic/cmd/restic/testdata/repo-restore-permissions-test.tar.gz
Vendored
BIN
View File
Binary file not shown.
src/github.com/restic/restic/cmd/restic/testdata/small-repo.tar.gz
Vendored
BIN
View File
Binary file not shown.
src/github.com/restic/restic/crypto/buffer_pool.go
+19
View File
@@ -0,0 +1,19 @@
package crypto
import "sync"
const defaultBufSize = 32 * 1024 // 32KiB
var bufPool = sync.Pool{
New: func() interface{} {
return make([]byte, defaultBufSize)
},
}
func getBuffer() []byte {
return bufPool.Get().([]byte)
}
func freeBuffer(buf []byte) {
bufPool.Put(buf)
}

Some files were not shown because too many files have changed in this diff Show More