Compare commits

..

9 Commits

Author SHA1 Message Date
Alexander Neumann
fe9f142b52 Add version for 0.16.5 2024-07-01 21:25:34 +02:00
Alexander Neumann
6ae760751a Generate CHANGELOG.md for 0.16.5 2024-07-01 21:25:33 +02:00
Alexander Neumann
2fa1b42706 Prepare changelog for 0.16.5 2024-07-01 21:25:33 +02:00
Michael Eischer
ca04a88e65 Merge pull request #4879 from restic/backport-azure-cli-option
Backport azure cli option
2024-06-26 21:07:16 +02:00
Michael Eischer
12e858b7af azure: deduplicate cli and default credentials case 2024-06-26 20:59:51 +02:00
Maik Riechert
834f08fe2d Azure: add option to force use of CLI credential 2024-06-26 20:59:51 +02:00
Michael Eischer
814ef4901f Merge pull request #4878 from restic/patch-deps-bump
Bump dependencies
2024-06-26 20:58:29 +02:00
Michael Eischer
84bc9432de update release verification script for latest docker 2024-06-26 20:49:23 +02:00
Michael Eischer
e9d711422a bump azure, golang and gcs dependencies 2024-06-26 20:49:23 +02:00
268 changed files with 4830 additions and 7691 deletions

View File

@@ -13,7 +13,7 @@ permissions:
contents: read
env:
latest_go: "1.22.x"
latest_go: "1.21.x"
GO111MODULE: on
jobs:
@@ -23,31 +23,26 @@ jobs:
# list of jobs to run:
include:
- job_name: Windows
go: 1.22.x
go: 1.21.x
os: windows-latest
- job_name: macOS
go: 1.22.x
go: 1.21.x
os: macOS-latest
test_fuse: false
- job_name: Linux
go: 1.22.x
go: 1.21.x
os: ubuntu-latest
test_cloud_backends: true
test_fuse: true
check_changelog: true
- job_name: Linux (race)
go: 1.22.x
os: ubuntu-latest
test_fuse: true
test_opts: "-race"
- job_name: Linux
go: 1.21.x
os: ubuntu-latest
test_fuse: true
test_opts: "-race"
- job_name: Linux
go: 1.20.x
@@ -260,7 +255,7 @@ jobs:
uses: golangci/golangci-lint-action@v3
with:
# Required: the version of golangci-lint is required and must be specified without patch version: we always use the latest patch version.
version: v1.56.1
version: v1.55.2
args: --verbose --timeout 5m
# only run golangci-lint for pull requests, otherwise ALL hints get

1
.gitignore vendored
View File

@@ -1,4 +1,3 @@
/.idea
/restic
/restic.exe
/.vagrant

View File

@@ -35,9 +35,6 @@ linters:
# parse and typecheck code
- typecheck
# ensure that http response bodies are closed
- bodyclose
issues:
# don't use the default exclude rules, this hides (among others) ignored
# errors from Close() calls
@@ -54,8 +51,3 @@ issues:
# staticcheck: there's no easy way to replace these packages
- "SA1019: \"golang.org/x/crypto/poly1305\" is deprecated"
- "SA1019: \"golang.org/x/crypto/openpgp\" is deprecated"
exclude-rules:
# revive: ignore unused parameters in tests
- path: (_test\.go|testing\.go|backend/.*/tests\.go)
text: "unused-parameter:"

View File

@@ -8,10 +8,6 @@ build:
tools:
python: "3.11"
# Build HTMLZip
formats:
- htmlzip
# Build documentation in the docs/ directory with Sphinx
sphinx:
configuration: doc/conf.py

File diff suppressed because it is too large Load Diff

View File

@@ -6,8 +6,7 @@ Ways to Help Out
Thank you for your contribution! Please **open an issue first** (or add a
comment to an existing issue) if you plan to work on any code or add a new
feature. This way, duplicate work is prevented and we can discuss your ideas
and design first. Small bugfixes are an exception to this rule, just open a
pull request in this case.
and design first.
There are several ways you can help us out. First of all code contributions and
bug fixes are most welcome. However even "minor" details as fixing spelling
@@ -62,7 +61,7 @@ uploading it somewhere or post only the parts that are really relevant.
If restic gets stuck, please also include a stacktrace in the description.
On non-Windows systems, you can send a SIGQUIT signal to restic or press
`Ctrl-\` to achieve the same result. This causes restic to print a stacktrace
and then exit immediately. This will not damage your repository, however,
and then exit immediatelly. This will not damage your repository, however,
it might be necessary to manually clean up stale lock files using
`restic unlock`.

View File

@@ -1 +1 @@
0.16.4
0.16.5

View File

@@ -1,6 +1,6 @@
Bugfix: Don't abort the stats command when data blobs are missing
Running the stats command in the blobs-per-file mode on a repository with
Runing the stats command in the blobs-per-file mode on a repository with
missing data blobs previously resulted in a crash.
https://github.com/restic/restic/pull/2668

View File

@@ -1,6 +1,6 @@
Bugfix: Mark repository files as read-only when using the local backend
Files stored in a local repository were marked as writable on the
Files stored in a local repository were marked as writeable on the
filesystem for non-Windows systems, which did not prevent accidental file
modifications outside of restic. In addition, the local backend did not work
with certain filesystems and network mounts which do not permit modifications

View File

@@ -5,7 +5,7 @@ another process using an exclusive lock through a filesystem snapshot. Restic
was unable to backup those files before. This update enables backing up these
files.
This needs to be enabled explicitly using the --use-fs-snapshot option of the
This needs to be enabled explicitely using the --use-fs-snapshot option of the
backup command.
https://github.com/restic/restic/issues/340

View File

@@ -2,7 +2,7 @@ Enhancement: Parallelize scan of snapshot content in `copy` and `prune`
The `copy` and `prune` commands used to traverse the directories of
snapshots one by one to find used data. This snapshot traversal is
now parallelized which can speed up this step several times.
now parallized which can speed up this step several times.
In addition the `check` command now reports how many snapshots have
already been processed.

View File

@@ -0,0 +1,6 @@
Enhancement: Update dependencies
A few potentially vulnerable dependencies were updated.
https://github.com/restic/restic/issues/4873
https://github.com/restic/restic/pull/4878

View File

@@ -0,0 +1,5 @@
Enhancement: Add option to force use of Azure CLI credential
A new environment variable `AZURE_FORCE_CLI_CREDENTIAL=true` allows forcing the use of Azure CLI credential, ignoring other credentials like managed identity.
https://github.com/restic/restic/pull/4799

View File

@@ -3,7 +3,7 @@ Enhancement: Add local metadata cache
We've added a local cache for metadata so that restic doesn't need to load
all metadata (snapshots, indexes, ...) from the repo each time it starts. By
default the cache is active, but there's a new global option `--no-cache`
that can be used to disable the cache. By default, the cache a standard
that can be used to disable the cache. By deafult, the cache a standard
cache folder for the OS, which can be overridden with `--cache-dir`. The
cache will automatically populate, indexes and snapshots are saved as they
are loaded. Cache directories for repos that haven't been used recently can

View File

@@ -1,6 +1,6 @@
Enhancement: Make `check` print `no errors found` explicitly
The `check` command now explicitly prints `No errors were found` when no errors
The `check` command now explicetly prints `No errors were found` when no errors
could be found.
https://github.com/restic/restic/pull/1319

View File

@@ -1,4 +1,4 @@
Bugfix: Limit bandwidth at the http.RoundTripper for HTTP based backends
Bugfix: Limit bandwith at the http.RoundTripper for HTTP based backends
https://github.com/restic/restic/issues/1506
https://github.com/restic/restic/pull/1511

View File

@@ -1,7 +1,7 @@
Bugfix: backup: Remove bandwidth display
This commit removes the bandwidth displayed during backup process. It is
misleading and seldom correct, because it's neither the "read
misleading and seldomly correct, because it's neither the "read
bandwidth" (only for the very first backup) nor the "upload bandwidth".
Many users are confused about (and rightly so), c.f. #1581, #1033, #1591

View File

@@ -6,7 +6,7 @@ that means making a request (e.g. via HTTP) and returning an error when the
file already exists.
This is not accurate, the file could have been created between the HTTP request
testing for it, and when writing starts, so we've relaxed this requirement,
testing for it, and when writing starts, so we've relaxed this requeriment,
which saves one additional HTTP request per newly added file.
https://github.com/restic/restic/pull/1623

View File

@@ -1,4 +1,4 @@
Enhancement: Allow keeping a time range of snapshots
Enhancement: Allow keeping a time range of snaphots
We've added the `--keep-within` option to the `forget` command. It instructs
restic to keep all snapshots within the given duration since the newest

View File

@@ -1,7 +1,7 @@
Enhancement: Display reason why forget keeps snapshots
We've added a column to the list of snapshots `forget` keeps which details the
reasons to keep a particular snapshot. This makes debugging policies for
reasons to keep a particuliar snapshot. This makes debugging policies for
forget much easier. Please remember to always try things out with `--dry-run`!
https://github.com/restic/restic/pull/1876

View File

@@ -9,7 +9,7 @@ file should be noticed, and the modified file will be backed up. The ctime check
will be disabled if the --ignore-inode flag was given.
If this change causes problems for you, please open an issue, and we can look in
to adding a separate flag to disable just the ctime check.
to adding a seperate flag to disable just the ctime check.
https://github.com/restic/restic/issues/2179
https://github.com/restic/restic/pull/2212

View File

@@ -1,16 +0,0 @@
Enhancement: Support reading backup from a commands's standard output
The `backup` command now supports the `--stdin-from-command` option. When using
this option, the arguments to `backup` are interpreted as a command instead of
paths to back up. `backup` then executes the given command and stores the
standard output from it in the backup, similar to the what the `--stdin` option
does. This also enables restic to verify that the command completes with exit
code zero. A non-zero exit code causes the backup to fail.
Note that the `--stdin` option does not have to be specified at the same time,
and that the `--stdin-filename` option also applies to `--stdin-from-command`.
Example: `restic backup --stdin-from-command --stdin-filename dump.sql mysqldump [...]`
https://github.com/restic/restic/issues/4251
https://github.com/restic/restic/pull/4410

View File

@@ -1,18 +0,0 @@
Enhancement: Allow AWS Assume Role to be used for S3 backend
Previously only credentials discovered via the Minio discovery methods
were used to authenticate.
However, there are many circumstances where the discovered credentials have
lower permissions and need to assume a specific role. This is now possible
using the following new environment variables.
- RESTIC_AWS_ASSUME_ROLE_ARN
- RESTIC_AWS_ASSUME_ROLE_SESSION_NAME
- RESTIC_AWS_ASSUME_ROLE_EXTERNAL_ID
- RESTIC_AWS_ASSUME_ROLE_REGION (defaults to us-east-1)
- RESTIC_AWS_ASSUME_ROLE_POLICY
- RESTIC_AWS_ASSUME_ROLE_STS_ENDPOINT
https://github.com/restic/restic/issues/4472
https://github.com/restic/restic/pull/4474

View File

@@ -1,8 +0,0 @@
Change: Don't retry to load files that don't exist
Restic used to always retry to load files. It now only retries to load
files if they exist.
https://github.com/restic/restic/issues/4515
https://github.com/restic/restic/issues/1523
https://github.com/restic/restic/pull/4520

View File

@@ -1,6 +0,0 @@
Change: Require at least ARMv6 for ARM binaries
The official release binaries of restic now require at least ARMv6 support for ARM platforms.
https://github.com/restic/restic/issues/4540
https://github.com/restic/restic/pull/4542

View File

@@ -1,7 +0,0 @@
Enhancement: Add support for `--json` option to `version` command
Restic now supports outputting restic version and used go version, OS and
architecture via JSON when using the version command.
https://github.com/restic/restic/issues/4547
https://github.com/restic/restic/pull/4553

View File

@@ -1,11 +0,0 @@
Enhancement: Add `--ncdu` option to `ls` command
NCDU (NCurses Disk Usage) is a tool to analyse disk usage of directories.
It has an option to save a directory tree and analyse it later.
The `ls` command now supports the `--ncdu` option which outputs information
about a snapshot in the NCDU format.
You can use it as follows: `restic ls latest --ncdu | ncdu -f -`
https://github.com/restic/restic/issues/4549
https://github.com/restic/restic/pull/4550

View File

@@ -1,12 +0,0 @@
Enhancement: Ignore s3.storage-class for metadata if archive tier is specified
There is no official cold storage support in restic, use this option at your
own risk.
Restic always stored all files on s3 using the specified `s3.storage-class`.
Now, restic will store metadata using a non-archive storage tier to avoid
problems when accessing a repository. To restore any data, it is still
necessary to manually warm up the required data beforehand.
https://github.com/restic/restic/issues/4583
https://github.com/restic/restic/pull/4584

View File

@@ -1,7 +0,0 @@
Bugfix: Properly report the ID of newly added keys
`restic key add` now reports the ID of a newly added key. This simplifies
selecting a specific key using the `--key-hint key` option.
https://github.com/restic/restic/issues/4656
https://github.com/restic/restic/pull/4657

View File

@@ -1,8 +0,0 @@
Enhancement: Move key add, list, remove and passwd as separate sub-commands
Restic now provides usage documentation for the `key` command. Each sub-command;
`add`, `list`, `remove` and `passwd` now have their own sub-command documentation
which can be invoked using `restic key <add|list|remove|passwd> --help`.
https://github.com/restic/restic/issues/4676
https://github.com/restic/restic/pull/4685

View File

@@ -1,8 +0,0 @@
Enhancement: Add --target flag to the dump command
Restic `dump` always printed to the standard output. It now permits to select a
`--target` file to write the output to.
https://github.com/restic/restic/issues/4678
https://github.com/restic/restic/pull/4682
https://github.com/restic/restic/pull/4692

View File

@@ -1,7 +0,0 @@
Bugfix: Correct hardlink handling in `stats` command
If files on different devices had the same inode id, then the `stats` command
did not correctly calculate the snapshot size. This has been fixed.
https://github.com/restic/restic/pull/4503
https://forum.restic.net/t/possible-bug-in-stats/6461/8

View File

@@ -1,11 +0,0 @@
Enhancement: Add bitrot detection to `diff` command
The output of the `diff` command now includes the modifier `?` for files
to indicate bitrot in backed up files. It will appear whenever there is a
difference in content while the metadata is exactly the same. Since files with
unchanged metadata are normally not read again when creating a backup, the
detection is only effective if the right-hand side of the diff has been created
with "backup --force".
https://github.com/restic/restic/issues/805
https://github.com/restic/restic/pull/4526

View File

@@ -1,5 +0,0 @@
Enhancement: Add `--new-host` and `--new-time` options to `rewrite` command
`restic rewrite` now allows rewriting the host and / or time metadata of a snapshot.
https://github.com/restic/restic/pull/4573

View File

@@ -1,7 +0,0 @@
Enhancement: `mount` tests mountpoint existence before opening the repository
The restic `mount` command now checks for the existence of the
mountpoint before opening the repository, leading to quicker error
detection.
https://github.com/restic/restic/pull/4590

View File

@@ -1,6 +0,0 @@
Bugfix: `find` ignored directories in some cases
In some cases, the `find` command ignored empty or moved directories. This has
been fixed.
https://github.com/restic/restic/pull/4615

View File

@@ -1,10 +0,0 @@
Enhancement: Improve `repair packs` command
The `repair packs` command has been improved to also be able to process
truncated pack files. The `check --read-data` command will provide instructions
on using the command if necessary to repair a repository. See the guide at
https://restic.readthedocs.io/en/stable/077_troubleshooting.html for further
instructions.
https://github.com/restic/restic/pull/4644
https://github.com/restic/restic/pull/4655

View File

@@ -12,6 +12,7 @@ import (
"runtime"
"strconv"
"strings"
"sync"
"time"
"github.com/spf13/cobra"
@@ -24,6 +25,7 @@ import (
"github.com/restic/restic/internal/repository"
"github.com/restic/restic/internal/restic"
"github.com/restic/restic/internal/textfile"
"github.com/restic/restic/internal/ui"
"github.com/restic/restic/internal/ui/backup"
"github.com/restic/restic/internal/ui/termstatus"
)
@@ -42,7 +44,7 @@ Exit status is 0 if the command was successful.
Exit status is 1 if there was a fatal error (no snapshot created).
Exit status is 3 if some source data could not be read (incomplete snapshot created).
`,
PreRun: func(_ *cobra.Command, _ []string) {
PreRun: func(cmd *cobra.Command, args []string) {
if backupOptions.Host == "" {
hostname, err := os.Hostname()
if err != nil {
@@ -54,9 +56,31 @@ Exit status is 3 if some source data could not be read (incomplete snapshot crea
},
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, args []string) error {
term, cancel := setupTermstatus()
defer cancel()
return runBackup(cmd.Context(), backupOptions, globalOptions, term, args)
ctx := cmd.Context()
var wg sync.WaitGroup
cancelCtx, cancel := context.WithCancel(ctx)
defer func() {
// shutdown termstatus
cancel()
wg.Wait()
}()
term := termstatus.New(globalOptions.stdout, globalOptions.stderr, globalOptions.Quiet)
wg.Add(1)
go func() {
defer wg.Done()
term.Run(cancelCtx)
}()
// use the terminal for stdout/stderr
prevStdout, prevStderr := globalOptions.stdout, globalOptions.stderr
defer func() {
globalOptions.stdout, globalOptions.stderr = prevStdout, prevStderr
}()
stdioWrapper := ui.NewStdioWrapper(term)
globalOptions.stdout, globalOptions.stderr = stdioWrapper.Stdout(), stdioWrapper.Stderr()
return runBackup(ctx, backupOptions, globalOptions, term, args)
},
}
@@ -73,7 +97,6 @@ type BackupOptions struct {
ExcludeLargerThan string
Stdin bool
StdinFilename string
StdinCommand bool
Tags restic.TagLists
Host string
FilesFrom []string
@@ -111,7 +134,6 @@ func init() {
f.StringVar(&backupOptions.ExcludeLargerThan, "exclude-larger-than", "", "max `size` of the files to be backed up (allowed suffixes: k/K, m/M, g/G, t/T)")
f.BoolVar(&backupOptions.Stdin, "stdin", false, "read backup from stdin")
f.StringVar(&backupOptions.StdinFilename, "stdin-filename", "stdin", "`filename` to use when reading from stdin")
f.BoolVar(&backupOptions.StdinCommand, "stdin-from-command", false, "interpret arguments as command to execute and store its stdout")
f.Var(&backupOptions.Tags, "tag", "add `tags` for the new snapshot in the format `tag[,tag,...]` (can be specified multiple times)")
f.UintVar(&backupOptions.ReadConcurrency, "read-concurrency", 0, "read `n` files concurrently (default: $RESTIC_READ_CONCURRENCY or 2)")
f.StringVarP(&backupOptions.Host, "host", "H", "", "set the `hostname` for the snapshot manually. To prevent an expensive rescan use the \"parent\" flag")
@@ -126,7 +148,7 @@ func init() {
f.StringArrayVar(&backupOptions.FilesFromRaw, "files-from-raw", nil, "read the files to backup from `file` (can be combined with file args; can be specified multiple times)")
f.StringVar(&backupOptions.TimeStamp, "time", "", "`time` of the backup (ex. '2012-11-01 22:08:41') (default: now)")
f.BoolVar(&backupOptions.WithAtime, "with-atime", false, "store the atime for all files and directories")
f.BoolVar(&backupOptions.IgnoreInode, "ignore-inode", false, "ignore inode number and ctime changes when checking for modified files")
f.BoolVar(&backupOptions.IgnoreInode, "ignore-inode", false, "ignore inode number changes when checking for modified files")
f.BoolVar(&backupOptions.IgnoreCtime, "ignore-ctime", false, "ignore ctime changes when checking for modified files")
f.BoolVarP(&backupOptions.DryRun, "dry-run", "n", false, "do not upload or write any data, just show what would be done")
f.BoolVar(&backupOptions.NoScan, "no-scan", false, "do not run scanner to estimate size of backup")
@@ -265,7 +287,7 @@ func (opts BackupOptions) Check(gopts GlobalOptions, args []string) error {
}
}
if opts.Stdin || opts.StdinCommand {
if opts.Stdin {
if len(opts.FilesFrom) > 0 {
return errors.Fatal("--stdin and --files-from cannot be used together")
}
@@ -276,7 +298,7 @@ func (opts BackupOptions) Check(gopts GlobalOptions, args []string) error {
return errors.Fatal("--stdin and --files-from-raw cannot be used together")
}
if len(args) > 0 && !opts.StdinCommand {
if len(args) > 0 {
return errors.Fatal("--stdin was specified and files/dirs were listed as arguments")
}
}
@@ -344,7 +366,7 @@ func collectRejectFuncs(opts BackupOptions, targets []string) (fs []RejectFunc,
// collectTargets returns a list of target files/dirs from several sources.
func collectTargets(opts BackupOptions, args []string) (targets []string, err error) {
if opts.Stdin || opts.StdinCommand {
if opts.Stdin {
return nil, nil
}
@@ -411,7 +433,7 @@ func collectTargets(opts BackupOptions, args []string) (targets []string, err er
// parent returns the ID of the parent snapshot. If there is none, nil is
// returned.
func findParentSnapshot(ctx context.Context, repo restic.ListerLoaderUnpacked, opts BackupOptions, targets []string, timeStampLimit time.Time) (*restic.Snapshot, error) {
func findParentSnapshot(ctx context.Context, repo restic.Repository, opts BackupOptions, targets []string, timeStampLimit time.Time) (*restic.Snapshot, error) {
if opts.Force {
return nil, nil
}
@@ -431,7 +453,7 @@ func findParentSnapshot(ctx context.Context, repo restic.ListerLoaderUnpacked, o
f.Tags = []restic.TagList{opts.Tags.Flatten()}
}
sn, _, err := f.FindLatest(ctx, repo, repo, snName)
sn, _, err := f.FindLatest(ctx, repo.Backend(), repo, snName)
// Snapshot not found is ok if no explicit parent was set
if opts.Parent == "" && errors.Is(err, restic.ErrNoSnapshotFound) {
err = nil
@@ -570,24 +592,16 @@ func runBackup(ctx context.Context, opts BackupOptions, gopts GlobalOptions, ter
defer localVss.DeleteSnapshots()
targetFS = localVss
}
if opts.Stdin || opts.StdinCommand {
if opts.Stdin {
if !gopts.JSON {
progressPrinter.V("read data from stdin")
}
filename := path.Join("/", opts.StdinFilename)
var source io.ReadCloser = os.Stdin
if opts.StdinCommand {
source, err = fs.NewCommandReader(ctx, args, globalOptions.stderr)
if err != nil {
return err
}
}
targetFS = &fs.Reader{
ModTime: timeStamp,
Name: filename,
Mode: 0644,
ReadCloser: source,
ReadCloser: os.Stdin,
}
targets = []string{filename}
}
@@ -609,20 +623,14 @@ func runBackup(ctx context.Context, opts BackupOptions, gopts GlobalOptions, ter
wg.Go(func() error { return sc.Scan(cancelCtx, targets) })
}
arch := archiver.New(repo, targetFS, archiver.Options{ReadConcurrency: opts.ReadConcurrency})
arch := archiver.New(repo, targetFS, archiver.Options{ReadConcurrency: backupOptions.ReadConcurrency})
arch.SelectByName = selectByNameFilter
arch.Select = selectFilter
arch.WithAtime = opts.WithAtime
success := true
arch.Error = func(item string, err error) error {
success = false
reterr := progressReporter.Error(item, err)
// If we receive a fatal error during the execution of the snapshot,
// we abort the snapshot.
if reterr == nil && errors.IsFatal(err) {
reterr = err
}
return reterr
return progressReporter.Error(item, err)
}
arch.CompleteItem = progressReporter.CompleteItem
arch.StartFile = progressReporter.StartFile

View File

@@ -9,7 +9,6 @@ import (
"runtime"
"testing"
"github.com/restic/restic/internal/backend"
"github.com/restic/restic/internal/fs"
"github.com/restic/restic/internal/restic"
rtest "github.com/restic/restic/internal/test"
@@ -266,7 +265,7 @@ func TestBackupTreeLoadError(t *testing.T) {
// delete the subdirectory pack first
for id := range treePacks {
rtest.OK(t, r.Backend().Remove(context.TODO(), backend.Handle{Type: restic.PackFile, Name: id.String()}))
rtest.OK(t, r.Backend().Remove(context.TODO(), restic.Handle{Type: restic.PackFile, Name: id.String()}))
}
testRunRebuildIndex(t, env.gopts)
// now the repo is missing the tree blob in the index; check should report this
@@ -568,72 +567,3 @@ func linkEqual(source, dest []string) bool {
return true
}
func TestStdinFromCommand(t *testing.T) {
env, cleanup := withTestEnvironment(t)
defer cleanup()
testSetupBackupData(t, env)
opts := BackupOptions{
StdinCommand: true,
StdinFilename: "stdin",
}
testRunBackup(t, filepath.Dir(env.testdata), []string{"python", "-c", "import sys; print('something'); sys.exit(0)"}, opts, env.gopts)
testListSnapshots(t, env.gopts, 1)
testRunCheck(t, env.gopts)
}
func TestStdinFromCommandNoOutput(t *testing.T) {
env, cleanup := withTestEnvironment(t)
defer cleanup()
testSetupBackupData(t, env)
opts := BackupOptions{
StdinCommand: true,
StdinFilename: "stdin",
}
err := testRunBackupAssumeFailure(t, filepath.Dir(env.testdata), []string{"python", "-c", "import sys; sys.exit(0)"}, opts, env.gopts)
rtest.Assert(t, err != nil && err.Error() == "at least one source file could not be read", "No data error expected")
testListSnapshots(t, env.gopts, 1)
testRunCheck(t, env.gopts)
}
func TestStdinFromCommandFailExitCode(t *testing.T) {
env, cleanup := withTestEnvironment(t)
defer cleanup()
testSetupBackupData(t, env)
opts := BackupOptions{
StdinCommand: true,
StdinFilename: "stdin",
}
err := testRunBackupAssumeFailure(t, filepath.Dir(env.testdata), []string{"python", "-c", "import sys; print('test'); sys.exit(1)"}, opts, env.gopts)
rtest.Assert(t, err != nil, "Expected error while backing up")
testListSnapshots(t, env.gopts, 0)
testRunCheck(t, env.gopts)
}
func TestStdinFromCommandFailNoOutputAndExitCode(t *testing.T) {
env, cleanup := withTestEnvironment(t)
defer cleanup()
testSetupBackupData(t, env)
opts := BackupOptions{
StdinCommand: true,
StdinFilename: "stdin",
}
err := testRunBackupAssumeFailure(t, filepath.Dir(env.testdata), []string{"python", "-c", "import sys; sys.exit(1)"}, opts, env.gopts)
rtest.Assert(t, err != nil, "Expected error while backing up")
testListSnapshots(t, env.gopts, 0)
testRunCheck(t, env.gopts)
}

View File

@@ -28,7 +28,7 @@ EXIT STATUS
Exit status is 0 if the command was successful, and non-zero if there was any error.
`,
DisableAutoGenTag: true,
RunE: func(_ *cobra.Command, args []string) error {
RunE: func(cmd *cobra.Command, args []string) error {
return runCache(cacheOptions, globalOptions, args)
},
}

View File

@@ -106,7 +106,7 @@ func runCat(ctx context.Context, gopts GlobalOptions, args []string) error {
Println(string(buf))
return nil
case "snapshot":
sn, _, err := restic.FindSnapshot(ctx, repo, repo, args[1])
sn, _, err := restic.FindSnapshot(ctx, repo.Backend(), repo, args[1])
if err != nil {
return errors.Fatalf("could not find snapshot: %v\n", err)
}
@@ -154,7 +154,7 @@ func runCat(ctx context.Context, gopts GlobalOptions, args []string) error {
return nil
case "pack":
h := backend.Handle{Type: restic.PackFile, Name: id.String()}
h := restic.Handle{Type: restic.PackFile, Name: id.String()}
buf, err := backend.LoadAll(ctx, nil, repo.Backend(), h)
if err != nil {
return err
@@ -193,7 +193,7 @@ func runCat(ctx context.Context, gopts GlobalOptions, args []string) error {
return errors.Fatal("blob not found")
case "tree":
sn, subfolder, err := restic.FindSnapshot(ctx, repo, repo, args[1])
sn, subfolder, err := restic.FindSnapshot(ctx, repo.Backend(), repo, args[1])
if err != nil {
return errors.Fatalf("could not find snapshot: %v\n", err)
}

View File

@@ -38,7 +38,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
RunE: func(cmd *cobra.Command, args []string) error {
return runCheck(cmd.Context(), checkOptions, globalOptions, args)
},
PreRunE: func(_ *cobra.Command, _ []string) error {
PreRunE: func(cmd *cobra.Command, args []string) error {
return checkFlags(checkOptions)
},
}
@@ -336,18 +336,20 @@ func runCheck(ctx context.Context, opts CheckOptions, gopts GlobalOptions, args
errorsFound = true
Warnf("%v\n", err)
if err, ok := err.(*checker.ErrPackData); ok {
salvagePacks = append(salvagePacks, err.PackID)
if strings.Contains(err.Error(), "wrong data returned, hash is") {
salvagePacks = append(salvagePacks, err.PackID)
}
}
}
p.Done()
if len(salvagePacks) > 0 {
Warnf("\nThe repository contains pack files with damaged blobs. These blobs must be removed to repair the repository. This can be done using the following commands. Please read the troubleshooting guide at https://restic.readthedocs.io/en/stable/077_troubleshooting.html first.\n\n")
var strIDs []string
Warnf("\nThe repository contains pack files with damaged blobs. These blobs must be removed to repair the repository. This can be done using the following commands:\n\n")
var strIds []string
for _, id := range salvagePacks {
strIDs = append(strIDs, id.String())
strIds = append(strIds, id.String())
}
Warnf("restic repair packs %v\nrestic repair snapshots --forget\n\n", strings.Join(strIDs, " "))
Warnf("RESTIC_FEATURES=repair-packs-v1 restic repair packs %v\nrestic repair snapshots --forget\n\n", strings.Join(strIds, " "))
Warnf("Corrupted blobs are either caused by hardware problems or bugs in restic. Please open an issue at https://github.com/restic/restic/issues/new/choose for further troubleshooting!\n")
}
}
@@ -415,7 +417,7 @@ func selectPacksByBucket(allPacks map[restic.ID]int64, bucket, totalBuckets uint
return packs
}
// selectRandomPacksByPercentage selects the given percentage of packs which are randomly chosen.
// selectRandomPacksByPercentage selects the given percentage of packs which are randomly choosen.
func selectRandomPacksByPercentage(allPacks map[restic.ID]int64, percentage float64) map[restic.ID]int64 {
packCount := len(allPacks)
packsToCheck := int(float64(packCount) * (percentage / 100.0))

View File

@@ -71,7 +71,7 @@ func TestSelectPacksByBucket(t *testing.T) {
var testPacks = make(map[restic.ID]int64)
for i := 1; i <= 10; i++ {
id := restic.NewRandomID()
// ensure relevant part of generated id is reproducible
// ensure relevant part of generated id is reproducable
id[0] = byte(i)
testPacks[id] = 0
}
@@ -124,7 +124,7 @@ func TestSelectRandomPacksByPercentage(t *testing.T) {
}
func TestSelectNoRandomPacksByPercentage(t *testing.T) {
// that the repository without pack files works
// that the a repository without pack files works
var testPacks = make(map[restic.ID]int64)
selectedPacks := selectRandomPacksByPercentage(testPacks, 10.0)
rtest.Assert(t, len(selectedPacks) == 0, "Expected 0 selected packs")
@@ -158,7 +158,7 @@ func TestSelectRandomPacksByFileSize(t *testing.T) {
}
func TestSelectNoRandomPacksByFileSize(t *testing.T) {
// that the repository without pack files works
// that the a repository without pack files works
var testPacks = make(map[restic.ID]int64)
selectedPacks := selectRandomPacksByFileSize(testPacks, 10, 500)
rtest.Assert(t, len(selectedPacks) == 0, "Expected 0 selected packs")

View File

@@ -4,6 +4,7 @@ import (
"context"
"fmt"
"github.com/restic/restic/internal/backend"
"github.com/restic/restic/internal/debug"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/repository"
@@ -87,12 +88,12 @@ func runCopy(ctx context.Context, opts CopyOptions, gopts GlobalOptions, args []
return err
}
srcSnapshotLister, err := restic.MemorizeList(ctx, srcRepo, restic.SnapshotFile)
srcSnapshotLister, err := backend.MemorizeList(ctx, srcRepo.Backend(), restic.SnapshotFile)
if err != nil {
return err
}
dstSnapshotLister, err := restic.MemorizeList(ctx, dstRepo, restic.SnapshotFile)
dstSnapshotLister, err := backend.MemorizeList(ctx, dstRepo.Backend(), restic.SnapshotFile)
if err != nil {
return err
}
@@ -126,12 +127,11 @@ func runCopy(ctx context.Context, opts CopyOptions, gopts GlobalOptions, args []
if sn.Original != nil {
srcOriginal = *sn.Original
}
if originalSns, ok := dstSnapshotByOriginal[srcOriginal]; ok {
isCopy := false
for _, originalSn := range originalSns {
if similarSnapshots(originalSn, sn) {
Verboseff("\n%v\n", sn)
Verboseff("\nsnapshot %s of %v at %s)\n", sn.ID().Str(), sn.Paths, sn.Time)
Verboseff("skipping source snapshot %s, was already copied to snapshot %s\n", sn.ID().Str(), originalSn.ID().Str())
isCopy = true
break
@@ -141,7 +141,7 @@ func runCopy(ctx context.Context, opts CopyOptions, gopts GlobalOptions, args []
continue
}
}
Verbosef("\n%v\n", sn)
Verbosef("\nsnapshot %s of %v at %s)\n", sn.ID().Str(), sn.Paths, sn.Time)
Verbosef(" copy started, this may take a while...\n")
if err := copyTree(ctx, srcRepo, dstRepo, visitedTrees, *sn.Tree, gopts.Quiet); err != nil {
return err

View File

@@ -52,23 +52,19 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
},
}
type DebugExamineOptions struct {
TryRepair bool
RepairByte bool
ExtractPack bool
ReuploadBlobs bool
}
var debugExamineOpts DebugExamineOptions
var tryRepair bool
var repairByte bool
var extractPack bool
var reuploadBlobs bool
func init() {
cmdRoot.AddCommand(cmdDebug)
cmdDebug.AddCommand(cmdDebugDump)
cmdDebug.AddCommand(cmdDebugExamine)
cmdDebugExamine.Flags().BoolVar(&debugExamineOpts.ExtractPack, "extract-pack", false, "write blobs to the current directory")
cmdDebugExamine.Flags().BoolVar(&debugExamineOpts.ReuploadBlobs, "reupload-blobs", false, "reupload blobs to the repository")
cmdDebugExamine.Flags().BoolVar(&debugExamineOpts.TryRepair, "try-repair", false, "try to repair broken blobs with single bit flips")
cmdDebugExamine.Flags().BoolVar(&debugExamineOpts.RepairByte, "repair-byte", false, "try to repair broken blobs by trying bytes")
cmdDebugExamine.Flags().BoolVar(&extractPack, "extract-pack", false, "write blobs to the current directory")
cmdDebugExamine.Flags().BoolVar(&reuploadBlobs, "reupload-blobs", false, "reupload blobs to the repository")
cmdDebugExamine.Flags().BoolVar(&tryRepair, "try-repair", false, "try to repair broken blobs with single bit flips")
cmdDebugExamine.Flags().BoolVar(&repairByte, "repair-byte", false, "try to repair broken blobs by trying bytes")
}
func prettyPrintJSON(wr io.Writer, item interface{}) error {
@@ -82,7 +78,7 @@ func prettyPrintJSON(wr io.Writer, item interface{}) error {
}
func debugPrintSnapshots(ctx context.Context, repo *repository.Repository, wr io.Writer) error {
return restic.ForAllSnapshots(ctx, repo, repo, nil, func(id restic.ID, snapshot *restic.Snapshot, err error) error {
return restic.ForAllSnapshots(ctx, repo.Backend(), repo, nil, func(id restic.ID, snapshot *restic.Snapshot, err error) error {
if err != nil {
return err
}
@@ -111,7 +107,7 @@ type Blob struct {
func printPacks(ctx context.Context, repo *repository.Repository, wr io.Writer) error {
var m sync.Mutex
return restic.ParallelList(ctx, repo, restic.PackFile, repo.Connections(), func(ctx context.Context, id restic.ID, size int64) error {
return restic.ParallelList(ctx, repo.Backend(), restic.PackFile, repo.Connections(), func(ctx context.Context, id restic.ID, size int64) error {
blobs, _, err := repo.ListPack(ctx, id, size)
if err != nil {
Warnf("error for pack %v: %v\n", id.Str(), err)
@@ -137,8 +133,8 @@ func printPacks(ctx context.Context, repo *repository.Repository, wr io.Writer)
})
}
func dumpIndexes(ctx context.Context, repo restic.ListerLoaderUnpacked, wr io.Writer) error {
return index.ForAllIndexes(ctx, repo, repo, func(id restic.ID, idx *index.Index, oldFormat bool, err error) error {
func dumpIndexes(ctx context.Context, repo restic.Repository, wr io.Writer) error {
return index.ForAllIndexes(ctx, repo.Backend(), repo, func(id restic.ID, idx *index.Index, oldFormat bool, err error) error {
Printf("index_id: %v\n", id)
if err != nil {
return err
@@ -200,7 +196,7 @@ var cmdDebugExamine = &cobra.Command{
Short: "Examine a pack file",
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, args []string) error {
return runDebugExamine(cmd.Context(), globalOptions, debugExamineOpts, args)
return runDebugExamine(cmd.Context(), globalOptions, args)
},
}
@@ -294,7 +290,7 @@ func tryRepairWithBitflip(ctx context.Context, key *crypto.Key, input []byte, by
})
err := wg.Wait()
if err != nil {
panic("all go routines can only return nil")
panic("all go rountines can only return nil")
}
if !found {
@@ -319,20 +315,20 @@ func decryptUnsigned(ctx context.Context, k *crypto.Key, buf []byte) []byte {
return out
}
func loadBlobs(ctx context.Context, opts DebugExamineOptions, repo restic.Repository, packID restic.ID, list []restic.Blob) error {
func loadBlobs(ctx context.Context, repo restic.Repository, packID restic.ID, list []restic.Blob) error {
dec, err := zstd.NewReader(nil)
if err != nil {
panic(err)
}
be := repo.Backend()
h := backend.Handle{
h := restic.Handle{
Name: packID.String(),
Type: restic.PackFile,
}
wg, ctx := errgroup.WithContext(ctx)
if opts.ReuploadBlobs {
if reuploadBlobs {
repo.StartPackUploader(ctx, wg)
}
@@ -360,8 +356,8 @@ func loadBlobs(ctx context.Context, opts DebugExamineOptions, repo restic.Reposi
filePrefix := ""
if err != nil {
Warnf("error decrypting blob: %v\n", err)
if opts.TryRepair || opts.RepairByte {
plaintext = tryRepairWithBitflip(ctx, key, buf, opts.RepairByte)
if tryRepair || repairByte {
plaintext = tryRepairWithBitflip(ctx, key, buf, repairByte)
}
if plaintext != nil {
outputPrefix = "repaired "
@@ -395,13 +391,13 @@ func loadBlobs(ctx context.Context, opts DebugExamineOptions, repo restic.Reposi
Printf(" successfully %vdecrypted blob (length %v), hash is %v, ID matches\n", outputPrefix, len(plaintext), id)
prefix = "correct-"
}
if opts.ExtractPack {
if extractPack {
err = storePlainBlob(id, filePrefix+prefix, plaintext)
if err != nil {
return err
}
}
if opts.ReuploadBlobs {
if reuploadBlobs {
_, _, _, err := repo.SaveBlob(ctx, blob.Type, plaintext, id, true)
if err != nil {
return err
@@ -410,7 +406,7 @@ func loadBlobs(ctx context.Context, opts DebugExamineOptions, repo restic.Reposi
}
}
if opts.ReuploadBlobs {
if reuploadBlobs {
return repo.Flush(ctx)
}
return nil
@@ -441,7 +437,7 @@ func storePlainBlob(id restic.ID, prefix string, plain []byte) error {
return nil
}
func runDebugExamine(ctx context.Context, gopts GlobalOptions, opts DebugExamineOptions, args []string) error {
func runDebugExamine(ctx context.Context, gopts GlobalOptions, args []string) error {
repo, err := OpenRepository(ctx, gopts)
if err != nil {
return err
@@ -451,7 +447,7 @@ func runDebugExamine(ctx context.Context, gopts GlobalOptions, opts DebugExamine
for _, name := range args {
id, err := restic.ParseID(name)
if err != nil {
id, err = restic.Find(ctx, repo, restic.PackFile, name)
id, err = restic.Find(ctx, repo.Backend(), restic.PackFile, name)
if err != nil {
Warnf("error: %v\n", err)
continue
@@ -480,7 +476,7 @@ func runDebugExamine(ctx context.Context, gopts GlobalOptions, opts DebugExamine
}
for _, id := range ids {
err := examinePack(ctx, opts, repo, id)
err := examinePack(ctx, repo, id)
if err != nil {
Warnf("error: %v\n", err)
}
@@ -491,10 +487,10 @@ func runDebugExamine(ctx context.Context, gopts GlobalOptions, opts DebugExamine
return nil
}
func examinePack(ctx context.Context, opts DebugExamineOptions, repo restic.Repository, id restic.ID) error {
func examinePack(ctx context.Context, repo restic.Repository, id restic.ID) error {
Printf("examine %v\n", id)
h := backend.Handle{
h := restic.Handle{
Type: restic.PackFile,
Name: id.String(),
}
@@ -528,7 +524,7 @@ func examinePack(ctx context.Context, opts DebugExamineOptions, repo restic.Repo
checkPackSize(blobs, fi.Size)
err = loadBlobs(ctx, opts, repo, id, blobs)
err = loadBlobs(ctx, repo, id, blobs)
if err != nil {
Warnf("error: %v\n", err)
} else {
@@ -546,7 +542,7 @@ func examinePack(ctx context.Context, opts DebugExamineOptions, repo restic.Repo
checkPackSize(blobs, fi.Size)
if !blobsLoaded {
return loadBlobs(ctx, opts, repo, id, blobs)
return loadBlobs(ctx, repo, id, blobs)
}
return nil
}

View File

@@ -7,6 +7,7 @@ import (
"reflect"
"sort"
"github.com/restic/restic/internal/backend"
"github.com/restic/restic/internal/debug"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/restic"
@@ -27,10 +28,6 @@ directory:
* U The metadata (access mode, timestamps, ...) for the item was updated
* M The file's content was modified
* T The type was changed, e.g. a file was made a symlink
* ? Bitrot detected: The file's content has changed but all metadata is the same
Metadata comparison will likely not work if a backup was created using the
'--ignore-inode' or '--ignore-ctime' option.
To only compare files in specific subfolders, you can use the
"<snapshotID>:<subfolder>" syntax, where "subfolder" is a path within the
@@ -61,7 +58,7 @@ func init() {
f.BoolVar(&diffOptions.ShowMetadata, "metadata", false, "print changes in metadata")
}
func loadSnapshot(ctx context.Context, be restic.Lister, repo restic.LoaderUnpacked, desc string) (*restic.Snapshot, string, error) {
func loadSnapshot(ctx context.Context, be restic.Lister, repo restic.Repository, desc string) (*restic.Snapshot, string, error) {
sn, subfolder, err := restic.FindSnapshot(ctx, be, repo, desc)
if err != nil {
return nil, "", errors.Fatal(err.Error())
@@ -71,7 +68,7 @@ func loadSnapshot(ctx context.Context, be restic.Lister, repo restic.LoaderUnpac
// Comparer collects all things needed to compare two snapshots.
type Comparer struct {
repo restic.BlobLoader
repo restic.Repository
opts DiffOptions
printChange func(change *Change)
}
@@ -147,7 +144,7 @@ type DiffStatsContainer struct {
}
// updateBlobs updates the blob counters in the stats struct.
func updateBlobs(repo restic.Loader, blobs restic.BlobSet, stats *DiffStat) {
func updateBlobs(repo restic.Repository, blobs restic.BlobSet, stats *DiffStat) {
for h := range blobs {
switch h.Type {
case restic.DataBlob:
@@ -276,16 +273,6 @@ func (c *Comparer) diffTree(ctx context.Context, stats *DiffStatsContainer, pref
!reflect.DeepEqual(node1.Content, node2.Content) {
mod += "M"
stats.ChangedFiles++
node1NilContent := *node1
node2NilContent := *node2
node1NilContent.Content = nil
node2NilContent.Content = nil
// the bitrot detection may not work if `backup --ignore-inode` or `--ignore-ctime` were used
if node1NilContent.Equals(node2NilContent) {
// probable bitrot detected
mod += "?"
}
} else if c.opts.ShowMetadata && !node1.Equals(*node2) {
mod += "U"
}
@@ -359,7 +346,7 @@ func runDiff(ctx context.Context, opts DiffOptions, gopts GlobalOptions, args []
}
// cache snapshots listing
be, err := restic.MemorizeList(ctx, repo, restic.SnapshotFile)
be, err := backend.MemorizeList(ctx, repo.Backend(), restic.SnapshotFile)
if err != nil {
return err
}
@@ -401,7 +388,7 @@ func runDiff(ctx context.Context, opts DiffOptions, gopts GlobalOptions, args []
c := &Comparer{
repo: repo,
opts: opts,
opts: diffOptions,
printChange: func(change *Change) {
Printf("%-5s%v\n", change.Modifier, change.Path)
},
@@ -418,7 +405,7 @@ func runDiff(ctx context.Context, opts DiffOptions, gopts GlobalOptions, args []
}
if gopts.Quiet {
c.printChange = func(_ *Change) {}
c.printChange = func(change *Change) {}
}
stats := &DiffStatsContainer{

View File

@@ -46,7 +46,6 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
type DumpOptions struct {
restic.SnapshotFilter
Archive string
Target string
}
var dumpOptions DumpOptions
@@ -57,7 +56,6 @@ func init() {
flags := cmdDump.Flags()
initSingleSnapshotFilter(flags, &dumpOptions.SnapshotFilter)
flags.StringVarP(&dumpOptions.Archive, "archive", "a", "tar", "set archive `format` as \"tar\" or \"zip\"")
flags.StringVarP(&dumpOptions.Target, "target", "t", "", "write the output to target `path`")
}
func splitPath(p string) []string {
@@ -69,11 +67,11 @@ func splitPath(p string) []string {
return append(s, f)
}
func printFromTree(ctx context.Context, tree *restic.Tree, repo restic.BlobLoader, prefix string, pathComponents []string, d *dump.Dumper, canWriteArchiveFunc func() error) error {
func printFromTree(ctx context.Context, tree *restic.Tree, repo restic.Repository, prefix string, pathComponents []string, d *dump.Dumper) error {
// If we print / we need to assume that there are multiple nodes at that
// level in the tree.
if pathComponents[0] == "" {
if err := canWriteArchiveFunc(); err != nil {
if err := checkStdoutArchive(); err != nil {
return err
}
return d.DumpTree(ctx, tree, "/")
@@ -93,9 +91,9 @@ func printFromTree(ctx context.Context, tree *restic.Tree, repo restic.BlobLoade
if err != nil {
return errors.Wrapf(err, "cannot load subtree for %q", item)
}
return printFromTree(ctx, subtree, repo, item, pathComponents[1:], d, canWriteArchiveFunc)
return printFromTree(ctx, subtree, repo, item, pathComponents[1:], d)
case dump.IsDir(node):
if err := canWriteArchiveFunc(); err != nil {
if err := checkStdoutArchive(); err != nil {
return err
}
subtree, err := restic.LoadTree(ctx, repo, *node.Subtree)
@@ -149,7 +147,7 @@ func runDump(ctx context.Context, opts DumpOptions, gopts GlobalOptions, args []
Hosts: opts.Hosts,
Paths: opts.Paths,
Tags: opts.Tags,
}).FindLatest(ctx, repo, repo, snapshotIDString)
}).FindLatest(ctx, repo.Backend(), repo, snapshotIDString)
if err != nil {
return errors.Fatalf("failed to find snapshot: %v", err)
}
@@ -170,24 +168,8 @@ func runDump(ctx context.Context, opts DumpOptions, gopts GlobalOptions, args []
return errors.Fatalf("loading tree for snapshot %q failed: %v", snapshotIDString, err)
}
outputFileWriter := os.Stdout
canWriteArchiveFunc := checkStdoutArchive
if opts.Target != "" {
file, err := os.Create(opts.Target)
if err != nil {
return fmt.Errorf("cannot dump to file: %w", err)
}
defer func() {
_ = file.Close()
}()
outputFileWriter = file
canWriteArchiveFunc = func() error { return nil }
}
d := dump.New(opts.Archive, repo, outputFileWriter)
err = printFromTree(ctx, tree, repo, "/", splittedPath, d, canWriteArchiveFunc)
d := dump.New(opts.Archive, repo, os.Stdout)
err = printFromTree(ctx, tree, repo, "/", splittedPath, d)
if err != nil {
return errors.Fatalf("cannot dump file: %v", err)
}

View File

@@ -9,6 +9,7 @@ import (
"github.com/spf13/cobra"
"github.com/restic/restic/internal/backend"
"github.com/restic/restic/internal/debug"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/filter"
@@ -244,12 +245,13 @@ func (s *statefulOutput) Finish() {
// Finder bundles information needed to find a file or directory.
type Finder struct {
repo restic.Repository
pat findPattern
out statefulOutput
blobIDs map[string]struct{}
treeIDs map[string]struct{}
itemsFound int
repo restic.Repository
pat findPattern
out statefulOutput
ignoreTrees restic.IDSet
blobIDs map[string]struct{}
treeIDs map[string]struct{}
itemsFound int
}
func (f *Finder) findInSnapshot(ctx context.Context, sn *restic.Snapshot) error {
@@ -260,17 +262,17 @@ func (f *Finder) findInSnapshot(ctx context.Context, sn *restic.Snapshot) error
}
f.out.newsn = sn
return walker.Walk(ctx, f.repo, *sn.Tree, walker.WalkVisitor{ProcessNode: func(parentTreeID restic.ID, nodepath string, node *restic.Node, err error) error {
return walker.Walk(ctx, f.repo, *sn.Tree, f.ignoreTrees, func(parentTreeID restic.ID, nodepath string, node *restic.Node, err error) (bool, error) {
if err != nil {
debug.Log("Error loading tree %v: %v", parentTreeID, err)
Printf("Unable to load tree %s\n ... which belongs to snapshot %s\n", parentTreeID, sn.ID())
return walker.ErrSkipNode
return false, walker.ErrSkipNode
}
if node == nil {
return nil
return false, nil
}
normalizedNodepath := nodepath
@@ -283,7 +285,7 @@ func (f *Finder) findInSnapshot(ctx context.Context, sn *restic.Snapshot) error
for _, pat := range f.pat.pattern {
found, err := filter.Match(pat, normalizedNodepath)
if err != nil {
return err
return false, err
}
if found {
foundMatch = true
@@ -291,13 +293,16 @@ func (f *Finder) findInSnapshot(ctx context.Context, sn *restic.Snapshot) error
}
}
var errIfNoMatch error
var (
ignoreIfNoMatch = true
errIfNoMatch error
)
if node.Type == "dir" {
var childMayMatch bool
for _, pat := range f.pat.pattern {
mayMatch, err := filter.ChildMatch(pat, normalizedNodepath)
if err != nil {
return err
return false, err
}
if mayMatch {
childMayMatch = true
@@ -306,28 +311,31 @@ func (f *Finder) findInSnapshot(ctx context.Context, sn *restic.Snapshot) error
}
if !childMayMatch {
ignoreIfNoMatch = true
errIfNoMatch = walker.ErrSkipNode
} else {
ignoreIfNoMatch = false
}
}
if !foundMatch {
return errIfNoMatch
return ignoreIfNoMatch, errIfNoMatch
}
if !f.pat.oldest.IsZero() && node.ModTime.Before(f.pat.oldest) {
debug.Log(" ModTime is older than %s\n", f.pat.oldest)
return errIfNoMatch
return ignoreIfNoMatch, errIfNoMatch
}
if !f.pat.newest.IsZero() && node.ModTime.After(f.pat.newest) {
debug.Log(" ModTime is newer than %s\n", f.pat.newest)
return errIfNoMatch
return ignoreIfNoMatch, errIfNoMatch
}
debug.Log(" found match\n")
f.out.PrintPattern(nodepath, node)
return nil
}})
return false, nil
})
}
func (f *Finder) findIDs(ctx context.Context, sn *restic.Snapshot) error {
@@ -338,17 +346,17 @@ func (f *Finder) findIDs(ctx context.Context, sn *restic.Snapshot) error {
}
f.out.newsn = sn
return walker.Walk(ctx, f.repo, *sn.Tree, walker.WalkVisitor{ProcessNode: func(parentTreeID restic.ID, nodepath string, node *restic.Node, err error) error {
return walker.Walk(ctx, f.repo, *sn.Tree, f.ignoreTrees, func(parentTreeID restic.ID, nodepath string, node *restic.Node, err error) (bool, error) {
if err != nil {
debug.Log("Error loading tree %v: %v", parentTreeID, err)
Printf("Unable to load tree %s\n ... which belongs to snapshot %s\n", parentTreeID, sn.ID())
return walker.ErrSkipNode
return false, walker.ErrSkipNode
}
if node == nil {
return nil
return false, nil
}
if node.Type == "dir" && f.treeIDs != nil {
@@ -366,7 +374,7 @@ func (f *Finder) findIDs(ctx context.Context, sn *restic.Snapshot) error {
// looking for blobs)
if f.itemsFound >= len(f.treeIDs) && f.blobIDs == nil {
// Return an error to terminate the Walk
return errors.New("OK")
return true, errors.New("OK")
}
}
}
@@ -387,8 +395,8 @@ func (f *Finder) findIDs(ctx context.Context, sn *restic.Snapshot) error {
}
}
return nil
}})
return false, nil
})
}
var errAllPacksFound = errors.New("all packs found")
@@ -576,7 +584,7 @@ func runFind(ctx context.Context, opts FindOptions, gopts GlobalOptions, args []
}
}
snapshotLister, err := restic.MemorizeList(ctx, repo, restic.SnapshotFile)
snapshotLister, err := backend.MemorizeList(ctx, repo.Backend(), restic.SnapshotFile)
if err != nil {
return err
}
@@ -586,9 +594,10 @@ func runFind(ctx context.Context, opts FindOptions, gopts GlobalOptions, args []
}
f := &Finder{
repo: repo,
pat: pat,
out: statefulOutput{ListLong: opts.ListLong, HumanReadable: opts.HumanReadable, JSON: gopts.JSON},
repo: repo,
pat: pat,
out: statefulOutput{ListLong: opts.ListLong, HumanReadable: opts.HumanReadable, JSON: gopts.JSON},
ignoreTrees: restic.NewIDSet(),
}
if opts.BlobID {

View File

@@ -33,7 +33,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
`,
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, args []string) error {
return runForget(cmd.Context(), forgetOptions, forgetPruneOptions, globalOptions, args)
return runForget(cmd.Context(), forgetOptions, globalOptions, args)
},
}
@@ -98,7 +98,6 @@ type ForgetOptions struct {
}
var forgetOptions ForgetOptions
var forgetPruneOptions PruneOptions
func init() {
cmdRoot.AddCommand(cmdForget)
@@ -133,7 +132,7 @@ func init() {
f.BoolVar(&forgetOptions.Prune, "prune", false, "automatically run the 'prune' command if snapshots have been removed")
f.SortFlags = false
addPruneOptions(cmdForget, &forgetPruneOptions)
addPruneOptions(cmdForget)
}
func verifyForgetOptions(opts *ForgetOptions) error {
@@ -152,7 +151,7 @@ func verifyForgetOptions(opts *ForgetOptions) error {
return nil
}
func runForget(ctx context.Context, opts ForgetOptions, pruneOptions PruneOptions, gopts GlobalOptions, args []string) error {
func runForget(ctx context.Context, opts ForgetOptions, gopts GlobalOptions, args []string) error {
err := verifyForgetOptions(&opts)
if err != nil {
return err
@@ -184,7 +183,7 @@ func runForget(ctx context.Context, opts ForgetOptions, pruneOptions PruneOption
var snapshots restic.Snapshots
removeSnIDs := restic.NewIDSet()
for sn := range FindFilteredSnapshots(ctx, repo, repo, &opts.SnapshotFilter, args) {
for sn := range FindFilteredSnapshots(ctx, repo.Backend(), repo, &opts.SnapshotFilter, args) {
snapshots = append(snapshots, sn)
}

View File

@@ -9,8 +9,5 @@ import (
func testRunForget(t testing.TB, gopts GlobalOptions, args ...string) {
opts := ForgetOptions{}
pruneOpts := PruneOptions{
MaxUnused: "5%",
}
rtest.OK(t, runForget(context.TODO(), opts, pruneOpts, gopts, args))
rtest.OK(t, runForget(context.TODO(), opts, gopts, args))
}

View File

@@ -21,9 +21,7 @@ EXIT STATUS
Exit status is 0 if the command was successful, and non-zero if there was any error.
`,
DisableAutoGenTag: true,
RunE: func(_ *cobra.Command, args []string) error {
return runGenerate(genOpts, args)
},
RunE: runGenerate,
}
type generateOptions struct {
@@ -92,48 +90,48 @@ func writePowerShellCompletion(file string) error {
return cmdRoot.GenPowerShellCompletionFile(file)
}
func runGenerate(opts generateOptions, args []string) error {
func runGenerate(_ *cobra.Command, args []string) error {
if len(args) > 0 {
return errors.Fatal("the generate command expects no arguments, only options - please see `restic help generate` for usage and flags")
}
if opts.ManDir != "" {
err := writeManpages(opts.ManDir)
if genOpts.ManDir != "" {
err := writeManpages(genOpts.ManDir)
if err != nil {
return err
}
}
if opts.BashCompletionFile != "" {
err := writeBashCompletion(opts.BashCompletionFile)
if genOpts.BashCompletionFile != "" {
err := writeBashCompletion(genOpts.BashCompletionFile)
if err != nil {
return err
}
}
if opts.FishCompletionFile != "" {
err := writeFishCompletion(opts.FishCompletionFile)
if genOpts.FishCompletionFile != "" {
err := writeFishCompletion(genOpts.FishCompletionFile)
if err != nil {
return err
}
}
if opts.ZSHCompletionFile != "" {
err := writeZSHCompletion(opts.ZSHCompletionFile)
if genOpts.ZSHCompletionFile != "" {
err := writeZSHCompletion(genOpts.ZSHCompletionFile)
if err != nil {
return err
}
}
if opts.PowerShellCompletionFile != "" {
err := writePowerShellCompletion(opts.PowerShellCompletionFile)
if genOpts.PowerShellCompletionFile != "" {
err := writePowerShellCompletion(genOpts.PowerShellCompletionFile)
if err != nil {
return err
}
}
var empty generateOptions
if opts == empty {
if genOpts == empty {
return errors.Fatal("nothing to do, please specify at least one output file/dir")
}

View File

@@ -1,18 +1,265 @@
package main
import (
"context"
"encoding/json"
"os"
"strings"
"sync"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/repository"
"github.com/restic/restic/internal/restic"
"github.com/restic/restic/internal/ui/table"
"github.com/spf13/cobra"
)
var cmdKey = &cobra.Command{
Use: "key",
Use: "key [flags] [list|add|remove|passwd] [ID]",
Short: "Manage keys (passwords)",
Long: `
The "key" command allows you to set multiple access keys or passwords
per repository.
`,
The "key" command manages keys (passwords) for accessing the repository.
EXIT STATUS
===========
Exit status is 0 if the command was successful, and non-zero if there was any error.
`,
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, args []string) error {
return runKey(cmd.Context(), globalOptions, args)
},
}
var (
newPasswordFile string
keyUsername string
keyHostname string
)
func init() {
cmdRoot.AddCommand(cmdKey)
flags := cmdKey.Flags()
flags.StringVarP(&newPasswordFile, "new-password-file", "", "", "`file` from which to read the new password")
flags.StringVarP(&keyUsername, "user", "", "", "the username for new keys")
flags.StringVarP(&keyHostname, "host", "", "", "the hostname for new keys")
}
func listKeys(ctx context.Context, s *repository.Repository, gopts GlobalOptions) error {
type keyInfo struct {
Current bool `json:"current"`
ID string `json:"id"`
UserName string `json:"userName"`
HostName string `json:"hostName"`
Created string `json:"created"`
}
var m sync.Mutex
var keys []keyInfo
err := restic.ParallelList(ctx, s.Backend(), restic.KeyFile, s.Connections(), func(ctx context.Context, id restic.ID, size int64) error {
k, err := repository.LoadKey(ctx, s, id)
if err != nil {
Warnf("LoadKey() failed: %v\n", err)
return nil
}
key := keyInfo{
Current: id == s.KeyID(),
ID: id.Str(),
UserName: k.Username,
HostName: k.Hostname,
Created: k.Created.Local().Format(TimeFormat),
}
m.Lock()
defer m.Unlock()
keys = append(keys, key)
return nil
})
if err != nil {
return err
}
if gopts.JSON {
return json.NewEncoder(globalOptions.stdout).Encode(keys)
}
tab := table.New()
tab.AddColumn(" ID", "{{if .Current}}*{{else}} {{end}}{{ .ID }}")
tab.AddColumn("User", "{{ .UserName }}")
tab.AddColumn("Host", "{{ .HostName }}")
tab.AddColumn("Created", "{{ .Created }}")
for _, key := range keys {
tab.AddRow(key)
}
return tab.Write(globalOptions.stdout)
}
// testKeyNewPassword is used to set a new password during integration testing.
var testKeyNewPassword string
func getNewPassword(gopts GlobalOptions) (string, error) {
if testKeyNewPassword != "" {
return testKeyNewPassword, nil
}
if newPasswordFile != "" {
return loadPasswordFromFile(newPasswordFile)
}
// Since we already have an open repository, temporary remove the password
// to prompt the user for the passwd.
newopts := gopts
newopts.password = ""
return ReadPasswordTwice(newopts,
"enter new password: ",
"enter password again: ")
}
func addKey(ctx context.Context, repo *repository.Repository, gopts GlobalOptions) error {
pw, err := getNewPassword(gopts)
if err != nil {
return err
}
id, err := repository.AddKey(ctx, repo, pw, keyUsername, keyHostname, repo.Key())
if err != nil {
return errors.Fatalf("creating new key failed: %v\n", err)
}
err = switchToNewKeyAndRemoveIfBroken(ctx, repo, id, pw)
if err != nil {
return err
}
Verbosef("saved new key as %s\n", id)
return nil
}
func deleteKey(ctx context.Context, repo *repository.Repository, id restic.ID) error {
if id == repo.KeyID() {
return errors.Fatal("refusing to remove key currently used to access repository")
}
h := restic.Handle{Type: restic.KeyFile, Name: id.String()}
err := repo.Backend().Remove(ctx, h)
if err != nil {
return err
}
Verbosef("removed key %v\n", id)
return nil
}
func changePassword(ctx context.Context, repo *repository.Repository, gopts GlobalOptions) error {
pw, err := getNewPassword(gopts)
if err != nil {
return err
}
id, err := repository.AddKey(ctx, repo, pw, "", "", repo.Key())
if err != nil {
return errors.Fatalf("creating new key failed: %v\n", err)
}
oldID := repo.KeyID()
err = switchToNewKeyAndRemoveIfBroken(ctx, repo, id, pw)
if err != nil {
return err
}
h := restic.Handle{Type: restic.KeyFile, Name: oldID.String()}
err = repo.Backend().Remove(ctx, h)
if err != nil {
return err
}
Verbosef("saved new key as %s\n", id)
return nil
}
func switchToNewKeyAndRemoveIfBroken(ctx context.Context, repo *repository.Repository, key *repository.Key, pw string) error {
// Verify new key to make sure it really works. A broken key can render the
// whole repository inaccessible
err := repo.SearchKey(ctx, pw, 0, key.ID().String())
if err != nil {
// the key is invalid, try to remove it
h := restic.Handle{Type: restic.KeyFile, Name: key.ID().String()}
_ = repo.Backend().Remove(ctx, h)
return errors.Fatalf("failed to access repository with new key: %v", err)
}
return nil
}
func runKey(ctx context.Context, gopts GlobalOptions, args []string) error {
if len(args) < 1 || (args[0] == "remove" && len(args) != 2) || (args[0] != "remove" && len(args) != 1) {
return errors.Fatal("wrong number of arguments")
}
repo, err := OpenRepository(ctx, gopts)
if err != nil {
return err
}
switch args[0] {
case "list":
if !gopts.NoLock {
var lock *restic.Lock
lock, ctx, err = lockRepo(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err
}
}
return listKeys(ctx, repo, gopts)
case "add":
lock, ctx, err := lockRepo(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err
}
return addKey(ctx, repo, gopts)
case "remove":
lock, ctx, err := lockRepoExclusive(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err
}
id, err := restic.Find(ctx, repo.Backend(), restic.KeyFile, args[1])
if err != nil {
return err
}
return deleteKey(ctx, repo, id)
case "passwd":
lock, ctx, err := lockRepoExclusive(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err
}
return changePassword(ctx, repo, gopts)
}
return nil
}
func loadPasswordFromFile(pwdFile string) (string, error) {
s, err := os.ReadFile(pwdFile)
if os.IsNotExist(err) {
return "", errors.Fatalf("%s does not exist", pwdFile)
}
return strings.TrimSpace(string(s)), errors.Wrap(err, "Readfile")
}

View File

@@ -1,128 +0,0 @@
package main
import (
"context"
"fmt"
"os"
"strings"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/repository"
"github.com/spf13/cobra"
)
var cmdKeyAdd = &cobra.Command{
Use: "add",
Short: "Add a new key (password) to the repository; returns the new key ID",
Long: `
The "add" sub-command creates a new key and validates the key. Returns the new key ID.
EXIT STATUS
===========
Exit status is 0 if the command is successful, and non-zero if there was any error.
`,
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, args []string) error {
return runKeyAdd(cmd.Context(), globalOptions, keyAddOpts, args)
},
}
type KeyAddOptions struct {
NewPasswordFile string
Username string
Hostname string
}
var keyAddOpts KeyAddOptions
func init() {
cmdKey.AddCommand(cmdKeyAdd)
flags := cmdKeyAdd.Flags()
flags.StringVarP(&keyAddOpts.NewPasswordFile, "new-password-file", "", "", "`file` from which to read the new password")
flags.StringVarP(&keyAddOpts.Username, "user", "", "", "the username for new key")
flags.StringVarP(&keyAddOpts.Hostname, "host", "", "", "the hostname for new key")
}
func runKeyAdd(ctx context.Context, gopts GlobalOptions, opts KeyAddOptions, args []string) error {
if len(args) > 0 {
return fmt.Errorf("the key add command expects no arguments, only options - please see `restic help key add` for usage and flags")
}
repo, err := OpenRepository(ctx, gopts)
if err != nil {
return err
}
lock, ctx, err := lockRepo(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err
}
return addKey(ctx, repo, gopts, opts)
}
func addKey(ctx context.Context, repo *repository.Repository, gopts GlobalOptions, opts KeyAddOptions) error {
pw, err := getNewPassword(gopts, opts.NewPasswordFile)
if err != nil {
return err
}
id, err := repository.AddKey(ctx, repo, pw, opts.Username, opts.Hostname, repo.Key())
if err != nil {
return errors.Fatalf("creating new key failed: %v\n", err)
}
err = switchToNewKeyAndRemoveIfBroken(ctx, repo, id, pw)
if err != nil {
return err
}
Verbosef("saved new key with ID %s\n", id.ID())
return nil
}
// testKeyNewPassword is used to set a new password during integration testing.
var testKeyNewPassword string
func getNewPassword(gopts GlobalOptions, newPasswordFile string) (string, error) {
if testKeyNewPassword != "" {
return testKeyNewPassword, nil
}
if newPasswordFile != "" {
return loadPasswordFromFile(newPasswordFile)
}
// Since we already have an open repository, temporary remove the password
// to prompt the user for the passwd.
newopts := gopts
newopts.password = ""
return ReadPasswordTwice(newopts,
"enter new password: ",
"enter password again: ")
}
func loadPasswordFromFile(pwdFile string) (string, error) {
s, err := os.ReadFile(pwdFile)
if os.IsNotExist(err) {
return "", errors.Fatalf("%s does not exist", pwdFile)
}
return strings.TrimSpace(string(s)), errors.Wrap(err, "Readfile")
}
func switchToNewKeyAndRemoveIfBroken(ctx context.Context, repo *repository.Repository, key *repository.Key, pw string) error {
// Verify new key to make sure it really works. A broken key can render the
// whole repository inaccessible
err := repo.SearchKey(ctx, pw, 0, key.ID().String())
if err != nil {
// the key is invalid, try to remove it
_ = repository.RemoveKey(ctx, repo, key.ID())
return errors.Fatalf("failed to access repository with new key: %v", err)
}
return nil
}

View File

@@ -4,17 +4,16 @@ import (
"bufio"
"context"
"regexp"
"strings"
"testing"
"github.com/restic/restic/internal/backend"
"github.com/restic/restic/internal/repository"
"github.com/restic/restic/internal/restic"
rtest "github.com/restic/restic/internal/test"
)
func testRunKeyListOtherIDs(t testing.TB, gopts GlobalOptions) []string {
buf, err := withCaptureStdout(func() error {
return runKeyList(context.TODO(), gopts, []string{})
return runKey(context.TODO(), gopts, []string{"list"})
})
rtest.OK(t, err)
@@ -37,20 +36,21 @@ func testRunKeyAddNewKey(t testing.TB, newPassword string, gopts GlobalOptions)
testKeyNewPassword = ""
}()
rtest.OK(t, runKeyAdd(context.TODO(), gopts, KeyAddOptions{}, []string{}))
rtest.OK(t, runKey(context.TODO(), gopts, []string{"add"}))
}
func testRunKeyAddNewKeyUserHost(t testing.TB, gopts GlobalOptions) {
testKeyNewPassword = "john's geheimnis"
defer func() {
testKeyNewPassword = ""
keyUsername = ""
keyHostname = ""
}()
rtest.OK(t, cmdKey.Flags().Parse([]string{"--user=john", "--host=example.com"}))
t.Log("adding key for john@example.com")
rtest.OK(t, runKeyAdd(context.TODO(), gopts, KeyAddOptions{
Username: "john",
Hostname: "example.com",
}, []string{}))
rtest.OK(t, runKey(context.TODO(), gopts, []string{"add"}))
repo, err := OpenRepository(context.TODO(), gopts)
rtest.OK(t, err)
@@ -67,13 +67,13 @@ func testRunKeyPasswd(t testing.TB, newPassword string, gopts GlobalOptions) {
testKeyNewPassword = ""
}()
rtest.OK(t, runKeyPasswd(context.TODO(), gopts, KeyPasswdOptions{}, []string{}))
rtest.OK(t, runKey(context.TODO(), gopts, []string{"passwd"}))
}
func testRunKeyRemove(t testing.TB, gopts GlobalOptions, IDs []string) {
t.Logf("remove %d keys: %q\n", len(IDs), IDs)
for _, id := range IDs {
rtest.OK(t, runKeyRemove(context.TODO(), gopts, []string{id}))
rtest.OK(t, runKey(context.TODO(), gopts, []string{"remove", id}))
}
}
@@ -103,18 +103,18 @@ func TestKeyAddRemove(t *testing.T) {
env.gopts.password = passwordList[len(passwordList)-1]
t.Logf("testing access with last password %q\n", env.gopts.password)
rtest.OK(t, runKeyList(context.TODO(), env.gopts, []string{}))
rtest.OK(t, runKey(context.TODO(), env.gopts, []string{"list"}))
testRunCheck(t, env.gopts)
testRunKeyAddNewKeyUserHost(t, env.gopts)
}
type emptySaveBackend struct {
backend.Backend
restic.Backend
}
func (b *emptySaveBackend) Save(ctx context.Context, h backend.Handle, _ backend.RewindReader) error {
return b.Backend.Save(ctx, h, backend.NewByteReader([]byte{}, nil))
func (b *emptySaveBackend) Save(ctx context.Context, h restic.Handle, _ restic.RewindReader) error {
return b.Backend.Save(ctx, h, restic.NewByteReader([]byte{}, nil))
}
func TestKeyProblems(t *testing.T) {
@@ -122,7 +122,7 @@ func TestKeyProblems(t *testing.T) {
defer cleanup()
testRunInit(t, env.gopts)
env.gopts.backendTestHook = func(r backend.Backend) (backend.Backend, error) {
env.gopts.backendTestHook = func(r restic.Backend) (restic.Backend, error) {
return &emptySaveBackend{r}, nil
}
@@ -131,45 +131,15 @@ func TestKeyProblems(t *testing.T) {
testKeyNewPassword = ""
}()
err := runKeyPasswd(context.TODO(), env.gopts, KeyPasswdOptions{}, []string{})
err := runKey(context.TODO(), env.gopts, []string{"passwd"})
t.Log(err)
rtest.Assert(t, err != nil, "expected passwd change to fail")
err = runKeyAdd(context.TODO(), env.gopts, KeyAddOptions{}, []string{})
err = runKey(context.TODO(), env.gopts, []string{"add"})
t.Log(err)
rtest.Assert(t, err != nil, "expected key adding to fail")
t.Logf("testing access with initial password %q\n", env.gopts.password)
rtest.OK(t, runKeyList(context.TODO(), env.gopts, []string{}))
rtest.OK(t, runKey(context.TODO(), env.gopts, []string{"list"}))
testRunCheck(t, env.gopts)
}
func TestKeyCommandInvalidArguments(t *testing.T) {
env, cleanup := withTestEnvironment(t)
defer cleanup()
testRunInit(t, env.gopts)
env.gopts.backendTestHook = func(r backend.Backend) (backend.Backend, error) {
return &emptySaveBackend{r}, nil
}
err := runKeyAdd(context.TODO(), env.gopts, KeyAddOptions{}, []string{"johndoe"})
t.Log(err)
rtest.Assert(t, err != nil && strings.Contains(err.Error(), "no arguments"), "unexpected error for key add: %v", err)
err = runKeyPasswd(context.TODO(), env.gopts, KeyPasswdOptions{}, []string{"johndoe"})
t.Log(err)
rtest.Assert(t, err != nil && strings.Contains(err.Error(), "no arguments"), "unexpected error for key passwd: %v", err)
err = runKeyList(context.TODO(), env.gopts, []string{"johndoe"})
t.Log(err)
rtest.Assert(t, err != nil && strings.Contains(err.Error(), "no arguments"), "unexpected error for key list: %v", err)
err = runKeyRemove(context.TODO(), env.gopts, []string{})
t.Log(err)
rtest.Assert(t, err != nil && strings.Contains(err.Error(), "one argument"), "unexpected error for key remove: %v", err)
err = runKeyRemove(context.TODO(), env.gopts, []string{"john", "doe"})
t.Log(err)
rtest.Assert(t, err != nil && strings.Contains(err.Error(), "one argument"), "unexpected error for key remove: %v", err)
}

View File

@@ -1,112 +0,0 @@
package main
import (
"context"
"encoding/json"
"fmt"
"sync"
"github.com/restic/restic/internal/repository"
"github.com/restic/restic/internal/restic"
"github.com/restic/restic/internal/ui/table"
"github.com/spf13/cobra"
)
var cmdKeyList = &cobra.Command{
Use: "list",
Short: "List keys (passwords)",
Long: `
The "list" sub-command lists all the keys (passwords) associated with the repository.
Returns the key ID, username, hostname, created time and if it's the current key being
used to access the repository.
EXIT STATUS
===========
Exit status is 0 if the command is successful, and non-zero if there was any error.
`,
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, args []string) error {
return runKeyList(cmd.Context(), globalOptions, args)
},
}
func init() {
cmdKey.AddCommand(cmdKeyList)
}
func runKeyList(ctx context.Context, gopts GlobalOptions, args []string) error {
if len(args) > 0 {
return fmt.Errorf("the key list command expects no arguments, only options - please see `restic help key list` for usage and flags")
}
repo, err := OpenRepository(ctx, gopts)
if err != nil {
return err
}
if !gopts.NoLock {
var lock *restic.Lock
lock, ctx, err = lockRepo(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err
}
}
return listKeys(ctx, repo, gopts)
}
func listKeys(ctx context.Context, s *repository.Repository, gopts GlobalOptions) error {
type keyInfo struct {
Current bool `json:"current"`
ID string `json:"id"`
UserName string `json:"userName"`
HostName string `json:"hostName"`
Created string `json:"created"`
}
var m sync.Mutex
var keys []keyInfo
err := restic.ParallelList(ctx, s, restic.KeyFile, s.Connections(), func(ctx context.Context, id restic.ID, _ int64) error {
k, err := repository.LoadKey(ctx, s, id)
if err != nil {
Warnf("LoadKey() failed: %v\n", err)
return nil
}
key := keyInfo{
Current: id == s.KeyID(),
ID: id.Str(),
UserName: k.Username,
HostName: k.Hostname,
Created: k.Created.Local().Format(TimeFormat),
}
m.Lock()
defer m.Unlock()
keys = append(keys, key)
return nil
})
if err != nil {
return err
}
if gopts.JSON {
return json.NewEncoder(globalOptions.stdout).Encode(keys)
}
tab := table.New()
tab.AddColumn(" ID", "{{if .Current}}*{{else}} {{end}}{{ .ID }}")
tab.AddColumn("User", "{{ .UserName }}")
tab.AddColumn("Host", "{{ .HostName }}")
tab.AddColumn("Created", "{{ .Created }}")
for _, key := range keys {
tab.AddRow(key)
}
return tab.Write(globalOptions.stdout)
}

View File

@@ -1,89 +0,0 @@
package main
import (
"context"
"fmt"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/repository"
"github.com/spf13/cobra"
)
var cmdKeyPasswd = &cobra.Command{
Use: "passwd",
Short: "Change key (password); creates a new key ID and removes the old key ID, returns new key ID",
Long: `
The "passwd" sub-command creates a new key, validates the key and remove the old key ID.
Returns the new key ID.
EXIT STATUS
===========
Exit status is 0 if the command is successful, and non-zero if there was any error.
`,
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, args []string) error {
return runKeyPasswd(cmd.Context(), globalOptions, keyPasswdOpts, args)
},
}
type KeyPasswdOptions struct {
KeyAddOptions
}
var keyPasswdOpts KeyPasswdOptions
func init() {
cmdKey.AddCommand(cmdKeyPasswd)
flags := cmdKeyPasswd.Flags()
flags.StringVarP(&keyPasswdOpts.NewPasswordFile, "new-password-file", "", "", "`file` from which to read the new password")
flags.StringVarP(&keyPasswdOpts.Username, "user", "", "", "the username for new key")
flags.StringVarP(&keyPasswdOpts.Hostname, "host", "", "", "the hostname for new key")
}
func runKeyPasswd(ctx context.Context, gopts GlobalOptions, opts KeyPasswdOptions, args []string) error {
if len(args) > 0 {
return fmt.Errorf("the key passwd command expects no arguments, only options - please see `restic help key passwd` for usage and flags")
}
repo, err := OpenRepository(ctx, gopts)
if err != nil {
return err
}
lock, ctx, err := lockRepoExclusive(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err
}
return changePassword(ctx, repo, gopts, opts)
}
func changePassword(ctx context.Context, repo *repository.Repository, gopts GlobalOptions, opts KeyPasswdOptions) error {
pw, err := getNewPassword(gopts, opts.NewPasswordFile)
if err != nil {
return err
}
id, err := repository.AddKey(ctx, repo, pw, "", "", repo.Key())
if err != nil {
return errors.Fatalf("creating new key failed: %v\n", err)
}
oldID := repo.KeyID()
err = switchToNewKeyAndRemoveIfBroken(ctx, repo, id, pw)
if err != nil {
return err
}
err = repository.RemoveKey(ctx, repo, oldID)
if err != nil {
return err
}
Verbosef("saved new key as %s\n", id)
return nil
}

View File

@@ -1,73 +0,0 @@
package main
import (
"context"
"fmt"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/repository"
"github.com/restic/restic/internal/restic"
"github.com/spf13/cobra"
)
var cmdKeyRemove = &cobra.Command{
Use: "remove [ID]",
Short: "Remove key ID (password) from the repository.",
Long: `
The "remove" sub-command removes the selected key ID. The "remove" command does not allow
removing the current key being used to access the repository.
EXIT STATUS
===========
Exit status is 0 if the command is successful, and non-zero if there was any error.
`,
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, args []string) error {
return runKeyRemove(cmd.Context(), globalOptions, args)
},
}
func init() {
cmdKey.AddCommand(cmdKeyRemove)
}
func runKeyRemove(ctx context.Context, gopts GlobalOptions, args []string) error {
if len(args) != 1 {
return fmt.Errorf("key remove expects one argument as the key id")
}
repo, err := OpenRepository(ctx, gopts)
if err != nil {
return err
}
lock, ctx, err := lockRepoExclusive(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err
}
idPrefix := args[0]
return deleteKey(ctx, repo, idPrefix)
}
func deleteKey(ctx context.Context, repo *repository.Repository, idPrefix string) error {
id, err := restic.Find(ctx, repo, restic.KeyFile, idPrefix)
if err != nil {
return err
}
if id == repo.KeyID() {
return errors.Fatal("refusing to remove key currently used to access repository")
}
err = repository.RemoveKey(ctx, repo, id)
if err != nil {
return err
}
Verbosef("removed key %v\n", id)
return nil
}

View File

@@ -23,7 +23,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
`,
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, args []string) error {
return runList(cmd.Context(), globalOptions, args)
return runList(cmd.Context(), cmd, globalOptions, args)
},
}
@@ -31,9 +31,9 @@ func init() {
cmdRoot.AddCommand(cmdList)
}
func runList(ctx context.Context, gopts GlobalOptions, args []string) error {
func runList(ctx context.Context, cmd *cobra.Command, gopts GlobalOptions, args []string) error {
if len(args) != 1 {
return errors.Fatal("type not specified")
return errors.Fatal("type not specified, usage: " + cmd.Use)
}
repo, err := OpenRepository(ctx, gopts)
@@ -63,7 +63,7 @@ func runList(ctx context.Context, gopts GlobalOptions, args []string) error {
case "locks":
t = restic.LockFile
case "blobs":
return index.ForAllIndexes(ctx, repo, repo, func(_ restic.ID, idx *index.Index, _ bool, err error) error {
return index.ForAllIndexes(ctx, repo.Backend(), repo, func(id restic.ID, idx *index.Index, oldFormat bool, err error) error {
if err != nil {
return err
}
@@ -76,7 +76,7 @@ func runList(ctx context.Context, gopts GlobalOptions, args []string) error {
return errors.Fatal("invalid type")
}
return repo.List(ctx, t, func(id restic.ID, _ int64) error {
return repo.List(ctx, t, func(id restic.ID, size int64) error {
Printf("%s\n", id)
return nil
})

View File

@@ -12,7 +12,7 @@ import (
func testRunList(t testing.TB, tpe string, opts GlobalOptions) restic.IDs {
buf, err := withCaptureStdout(func() error {
return runList(context.TODO(), opts, []string{tpe})
return runList(context.TODO(), cmdList, opts, []string{tpe})
})
rtest.OK(t, err)
return parseIDsFromReader(t, buf)

View File

@@ -3,14 +3,13 @@ package main
import (
"context"
"encoding/json"
"fmt"
"io"
"os"
"strings"
"time"
"github.com/spf13/cobra"
"github.com/restic/restic/internal/backend"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/fs"
"github.com/restic/restic/internal/restic"
@@ -53,7 +52,6 @@ type LsOptions struct {
restic.SnapshotFilter
Recursive bool
HumanReadable bool
Ncdu bool
}
var lsOptions LsOptions
@@ -66,47 +64,16 @@ func init() {
flags.BoolVarP(&lsOptions.ListLong, "long", "l", false, "use a long listing format showing size and mode")
flags.BoolVar(&lsOptions.Recursive, "recursive", false, "include files in subfolders of the listed directories")
flags.BoolVar(&lsOptions.HumanReadable, "human-readable", false, "print sizes in human readable format")
flags.BoolVar(&lsOptions.Ncdu, "ncdu", false, "output NCDU export format (pipe into 'ncdu -f -')")
}
type lsPrinter interface {
Snapshot(sn *restic.Snapshot)
Node(path string, node *restic.Node)
LeaveDir(path string)
Close()
}
type jsonLsPrinter struct {
enc *json.Encoder
}
func (p *jsonLsPrinter) Snapshot(sn *restic.Snapshot) {
type lsSnapshot struct {
*restic.Snapshot
ID *restic.ID `json:"id"`
ShortID string `json:"short_id"`
StructType string `json:"struct_type"` // "snapshot"
}
err := p.enc.Encode(lsSnapshot{
Snapshot: sn,
ID: sn.ID(),
ShortID: sn.ID().Str(),
StructType: "snapshot",
})
if err != nil {
Warnf("JSON encode failed: %v\n", err)
}
type lsSnapshot struct {
*restic.Snapshot
ID *restic.ID `json:"id"`
ShortID string `json:"short_id"`
StructType string `json:"struct_type"` // "snapshot"
}
// Print node in our custom JSON format, followed by a newline.
func (p *jsonLsPrinter) Node(path string, node *restic.Node) {
err := lsNodeJSON(p.enc, path, node)
if err != nil {
Warnf("JSON encode failed: %v\n", err)
}
}
func lsNodeJSON(enc *json.Encoder, path string, node *restic.Node) error {
n := &struct {
Name string `json:"name"`
@@ -148,117 +115,10 @@ func lsNodeJSON(enc *json.Encoder, path string, node *restic.Node) error {
return enc.Encode(n)
}
func (p *jsonLsPrinter) LeaveDir(_ string) {}
func (p *jsonLsPrinter) Close() {}
type ncduLsPrinter struct {
out io.Writer
depth int
}
// lsSnapshotNcdu prints a restic snapshot in Ncdu save format.
// It opens the JSON list. Nodes are added with lsNodeNcdu and the list is closed by lsCloseNcdu.
// Format documentation: https://dev.yorhel.nl/ncdu/jsonfmt
func (p *ncduLsPrinter) Snapshot(sn *restic.Snapshot) {
const NcduMajorVer = 1
const NcduMinorVer = 2
snapshotBytes, err := json.Marshal(sn)
if err != nil {
Warnf("JSON encode failed: %v\n", err)
}
p.depth++
fmt.Fprintf(p.out, "[%d, %d, %s", NcduMajorVer, NcduMinorVer, string(snapshotBytes))
}
func lsNcduNode(_ string, node *restic.Node) ([]byte, error) {
type NcduNode struct {
Name string `json:"name"`
Asize uint64 `json:"asize"`
Dsize uint64 `json:"dsize"`
Dev uint64 `json:"dev"`
Ino uint64 `json:"ino"`
NLink uint64 `json:"nlink"`
NotReg bool `json:"notreg"`
UID uint32 `json:"uid"`
GID uint32 `json:"gid"`
Mode uint16 `json:"mode"`
Mtime int64 `json:"mtime"`
}
outNode := NcduNode{
Name: node.Name,
Asize: node.Size,
Dsize: node.Size,
Dev: node.DeviceID,
Ino: node.Inode,
NLink: node.Links,
NotReg: node.Type != "dir" && node.Type != "file",
UID: node.UID,
GID: node.GID,
Mode: uint16(node.Mode & os.ModePerm),
Mtime: node.ModTime.Unix(),
}
// bits according to inode(7) manpage
if node.Mode&os.ModeSetuid != 0 {
outNode.Mode |= 0o4000
}
if node.Mode&os.ModeSetgid != 0 {
outNode.Mode |= 0o2000
}
if node.Mode&os.ModeSticky != 0 {
outNode.Mode |= 0o1000
}
return json.Marshal(outNode)
}
func (p *ncduLsPrinter) Node(path string, node *restic.Node) {
out, err := lsNcduNode(path, node)
if err != nil {
Warnf("JSON encode failed: %v\n", err)
}
if node.Type == "dir" {
fmt.Fprintf(p.out, ",\n%s[\n%s%s", strings.Repeat(" ", p.depth), strings.Repeat(" ", p.depth+1), string(out))
p.depth++
} else {
fmt.Fprintf(p.out, ",\n%s%s", strings.Repeat(" ", p.depth), string(out))
}
}
func (p *ncduLsPrinter) LeaveDir(_ string) {
p.depth--
fmt.Fprintf(p.out, "\n%s]", strings.Repeat(" ", p.depth))
}
func (p *ncduLsPrinter) Close() {
fmt.Fprint(p.out, "\n]\n")
}
type textLsPrinter struct {
dirs []string
ListLong bool
HumanReadable bool
}
func (p *textLsPrinter) Snapshot(sn *restic.Snapshot) {
Verbosef("%v filtered by %v:\n", sn, p.dirs)
}
func (p *textLsPrinter) Node(path string, node *restic.Node) {
Printf("%s\n", formatNode(path, node, p.ListLong, p.HumanReadable))
}
func (p *textLsPrinter) LeaveDir(_ string) {}
func (p *textLsPrinter) Close() {}
func runLs(ctx context.Context, opts LsOptions, gopts GlobalOptions, args []string) error {
if len(args) == 0 {
return errors.Fatal("no snapshot ID specified, specify snapshot ID or use special ID 'latest'")
}
if opts.Ncdu && gopts.JSON {
return errors.Fatal("only either '--json' or '--ncdu' can be specified")
}
// extract any specific directories to walk
var dirs []string
@@ -310,7 +170,7 @@ func runLs(ctx context.Context, opts LsOptions, gopts GlobalOptions, args []stri
return err
}
snapshotLister, err := restic.MemorizeList(ctx, repo, restic.SnapshotFile)
snapshotLister, err := backend.MemorizeList(ctx, repo.Backend(), restic.SnapshotFile)
if err != nil {
return err
}
@@ -320,21 +180,38 @@ func runLs(ctx context.Context, opts LsOptions, gopts GlobalOptions, args []stri
return err
}
var printer lsPrinter
var (
printSnapshot func(sn *restic.Snapshot)
printNode func(path string, node *restic.Node)
)
if gopts.JSON {
printer = &jsonLsPrinter{
enc: json.NewEncoder(globalOptions.stdout),
enc := json.NewEncoder(globalOptions.stdout)
printSnapshot = func(sn *restic.Snapshot) {
err := enc.Encode(lsSnapshot{
Snapshot: sn,
ID: sn.ID(),
ShortID: sn.ID().Str(),
StructType: "snapshot",
})
if err != nil {
Warnf("JSON encode failed: %v\n", err)
}
}
} else if opts.Ncdu {
printer = &ncduLsPrinter{
out: globalOptions.stdout,
printNode = func(path string, node *restic.Node) {
err := lsNodeJSON(enc, path, node)
if err != nil {
Warnf("JSON encode failed: %v\n", err)
}
}
} else {
printer = &textLsPrinter{
dirs: dirs,
ListLong: opts.ListLong,
HumanReadable: opts.HumanReadable,
printSnapshot = func(sn *restic.Snapshot) {
Verbosef("snapshot %s of %v filtered by %v at %s):\n", sn.ID().Str(), sn.Paths, dirs, sn.Time)
}
printNode = func(path string, node *restic.Node) {
Printf("%s\n", formatNode(path, node, lsOptions.ListLong, lsOptions.HumanReadable))
}
}
@@ -352,55 +229,44 @@ func runLs(ctx context.Context, opts LsOptions, gopts GlobalOptions, args []stri
return err
}
printer.Snapshot(sn)
printSnapshot(sn)
processNode := func(_ restic.ID, nodepath string, node *restic.Node, err error) error {
err = walker.Walk(ctx, repo, *sn.Tree, nil, func(_ restic.ID, nodepath string, node *restic.Node, err error) (bool, error) {
if err != nil {
return err
return false, err
}
if node == nil {
return nil
return false, nil
}
if withinDir(nodepath) {
// if we're within a dir, print the node
printer.Node(nodepath, node)
printNode(nodepath, node)
// if recursive listing is requested, signal the walker that it
// should continue walking recursively
if opts.Recursive {
return nil
return false, nil
}
}
// if there's an upcoming match deeper in the tree (but we're not
// there yet), signal the walker to descend into any subdirs
if approachingMatchingTree(nodepath) {
return nil
return false, nil
}
// otherwise, signal the walker to not walk recursively into any
// subdirs
if node.Type == "dir" {
return walker.ErrSkipNode
return false, walker.ErrSkipNode
}
return nil
}
err = walker.Walk(ctx, repo, *sn.Tree, walker.WalkVisitor{
ProcessNode: processNode,
LeaveDir: func(path string) {
// the root path `/` has no corresponding node and is thus also skipped by processNode
if withinDir(path) && path != "/" {
printer.LeaveDir(path)
}
},
return false, nil
})
if err != nil {
return err
}
printer.Close()
return nil
}

View File

@@ -2,46 +2,18 @@ package main
import (
"context"
"encoding/json"
"path/filepath"
"strings"
"testing"
rtest "github.com/restic/restic/internal/test"
)
func testRunLsWithOpts(t testing.TB, gopts GlobalOptions, opts LsOptions, args []string) []byte {
func testRunLs(t testing.TB, gopts GlobalOptions, snapshotID string) []string {
buf, err := withCaptureStdout(func() error {
gopts.Quiet = true
return runLs(context.TODO(), opts, gopts, args)
opts := LsOptions{}
return runLs(context.TODO(), opts, gopts, []string{snapshotID})
})
rtest.OK(t, err)
return buf.Bytes()
}
func testRunLs(t testing.TB, gopts GlobalOptions, snapshotID string) []string {
out := testRunLsWithOpts(t, gopts, LsOptions{}, []string{snapshotID})
return strings.Split(string(out), "\n")
}
func assertIsValidJSON(t *testing.T, data []byte) {
// Sanity check: output must be valid JSON.
var v interface{}
err := json.Unmarshal(data, &v)
rtest.OK(t, err)
}
func TestRunLsNcdu(t *testing.T) {
env, cleanup := withTestEnvironment(t)
defer cleanup()
testRunInit(t, env.gopts)
opts := BackupOptions{}
testRunBackup(t, filepath.Dir(env.testdata), []string{"testdata"}, opts, env.gopts)
ncdu := testRunLsWithOpts(t, env.gopts, LsOptions{Ncdu: true}, []string{"latest"})
assertIsValidJSON(t, ncdu)
ncdu = testRunLsWithOpts(t, env.gopts, LsOptions{Ncdu: true}, []string{"latest", "/testdata"})
assertIsValidJSON(t, ncdu)
return strings.Split(buf.String(), "\n")
}

View File

@@ -11,94 +11,78 @@ import (
rtest "github.com/restic/restic/internal/test"
)
type lsTestNode struct {
path string
restic.Node
}
var lsTestNodes = []lsTestNode{
// Mode is omitted when zero.
// Permissions, by convention is "-" per mode bit
{
path: "/bar/baz",
Node: restic.Node{
Name: "baz",
Type: "file",
Size: 12345,
UID: 10000000,
GID: 20000000,
User: "nobody",
Group: "nobodies",
Links: 1,
},
},
// Even empty files get an explicit size.
{
path: "/foo/empty",
Node: restic.Node{
Name: "empty",
Type: "file",
Size: 0,
UID: 1001,
GID: 1001,
User: "not printed",
Group: "not printed",
Links: 0xF00,
},
},
// Non-regular files do not get a size.
// Mode is printed in decimal, including the type bits.
{
path: "/foo/link",
Node: restic.Node{
Name: "link",
Type: "symlink",
Mode: os.ModeSymlink | 0777,
LinkTarget: "not printed",
},
},
{
path: "/some/directory",
Node: restic.Node{
Name: "directory",
Type: "dir",
Mode: os.ModeDir | 0755,
ModTime: time.Date(2020, 1, 2, 3, 4, 5, 0, time.UTC),
AccessTime: time.Date(2021, 2, 3, 4, 5, 6, 7, time.UTC),
ChangeTime: time.Date(2022, 3, 4, 5, 6, 7, 8, time.UTC),
},
},
// Test encoding of setuid/setgid/sticky bit
{
path: "/some/sticky",
Node: restic.Node{
Name: "sticky",
Type: "dir",
Mode: os.ModeDir | 0755 | os.ModeSetuid | os.ModeSetgid | os.ModeSticky,
},
},
}
func TestLsNodeJSON(t *testing.T) {
for i, expect := range []string{
`{"name":"baz","type":"file","path":"/bar/baz","uid":10000000,"gid":20000000,"size":12345,"permissions":"----------","mtime":"0001-01-01T00:00:00Z","atime":"0001-01-01T00:00:00Z","ctime":"0001-01-01T00:00:00Z","struct_type":"node"}`,
`{"name":"empty","type":"file","path":"/foo/empty","uid":1001,"gid":1001,"size":0,"permissions":"----------","mtime":"0001-01-01T00:00:00Z","atime":"0001-01-01T00:00:00Z","ctime":"0001-01-01T00:00:00Z","struct_type":"node"}`,
`{"name":"link","type":"symlink","path":"/foo/link","uid":0,"gid":0,"mode":134218239,"permissions":"Lrwxrwxrwx","mtime":"0001-01-01T00:00:00Z","atime":"0001-01-01T00:00:00Z","ctime":"0001-01-01T00:00:00Z","struct_type":"node"}`,
`{"name":"directory","type":"dir","path":"/some/directory","uid":0,"gid":0,"mode":2147484141,"permissions":"drwxr-xr-x","mtime":"2020-01-02T03:04:05Z","atime":"2021-02-03T04:05:06.000000007Z","ctime":"2022-03-04T05:06:07.000000008Z","struct_type":"node"}`,
`{"name":"sticky","type":"dir","path":"/some/sticky","uid":0,"gid":0,"mode":2161115629,"permissions":"dugtrwxr-xr-x","mtime":"0001-01-01T00:00:00Z","atime":"0001-01-01T00:00:00Z","ctime":"0001-01-01T00:00:00Z","struct_type":"node"}`,
for _, c := range []struct {
path string
restic.Node
expect string
}{
// Mode is omitted when zero.
// Permissions, by convention is "-" per mode bit
{
path: "/bar/baz",
Node: restic.Node{
Name: "baz",
Type: "file",
Size: 12345,
UID: 10000000,
GID: 20000000,
User: "nobody",
Group: "nobodies",
Links: 1,
},
expect: `{"name":"baz","type":"file","path":"/bar/baz","uid":10000000,"gid":20000000,"size":12345,"permissions":"----------","mtime":"0001-01-01T00:00:00Z","atime":"0001-01-01T00:00:00Z","ctime":"0001-01-01T00:00:00Z","struct_type":"node"}`,
},
// Even empty files get an explicit size.
{
path: "/foo/empty",
Node: restic.Node{
Name: "empty",
Type: "file",
Size: 0,
UID: 1001,
GID: 1001,
User: "not printed",
Group: "not printed",
Links: 0xF00,
},
expect: `{"name":"empty","type":"file","path":"/foo/empty","uid":1001,"gid":1001,"size":0,"permissions":"----------","mtime":"0001-01-01T00:00:00Z","atime":"0001-01-01T00:00:00Z","ctime":"0001-01-01T00:00:00Z","struct_type":"node"}`,
},
// Non-regular files do not get a size.
// Mode is printed in decimal, including the type bits.
{
path: "/foo/link",
Node: restic.Node{
Name: "link",
Type: "symlink",
Mode: os.ModeSymlink | 0777,
LinkTarget: "not printed",
},
expect: `{"name":"link","type":"symlink","path":"/foo/link","uid":0,"gid":0,"mode":134218239,"permissions":"Lrwxrwxrwx","mtime":"0001-01-01T00:00:00Z","atime":"0001-01-01T00:00:00Z","ctime":"0001-01-01T00:00:00Z","struct_type":"node"}`,
},
{
path: "/some/directory",
Node: restic.Node{
Name: "directory",
Type: "dir",
Mode: os.ModeDir | 0755,
ModTime: time.Date(2020, 1, 2, 3, 4, 5, 0, time.UTC),
AccessTime: time.Date(2021, 2, 3, 4, 5, 6, 7, time.UTC),
ChangeTime: time.Date(2022, 3, 4, 5, 6, 7, 8, time.UTC),
},
expect: `{"name":"directory","type":"dir","path":"/some/directory","uid":0,"gid":0,"mode":2147484141,"permissions":"drwxr-xr-x","mtime":"2020-01-02T03:04:05Z","atime":"2021-02-03T04:05:06.000000007Z","ctime":"2022-03-04T05:06:07.000000008Z","struct_type":"node"}`,
},
} {
c := lsTestNodes[i]
buf := new(bytes.Buffer)
enc := json.NewEncoder(buf)
err := lsNodeJSON(enc, c.path, &c.Node)
rtest.OK(t, err)
rtest.Equals(t, expect+"\n", buf.String())
rtest.Equals(t, c.expect+"\n", buf.String())
// Sanity check: output must be valid JSON.
var v interface{}
@@ -106,54 +90,3 @@ func TestLsNodeJSON(t *testing.T) {
rtest.OK(t, err)
}
}
func TestLsNcduNode(t *testing.T) {
for i, expect := range []string{
`{"name":"baz","asize":12345,"dsize":12345,"dev":0,"ino":0,"nlink":1,"notreg":false,"uid":10000000,"gid":20000000,"mode":0,"mtime":-62135596800}`,
`{"name":"empty","asize":0,"dsize":0,"dev":0,"ino":0,"nlink":3840,"notreg":false,"uid":1001,"gid":1001,"mode":0,"mtime":-62135596800}`,
`{"name":"link","asize":0,"dsize":0,"dev":0,"ino":0,"nlink":0,"notreg":true,"uid":0,"gid":0,"mode":511,"mtime":-62135596800}`,
`{"name":"directory","asize":0,"dsize":0,"dev":0,"ino":0,"nlink":0,"notreg":false,"uid":0,"gid":0,"mode":493,"mtime":1577934245}`,
`{"name":"sticky","asize":0,"dsize":0,"dev":0,"ino":0,"nlink":0,"notreg":false,"uid":0,"gid":0,"mode":4077,"mtime":-62135596800}`,
} {
c := lsTestNodes[i]
out, err := lsNcduNode(c.path, &c.Node)
rtest.OK(t, err)
rtest.Equals(t, expect, string(out))
// Sanity check: output must be valid JSON.
var v interface{}
err = json.Unmarshal(out, &v)
rtest.OK(t, err)
}
}
func TestLsNcdu(t *testing.T) {
var buf bytes.Buffer
printer := &ncduLsPrinter{
out: &buf,
}
printer.Snapshot(&restic.Snapshot{
Hostname: "host",
Paths: []string{"/example"},
})
printer.Node("/directory", &restic.Node{
Type: "dir",
Name: "directory",
})
printer.Node("/directory/data", &restic.Node{
Type: "file",
Name: "data",
Size: 42,
})
printer.LeaveDir("/directory")
printer.Close()
rtest.Equals(t, `[1, 2, {"time":"0001-01-01T00:00:00Z","tree":null,"paths":["/example"],"hostname":"host"},
[
{"name":"directory","asize":0,"dsize":0,"dev":0,"ino":0,"nlink":0,"notreg":false,"uid":0,"gid":0,"mode":0,"mtime":-62135596800},
{"name":"data","asize":42,"dsize":42,"dev":0,"ino":0,"nlink":0,"notreg":false,"uid":0,"gid":0,"mode":0,"mtime":-62135596800}
]
]
`, buf.String())
}

View File

@@ -113,15 +113,6 @@ func runMount(ctx context.Context, opts MountOptions, gopts GlobalOptions, args
return errors.Fatal("wrong number of parameters")
}
mountpoint := args[0]
// Check the existence of the mount point at the earliest stage to
// prevent unnecessary computations while opening the repository.
if _, err := resticfs.Stat(mountpoint); errors.Is(err, os.ErrNotExist) {
Verbosef("Mountpoint %s doesn't exist\n", mountpoint)
return err
}
debug.Log("start mount")
defer debug.Log("finish mount")
@@ -145,6 +136,12 @@ func runMount(ctx context.Context, opts MountOptions, gopts GlobalOptions, args
return err
}
mountpoint := args[0]
if _, err := resticfs.Stat(mountpoint); errors.Is(err, os.ErrNotExist) {
Verbosef("Mountpoint %s doesn't exist\n", mountpoint)
return err
}
mountOptions := []systemFuse.MountOption{
systemFuse.ReadOnly(),
systemFuse.FSName("restic"),

View File

@@ -21,7 +21,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
`,
Hidden: true,
DisableAutoGenTag: true,
Run: func(_ *cobra.Command, _ []string) {
Run: func(cmd *cobra.Command, args []string) {
fmt.Printf("All Extended Options:\n")
var maxLen int
for _, opt := range options.List() {

View File

@@ -15,7 +15,6 @@ import (
"github.com/restic/restic/internal/repository"
"github.com/restic/restic/internal/restic"
"github.com/restic/restic/internal/ui"
"github.com/restic/restic/internal/ui/progress"
"github.com/spf13/cobra"
)
@@ -37,7 +36,7 @@ EXIT STATUS
Exit status is 0 if the command was successful, and non-zero if there was any error.
`,
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, _ []string) error {
RunE: func(cmd *cobra.Command, args []string) error {
return runPrune(cmd.Context(), pruneOptions, globalOptions)
},
}
@@ -67,10 +66,10 @@ func init() {
f := cmdPrune.Flags()
f.BoolVarP(&pruneOptions.DryRun, "dry-run", "n", false, "do not modify the repository, just print what would be done")
f.StringVarP(&pruneOptions.UnsafeNoSpaceRecovery, "unsafe-recover-no-free-space", "", "", "UNSAFE, READ THE DOCUMENTATION BEFORE USING! Try to recover a repository stuck with no free space. Do not use without trying out 'prune --max-repack-size 0' first.")
addPruneOptions(cmdPrune, &pruneOptions)
addPruneOptions(cmdPrune)
}
func addPruneOptions(c *cobra.Command, pruneOptions *PruneOptions) {
func addPruneOptions(c *cobra.Command) {
f := c.Flags()
f.StringVar(&pruneOptions.MaxUnused, "max-unused", "5%", "tolerate given `limit` of unused data (absolute value in bytes with suffixes k/K, m/M, g/G, t/T, a value in % or the word 'unlimited')")
f.StringVar(&pruneOptions.MaxRepackSize, "max-repack-size", "", "maximum `size` to repack (allowed suffixes: k/K, m/M, g/G, t/T)")
@@ -101,7 +100,7 @@ func verifyPruneOptions(opts *PruneOptions) error {
// parse MaxUnused either as unlimited, a percentage, or an absolute number of bytes
switch {
case maxUnused == "unlimited":
opts.maxUnusedBytes = func(_ uint64) uint64 {
opts.maxUnusedBytes = func(used uint64) uint64 {
return math.MaxUint64
}
@@ -130,7 +129,7 @@ func verifyPruneOptions(opts *PruneOptions) error {
return errors.Fatalf("invalid number of bytes %q for --max-unused: %v", opts.MaxUnused, err)
}
opts.maxUnusedBytes = func(_ uint64) uint64 {
opts.maxUnusedBytes = func(used uint64) uint64 {
return uint64(size)
}
}
@@ -153,7 +152,7 @@ func runPrune(ctx context.Context, opts PruneOptions, gopts GlobalOptions) error
return err
}
if repo.Connections() < 2 {
if repo.Backend().Connections() < 2 {
return errors.Fatal("prune requires a backend connection limit of at least two")
}
@@ -407,7 +406,7 @@ func packInfoFromIndex(ctx context.Context, idx restic.MasterIndex, usedBlobs re
})
// if duplicate blobs exist, those will be set to either "used" or "unused":
// - mark only one occurrence of duplicate blobs as used
// - mark only one occurence of duplicate blobs as used
// - if there are already some used blobs in a pack, possibly mark duplicates in this pack as "used"
// - if there are no used blobs in a pack, possibly mark duplicates as "unused"
if hasDuplicates {
@@ -416,7 +415,7 @@ func packInfoFromIndex(ctx context.Context, idx restic.MasterIndex, usedBlobs re
bh := blob.BlobHandle
count, ok := usedBlobs[bh]
// skip non-duplicate, aka. normal blobs
// count == 0 is used to mark that this was a duplicate blob with only a single occurrence remaining
// count == 0 is used to mark that this was a duplicate blob with only a single occurence remaining
if !ok || count == 1 {
return
}
@@ -425,7 +424,7 @@ func packInfoFromIndex(ctx context.Context, idx restic.MasterIndex, usedBlobs re
size := uint64(blob.Length)
switch {
case ip.usedBlobs > 0, count == 0:
// other used blobs in pack or "last" occurrence -> transition to used
// other used blobs in pack or "last" occurence -> transition to used
ip.usedSize += size
ip.usedBlobs++
ip.unusedSize -= size
@@ -435,7 +434,7 @@ func packInfoFromIndex(ctx context.Context, idx restic.MasterIndex, usedBlobs re
stats.blobs.used++
stats.size.duplicate -= size
stats.blobs.duplicate--
// let other occurrences remain marked as unused
// let other occurences remain marked as unused
usedBlobs[bh] = 1
default:
// remain unused and decrease counter
@@ -767,7 +766,7 @@ func doPrune(ctx context.Context, opts PruneOptions, gopts GlobalOptions, repo r
return errors.Fatalf("%s", err)
}
} else if len(plan.ignorePacks) != 0 {
err = rebuildIndexFiles(ctx, gopts, repo, plan.ignorePacks, nil, false)
err = rebuildIndexFiles(ctx, gopts, repo, plan.ignorePacks, nil)
if err != nil {
return errors.Fatalf("%s", err)
}
@@ -779,7 +778,7 @@ func doPrune(ctx context.Context, opts PruneOptions, gopts GlobalOptions, repo r
}
if opts.unsafeRecovery {
err = rebuildIndexFiles(ctx, gopts, repo, plan.ignorePacks, nil, true)
_, err = writeIndexFiles(ctx, gopts, repo, plan.ignorePacks, nil)
if err != nil {
return errors.Fatalf("%s", err)
}
@@ -789,28 +788,29 @@ func doPrune(ctx context.Context, opts PruneOptions, gopts GlobalOptions, repo r
return nil
}
func rebuildIndexFiles(ctx context.Context, gopts GlobalOptions, repo restic.Repository, removePacks restic.IDSet, extraObsolete restic.IDs, skipDeletion bool) error {
func writeIndexFiles(ctx context.Context, gopts GlobalOptions, repo restic.Repository, removePacks restic.IDSet, extraObsolete restic.IDs) (restic.IDSet, error) {
Verbosef("rebuilding index\n")
bar := newProgressMax(!gopts.Quiet, 0, "packs processed")
return repo.Index().Save(ctx, repo, removePacks, extraObsolete, restic.MasterIndexSaveOpts{
SaveProgress: bar,
DeleteProgress: func() *progress.Counter {
return newProgressMax(!gopts.Quiet, 0, "old indexes deleted")
},
DeleteReport: func(id restic.ID, _ error) {
if gopts.verbosity > 2 {
Verbosef("removed index %v\n", id.String())
}
},
SkipDeletion: skipDeletion,
})
obsoleteIndexes, err := repo.Index().Save(ctx, repo, removePacks, extraObsolete, bar)
bar.Done()
return obsoleteIndexes, err
}
func rebuildIndexFiles(ctx context.Context, gopts GlobalOptions, repo restic.Repository, removePacks restic.IDSet, extraObsolete restic.IDs) error {
obsoleteIndexes, err := writeIndexFiles(ctx, gopts, repo, removePacks, extraObsolete)
if err != nil {
return err
}
Verbosef("deleting obsolete index files\n")
return DeleteFilesChecked(ctx, gopts, repo, obsoleteIndexes, restic.IndexFile)
}
func getUsedBlobs(ctx context.Context, repo restic.Repository, ignoreSnapshots restic.IDSet, quiet bool) (usedBlobs restic.CountedBlobSet, err error) {
var snapshotTrees restic.IDs
Verbosef("loading all snapshots...\n")
err = restic.ForAllSnapshots(ctx, repo, repo, ignoreSnapshots,
err = restic.ForAllSnapshots(ctx, repo.Backend(), repo, ignoreSnapshots,
func(id restic.ID, sn *restic.Snapshot, err error) error {
if err != nil {
debug.Log("failed to load snapshot %v (error %v)", id, err)

View File

@@ -6,13 +6,13 @@ import (
"path/filepath"
"testing"
"github.com/restic/restic/internal/backend"
"github.com/restic/restic/internal/restic"
rtest "github.com/restic/restic/internal/test"
)
func testRunPrune(t testing.TB, gopts GlobalOptions, opts PruneOptions) {
oldHook := gopts.backendTestHook
gopts.backendTestHook = func(r backend.Backend) (backend.Backend, error) { return newListOnceBackend(r), nil }
gopts.backendTestHook = func(r restic.Backend) (restic.Backend, error) { return newListOnceBackend(r), nil }
defer func() {
gopts.backendTestHook = oldHook
}()
@@ -81,10 +81,7 @@ func testRunForgetJSON(t testing.TB, gopts GlobalOptions, args ...string) {
DryRun: true,
Last: 1,
}
pruneOpts := PruneOptions{
MaxUnused: "5%",
}
return runForget(context.TODO(), opts, pruneOpts, gopts, args)
return runForget(context.TODO(), opts, gopts, args)
})
rtest.OK(t, err)
@@ -133,7 +130,7 @@ func TestPruneWithDamagedRepository(t *testing.T) {
removePacksExcept(env.gopts, t, oldPacks, false)
oldHook := env.gopts.backendTestHook
env.gopts.backendTestHook = func(r backend.Backend) (backend.Backend, error) { return newListOnceBackend(r), nil }
env.gopts.backendTestHook = func(r restic.Backend) (restic.Backend, error) { return newListOnceBackend(r), nil }
defer func() {
env.gopts.backendTestHook = oldHook
}()

View File

@@ -5,6 +5,7 @@ import (
"os"
"time"
"github.com/restic/restic/internal/backend"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/restic"
"github.com/spf13/cobra"
@@ -25,7 +26,7 @@ EXIT STATUS
Exit status is 0 if the command was successful, and non-zero if there was any error.
`,
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, _ []string) error {
RunE: func(cmd *cobra.Command, args []string) error {
return runRecover(cmd.Context(), globalOptions)
},
}
@@ -51,7 +52,7 @@ func runRecover(ctx context.Context, gopts GlobalOptions) error {
return err
}
snapshotLister, err := restic.MemorizeList(ctx, repo, restic.SnapshotFile)
snapshotLister, err := backend.MemorizeList(ctx, repo.Backend(), restic.SnapshotFile)
if err != nil {
return err
}
@@ -91,7 +92,7 @@ func runRecover(ctx context.Context, gopts GlobalOptions) error {
bar.Done()
Verbosef("load snapshots\n")
err = restic.ForAllSnapshots(ctx, snapshotLister, repo, nil, func(_ restic.ID, sn *restic.Snapshot, _ error) error {
err = restic.ForAllSnapshots(ctx, snapshotLister, repo, nil, func(id restic.ID, sn *restic.Snapshot, err error) error {
trees[*sn.Tree] = true
return nil
})
@@ -158,7 +159,7 @@ func runRecover(ctx context.Context, gopts GlobalOptions) error {
}
func createSnapshot(ctx context.Context, name, hostname string, tags []string, repo restic.SaverUnpacked, tree *restic.ID) error {
func createSnapshot(ctx context.Context, name, hostname string, tags []string, repo restic.Repository, tree *restic.ID) error {
sn, err := restic.NewSnapshot([]string{name}, tags, hostname, time.Now())
if err != nil {
return errors.Fatalf("unable to save snapshot: %v", err)

View File

@@ -24,7 +24,7 @@ EXIT STATUS
Exit status is 0 if the command was successful, and non-zero if there was any error.
`,
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, _ []string) error {
RunE: func(cmd *cobra.Command, args []string) error {
return runRebuildIndex(cmd.Context(), repairIndexOptions, globalOptions)
},
}
@@ -78,7 +78,7 @@ func rebuildIndex(ctx context.Context, opts RepairIndexOptions, gopts GlobalOpti
if opts.ReadAllPacks {
// get list of old index files but start with empty index
err := repo.List(ctx, restic.IndexFile, func(id restic.ID, _ int64) error {
err := repo.List(ctx, restic.IndexFile, func(id restic.ID, size int64) error {
obsoleteIndexes = append(obsoleteIndexes, id)
return nil
})
@@ -88,7 +88,7 @@ func rebuildIndex(ctx context.Context, opts RepairIndexOptions, gopts GlobalOpti
} else {
Verbosef("loading indexes...\n")
mi := index.NewMasterIndex()
err := index.ForAllIndexes(ctx, repo, repo, func(id restic.ID, idx *index.Index, _ bool, err error) error {
err := index.ForAllIndexes(ctx, repo.Backend(), repo, func(id restic.ID, idx *index.Index, oldFormat bool, err error) error {
if err != nil {
Warnf("removing invalid index %v: %v\n", id, err)
obsoleteIndexes = append(obsoleteIndexes, id)
@@ -154,7 +154,7 @@ func rebuildIndex(ctx context.Context, opts RepairIndexOptions, gopts GlobalOpti
}
}
err = rebuildIndexFiles(ctx, gopts, repo, removePacks, obsoleteIndexes, false)
err = rebuildIndexFiles(ctx, gopts, repo, removePacks, obsoleteIndexes)
if err != nil {
return err
}

View File

@@ -8,7 +8,6 @@ import (
"sync"
"testing"
"github.com/restic/restic/internal/backend"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/index"
"github.com/restic/restic/internal/restic"
@@ -71,12 +70,12 @@ func TestRebuildIndexAlwaysFull(t *testing.T) {
// indexErrorBackend modifies the first index after reading.
type indexErrorBackend struct {
backend.Backend
restic.Backend
lock sync.Mutex
hasErred bool
}
func (b *indexErrorBackend) Load(ctx context.Context, h backend.Handle, length int, offset int64, consumer func(rd io.Reader) error) error {
func (b *indexErrorBackend) Load(ctx context.Context, h restic.Handle, length int, offset int64, consumer func(rd io.Reader) error) error {
return b.Backend.Load(ctx, h, length, offset, func(rd io.Reader) error {
// protect hasErred
b.lock.Lock()
@@ -102,7 +101,7 @@ func (erd errorReadCloser) Read(p []byte) (int, error) {
}
func TestRebuildIndexDamage(t *testing.T) {
testRebuildIndex(t, func(r backend.Backend) (backend.Backend, error) {
testRebuildIndex(t, func(r restic.Backend) (restic.Backend, error) {
return &indexErrorBackend{
Backend: r,
}, nil
@@ -110,11 +109,11 @@ func TestRebuildIndexDamage(t *testing.T) {
}
type appendOnlyBackend struct {
backend.Backend
restic.Backend
}
// called via repo.Backend().Remove()
func (b *appendOnlyBackend) Remove(_ context.Context, h backend.Handle) error {
func (b *appendOnlyBackend) Remove(_ context.Context, h restic.Handle) error {
return errors.Errorf("Failed to remove %v", h)
}
@@ -128,7 +127,7 @@ func TestRebuildIndexFailsOnAppendOnly(t *testing.T) {
err := withRestoreGlobalOptions(func() error {
globalOptions.stdout = io.Discard
env.gopts.backendTestHook = func(r backend.Backend) (backend.Backend, error) {
env.gopts.backendTestHook = func(r restic.Backend) (restic.Backend, error) {
return &appendOnlyBackend{r}, nil
}
return runRebuildIndex(context.TODO(), RepairIndexOptions{}, env.gopts)

View File

@@ -5,12 +5,11 @@ import (
"io"
"os"
"github.com/restic/restic/internal/backend"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/repository"
"github.com/restic/restic/internal/restic"
"github.com/restic/restic/internal/ui/termstatus"
"github.com/spf13/cobra"
"golang.org/x/sync/errgroup"
)
var cmdRepairPacks = &cobra.Command{
@@ -29,9 +28,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
`,
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, args []string) error {
term, cancel := setupTermstatus()
defer cancel()
return runRepairPacks(cmd.Context(), globalOptions, term, args)
return runRepairPacks(cmd.Context(), globalOptions, args)
},
}
@@ -39,7 +36,14 @@ func init() {
cmdRepair.AddCommand(cmdRepairPacks)
}
func runRepairPacks(ctx context.Context, gopts GlobalOptions, term *termstatus.Terminal, args []string) error {
func runRepairPacks(ctx context.Context, gopts GlobalOptions, args []string) error {
// FIXME discuss and add proper feature flag mechanism
flag, _ := os.LookupEnv("RESTIC_FEATURES")
if flag != "repair-packs-v1" {
return errors.Fatal("This command is experimental and may change/be removed without notice between restic versions. " +
"Set the environment variable 'RESTIC_FEATURES=repair-packs-v1' to enable it.")
}
ids := restic.NewIDSet()
for _, arg := range args {
id, err := restic.ParseID(arg)
@@ -63,22 +67,24 @@ func runRepairPacks(ctx context.Context, gopts GlobalOptions, term *termstatus.T
return err
}
return repairPacks(ctx, gopts, repo, ids)
}
func repairPacks(ctx context.Context, gopts GlobalOptions, repo *repository.Repository, ids restic.IDSet) error {
bar := newIndexProgress(gopts.Quiet, gopts.JSON)
err = repo.LoadIndex(ctx, bar)
err := repo.LoadIndex(ctx, bar)
if err != nil {
return errors.Fatalf("%s", err)
}
printer := newTerminalProgressPrinter(gopts.verbosity, term)
printer.P("saving backup copies of pack files to current folder")
Warnf("saving backup copies of pack files in current folder\n")
for id := range ids {
f, err := os.OpenFile("pack-"+id.String(), os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o666)
if err != nil {
return err
return errors.Fatalf("%s", err)
}
err = repo.Backend().Load(ctx, backend.Handle{Type: restic.PackFile, Name: id.String()}, 0, 0, func(rd io.Reader) error {
err = repo.Backend().Load(ctx, restic.Handle{Type: restic.PackFile, Name: id.String()}, 0, 0, func(rd io.Reader) error {
_, err := f.Seek(0, 0)
if err != nil {
return err
@@ -87,15 +93,66 @@ func runRepairPacks(ctx context.Context, gopts GlobalOptions, term *termstatus.T
return err
})
if err != nil {
return err
return errors.Fatalf("%s", err)
}
}
err = repository.RepairPacks(ctx, repo, ids, printer)
wg, wgCtx := errgroup.WithContext(ctx)
repo.StartPackUploader(wgCtx, wg)
repo.DisableAutoIndexUpdate()
Warnf("salvaging intact data from specified pack files\n")
bar = newProgressMax(!gopts.Quiet, uint64(len(ids)), "pack files")
defer bar.Done()
wg.Go(func() error {
// examine all data the indexes have for the pack file
for b := range repo.Index().ListPacks(wgCtx, ids) {
blobs := b.Blobs
if len(blobs) == 0 {
Warnf("no blobs found for pack %v\n", b.PackID)
bar.Add(1)
continue
}
err = repository.StreamPack(wgCtx, repo.Backend().Load, repo.Key(), b.PackID, blobs, func(blob restic.BlobHandle, buf []byte, err error) error {
if err != nil {
// Fallback path
buf, err = repo.LoadBlob(wgCtx, blob.Type, blob.ID, nil)
if err != nil {
Warnf("failed to load blob %v: %v\n", blob.ID, err)
return nil
}
}
id, _, _, err := repo.SaveBlob(wgCtx, blob.Type, buf, restic.ID{}, true)
if !id.Equal(blob.ID) {
panic("pack id mismatch during upload")
}
return err
})
if err != nil {
return err
}
bar.Add(1)
}
return repo.Flush(wgCtx)
})
if err := wg.Wait(); err != nil {
return errors.Fatalf("%s", err)
}
bar.Done()
// remove salvaged packs from index
err = rebuildIndexFiles(ctx, gopts, repo, ids, nil)
if err != nil {
return errors.Fatalf("%s", err)
}
// cleanup
Warnf("removing salvaged pack files\n")
DeleteFiles(ctx, gopts, repo, ids, restic.PackFile)
Warnf("\nUse `restic repair snapshots --forget` to remove the corrupted data blobs from all snapshots\n")
return nil
}

View File

@@ -3,6 +3,7 @@ package main
import (
"context"
"github.com/restic/restic/internal/backend"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/restic"
"github.com/restic/restic/internal/walker"
@@ -83,7 +84,7 @@ func runRepairSnapshots(ctx context.Context, gopts GlobalOptions, opts RepairOpt
repo.SetDryRun()
}
snapshotLister, err := restic.MemorizeList(ctx, repo, restic.SnapshotFile)
snapshotLister, err := backend.MemorizeList(ctx, repo.Backend(), restic.SnapshotFile)
if err != nil {
return err
}
@@ -125,7 +126,7 @@ func runRepairSnapshots(ctx context.Context, gopts GlobalOptions, opts RepairOpt
node.Size = newSize
return node
},
RewriteFailedTree: func(_ restic.ID, path string, _ error) (restic.ID, error) {
RewriteFailedTree: func(nodeID restic.ID, path string, _ error) (restic.ID, error) {
if path == "/" {
Verbosef(" dir %q: not readable\n", path)
// remove snapshots with invalid root node
@@ -144,11 +145,11 @@ func runRepairSnapshots(ctx context.Context, gopts GlobalOptions, opts RepairOpt
changedCount := 0
for sn := range FindFilteredSnapshots(ctx, snapshotLister, repo, &opts.SnapshotFilter, args) {
Verbosef("\n%v\n", sn)
Verbosef("\nsnapshot %s of %v at %s)\n", sn.ID().Str(), sn.Paths, sn.Time)
changed, err := filterAndReplaceSnapshot(ctx, repo, sn,
func(ctx context.Context, sn *restic.Snapshot) (restic.ID, error) {
return rewriter.RewriteTree(ctx, repo, "/", *sn.Tree)
}, opts.DryRun, opts.Forget, nil, "repaired")
}, opts.DryRun, opts.Forget, "repaired")
if err != nil {
return errors.Fatalf("unable to rewrite snapshot ID %q: %v", sn.ID().Str(), err)
}

View File

@@ -3,6 +3,7 @@ package main
import (
"context"
"strings"
"sync"
"time"
"github.com/restic/restic/internal/debug"
@@ -37,9 +38,31 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
`,
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, args []string) error {
term, cancel := setupTermstatus()
defer cancel()
return runRestore(cmd.Context(), restoreOptions, globalOptions, term, args)
ctx := cmd.Context()
var wg sync.WaitGroup
cancelCtx, cancel := context.WithCancel(ctx)
defer func() {
// shutdown termstatus
cancel()
wg.Wait()
}()
term := termstatus.New(globalOptions.stdout, globalOptions.stderr, globalOptions.Quiet)
wg.Add(1)
go func() {
defer wg.Done()
term.Run(cancelCtx)
}()
// allow usage of warnf / verbosef
prevStdout, prevStderr := globalOptions.stdout, globalOptions.stderr
defer func() {
globalOptions.stdout, globalOptions.stderr = prevStdout, prevStderr
}()
stdioWrapper := ui.NewStdioWrapper(term)
globalOptions.stdout, globalOptions.stderr = stdioWrapper.Stdout(), stdioWrapper.Stderr()
return runRestore(ctx, restoreOptions, globalOptions, term, args)
},
}
@@ -145,7 +168,7 @@ func runRestore(ctx context.Context, opts RestoreOptions, gopts GlobalOptions,
Hosts: opts.Hosts,
Paths: opts.Paths,
Tags: opts.Tags,
}).FindLatest(ctx, repo, repo, snapshotIDString)
}).FindLatest(ctx, repo.Backend(), repo, snapshotIDString)
if err != nil {
return errors.Fatalf("failed to find snapshot: %v", err)
}
@@ -181,7 +204,7 @@ func runRestore(ctx context.Context, opts RestoreOptions, gopts GlobalOptions,
excludePatterns := filter.ParsePatterns(opts.Exclude)
insensitiveExcludePatterns := filter.ParsePatterns(opts.InsensitiveExclude)
selectExcludeFilter := func(item string, _ string, node *restic.Node) (selectedForRestore bool, childMayBeSelected bool) {
selectExcludeFilter := func(item string, dstpath string, node *restic.Node) (selectedForRestore bool, childMayBeSelected bool) {
matched, err := filter.List(excludePatterns, item)
if err != nil {
msg.E("error for exclude pattern: %v", err)
@@ -204,7 +227,7 @@ func runRestore(ctx context.Context, opts RestoreOptions, gopts GlobalOptions,
includePatterns := filter.ParsePatterns(opts.Include)
insensitiveIncludePatterns := filter.ParsePatterns(opts.InsensitiveInclude)
selectIncludeFilter := func(item string, _ string, node *restic.Node) (selectedForRestore bool, childMayBeSelected bool) {
selectIncludeFilter := func(item string, dstpath string, node *restic.Node) (selectedForRestore bool, childMayBeSelected bool) {
matched, childMayMatch, err := filter.ListWithChild(includePatterns, item)
if err != nil {
msg.E("error for include pattern: %v", err)

View File

@@ -3,7 +3,6 @@ package main
import (
"context"
"fmt"
"time"
"github.com/spf13/cobra"
"golang.org/x/sync/errgroup"
@@ -47,42 +46,11 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
},
}
type snapshotMetadata struct {
Hostname string
Time *time.Time
}
type snapshotMetadataArgs struct {
Hostname string
Time string
}
func (sma snapshotMetadataArgs) empty() bool {
return sma.Hostname == "" && sma.Time == ""
}
func (sma snapshotMetadataArgs) convert() (*snapshotMetadata, error) {
if sma.empty() {
return nil, nil
}
var timeStamp *time.Time
if sma.Time != "" {
t, err := time.ParseInLocation(TimeFormat, sma.Time, time.Local)
if err != nil {
return nil, errors.Fatalf("error in time option: %v\n", err)
}
timeStamp = &t
}
return &snapshotMetadata{Hostname: sma.Hostname, Time: timeStamp}, nil
}
// RewriteOptions collects all options for the rewrite command.
type RewriteOptions struct {
Forget bool
DryRun bool
Metadata snapshotMetadataArgs
restic.SnapshotFilter
excludePatternOptions
}
@@ -95,15 +63,11 @@ func init() {
f := cmdRewrite.Flags()
f.BoolVarP(&rewriteOptions.Forget, "forget", "", false, "remove original snapshots after creating new ones")
f.BoolVarP(&rewriteOptions.DryRun, "dry-run", "n", false, "do not do anything, just print what would be done")
f.StringVar(&rewriteOptions.Metadata.Hostname, "new-host", "", "replace hostname")
f.StringVar(&rewriteOptions.Metadata.Time, "new-time", "", "replace time of the backup")
initMultiSnapshotFilter(f, &rewriteOptions.SnapshotFilter, true)
initExcludePatternOptions(f, &rewriteOptions.excludePatternOptions)
}
type rewriteFilterFunc func(ctx context.Context, sn *restic.Snapshot) (restic.ID, error)
func rewriteSnapshot(ctx context.Context, repo *repository.Repository, sn *restic.Snapshot, opts RewriteOptions) (bool, error) {
if sn.Tree == nil {
return false, errors.Errorf("snapshot %v has nil tree", sn.ID().Str())
@@ -114,50 +78,33 @@ func rewriteSnapshot(ctx context.Context, repo *repository.Repository, sn *resti
return false, err
}
metadata, err := opts.Metadata.convert()
if err != nil {
return false, err
}
var filter rewriteFilterFunc
if len(rejectByNameFuncs) > 0 {
selectByName := func(nodepath string) bool {
for _, reject := range rejectByNameFuncs {
if reject(nodepath) {
return false
}
selectByName := func(nodepath string) bool {
for _, reject := range rejectByNameFuncs {
if reject(nodepath) {
return false
}
return true
}
rewriter := walker.NewTreeRewriter(walker.RewriteOpts{
RewriteNode: func(node *restic.Node, path string) *restic.Node {
if selectByName(path) {
return node
}
Verbosef(fmt.Sprintf("excluding %s\n", path))
return nil
},
DisableNodeCache: true,
})
filter = func(ctx context.Context, sn *restic.Snapshot) (restic.ID, error) {
return rewriter.RewriteTree(ctx, repo, "/", *sn.Tree)
}
} else {
filter = func(_ context.Context, sn *restic.Snapshot) (restic.ID, error) {
return *sn.Tree, nil
}
return true
}
rewriter := walker.NewTreeRewriter(walker.RewriteOpts{
RewriteNode: func(node *restic.Node, path string) *restic.Node {
if selectByName(path) {
return node
}
Verbosef(fmt.Sprintf("excluding %s\n", path))
return nil
},
DisableNodeCache: true,
})
return filterAndReplaceSnapshot(ctx, repo, sn,
filter, opts.DryRun, opts.Forget, metadata, "rewrite")
func(ctx context.Context, sn *restic.Snapshot) (restic.ID, error) {
return rewriter.RewriteTree(ctx, repo, "/", *sn.Tree)
}, opts.DryRun, opts.Forget, "rewrite")
}
func filterAndReplaceSnapshot(ctx context.Context, repo restic.Repository, sn *restic.Snapshot,
filter rewriteFilterFunc, dryRun bool, forget bool, newMetadata *snapshotMetadata, addTag string) (bool, error) {
func filterAndReplaceSnapshot(ctx context.Context, repo restic.Repository, sn *restic.Snapshot, filter func(ctx context.Context, sn *restic.Snapshot) (restic.ID, error), dryRun bool, forget bool, addTag string) (bool, error) {
wg, wgCtx := errgroup.WithContext(ctx)
repo.StartPackUploader(wgCtx, wg)
@@ -181,7 +128,7 @@ func filterAndReplaceSnapshot(ctx context.Context, repo restic.Repository, sn *r
if dryRun {
Verbosef("would delete empty snapshot\n")
} else {
h := backend.Handle{Type: restic.SnapshotFile, Name: sn.ID().String()}
h := restic.Handle{Type: restic.SnapshotFile, Name: sn.ID().String()}
if err = repo.Backend().Remove(ctx, h); err != nil {
return false, err
}
@@ -191,7 +138,7 @@ func filterAndReplaceSnapshot(ctx context.Context, repo restic.Repository, sn *r
return true, nil
}
if filteredTree == *sn.Tree && newMetadata == nil {
if filteredTree == *sn.Tree {
debug.Log("Snapshot %v not modified", sn)
return false, nil
}
@@ -204,14 +151,6 @@ func filterAndReplaceSnapshot(ctx context.Context, repo restic.Repository, sn *r
Verbosef("would remove old snapshot\n")
}
if newMetadata != nil && newMetadata.Time != nil {
Verbosef("would set time to %s\n", newMetadata.Time)
}
if newMetadata != nil && newMetadata.Hostname != "" {
Verbosef("would set hostname to %s\n", newMetadata.Hostname)
}
return true, nil
}
@@ -223,16 +162,6 @@ func filterAndReplaceSnapshot(ctx context.Context, repo restic.Repository, sn *r
sn.AddTags([]string{addTag})
}
if newMetadata != nil && newMetadata.Time != nil {
Verbosef("setting time to %s\n", *newMetadata.Time)
sn.Time = *newMetadata.Time
}
if newMetadata != nil && newMetadata.Hostname != "" {
Verbosef("setting host to %s\n", newMetadata.Hostname)
sn.Hostname = newMetadata.Hostname
}
// Save the new snapshot.
id, err := restic.SaveSnapshot(ctx, repo, sn)
if err != nil {
@@ -241,7 +170,7 @@ func filterAndReplaceSnapshot(ctx context.Context, repo restic.Repository, sn *r
Verbosef("saved new snapshot %v\n", id.Str())
if forget {
h := backend.Handle{Type: restic.SnapshotFile, Name: sn.ID().String()}
h := restic.Handle{Type: restic.SnapshotFile, Name: sn.ID().String()}
if err = repo.Backend().Remove(ctx, h); err != nil {
return false, err
}
@@ -252,8 +181,8 @@ func filterAndReplaceSnapshot(ctx context.Context, repo restic.Repository, sn *r
}
func runRewrite(ctx context.Context, opts RewriteOptions, gopts GlobalOptions, args []string) error {
if opts.excludePatternOptions.Empty() && opts.Metadata.empty() {
return errors.Fatal("Nothing to do: no excludes provided and no new metadata provided")
if opts.excludePatternOptions.Empty() {
return errors.Fatal("Nothing to do: no excludes provided")
}
repo, err := OpenRepository(ctx, gopts)
@@ -278,7 +207,7 @@ func runRewrite(ctx context.Context, opts RewriteOptions, gopts GlobalOptions, a
repo.SetDryRun()
}
snapshotLister, err := restic.MemorizeList(ctx, repo, restic.SnapshotFile)
snapshotLister, err := backend.MemorizeList(ctx, repo.Backend(), restic.SnapshotFile)
if err != nil {
return err
}
@@ -290,7 +219,7 @@ func runRewrite(ctx context.Context, opts RewriteOptions, gopts GlobalOptions, a
changedCount := 0
for sn := range FindFilteredSnapshots(ctx, snapshotLister, repo, &opts.SnapshotFilter, args) {
Verbosef("\n%v\n", sn)
Verbosef("\nsnapshot %s of %v at %s)\n", sn.ID().Str(), sn.Paths, sn.Time)
changed, err := rewriteSnapshot(ctx, repo, sn, opts)
if err != nil {
return errors.Fatalf("unable to rewrite snapshot ID %q: %v", sn.ID().Str(), err)

View File

@@ -9,13 +9,12 @@ import (
rtest "github.com/restic/restic/internal/test"
)
func testRunRewriteExclude(t testing.TB, gopts GlobalOptions, excludes []string, forget bool, metadata snapshotMetadataArgs) {
func testRunRewriteExclude(t testing.TB, gopts GlobalOptions, excludes []string, forget bool) {
opts := RewriteOptions{
excludePatternOptions: excludePatternOptions{
Excludes: excludes,
},
Forget: forget,
Metadata: metadata,
Forget: forget,
}
rtest.OK(t, runRewrite(context.TODO(), opts, gopts, nil))
@@ -39,7 +38,7 @@ func TestRewrite(t *testing.T) {
createBasicRewriteRepo(t, env)
// exclude some data
testRunRewriteExclude(t, env.gopts, []string{"3"}, false, snapshotMetadataArgs{Hostname: "", Time: ""})
testRunRewriteExclude(t, env.gopts, []string{"3"}, false)
snapshotIDs := testRunList(t, "snapshots", env.gopts)
rtest.Assert(t, len(snapshotIDs) == 2, "expected two snapshots, got %v", snapshotIDs)
testRunCheck(t, env.gopts)
@@ -51,7 +50,7 @@ func TestRewriteUnchanged(t *testing.T) {
snapshotID := createBasicRewriteRepo(t, env)
// use an exclude that will not exclude anything
testRunRewriteExclude(t, env.gopts, []string{"3dflkhjgdflhkjetrlkhjgfdlhkj"}, false, snapshotMetadataArgs{Hostname: "", Time: ""})
testRunRewriteExclude(t, env.gopts, []string{"3dflkhjgdflhkjetrlkhjgfdlhkj"}, false)
newSnapshotIDs := testRunList(t, "snapshots", env.gopts)
rtest.Assert(t, len(newSnapshotIDs) == 1, "expected one snapshot, got %v", newSnapshotIDs)
rtest.Assert(t, snapshotID == newSnapshotIDs[0], "snapshot id changed unexpectedly")
@@ -64,44 +63,11 @@ func TestRewriteReplace(t *testing.T) {
snapshotID := createBasicRewriteRepo(t, env)
// exclude some data
testRunRewriteExclude(t, env.gopts, []string{"3"}, true, snapshotMetadataArgs{Hostname: "", Time: ""})
newSnapshotIDs := testListSnapshots(t, env.gopts, 1)
testRunRewriteExclude(t, env.gopts, []string{"3"}, true)
newSnapshotIDs := testRunList(t, "snapshots", env.gopts)
rtest.Assert(t, len(newSnapshotIDs) == 1, "expected one snapshot, got %v", newSnapshotIDs)
rtest.Assert(t, snapshotID != newSnapshotIDs[0], "snapshot id should have changed")
// check forbids unused blobs, thus remove them first
testRunPrune(t, env.gopts, PruneOptions{MaxUnused: "0"})
testRunCheck(t, env.gopts)
}
func testRewriteMetadata(t *testing.T, metadata snapshotMetadataArgs) {
env, cleanup := withTestEnvironment(t)
defer cleanup()
createBasicRewriteRepo(t, env)
testRunRewriteExclude(t, env.gopts, []string{}, true, metadata)
repo, _ := OpenRepository(context.TODO(), env.gopts)
snapshots, err := restic.TestLoadAllSnapshots(context.TODO(), repo, nil)
rtest.OK(t, err)
rtest.Assert(t, len(snapshots) == 1, "expected one snapshot, got %v", len(snapshots))
newSnapshot := snapshots[0]
if metadata.Time != "" {
rtest.Assert(t, newSnapshot.Time.Format(TimeFormat) == metadata.Time, "New snapshot should have time %s", metadata.Time)
}
if metadata.Hostname != "" {
rtest.Assert(t, newSnapshot.Hostname == metadata.Hostname, "New snapshot should have host %s", metadata.Hostname)
}
}
func TestRewriteMetadata(t *testing.T) {
newHost := "new host"
newTime := "1999-01-01 11:11:11"
for _, metadata := range []snapshotMetadataArgs{
{Hostname: "", Time: newTime},
{Hostname: newHost, Time: ""},
{Hostname: newHost, Time: newTime},
} {
testRewriteMetadata(t, metadata)
}
}

View File

@@ -73,7 +73,7 @@ func runSnapshots(ctx context.Context, opts SnapshotOptions, gopts GlobalOptions
}
var snapshots restic.Snapshots
for sn := range FindFilteredSnapshots(ctx, repo, repo, &opts.SnapshotFilter, args) {
for sn := range FindFilteredSnapshots(ctx, repo.Backend(), repo, &opts.SnapshotFilter, args) {
snapshots = append(snapshots, sn)
}
snapshotGroups, grouped, err := restic.GroupSnapshots(snapshots, opts.GroupBy)
@@ -290,7 +290,7 @@ func PrintSnapshotGroupHeader(stdout io.Writer, groupKeyJSON string) error {
return nil
}
// Snapshot helps to print Snapshots as JSON with their ID included.
// Snapshot helps to print Snaphots as JSON with their ID included.
type Snapshot struct {
*restic.Snapshot

View File

@@ -8,10 +8,10 @@ import (
"strings"
"github.com/restic/chunker"
"github.com/restic/restic/internal/backend"
"github.com/restic/restic/internal/crypto"
"github.com/restic/restic/internal/repository"
"github.com/restic/restic/internal/restic"
"github.com/restic/restic/internal/restorer"
"github.com/restic/restic/internal/ui"
"github.com/restic/restic/internal/ui/table"
"github.com/restic/restic/internal/walker"
@@ -94,7 +94,7 @@ func runStats(ctx context.Context, opts StatsOptions, gopts GlobalOptions, args
}
}
snapshotLister, err := restic.MemorizeList(ctx, repo, restic.SnapshotFile)
snapshotLister, err := backend.MemorizeList(ctx, repo.Backend(), restic.SnapshotFile)
if err != nil {
return err
}
@@ -189,7 +189,7 @@ func runStats(ctx context.Context, opts StatsOptions, gopts GlobalOptions, args
return nil
}
func statsWalkSnapshot(ctx context.Context, snapshot *restic.Snapshot, repo restic.Loader, opts StatsOptions, stats *statsContainer) error {
func statsWalkSnapshot(ctx context.Context, snapshot *restic.Snapshot, repo restic.Repository, opts StatsOptions, stats *statsContainer) error {
if snapshot.Tree == nil {
return fmt.Errorf("snapshot %s has nil tree", snapshot.ID().Str())
}
@@ -202,10 +202,8 @@ func statsWalkSnapshot(ctx context.Context, snapshot *restic.Snapshot, repo rest
return restic.FindUsedBlobs(ctx, repo, restic.IDs{*snapshot.Tree}, stats.blobs, nil)
}
hardLinkIndex := restorer.NewHardlinkIndex[struct{}]()
err := walker.Walk(ctx, repo, *snapshot.Tree, walker.WalkVisitor{
ProcessNode: statsWalkTree(repo, opts, stats, hardLinkIndex),
})
uniqueInodes := make(map[uint64]struct{})
err := walker.Walk(ctx, repo, *snapshot.Tree, restic.NewIDSet(), statsWalkTree(repo, opts, stats, uniqueInodes))
if err != nil {
return fmt.Errorf("walking tree %s: %v", *snapshot.Tree, err)
}
@@ -213,13 +211,13 @@ func statsWalkSnapshot(ctx context.Context, snapshot *restic.Snapshot, repo rest
return nil
}
func statsWalkTree(repo restic.Loader, opts StatsOptions, stats *statsContainer, hardLinkIndex *restorer.HardlinkIndex[struct{}]) walker.WalkFunc {
return func(parentTreeID restic.ID, npath string, node *restic.Node, nodeErr error) error {
func statsWalkTree(repo restic.Repository, opts StatsOptions, stats *statsContainer, uniqueInodes map[uint64]struct{}) walker.WalkFunc {
return func(parentTreeID restic.ID, npath string, node *restic.Node, nodeErr error) (bool, error) {
if nodeErr != nil {
return nodeErr
return true, nodeErr
}
if node == nil {
return nil
return true, nil
}
if opts.countMode == countModeUniqueFilesByContents || opts.countMode == countModeBlobsPerFile {
@@ -249,7 +247,7 @@ func statsWalkTree(repo restic.Loader, opts StatsOptions, stats *statsContainer,
// is always a data blob since we're accessing it via a file's Content array
blobSize, found := repo.LookupBlobSize(blobID, restic.DataBlob)
if !found {
return fmt.Errorf("blob %s not found for tree %s", blobID, parentTreeID)
return true, fmt.Errorf("blob %s not found for tree %s", blobID, parentTreeID)
}
// count the blob's size, then add this blob by this
@@ -272,13 +270,15 @@ func statsWalkTree(repo restic.Loader, opts StatsOptions, stats *statsContainer,
// if inodes are present, only count each inode once
// (hard links do not increase restore size)
if !hardLinkIndex.Has(node.Inode, node.DeviceID) || node.Inode == 0 {
hardLinkIndex.Add(node.Inode, node.DeviceID, struct{}{})
if _, ok := uniqueInodes[node.Inode]; !ok || node.Inode == 0 {
uniqueInodes[node.Inode] = struct{}{}
stats.TotalSize += node.Size
}
return false, nil
}
return nil
return true, nil
}
}
@@ -365,9 +365,9 @@ func statsDebug(ctx context.Context, repo restic.Repository) error {
return nil
}
func statsDebugFileType(ctx context.Context, repo restic.Lister, tpe restic.FileType) (*sizeHistogram, error) {
func statsDebugFileType(ctx context.Context, repo restic.Repository, tpe restic.FileType) (*sizeHistogram, error) {
hist := newSizeHistogram(2 * repository.MaxPackSize)
err := repo.List(ctx, tpe, func(_ restic.ID, size int64) error {
err := repo.List(ctx, tpe, func(id restic.ID, size int64) error {
hist.Add(uint64(size))
return nil
})

View File

@@ -5,7 +5,6 @@ import (
"github.com/spf13/cobra"
"github.com/restic/restic/internal/backend"
"github.com/restic/restic/internal/debug"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/repository"
@@ -86,7 +85,7 @@ func changeTags(ctx context.Context, repo *repository.Repository, sn *restic.Sna
debug.Log("new snapshot saved as %v", id)
// Remove the old snapshot.
h := backend.Handle{Type: restic.SnapshotFile, Name: sn.ID().String()}
h := restic.Handle{Type: restic.SnapshotFile, Name: sn.ID().String()}
if err = repo.Backend().Remove(ctx, h); err != nil {
return false, err
}
@@ -120,7 +119,7 @@ func runTag(ctx context.Context, opts TagOptions, gopts GlobalOptions, args []st
}
changeCnt := 0
for sn := range FindFilteredSnapshots(ctx, repo, repo, &opts.SnapshotFilter, args) {
for sn := range FindFilteredSnapshots(ctx, repo.Backend(), repo, &opts.SnapshotFilter, args) {
changed, err := changeTags(ctx, repo, sn, opts.SetTags.Flatten(), opts.AddTags.Flatten(), opts.RemoveTags.Flatten())
if err != nil {
Warnf("unable to modify the tags for snapshot ID %q, ignoring: %v\n", sn.ID(), err)

View File

@@ -19,7 +19,7 @@ EXIT STATUS
Exit status is 0 if the command was successful, and non-zero if there was any error.
`,
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, _ []string) error {
RunE: func(cmd *cobra.Command, args []string) error {
return runUnlock(cmd.Context(), unlockOptions, globalOptions)
},
}

View File

@@ -1,7 +1,6 @@
package main
import (
"encoding/json"
"fmt"
"runtime"
@@ -21,32 +20,9 @@ EXIT STATUS
Exit status is 0 if the command was successful, and non-zero if there was any error.
`,
DisableAutoGenTag: true,
Run: func(_ *cobra.Command, _ []string) {
if globalOptions.JSON {
type jsonVersion struct {
Version string `json:"version"`
GoVersion string `json:"go_version"`
GoOS string `json:"go_os"`
GoArch string `json:"go_arch"`
}
jsonS := jsonVersion{
Version: version,
GoVersion: runtime.Version(),
GoOS: runtime.GOOS,
GoArch: runtime.GOARCH,
}
err := json.NewEncoder(globalOptions.stdout).Encode(jsonS)
if err != nil {
Warnf("JSON encode failed: %v\n", err)
return
}
} else {
fmt.Printf("restic %s compiled with %v on %v/%v\n",
version, runtime.Version(), runtime.GOOS, runtime.GOARCH)
}
Run: func(cmd *cobra.Command, args []string) {
fmt.Printf("restic %s compiled with %v on %v/%v\n",
version, runtime.Version(), runtime.GOOS, runtime.GOARCH)
},
}

View File

@@ -1,126 +0,0 @@
package main
import (
"context"
"log"
"net/http"
"os"
"time"
"github.com/spf13/cobra"
"golang.org/x/net/webdav"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/restic"
"github.com/restic/restic/internal/server/rofs"
)
var cmdWebDAV = &cobra.Command{
Use: "webdav [flags]",
Short: "runs a WebDAV server for the repository",
Long: `
The webdav command runs a WebDAV server for the reposiotry that you can then access via a WebDAV client.
`,
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, args []string) error {
return runWebDAV(cmd.Context(), webdavOptions, globalOptions, args)
},
}
// WebDAVOptions collects all options for the webdav command.
type WebDAVOptions struct {
Listen string
restic.SnapshotFilter
TimeTemplate string
PathTemplates []string
}
var webdavOptions WebDAVOptions
func init() {
cmdRoot.AddCommand(cmdWebDAV)
fs := cmdWebDAV.Flags()
fs.StringVarP(&webdavOptions.Listen, "listen", "l", "localhost:3080", "set the listen host name and `address`")
initMultiSnapshotFilter(fs, &webdavOptions.SnapshotFilter, true)
fs.StringArrayVar(&webdavOptions.PathTemplates, "path-template", nil, "set `template` for path names (can be specified multiple times)")
fs.StringVar(&webdavOptions.TimeTemplate, "time-template", time.RFC3339, "set `template` to use for times")
}
func runWebDAV(ctx context.Context, opts WebDAVOptions, gopts GlobalOptions, args []string) error {
if len(args) > 0 {
return errors.Fatal("this command does not accept additional arguments")
}
repo, err := OpenRepository(ctx, gopts)
if err != nil {
return err
}
lock, ctx, err := lockRepo(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err
}
bar := newIndexProgress(gopts.Quiet, gopts.JSON)
err = repo.LoadIndex(ctx, bar)
if err != nil {
return err
}
errorLogger := log.New(os.Stderr, "error log: ", log.Flags())
cfg := rofs.Config{
Filter: opts.SnapshotFilter,
TimeTemplate: opts.TimeTemplate,
PathTemplates: opts.PathTemplates,
}
root, err := rofs.New(ctx, repo, cfg)
if err != nil {
return err
}
// root := os.DirFS(".")
// h, err := webdav.NewWebDAV(ctx, root)
// if err != nil {
// return err
// }
// root := fstest.MapFS{
// "foobar": &fstest.MapFile{
// Data: []byte("foobar test content"),
// Mode: 0644,
// ModTime: time.Now(),
// },
// "test.txt": &fstest.MapFile{
// Data: []byte("other file"),
// Mode: 0640,
// ModTime: time.Now(),
// },
// }
logRequest := func(req *http.Request, err error) {
errorLogger.Printf("req %v %v -> %v\n", req.Method, req.URL.Path, err)
}
srv := &http.Server{
ReadTimeout: 60 * time.Second,
WriteTimeout: 60 * time.Second,
Addr: opts.Listen,
// Handler: http.FileServer(http.FS(root)),
Handler: &webdav.Handler{
FileSystem: rofs.WebDAVFS(root),
LockSystem: webdav.NewMemLS(),
Logger: logRequest,
},
ErrorLog: errorLogger,
}
return srv.ListenAndServe()
}

View File

@@ -3,6 +3,8 @@ package main
import (
"context"
"golang.org/x/sync/errgroup"
"github.com/restic/restic/internal/restic"
)
@@ -21,21 +23,46 @@ func DeleteFilesChecked(ctx context.Context, gopts GlobalOptions, repo restic.Re
// deleteFiles deletes the given fileList of fileType in parallel
// if ignoreError=true, it will print a warning if there was an error, else it will abort.
func deleteFiles(ctx context.Context, gopts GlobalOptions, ignoreError bool, repo restic.Repository, fileList restic.IDSet, fileType restic.FileType) error {
bar := newProgressMax(!gopts.JSON && !gopts.Quiet, 0, "files deleted")
defer bar.Done()
return restic.ParallelRemove(ctx, repo, fileList, fileType, func(id restic.ID, err error) error {
if err != nil {
if !gopts.JSON {
Warnf("unable to remove %v/%v from the repository\n", fileType, id)
totalCount := len(fileList)
fileChan := make(chan restic.ID)
wg, ctx := errgroup.WithContext(ctx)
wg.Go(func() error {
defer close(fileChan)
for id := range fileList {
select {
case fileChan <- id:
case <-ctx.Done():
return ctx.Err()
}
if !ignoreError {
return err
}
}
if !gopts.JSON && gopts.verbosity > 2 {
Verbosef("removed %v/%v\n", fileType, id)
}
return nil
}, bar)
})
bar := newProgressMax(!gopts.JSON && !gopts.Quiet, uint64(totalCount), "files deleted")
defer bar.Done()
// deleting files is IO-bound
workerCount := repo.Connections()
for i := 0; i < int(workerCount); i++ {
wg.Go(func() error {
for id := range fileChan {
h := restic.Handle{Type: fileType, Name: id.String()}
err := repo.Backend().Remove(ctx, h)
if err != nil {
if !gopts.JSON {
Warnf("unable to remove %v from the repository\n", h)
}
if !ignoreError {
return err
}
}
if !gopts.JSON && gopts.verbosity > 2 {
Verbosef("removed %v\n", h)
}
bar.Add(1)
}
return nil
})
}
err := wg.Wait()
return err
}

View File

@@ -426,7 +426,7 @@ func readExcludePatternsFromFiles(excludeFiles []string) ([]string, error) {
return scanner.Err()
}()
if err != nil {
return nil, fmt.Errorf("failed to read excludes from file %q: %w", filename, err)
return nil, err
}
}
return excludes, nil

View File

@@ -3,6 +3,7 @@ package main
import (
"context"
"github.com/restic/restic/internal/backend"
"github.com/restic/restic/internal/restic"
"github.com/spf13/pflag"
)
@@ -32,7 +33,7 @@ func FindFilteredSnapshots(ctx context.Context, be restic.Lister, loader restic.
out := make(chan *restic.Snapshot)
go func() {
defer close(out)
be, err := restic.MemorizeList(ctx, be, restic.SnapshotFile)
be, err := backend.MemorizeList(ctx, be, restic.SnapshotFile)
if err != nil {
Warnf("could not load snapshots: %v\n", err)
return

View File

@@ -43,12 +43,12 @@ import (
"golang.org/x/term"
)
var version = "0.16.4-dev (compiled manually)"
var version = "0.16.5"
// TimeFormat is the format used for all timestamps printed by restic.
const TimeFormat = "2006-01-02 15:04:05"
type backendWrapper func(r backend.Backend) (backend.Backend, error)
type backendWrapper func(r restic.Backend) (restic.Backend, error)
// GlobalOptions hold all global options for restic.
type GlobalOptions struct {
@@ -128,7 +128,7 @@ func init() {
f.StringVarP(&globalOptions.KeyHint, "key-hint", "", "", "`key` ID of key to try decrypting first (default: $RESTIC_KEY_HINT)")
f.StringVarP(&globalOptions.PasswordCommand, "password-command", "", "", "shell `command` to obtain the repository password from (default: $RESTIC_PASSWORD_COMMAND)")
f.BoolVarP(&globalOptions.Quiet, "quiet", "q", false, "do not output comprehensive progress report")
// use empty parameter name as `-v, --verbose n` instead of the correct `--verbose=n` is confusing
// use empty paremeter name as `-v, --verbose n` instead of the correct `--verbose=n` is confusing
f.CountVarP(&globalOptions.Verbose, "verbose", "v", "be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)")
f.BoolVar(&globalOptions.NoLock, "no-lock", false, "do not lock the repository, this allows some operations on read-only repositories")
f.DurationVar(&globalOptions.RetryLock, "retry-lock", 0, "retry to lock the repository if it is already locked, takes a value like 5m or 2h (default: no retries)")
@@ -556,7 +556,7 @@ func OpenRepository(ctx context.Context, opts GlobalOptions) (*repository.Reposi
func parseConfig(loc location.Location, opts options.Options) (interface{}, error) {
cfg := loc.Config
if cfg, ok := cfg.(backend.ApplyEnvironmenter); ok {
if cfg, ok := cfg.(restic.ApplyEnvironmenter); ok {
cfg.ApplyEnvironment("")
}
@@ -571,14 +571,14 @@ func parseConfig(loc location.Location, opts options.Options) (interface{}, erro
}
// Open the backend specified by a location config.
func open(ctx context.Context, s string, gopts GlobalOptions, opts options.Options) (backend.Backend, error) {
func open(ctx context.Context, s string, gopts GlobalOptions, opts options.Options) (restic.Backend, error) {
debug.Log("parsing location %v", location.StripPassword(gopts.backends, s))
loc, err := location.Parse(gopts.backends, s)
if err != nil {
return nil, errors.Fatalf("parsing repository location failed: %v", err)
}
var be backend.Backend
var be restic.Backend
cfg, err := parseConfig(loc, opts)
if err != nil {
@@ -616,7 +616,7 @@ func open(ctx context.Context, s string, gopts GlobalOptions, opts options.Optio
}
// check if config is there
fi, err := be.Stat(ctx, backend.Handle{Type: restic.ConfigFile})
fi, err := be.Stat(ctx, restic.Handle{Type: restic.ConfigFile})
if err != nil {
return nil, errors.Fatalf("unable to open config file: %v\nIs there a repository at the following location?\n%v", err, location.StripPassword(gopts.backends, s))
}
@@ -629,7 +629,7 @@ func open(ctx context.Context, s string, gopts GlobalOptions, opts options.Optio
}
// Create the backend specified by URI.
func create(ctx context.Context, s string, gopts GlobalOptions, opts options.Options) (backend.Backend, error) {
func create(ctx context.Context, s string, gopts GlobalOptions, opts options.Options) (restic.Backend, error) {
debug.Log("parsing location %v", location.StripPassword(gopts.backends, s))
loc, err := location.Parse(gopts.backends, s)
if err != nil {

View File

@@ -12,7 +12,6 @@ import (
"sync"
"testing"
"github.com/restic/restic/internal/backend"
"github.com/restic/restic/internal/backend/retry"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/options"
@@ -205,7 +204,7 @@ func withTestEnvironment(t testing.TB) (env *testEnvironment, cleanup func()) {
extended: make(options.Options),
// replace this hook with "nil" if listing a filetype more than once is necessary
backendTestHook: func(r backend.Backend) (backend.Backend, error) { return newOrderedListOnceBackend(r), nil },
backendTestHook: func(r restic.Backend) (restic.Backend, error) { return newOrderedListOnceBackend(r), nil },
// start with default set of backends
backends: globalOptions.backends,
}
@@ -249,7 +248,7 @@ func removePacks(gopts GlobalOptions, t testing.TB, remove restic.IDSet) {
rtest.OK(t, err)
for id := range remove {
rtest.OK(t, r.Backend().Remove(context.TODO(), backend.Handle{Type: restic.PackFile, Name: id.String()}))
rtest.OK(t, r.Backend().Remove(context.TODO(), restic.Handle{Type: restic.PackFile, Name: id.String()}))
}
}
@@ -272,7 +271,7 @@ func removePacksExcept(gopts GlobalOptions, t testing.TB, keep restic.IDSet, rem
if treePacks.Has(id) != removeTreePacks || keep.Has(id) {
return nil
}
return r.Backend().Remove(context.TODO(), backend.Handle{Type: restic.PackFile, Name: id.String()})
return r.Backend().Remove(context.TODO(), restic.Handle{Type: restic.PackFile, Name: id.String()})
}))
}

View File

@@ -8,7 +8,6 @@ import (
"path/filepath"
"testing"
"github.com/restic/restic/internal/backend"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/restic"
rtest "github.com/restic/restic/internal/test"
@@ -42,12 +41,12 @@ func TestCheckRestoreNoLock(t *testing.T) {
// backends (like e.g. Amazon S3) as the second listing may be inconsistent to what
// is expected by the first listing + some operations.
type listOnceBackend struct {
backend.Backend
restic.Backend
listedFileType map[restic.FileType]bool
strictOrder bool
}
func newListOnceBackend(be backend.Backend) *listOnceBackend {
func newListOnceBackend(be restic.Backend) *listOnceBackend {
return &listOnceBackend{
Backend: be,
listedFileType: make(map[restic.FileType]bool),
@@ -55,7 +54,7 @@ func newListOnceBackend(be backend.Backend) *listOnceBackend {
}
}
func newOrderedListOnceBackend(be backend.Backend) *listOnceBackend {
func newOrderedListOnceBackend(be restic.Backend) *listOnceBackend {
return &listOnceBackend{
Backend: be,
listedFileType: make(map[restic.FileType]bool),
@@ -63,7 +62,7 @@ func newOrderedListOnceBackend(be backend.Backend) *listOnceBackend {
}
}
func (be *listOnceBackend) List(ctx context.Context, t restic.FileType, fn func(backend.FileInfo) error) error {
func (be *listOnceBackend) List(ctx context.Context, t restic.FileType, fn func(restic.FileInfo) error) error {
if t != restic.LockFile && be.listedFileType[t] {
return errors.Errorf("tried listing type %v the second time", t)
}
@@ -78,7 +77,7 @@ func TestListOnce(t *testing.T) {
env, cleanup := withTestEnvironment(t)
defer cleanup()
env.gopts.backendTestHook = func(r backend.Backend) (backend.Backend, error) {
env.gopts.backendTestHook = func(r restic.Backend) (restic.Backend, error) {
return newListOnceBackend(r), nil
}
pruneOpts := PruneOptions{MaxUnused: "0"}
@@ -105,10 +104,10 @@ func (r *writeToOnly) WriteTo(w io.Writer) (int64, error) {
}
type onlyLoadWithWriteToBackend struct {
backend.Backend
restic.Backend
}
func (be *onlyLoadWithWriteToBackend) Load(ctx context.Context, h backend.Handle,
func (be *onlyLoadWithWriteToBackend) Load(ctx context.Context, h restic.Handle,
length int, offset int64, fn func(rd io.Reader) error) error {
return be.Backend.Load(ctx, h, length, offset, func(rd io.Reader) error {
@@ -121,7 +120,7 @@ func TestBackendLoadWriteTo(t *testing.T) {
defer cleanup()
// setup backend which only works if it's WriteTo method is correctly propagated upwards
env.gopts.backendInnerTestHook = func(r backend.Backend) (backend.Backend, error) {
env.gopts.backendInnerTestHook = func(r restic.Backend) (restic.Backend, error) {
return &onlyLoadWithWriteToBackend{Backend: r}, nil
}
@@ -141,7 +140,7 @@ func TestFindListOnce(t *testing.T) {
env, cleanup := withTestEnvironment(t)
defer cleanup()
env.gopts.backendTestHook = func(r backend.Backend) (backend.Backend, error) {
env.gopts.backendTestHook = func(r restic.Backend) (restic.Backend, error) {
return newListOnceBackend(r), nil
}
@@ -159,7 +158,7 @@ func TestFindListOnce(t *testing.T) {
snapshotIDs := restic.NewIDSet()
// specify the two oldest snapshots explicitly and use "latest" to reference the newest one
for sn := range FindFilteredSnapshots(context.TODO(), repo, repo, &restic.SnapshotFilter{}, []string{
for sn := range FindFilteredSnapshots(context.TODO(), repo.Backend(), repo, &restic.SnapshotFilter{}, []string{
secondSnapshot[0].String(),
secondSnapshot[1].String()[:8],
"latest",

View File

@@ -6,7 +6,6 @@ import (
"sync"
"time"
"github.com/restic/restic/internal/backend"
"github.com/restic/restic/internal/debug"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/restic"
@@ -132,7 +131,7 @@ type refreshLockRequest struct {
result chan bool
}
func refreshLocks(ctx context.Context, backend backend.Backend, lockInfo *lockContext, refreshed chan<- struct{}, forceRefresh <-chan refreshLockRequest) {
func refreshLocks(ctx context.Context, backend restic.Backend, lockInfo *lockContext, refreshed chan<- struct{}, forceRefresh <-chan refreshLockRequest) {
debug.Log("start")
lock := lockInfo.lock
ticker := time.NewTicker(refreshInterval)
@@ -258,8 +257,8 @@ func monitorLockRefresh(ctx context.Context, lockInfo *lockContext, refreshed <-
}
}
func tryRefreshStaleLock(ctx context.Context, be backend.Backend, lock *restic.Lock, cancel context.CancelFunc) bool {
freeze := backend.AsBackend[backend.FreezeBackend](be)
func tryRefreshStaleLock(ctx context.Context, backend restic.Backend, lock *restic.Lock, cancel context.CancelFunc) bool {
freeze := restic.AsBackend[restic.FreezeBackend](backend)
if freeze != nil {
debug.Log("freezing backend")
freeze.Freeze()

View File

@@ -9,7 +9,6 @@ import (
"testing"
"time"
"github.com/restic/restic/internal/backend"
"github.com/restic/restic/internal/backend/location"
"github.com/restic/restic/internal/backend/mem"
"github.com/restic/restic/internal/debug"
@@ -105,11 +104,11 @@ func TestLockConflict(t *testing.T) {
}
type writeOnceBackend struct {
backend.Backend
restic.Backend
written bool
}
func (b *writeOnceBackend) Save(ctx context.Context, h backend.Handle, rd backend.RewindReader) error {
func (b *writeOnceBackend) Save(ctx context.Context, h restic.Handle, rd restic.RewindReader) error {
if b.written {
return fmt.Errorf("fail after first write")
}
@@ -118,7 +117,7 @@ func (b *writeOnceBackend) Save(ctx context.Context, h backend.Handle, rd backen
}
func TestLockFailedRefresh(t *testing.T) {
repo, cleanup, env := openLockTestRepo(t, func(r backend.Backend) (backend.Backend, error) {
repo, cleanup, env := openLockTestRepo(t, func(r restic.Backend) (restic.Backend, error) {
return &writeOnceBackend{Backend: r}, nil
})
defer cleanup()
@@ -144,11 +143,11 @@ func TestLockFailedRefresh(t *testing.T) {
}
type loggingBackend struct {
backend.Backend
restic.Backend
t *testing.T
}
func (b *loggingBackend) Save(ctx context.Context, h backend.Handle, rd backend.RewindReader) error {
func (b *loggingBackend) Save(ctx context.Context, h restic.Handle, rd restic.RewindReader) error {
b.t.Logf("save %v @ %v", h, time.Now())
err := b.Backend.Save(ctx, h, rd)
b.t.Logf("save finished %v @ %v", h, time.Now())
@@ -156,7 +155,7 @@ func (b *loggingBackend) Save(ctx context.Context, h backend.Handle, rd backend.
}
func TestLockSuccessfulRefresh(t *testing.T) {
repo, cleanup, env := openLockTestRepo(t, func(r backend.Backend) (backend.Backend, error) {
repo, cleanup, env := openLockTestRepo(t, func(r restic.Backend) (restic.Backend, error) {
return &loggingBackend{
Backend: r,
t: t,
@@ -194,12 +193,12 @@ func TestLockSuccessfulRefresh(t *testing.T) {
}
type slowBackend struct {
backend.Backend
restic.Backend
m sync.Mutex
sleep time.Duration
}
func (b *slowBackend) Save(ctx context.Context, h backend.Handle, rd backend.RewindReader) error {
func (b *slowBackend) Save(ctx context.Context, h restic.Handle, rd restic.RewindReader) error {
b.m.Lock()
sleep := b.sleep
b.m.Unlock()
@@ -209,7 +208,7 @@ func (b *slowBackend) Save(ctx context.Context, h backend.Handle, rd backend.Rew
func TestLockSuccessfulStaleRefresh(t *testing.T) {
var sb *slowBackend
repo, cleanup, env := openLockTestRepo(t, func(r backend.Backend) (backend.Backend, error) {
repo, cleanup, env := openLockTestRepo(t, func(r restic.Backend) (restic.Backend, error) {
sb = &slowBackend{Backend: r}
return sb, nil
})

View File

@@ -37,7 +37,7 @@ The full documentation can be found at https://restic.readthedocs.io/ .
SilenceUsage: true,
DisableAutoGenTag: true,
PersistentPreRunE: func(c *cobra.Command, _ []string) error {
PersistentPreRunE: func(c *cobra.Command, args []string) error {
// set verbosity, default is one
globalOptions.verbosity = 1
if globalOptions.Quiet && globalOptions.Verbose > 0 {

View File

@@ -30,7 +30,7 @@ func calculateProgressInterval(show bool, json bool) time.Duration {
}
// newTerminalProgressMax returns a progress.Counter that prints to stdout or terminal if provided.
func newGenericProgressMax(show bool, max uint64, description string, print func(status string, final bool)) *progress.Counter {
func newGenericProgressMax(show bool, max uint64, description string, print func(status string)) *progress.Counter {
if !show {
return nil
}
@@ -46,18 +46,16 @@ func newGenericProgressMax(show bool, max uint64, description string, print func
ui.FormatDuration(d), ui.FormatPercent(v, max), v, max, description)
}
print(status, final)
print(status)
if final {
fmt.Print("\n")
}
})
}
func newTerminalProgressMax(show bool, max uint64, description string, term *termstatus.Terminal) *progress.Counter {
return newGenericProgressMax(show, max, description, func(status string, final bool) {
if final {
term.SetStatus([]string{})
term.Print(status)
} else {
term.SetStatus([]string{status})
}
return newGenericProgressMax(show, max, description, func(status string) {
term.SetStatus([]string{status})
})
}
@@ -66,7 +64,7 @@ func newProgressMax(show bool, max uint64, description string) *progress.Counter
return newGenericProgressMax(show, max, description, printProgress)
}
func printProgress(status string, final bool) {
func printProgress(status string) {
canUpdateStatus := stdoutCanUpdateStatus()
@@ -97,9 +95,6 @@ func printProgress(status string, final bool) {
}
_, _ = os.Stdout.Write([]byte(clear + status + carriageControl))
if final {
_, _ = os.Stdout.Write([]byte("\n"))
}
}
func newIndexProgress(quiet bool, json bool) *progress.Counter {
@@ -109,21 +104,3 @@ func newIndexProgress(quiet bool, json bool) *progress.Counter {
func newIndexTerminalProgress(quiet bool, json bool, term *termstatus.Terminal) *progress.Counter {
return newTerminalProgressMax(!quiet && !json && stdoutIsTerminal(), 0, "index files loaded", term)
}
type terminalProgressPrinter struct {
term *termstatus.Terminal
ui.Message
show bool
}
func (t *terminalProgressPrinter) NewCounter(description string) *progress.Counter {
return newTerminalProgressMax(t.show, 0, description, t.term)
}
func newTerminalProgressPrinter(verbosity uint, term *termstatus.Terminal) progress.Printer {
return &terminalProgressPrinter{
term: term,
Message: *ui.NewMessage(term, verbosity),
show: verbosity > 0,
}
}

View File

@@ -1,43 +0,0 @@
package main
import (
"context"
"sync"
"github.com/restic/restic/internal/ui"
"github.com/restic/restic/internal/ui/termstatus"
)
// setupTermstatus creates a new termstatus and reroutes globalOptions.{stdout,stderr} to it
// The returned function must be called to shut down the termstatus,
//
// Expected usage:
// ```
// term, cancel := setupTermstatus()
// defer cancel()
// // do stuff
// ```
func setupTermstatus() (*termstatus.Terminal, func()) {
var wg sync.WaitGroup
// only shutdown once cancel is called to ensure that no output is lost
cancelCtx, cancel := context.WithCancel(context.Background())
term := termstatus.New(globalOptions.stdout, globalOptions.stderr, globalOptions.Quiet)
wg.Add(1)
go func() {
defer wg.Done()
term.Run(cancelCtx)
}()
// use the termstatus for stdout/stderr
prevStdout, prevStderr := globalOptions.stdout, globalOptions.stderr
stdioWrapper := ui.NewStdioWrapper(term)
globalOptions.stdout, globalOptions.stderr = stdioWrapper.Stdout(), stdioWrapper.Stderr()
return term, func() {
// shutdown termstatus
globalOptions.stdout, globalOptions.stderr = prevStdout, prevStderr
cancel()
wg.Wait()
}
}

View File

@@ -35,15 +35,15 @@ environment variable ``RESTIC_REPOSITORY_FILE``.
For automating the supply of the repository password to restic, several options
exist:
* Setting the environment variable ``RESTIC_PASSWORD``
* Setting the environment variable ``RESTIC_PASSWORD``
* Specifying the path to a file with the password via the option
``--password-file`` or the environment variable ``RESTIC_PASSWORD_FILE``
* Configuring a program to be called when the password is needed via the
option ``--password-command`` or the environment variable
``RESTIC_PASSWORD_COMMAND``
* Specifying the path to a file with the password via the option
``--password-file`` or the environment variable ``RESTIC_PASSWORD_FILE``
* Configuring a program to be called when the password is needed via the
option ``--password-command`` or the environment variable
``RESTIC_PASSWORD_COMMAND``
The ``init`` command has an option called ``--repository-version`` which can
be used to explicitly set the version of the new repository. By default, the
current stable version is used (see table below). The alias ``latest`` will
@@ -487,8 +487,7 @@ Backblaze B2
Different from the B2 backend, restic's S3 backend will only hide no longer
necessary files. Thus, make sure to setup lifecycle rules to eventually
delete hidden files. The lifecycle setting "Keep only the last version of the file"
will keep only the most current version of a file. Read the [Backblaze documentation](https://www.backblaze.com/docs/cloud-storage-lifecycle-rules).
delete hidden files.
Restic can backup data to any Backblaze B2 bucket. You need to first setup the
following environment variables with the credentials you can find in the
@@ -549,17 +548,23 @@ For authentication export one of the following variables:
# For SAS
$ export AZURE_ACCOUNT_SAS=<SAS_TOKEN>
For authentication using ``az login`` set the resource group name and ensure the user has
the minimum permissions of the role assignment ``Storage Blob Data Contributor`` on Azure RBAC.
For authentication using ``az login`` ensure the user has
the minimum permissions of the role assignment ``Storage Blob Data Contributor`` on Azure RBAC
for the storage account.
.. code-block:: console
$ export AZURE_RESOURCE_GROUP=<RESOURCE_GROUP_NAME>
$ az login
Alternatively, if run on Azure, restic will automatically uses service accounts configured
Alternatively, if run on Azure, restic will automatically use service accounts configured
via the standard environment variables or Workload / Managed Identities.
To enforce the use of the Azure CLI credential when other credentials are present, set the following environment variable:
.. code-block:: console
$ export AZURE_FORCE_CLI_CREDENTIAL=true
Restic will by default use Azure's global domain ``core.windows.net`` as endpoint suffix.
You can specify other suffixes as follows:
@@ -717,9 +722,9 @@ For debugging rclone, you can set the environment variable ``RCLONE_VERBOSE=2``.
The rclone backend has three additional options:
* ``-o rclone.program`` specifies the path to rclone, the default value is just ``rclone``
* ``-o rclone.args`` allows setting the arguments passed to rclone, by default this is ``serve restic --stdio --b2-hard-delete``
* ``-o rclone.timeout`` specifies timeout for waiting on repository opening, the default value is ``1m``
* ``-o rclone.program`` specifies the path to rclone, the default value is just ``rclone``
* ``-o rclone.args`` allows setting the arguments passed to rclone, by default this is ``serve restic --stdio --b2-hard-delete``
* ``-o rclone.timeout`` specifies timeout for waiting on repository opening, the default value is ``1m``
The reason for the ``--b2-hard-delete`` parameters can be found in the corresponding GitHub `issue #1657`_.

View File

@@ -170,10 +170,10 @@ On **Unix** (including Linux and Mac), given that a file lives at the same
location as a file in a previous backup, the following file metadata
attributes have to match for its contents to be presumed unchanged:
* Modification timestamp (mtime).
* Metadata change timestamp (ctime).
* File size.
* Inode number (internal number used to reference a file in a filesystem).
* Modification timestamp (mtime).
* Metadata change timestamp (ctime).
* File size.
* Inode number (internal number used to reference a file in a filesystem).
The reason for requiring both mtime and ctime to match is that Unix programs
can freely change mtime (and some do). In such cases, a ctime change may be
@@ -182,9 +182,9 @@ the only hint that a file did change.
The following ``restic backup`` command line flags modify the change detection
rules:
* ``--force``: turn off change detection and rescan all files.
* ``--ignore-ctime``: require mtime to match, but allow ctime to differ.
* ``--ignore-inode``: require mtime to match, but allow inode number
* ``--force``: turn off change detection and rescan all files.
* ``--ignore-ctime``: require mtime to match, but allow ctime to differ.
* ``--ignore-inode``: require mtime to match, but allow inode number
and ctime to differ.
The option ``--ignore-inode`` exists to support FUSE-based filesystems and
@@ -250,9 +250,9 @@ It can be used like this:
This instructs restic to exclude files matching the following criteria:
* All files matching ``*.c`` (parameter ``--exclude``)
* All files matching ``*.go`` (second line in ``excludes.txt``)
* All files and sub-directories named ``bar`` which reside somewhere below a directory called ``foo`` (fourth line in ``excludes.txt``)
* All files matching ``*.c`` (parameter ``--exclude``)
* All files matching ``*.go`` (second line in ``excludes.txt``)
* All files and sub-directories named ``bar`` which reside somewhere below a directory called ``foo`` (fourth line in ``excludes.txt``)
Patterns use the syntax of the Go function
`filepath.Match <https://pkg.go.dev/path/filepath#Match>`__
@@ -270,8 +270,8 @@ environment variable (depending on your operating system).
Patterns need to match on complete path components. For example, the pattern ``foo``:
* matches ``/dir1/foo/dir2/file`` and ``/dir/foo``
* does not match ``/dir/foobar`` or ``barfoo``
* matches ``/dir1/foo/dir2/file`` and ``/dir/foo``
* does not match ``/dir/foobar`` or ``barfoo``
A trailing ``/`` is ignored, a leading ``/`` anchors the pattern at the root directory.
This means, ``/bin`` matches ``/bin/bash`` but does not match ``/usr/bin/restic``.
@@ -281,9 +281,9 @@ e.g. ``b*ash`` matches ``/bin/bash`` but does not match ``/bin/ash``. For this,
the special wildcard ``**`` can be used to match arbitrary sub-directories: The
pattern ``foo/**/bar`` matches:
* ``/dir1/foo/dir2/bar/file``
* ``/foo/bar/file``
* ``/tmp/foo/bar``
* ``/dir1/foo/dir2/bar/file``
* ``/foo/bar/file``
* ``/tmp/foo/bar``
Spaces in patterns listed in an exclude file can be specified verbatim. That is,
in order to exclude a file named ``foo bar star.txt``, put that just as it reads
@@ -298,9 +298,9 @@ some escaping in order to pass the name/pattern as a single argument to restic.
On most Unixy shells, you can either quote or use backslashes. For example:
* ``--exclude='foo bar star/foo.txt'``
* ``--exclude="foo bar star/foo.txt"``
* ``--exclude=foo\ bar\ star/foo.txt``
* ``--exclude='foo bar star/foo.txt'``
* ``--exclude="foo bar star/foo.txt"``
* ``--exclude=foo\ bar\ star/foo.txt``
If a pattern starts with exclamation mark and matches a file that
was previously matched by a regular pattern, the match is cancelled.
@@ -381,8 +381,8 @@ contains one *pattern* per line. The file must be encoded as UTF-8, or UTF-16
with a byte-order mark. Leading and trailing whitespace is removed from the
patterns. Empty lines and lines starting with a ``#`` are ignored and each
pattern is expanded when read, such that special characters in it are expanded
according to the syntax described in the documentation of the Go function
`filepath.Match <https://pkg.go.dev/path/filepath#Match>`__.
using the Go function `filepath.Glob <https://pkg.go.dev/path/filepath#Glob>`__
- please see its documentation for the syntax you can use in the patterns.
The argument passed to ``--files-from-verbatim`` must be the name of a text file
that contains one *path* per line, e.g. as generated by GNU ``find`` with the
@@ -482,76 +482,43 @@ want to save the access time for files and directories, you can pass the
``--with-atime`` option to the ``backup`` command.
Note that ``restic`` does not back up some metadata associated with files. Of
particular note are:
particular note are::
* File creation date on Unix platforms
* Inode flags on Unix platforms
* File ownership and ACLs on Windows
* The "hidden" flag on Windows
Reading data from a command
***************************
Sometimes, it can be useful to directly save the output of a program, for example,
``mysqldump`` so that the SQL can later be restored. Restic supports this mode
of operation; just supply the option ``--stdin-from-command`` when using the
``backup`` action, and write the command in place of the files/directories:
.. code-block:: console
$ restic -r /srv/restic-repo backup --stdin-from-command mysqldump [...]
This command creates a new snapshot based on the standard output of ``mysqldump``.
By default, the command's standard output is saved in a file named ``stdin``.
A different name can be specified with ``--stdin-filename``:
.. code-block:: console
$ restic -r /srv/restic-repo backup --stdin-filename production.sql --stdin-from-command mysqldump [...]
Restic uses the command exit code to determine whether the command succeeded. A
non-zero exit code from the command causes restic to cancel the backup. This causes
restic to fail with exit code 1. No snapshot will be created in this case.
- file creation date on Unix platforms
- inode flags on Unix platforms
- file ownership and ACLs on Windows
- the "hidden" flag on Windows
Reading data from stdin
***********************
.. warning::
Restic cannot detect if data read from stdin is complete or not. As explained
below, this can cause incomplete backup unless additional checks (outside of
restic) are configured. If possible, use ``--stdin-from-command`` instead.
Alternatively, restic supports reading arbitrary data directly from the standard
input. Use the option ``--stdin`` of the ``backup`` command as follows:
Sometimes it can be nice to directly save the output of a program, e.g.
``mysqldump`` so that the SQL can later be restored. Restic supports
this mode of operation, just supply the option ``--stdin`` to the
``backup`` command like this:
.. code-block:: console
# Will not notice failures, see the warning below
$ gzip bigfile.dat | restic -r /srv/restic-repo backup --stdin
This creates a new snapshot of the content of ``bigfile.dat``.
As for ``--stdin-from-command``, the default file name is ``stdin``; a
different name can be specified with ``--stdin-filename``.
**Important**: while it is possible to pipe a command output to restic using
``--stdin``, doing so is discouraged as it will mask errors from the
command, leading to corrupted backups. For example, in the following code
block, if ``mysqldump`` fails to connect to the MySQL database, the restic
backup will nevertheless succeed in creating an _empty_ backup:
.. code-block:: console
# Will not notice failures, read the warning above
$ set -o pipefail
$ mysqldump [...] | restic -r /srv/restic-repo backup --stdin
A simple solution is to use ``--stdin-from-command`` (see above). If you
still need to use the ``--stdin`` flag, you must use the shell option ``set -o pipefail``
(so that a non-zero exit code from one of the programs in the pipe makes the
whole chain return a non-zero exit code) and you must check the exit code of
the pipe and act accordingly (e.g., remove the last backup). Refer to the
`Use the Unofficial Bash Strict Mode <http://redsymbol.net/articles/unofficial-bash-strict-mode/>`__
for more details on this.
This creates a new snapshot of the output of ``mysqldump``. You can then
use e.g. the fuse mounting option (see below) to mount the repository
and read the file.
By default, the file name ``stdin`` is used, a different name can be
specified with ``--stdin-filename``, e.g. like this:
.. code-block:: console
$ mysqldump [...] | restic -r /srv/restic-repo backup --stdin --stdin-filename production.sql
The option ``pipefail`` is highly recommended so that a non-zero exit code from
one of the programs in the pipe (e.g. ``mysqldump`` here) makes the whole chain
return a non-zero exit code. Refer to the `Use the Unofficial Bash Strict Mode
<http://redsymbol.net/articles/unofficial-bash-strict-mode/>`__ for more
details on this.
Tags for backup
***************
@@ -625,17 +592,12 @@ environment variables. The following lists these environment variables:
AWS_DEFAULT_REGION Amazon S3 default region
AWS_PROFILE Amazon credentials profile (alternative to specifying key and region)
AWS_SHARED_CREDENTIALS_FILE Location of the AWS CLI shared credentials file (default: ~/.aws/credentials)
RESTIC_AWS_ASSUME_ROLE_ARN Amazon IAM Role ARN to assume using discovered credentials
RESTIC_AWS_ASSUME_ROLE_SESSION_NAME Session Name to use with the role assumption
RESTIC_AWS_ASSUME_ROLE_EXTERNAL_ID External ID to use with the role assumption
RESTIC_AWS_ASSUME_ROLE_POLICY Inline Amazion IAM session policy
RESTIC_AWS_ASSUME_ROLE_REGION Region to use for IAM calls for the role assumption (default: us-east-1)
RESTIC_AWS_ASSUME_ROLE_STS_ENDPOINT URL to the STS endpoint (default is determined based on RESTIC_AWS_ASSUME_ROLE_REGION). You generally do not need to set this, advanced use only.
AZURE_ACCOUNT_NAME Account name for Azure
AZURE_ACCOUNT_KEY Account key for Azure
AZURE_ACCOUNT_SAS Shared access signatures (SAS) for Azure
AZURE_ENDPOINT_SUFFIX Endpoint suffix for Azure Storage (default: core.windows.net)
AZURE_FORCE_CLI_CREDENTIAL Force the use of Azure CLI credentials for authentication
B2_ACCOUNT_ID Account ID or applicationKeyId for Backblaze B2
B2_ACCOUNT_KEY Account Key or applicationKey for Backblaze B2
@@ -681,14 +643,15 @@ The external programs that restic may execute include ``rclone`` (for rclone
backends) and ``ssh`` (for the SFTP backend). These may respond to further
environment variables and configuration files; see their respective manuals.
Exit status codes
*****************
Restic returns one of the following exit status codes after the backup command is run:
* 0 when the backup was successful (snapshot with all source files created)
* 1 when there was a fatal error (no snapshot created)
* 3 when some source files could not be read (incomplete snapshot with remaining files created)
* 0 when the backup was successful (snapshot with all source files created)
* 1 when there was a fatal error (no snapshot created)
* 3 when some source files could not be read (incomplete snapshot with remaining files created)
Fatal errors occur for example when restic is unable to write to the backup destination, when
there are network connectivity issues preventing successful communication, or when an invalid

View File

@@ -82,76 +82,6 @@ Furthermore you can group the output by the same filters (host, paths, tags):
1 snapshots
Listing files in a snapshot
===========================
To get a list of the files in a specific snapshot you can use the ``ls`` command:
.. code-block:: console
$ restic ls 073a90db
snapshot 073a90db of [/home/user/work.txt] filtered by [] at 2024-01-21 16:51:18.474558607 +0100 CET):
/home
/home/user
/home/user/work.txt
The special snapshot ID ``latest`` can be used to list files and directories of the latest snapshot in the repository.
The ``--host`` flag can be used in conjunction to select the latest snapshot originating from a certain host only.
.. code-block:: console
$ restic ls --host kasimir latest
snapshot 073a90db of [/home/user/work.txt] filtered by [] at 2024-01-21 16:51:18.474558607 +0100 CET):
/home
/home/user
/home/user/work.txt
By default, ``ls`` prints all files in a snapshot.
File listings can optionally be filtered by directories. Any positional arguments after the snapshot ID are interpreted
as absolute directory paths, and only files inside those directories will be listed. Files in subdirectories are not
listed when filtering by directories. If the ``--recursive`` flag is used, then subdirectories are also included.
Any directory paths specified must be absolute (starting with a path separator); paths use the forward slash '/'
as separator.
.. code-block:: console
$ restic ls latest /home
snapshot 073a90db of [/home/user/work.txt] filtered by [/home] at 2024-01-21 16:51:18.474558607 +0100 CET):
/home
/home/user
.. code-block:: console
$ restic ls --recursive latest /home
snapshot 073a90db of [/home/user/work.txt] filtered by [/home] at 2024-01-21 16:51:18.474558607 +0100 CET):
/home
/home/user
/home/user/work.txt
To show more details about the files in a snapshot, you can use the ``--long`` option. The colums include
file permissions, UID, GID, file size, modification time and file path. For scripting usage, the
``ls`` command supports the ``--json`` flag; the JSON output format is described at :ref:`ls json`.
.. code-block:: console
$ restic ls --long latest
snapshot 073a90db of [/home/user/work.txt] filtered by [] at 2024-01-21 16:51:18.474558607 +0100 CET):
drwxr-xr-x 0 0 0 2024-01-21 16:50:52 /home
drwxr-xr-x 0 0 0 2024-01-21 16:51:03 /home/user
-rw-r--r-- 0 0 18 2024-01-21 16:51:03 /home/user/work.txt
NCDU (NCurses Disk Usage) is a tool to analyse disk usage of directories. The ``ls`` command supports
outputting information about a snapshot in the NCDU format using the ``--ncdu`` option.
You can use it as follows: ``restic ls latest --ncdu | ncdu -f -``
Copying snapshots between repositories
======================================
@@ -164,11 +94,11 @@ example from a local to a remote repository, you can use the ``copy`` command:
repository d6504c63 opened successfully, password is correct
repository 3dd0878c opened successfully, password is correct
snapshot 410b18a2 of [/home/user/work] at 2020-06-09 23:15:57.305305 +0200 CEST by user@kasimir
snapshot 410b18a2 of [/home/user/work] at 2020-06-09 23:15:57.305305 +0200 CEST)
copy started, this may take a while...
snapshot 7a746a07 saved
snapshot 4e5d5487 of [/home/user/work] at 2020-05-01 22:44:07.012113 +0200 CEST by user@kasimir
snapshot 4e5d5487 of [/home/user/work] at 2020-05-01 22:44:07.012113 +0200 CEST)
skipping snapshot 4e5d5487, was already copied to snapshot 50eb62b7
The example command copies all snapshots from the source repository
@@ -263,18 +193,18 @@ the unwanted files from affected snapshots by rewriting them using the
$ restic -r /srv/restic-repo rewrite --exclude secret-file
repository c881945a opened (repository version 2) successfully, password is correct
snapshot 6160ddb2 of [/home/user/work] at 2022-06-12 16:01:28.406630608 +0200 CEST by user@kasimir
snapshot 6160ddb2 of [/home/user/work] at 2022-06-12 16:01:28.406630608 +0200 CEST)
excluding /home/user/work/secret-file
saved new snapshot b6aee1ff
snapshot 4fbaf325 of [/home/user/work] at 2022-05-01 11:22:26.500093107 +0200 CEST by user@kasimir
snapshot 4fbaf325 of [/home/user/work] at 2022-05-01 11:22:26.500093107 +0200 CEST)
modified 1 snapshots
$ restic -r /srv/restic-repo rewrite --exclude secret-file 6160ddb2
repository c881945a opened (repository version 2) successfully, password is correct
snapshot 6160ddb2 of [/home/user/work] at 2022-06-12 16:01:28.406630608 +0200 CEST by user@kasimir
snapshot 6160ddb2 of [/home/user/work] at 2022-06-12 16:01:28.406630608 +0200 CEST)
excluding /home/user/work/secret-file
new snapshot saved as b6aee1ff
@@ -304,28 +234,6 @@ modifying the repository. Instead restic will only print the actions it would
perform.
Modifying metadata of snapshots
===============================
Sometimes it may be desirable to change the metadata of an existing snapshot.
Currently, rewriting the hostname and the time of the backup is supported.
This is possible using the ``rewrite`` command with the option ``--new-host`` followed by the desired new hostname or the option ``--new-time`` followed by the desired new timestamp.
.. code-block:: console
$ restic rewrite --new-host newhost --new-time "1999-01-01 11:11:11"
repository b7dbade3 opened (version 2, compression level auto)
[0:00] 100.00% 1 / 1 index files loaded
snapshot 8ed674f4 of [/path/to/abc.txt] at 2023-11-27 21:57:52.439139291 +0100 CET by user@kasimir
setting time to 1999-01-01 11:11:11 +0100 CET
setting host to newhost
saved new snapshot c05da643
modified 1 snapshots
.. _checking-integrity:
Checking integrity and consistency

View File

@@ -174,9 +174,3 @@ To include the folder content at the root of the archive, you can use the ``<sna
.. code-block:: console
$ restic -r /srv/restic-repo dump latest:/home/other/work / > restore.tar
It is also possible to ``dump`` the contents of a selected snapshot and folder
structure to a file using the ``--target`` flag.
.. code-block:: console
$ restic -r /srv/restic-repo dump latest / --target /home/linux.user/output.tar -a tar

View File

@@ -201,8 +201,7 @@ change
+------------------+--------------------------------------------------------------+
| ``modifier`` | Type of change, a concatenation of the following characters: |
| | "+" = added, "-" = removed, "T" = entry type changed, |
| | "M" = file content changed, "U" = metadata changed, |
| | "?" = bitrot detected |
| | "M" = file content changed, "U" = metadata changed |
+------------------+--------------------------------------------------------------+
statistics
@@ -409,8 +408,6 @@ The ``key list`` command returns an array of objects with the following structur
+--------------+------------------------------------+
.. _ls json:
ls
--
@@ -579,19 +576,3 @@ The stats command returns a single JSON object.
+------------------------------+-----------------------------------------------------+
| ``compression_space_saving`` | Overall space saving due to compression |
+------------------------------+-----------------------------------------------------+
version
-------
The version command returns a single JSON object.
+----------------+--------------------+
| ``version`` | restic version |
+----------------+--------------------+
| ``go_version`` | Go compile version |
+----------------+--------------------+
| ``go_os`` | Go OS |
+----------------+--------------------+
| ``go_arch`` | Go architecture |
+----------------+--------------------+

View File

@@ -76,10 +76,6 @@ Similarly, if a repository is repeatedly damaged, please open an `issue on Githu
somewhere. Please include the check output and additional information that might
help locate the problem.
If ``check`` detects damaged pack files, it will show instructions on how to repair
them using the ``repair pack`` command. Use that command instead of the "Repair the
index" section in this guide.
2. Backup the repository
************************
@@ -108,11 +104,6 @@ whether your issue is already known and solved. Please take a look at the
3. Repair the index
*******************
.. note::
If the `check` command tells you to run `restic repair pack`, then use that
command instead. It will repair the damaged pack files and also update the index.
Restic relies on its index to contain correct information about what data is
stored in the repository. Thus, the first step to repair a repository is to
repair the index:
@@ -162,7 +153,7 @@ command will automatically remove the original, damaged snapshots.
$ restic repair snapshots --forget
snapshot 6979421e of [/home/user/restic/restic] at 2022-11-02 20:59:18.617503315 +0100 CET by user@host
snapshot 6979421e of [/home/user/restic/restic] at 2022-11-02 20:59:18.617503315 +0100 CET)
file "/restic/internal/fuse/snapshots_dir.go": removed missing content
file "/restic/internal/restorer/restorer_unix_test.go": removed missing content
file "/restic/internal/walker/walker.go": removed missing content

View File

@@ -7,18 +7,18 @@ API.
The following values are valid for ``{type}``:
* ``data``
* ``keys``
* ``locks``
* ``snapshots``
* ``index``
* ``config``
* ``data``
* ``keys``
* ``locks``
* ``snapshots``
* ``index``
* ``config``
The API version is selected via the ``Accept`` HTTP header in the request. The
following values are defined:
* ``application/vnd.x.restic.rest.v1`` or empty: Select API version 1
* ``application/vnd.x.restic.rest.v2``: Select API version 2
* ``application/vnd.x.restic.rest.v1`` or empty: Select API version 1
* ``application/vnd.x.restic.rest.v2``: Select API version 2
The server will respond with the value of the highest version it supports in
the ``Content-Type`` HTTP response header for the HTTP requests which should

View File

@@ -48,7 +48,7 @@ be used instead of the complete filename.
Apart from the files stored within the ``keys`` and ``data`` directories,
all files are encrypted with AES-256 in counter mode (CTR). The integrity
of the encrypted data is secured by a Poly1305-AES message authentication
code (MAC).
code (sometimes also referred to as a "signature").
Files in the ``data`` directory ("pack files") consist of multiple parts
which are all independently encrypted and authenticated, see below.
@@ -824,4 +824,4 @@ Changes
Repository Version 2
--------------------
* Support compression for blobs (data/tree) and index / lock / snapshot files
* Support compression for blobs (data/tree) and index / lock / snapshot files

View File

@@ -9,14 +9,14 @@ restic for version 0.10.0 and later. For restic versions down to 0.9.3 please
refer to the documentation for the respective version. The binary produced
depends on the following things:
* The source code for the release
* The exact version of the official `Go compiler <https://go.dev>`__ used to produce the binaries (running ``restic version`` will print this)
* The architecture and operating system the Go compiler runs on (Linux, ``amd64``)
* The build tags (for official binaries, it's the tag ``selfupdate``)
* The path where the source code is extracted to (``/restic``)
* The path to the Go compiler (``/usr/local/go``)
* The path to the Go workspace (``GOPATH=/home/build/go``)
* Other environment variables (mostly ``$GOOS``, ``$GOARCH``, ``$CGO_ENABLED``)
* The source code for the release
* The exact version of the official `Go compiler <https://go.dev>`__ used to produce the binaries (running ``restic version`` will print this)
* The architecture and operating system the Go compiler runs on (Linux, ``amd64``)
* The build tags (for official binaries, it's the tag ``selfupdate``)
* The path where the source code is extracted to (``/restic``)
* The path to the Go compiler (``/usr/local/go``)
* The path to the Go workspace (``GOPATH=/home/build/go``)
* Other environment variables (mostly ``$GOOS``, ``$GOARCH``, ``$CGO_ENABLED``)
In addition, The compressed ZIP files for Windows depends on the modification
timestamp and filename of the binary contained in it. In order to reproduce the
@@ -69,9 +69,9 @@ container can be found in the `GitHub repository
<https://github.com/restic/builder>`__
The container serves the following goals:
* Have a very controlled environment which is independent from the local system
* Make it easy to have the correct version of the Go compiler at the right path
* Make it easy to pass in the source code to build at a well-defined path
* Have a very controlled environment which is independent from the local system
* Make it easy to have the correct version of the Go compiler at the right path
* Make it easy to pass in the source code to build at a well-defined path
The following steps are necessary to build the binaries:
@@ -113,26 +113,6 @@ The following steps are necessary to build the binaries:
restic/builder \
go run helpers/build-release-binaries/main.go --version 0.14.0 --verbose
Verifying the Official Binaries
*******************************
To verify the official binaries, you can either build them yourself using the above
instructions or use the ``helpers/verify-release-binaries.sh`` script from the restic
repository. Run it as ``helpers/verify-release-binaries.sh restic_version go_version``.
The specified go compiler version must match the one used to build the official
binaries. For example, for restic 0.16.2 the command would be
``helpers/verify-release-binaries.sh 0.16.2 1.21.3``.
The script requires bash, curl, docker, git, gpg, shasum and tar.
The script first downloads all release binaries, checks the SHASUM256 file and its
signature. Afterwards it checks that the tarball matches the restic git repository
contents, before first reproducing the builder docker container and finally the
restic binaries. As final step, the restic binary in both the docker hub images
and the GitHub container registry is verified. If any step fails, then the script
will issue a warning.
Prepare a New Release
*********************

Some files were not shown because too many files have changed in this diff Show More