Compare commits

..

4868 Commits

Author SHA1 Message Date
Alexander Neumann
40791fff64 Add version for 0.13.0 2022-03-26 20:09:59 +01:00
Alexander Neumann
a53a4a23fd Update manpages and auto-completion 2022-03-26 20:09:59 +01:00
Alexander Neumann
b567c08e80 Generate CHANGELOG.md for 0.13.0 2022-03-26 20:09:40 +01:00
Alexander Neumann
0ca89b6fec Prepare changelog for 0.13.0 2022-03-26 20:09:39 +01:00
Alexander Neumann
d7e46c187a Merge pull request #3678 from restic/changelogs
Review, reword and polish unreleased changelog entries
2022-03-26 19:44:30 +01:00
Alexander Neumann
6aefe3e111 Merge pull request #3677 from restic/3490-polish
check: Adjust help and documentation for check --read-data-subset
2022-03-26 19:34:51 +01:00
Leo R. Lundgren
03137a34db Review, reword and polish unreleased changelog entries 2022-03-26 13:01:53 +01:00
Leo R. Lundgren
c7d637ec39 check: Adjust help and documentation for check --read-data-subset 2022-03-26 00:11:04 +01:00
Alexander Neumann
6087c4ad75 Merge pull request #3656 from lgommans/forget-security
forget: Update docs for readability and append-only considerations
2022-03-24 21:36:19 +01:00
Leo R. Lundgren
cdf478c8f4 doc: More updates to forget documentation and security considerations 2022-03-23 23:12:19 +01:00
Luc Gommans
80969a6347 Update docs according to comments from MichaelEischer in PR #3656 2022-03-23 23:12:19 +01:00
Leo R. Lundgren
676d5d498c doc: Update forget security considerations and thread model 2022-03-23 23:12:19 +01:00
Luc Gommans
9c1d49e312 Document "forget" security considerations and add references
Removing data based on a policy when the attacker had the opportunity to
add data to your repository comes with some considerations. This is
added to the 060_forget.rst documentation.

That document is also updated to reflect that restic now considers
the current system time while running "forget".

References to the security considerations section are added:
- In `restic forget --help`
- In the threat model (design.rst)
- In the (030) setup section where an append-only setup is referenced

A reference is also to be added to the `rest-server` readme's
append-only paragraph (see my fork).

This commit also resolves a typo (amount->number for countable noun),
changes a password length recommendation into the metric that
actually matters when creating passwords (entropy) since I was editing
these doc files anyway, and updates the outdated copyright year in
`conf.py`.

Some wording in 060_forget (line 21..22) was changed to clarify what
"forget" and "prune" do, to try and avoid the apparent misconception
that "forget" does not remove any data.
2022-03-23 23:12:19 +01:00
Alexander Neumann
ca1e2316cf Merge pull request #3665 from MichaelEischer/sane-list-locks
list: Never lock the repository when listing lock files
2022-03-21 11:14:44 +01:00
Alexander Neumann
0b8b524f12 Merge pull request #3512 from MichaelEischer/cleaner-lock-refresh
Prevent lock refresh from leaving behind lots of stale locks
2022-03-21 11:10:37 +01:00
Alexander Neumann
a350625554 Merge pull request #3524 from MichaelEischer/atomic-sftp
sftp: Implement atomic uploads
2022-03-21 11:08:22 +01:00
Alexander Neumann
32e61f2620 Update changelog/unreleased/issue-1106
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2022-03-21 11:04:04 +01:00
Alexander Neumann
8388f66c4c Merge pull request #3668 from greatroar/symlink-size
Report symlink sizes from FUSE mount
2022-03-21 11:02:32 +01:00
Alexander Neumann
0937008648 Merge pull request #3654 from MichaelEischer/limit-huge-tree-streams
Limit number of large tree blobs loaded in parallel by StreamTrees
2022-03-21 11:01:04 +01:00
Alexander Neumann
3a285f91bc Merge pull request #2311 from vincentbernat/feature/negative-pattern
filter: ability to use negative patterns
2022-03-20 14:02:30 +01:00
Alexander Neumann
29a5778626 Improve wording 2022-03-20 13:46:16 +01:00
Michael Eischer
53656f019a filter: address review comments 2022-03-20 13:33:08 +01:00
Michael Eischer
cd190bee14 filter: short circuit if no negative patterns 2022-03-20 13:33:08 +01:00
Vincent Bernat
2ee07ded2b filter: ability to use negative patterns
This is quite similar to gitignore. If a pattern is suffixed by an
exclamation mark and match a file that was previously matched by a
regular pattern, the match is cancelled. Notably, this can be used
with `--exclude-file` to cancel the exclusion of some files.

Like for gitignore, once a directory is excluded, it is not possible
to include files inside the directory. For example, a user wanting to
only keep `*.c` in some directory should not use:

    ~/work
    !~/work/*.c

But:

    ~/work/*
    !~/work/*.c

I didn't write documentation or changelog entry. I would like to get
feedback if this is the right approach for excluding/including files
at will for backups. I use something like this as an exclude file to
backup my home:

    $HOME/**/*
    !$HOME/Documents
    !$HOME/code
    !$HOME/.emacs.d
    !$HOME/games
    # [...]
    node_modules
    *~
    *.o
    *.lo
    *.pyc
    # [...]
    $HOME/code/linux/*
    !$HOME/code/linux/.git
    # [...]

There are some limitations for this change:

 - Patterns are not mixed accross methods: patterns from file are
   handled first and if a file is excluded with this method, it's not
   possible to reinclude it with `--exclude !something`.

 - Patterns starting with `!` are now interpreted as a negative
   pattern. I don't think anyone was relying on that.

 - The whole list of patterns is walked for each match. We may
   optimize later by exiting early if we know no pattern is starting
   with `!`.

Fix #233
2022-03-20 13:33:08 +01:00
Michael Eischer
12606b575f filter: Cleanup variable naming 2022-03-20 13:33:08 +01:00
Michael Eischer
5f145f0c7e filter: introduce pattern struct 2022-03-20 13:33:08 +01:00
Vincent Bernat
13c40d4199 filter: additional tests for filter.List() 2022-03-20 13:33:08 +01:00
Alexander Neumann
13aae82635 Merge pull request #3673 from restic/update-go
Update go
2022-03-20 12:02:32 +01:00
Alexander Neumann
b85d035956 Fix calens install step 2022-03-20 11:36:45 +01:00
Alexander Neumann
47aa4613f7 Force Go to use Module Mode 2022-03-20 11:30:01 +01:00
Alexander Neumann
a9a5acb8ce Update golangci-lint 2022-03-20 11:26:56 +01:00
Alexander Neumann
6dee59b789 Install gox before checking out code
Otherwise newer Go versions complain that the hash for the installed
version of gox is not in the go.mod, which we don't want anyways because
the tests should use the latest version of gox.
2022-03-20 11:26:56 +01:00
Alexander Neumann
2e19d19216 Use latest Go version for cross-compile and lint 2022-03-20 11:26:56 +01:00
Alexander Neumann
18a1de0de1 Use "go get" or "go install" selectively
Go 1.18 dropped support for installing binaries via "go get", Go <= 1.16
does not support it. So we need to use the right verb depending on the
Go version.
2022-03-20 11:26:56 +01:00
Alexander Neumann
9b57fcc6b0 Fix build.go, minimum Go version is 1.14 2022-03-20 10:54:33 +01:00
Alexander Neumann
17878036d8 Update tests to Go 1.18 2022-03-20 10:54:24 +01:00
Jason Lenz
2b1932a258 Report symlink sizes from FUSE mount for snapshot dir
Fixes #3667.
2022-03-17 22:21:47 -05:00
greatroar
fdc738fb70 Report symlink sizes from FUSE mount
Fixes #3667.
2022-03-13 16:48:35 +01:00
MichaelEischer
daea461f15 Merge pull request #3663 from jimt/find-msgs
Remove period from find messages
2022-03-07 22:23:49 +01:00
Jim Tittsler
a3d99217a4 Remove period from find messages
Simplifies cut-and-paste of IDs (and makes stylistically
consistent with other messages) #3659
2022-03-07 11:16:04 +09:00
MichaelEischer
e0ab689ccd Merge pull request #3664 from DanielG/fix-doc-warning
doc: Fix block quote warning
2022-03-06 21:56:26 +01:00
Michael Eischer
7af69fd7b9 list: Never lock the repository when listing lock files
There's no point in locking the repository just to list the currently
existing lock files. This won't work for an exclusively locked
repository and is also confusing to users.
2022-03-06 21:44:51 +01:00
Daniel Gröber
49b67c8aaa doc: Fix block quote warning 2022-03-06 18:15:55 +01:00
rawtaz
44d543ede3 Merge pull request #3653 from MichaelEischer/fix-ls-option-help
ls: Fix description for --host, --tag and --path options
2022-02-19 23:06:24 +01:00
Michael Eischer
5ef4ee7760 ls: Fix description for --host, --tag and --path options 2022-02-19 22:36:02 +01:00
Michael Eischer
254c8743fc Limit number of large tree blobs loaded in parallel by StreamTrees
Load tree blobs with more than 50MB only from a single goroutine. Very
large tree blobs with for example 400 MB size can otherwise require
roughly 1GB * streamTreeParallelism memory.
2022-02-19 12:26:09 +01:00
MichaelEischer
ad4f4dbc7a Merge pull request #3645 from greatroar/stdin-parent
Don't set a parent for --stdin backups
2022-02-19 11:36:51 +01:00
greatroar
63f6a9b085 Don't set a parent for --stdin backups
Loading any parent tree for these only wastes time and memory.
Fixes #3641, where it was shown that the most recent tree will get
picked.

--parent is now implicitly ignored when --stdin is given.
2022-02-19 10:41:33 +01:00
MichaelEischer
4a2d5a146d Merge pull request #3507 from ahmgithubahm/document-AWS_PROFILE-support
Document AWS_PROFILE support
2022-02-18 23:40:25 +01:00
Michael Eischer
1efc26899d Update docs for AWS_PROFILE and AWS_SHARED_CREDENTIALS_FILE 2022-02-18 23:31:10 +01:00
MichaelEischer
8df246d0f3 Merge pull request #3628 from gum3ng/issue_3127
[#issue 3127] Add xattr support for Solaris
2022-02-17 22:07:39 +01:00
gum3ng
dd30083c2b [#issue 3127] Add xattr support for Solaris 2022-02-13 14:24:37 +05:30
MichaelEischer
fb4c5af5c4 Merge pull request #3642 from gco/master
Fix test failures on Solaris
2022-02-12 22:07:14 +01:00
MichaelEischer
18ec49ddfa Merge pull request #3644 from duritong/centos-epel-repo
add a note about installation via epel for RHEL / CentOS Stream 8 & 9
2022-02-12 21:41:46 +01:00
Michael Eischer
5ec312ca06 sftp: Implement atomic uploads
Create a temporary file with a sufficiently random name to essentially
avoid any chance of conflicts. Once the upload has finished remove the
temporary suffix. Interrupted upload thus will be ignored by restic.
2022-02-12 20:30:49 +01:00
Michael Eischer
aebd24e414 Add changelog 2022-02-12 20:21:58 +01:00
Michael Eischer
d72181c8c1 Ensure that the lock cleanup handler is run after the global one
cleanup handlers run in the order in which they are added. As Go calls
init() functions in lexical order, the cleanup handler from global.go
was registered before that from lock.go, which is the correct order.

Make this order explicit to ensure that this won't break accidentally.
2022-02-12 20:21:58 +01:00
Michael Eischer
c6fd13425b remember the refreshed lock file even if removal failed
This ensures that restic won't create lots of new lock files without
deleting them later on.

In some cases a Delete operation on a backend can return a "File does
not exist" error even though the Delete operation succeeded. This can
for example be caused by request retries. This caused restic to forget
about the new lock file and continue trying to remove the old (already
deleted) lock file.
2022-02-12 20:21:58 +01:00
MichaelEischer
cc90f2ba6b Merge pull request #2816 from greatroar/noatime
Set O_NOATIME flag on Linux
2022-02-07 21:38:31 +01:00
MichaelEischer
d8f58fb7bf Merge pull request #3592 from vgt/jsondiff
Add json output for diff command
2022-02-07 21:33:30 +01:00
duritong
a4786dda5a Update doc/020_installation.rst
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2022-02-06 21:31:18 +01:00
Michael Eischer
aaa7f94139 Add changelog for O_NOATIME 2022-02-06 15:00:37 +01:00
Michael Eischer
6b17a7110c backup: Set O_NOATIME in the right place
The archiver uses FS.OpenFile, where FS is an instance of the FS
interface. This is different from fs.OpenFile, which uses the OpenFile
method provided by the fs package.
2022-02-06 15:00:37 +01:00
greatroar
7080fed7ae Set O_NOATIME flag on Linux
Citing Kerrisk, The Linux Programming Interface:

    The O_NOATIME flag is intended for use by indexing and backup
    programs. Its use can significantly reduce the amount of disk
    activity, because repeated disk seeks back and forth across the
    disk are not required to read the contents of a file and to update
    the last access time in the file’s i-node[.]

restic used to do this, but the functionality was removed along with the
fadvise call in #670.
2022-02-06 15:00:34 +01:00
Michael Eischer
74f29ad09b diff: add basic test for json output format 2022-02-06 11:44:15 +01:00
Michael Eischer
5f34ad523f diff: fix test failure and add remark on quiet to changelog 2022-02-05 23:02:07 +01:00
MichaelEischer
58236ead12 Merge pull request #3619 from aneeshusa/avoid-time-travel-paradoxes-when-finding-parents
Avoid choosing parent snapshot newer than time of current snapshot
2022-02-05 22:51:24 +01:00
Michael Eischer
8ae4d86a84 rename snapshot timestamp filter variable 2022-02-05 22:42:38 +01:00
mh
3f0184ba2a add a note about installation via epel 2022-02-05 22:18:33 +01:00
rawtaz
90473ea9ff Merge pull request #3638 from leogott/patch-1
Documentation: Update msys2 wiki url after move
2022-01-27 17:07:45 +01:00
Leona 'leo' Gottfried
4e84e8ab3f Update msys2 wiki url after move
https://github.com/msys2/msys2/wiki/Porting was permanently moved and redirects to https://www.msys2.org/
I substituted the new location of the /wiki/Porting page in the docs
2022-01-27 11:09:50 +01:00
Greg
2e9180638e Fix test failures on Solaris
Add exceptions for symlinks, sticky bits, and device nodes in the same places where the BSDSs and/or Darwin have them.
2022-01-25 18:05:56 -08:00
Aneesh Agrawal
058dfc20da Avoid choosing parent snapshot newer than time of current snapshot
Currently, `restic backup` (if a `--parent` is not provided)
will choose the most recent matching snapshot as the parent snapshot.
This makes sense in the usual case,
where we tag the snapshot-being-created with the current time.

However, this doesn't make sense if the user has passed `--time`
and is currently creating a snapshot older than the latest snapshot.
Instead, choose the most recent snapshot
which is not newer than the snapshot-being-created's timestamp,
to avoid any time travel.

Impetus for this change:
I'm using restic for the first time!
I have a number of existing BTRFS snapshots
I am backing up via restic to serve as my initial set of backups.
I initially `restic backup`'d the most recent snapshot to test,
then started backing up each of the other snapshots.
I noticed in `restic cat snapshot <id>` output
that all the remaining snapshots have the most recent as the parent.
2022-01-23 23:55:00 -05:00
Aneesh Agrawal
502fc3281c Add CONTRIBUTING.md docs to not edit man pages
Document this code review feedback I got for other contributors.
2022-01-23 23:54:24 -05:00
David Vogt
77c850148a Add json output for diff command 2022-01-23 23:22:26 +01:00
MichaelEischer
df89aa0087 Merge pull request #3623 from invine/issue-3464
Skip lock file creation on forget with --no-lock and --dry-run
2022-01-23 18:12:34 +01:00
Pavel Frolov
792523b28b [issue 3464] skip lock creation in case of dry-run 2022-01-23 06:44:41 +03:00
MichaelEischer
f0a8182493 Merge pull request #3626 from fkusche/issue-3620-doc-unreferenced-packs
Update documentation regarding unreferenced packs
2022-01-22 15:15:25 +01:00
Florian Kusche
6183d0be53 Update output of restic check 2022-01-16 12:34:20 +01:00
rawtaz
7f6fc78f95 Merge pull request #3544 from restic/fix-b2-delete-retry
b2: Successful delete if file does not exist
2022-01-13 23:28:30 +01:00
rawtaz
abfbacf3d3 Merge pull request #3591 from MichaelEischer/prune-fix-max-repack
prune: Handle --max-repack-size=0 as expected
2022-01-13 03:53:20 +01:00
Florian Kusche
b0c1d0f9cd Update documentation regarding unreferenced packs
Also removes an unnecessary space at the end of the last line.
2022-01-09 11:45:03 +01:00
rawtaz
8b6fe845d4 Merge pull request #3618 from mattxtaz/master
Add missing colon in prune stats output and realign the fields
2022-01-06 23:07:56 +01:00
mattxtaz
6ff32ee4d3 Add missing colon in prune stats output and change padding to 14 chars to align the fields 2022-01-06 21:15:15 +00:00
rawtaz
2ff3b7d69c Merge pull request #3615 from gum3ng/issue_3558
doc: Add a FAQ section for invalid Windows filenames
2022-01-02 18:06:02 +01:00
gum3ng
9589de16db [issue 3558]: Add a FAQ section for invalid Windows filenames 2022-01-02 22:24:00 +05:30
MichaelEischer
2c3e5d943d Merge pull request #3593 from DarkKirb/parallelize-restic-copy
Parallelize blob upload/download for restic copy
2021-12-29 22:31:54 +01:00
Charlotte 🦝 Delenk
e2bb384a60 Parallelize blob upload/download for restic copy
Currently restic copy will copy each blob from every snapshot serially,
which has performance implications on high-latency backends such as b2.

This commit introduces 8x parallelism for blob downloads/uploads which
can improve restic copy operations up to 8x for repositories with many
small blobs on b2.

This commit also addresses the TODO comment in the copyTree function.

Related work:

A more thorough improvement of the restic copy performance can be found
in PR #3513
2021-12-29 18:59:09 +01:00
MichaelEischer
e5985e0d63 Merge pull request #3602 from cqjjjzr/fix-rclone-sigint
fix: rclone receiving SIGINT prematurely on Windows causing restic hang forever (#3601)
2021-12-29 16:56:37 +01:00
MichaelEischer
8832837a8a Merge pull request #3607 from jtagcat/quieter_cleanup
logging: quiet 'removing n old cache dirs'
2021-12-29 16:06:35 +01:00
jtagcat
f92130d878 logging: quiet 'removing n old cache dirs'
Closes #3595

Choosing to include `stdoutIsTerminal()` as:
 - all other instances with `!opts.JSON` do so
 - this likely will not affect anything, especially when autorun
 - this seems to not be a meaningful enough summary
     to include in auto-backup reports

JSON is still likely not guaranteed to work and this is a suboptimal
  solution to this. Ideally, #1804 should refactor all print statements,
  and define+document(+handle) when stdoutIsTerminal() should be used.
  Else, it may end up more inconsistent and bulky
  (duplicate lines, longer files).
2021-12-29 01:08:29 +02:00
Charlie Jiang
a5b0e0bef4 fix: rclone receiving SIGINT prematurely on Windows causing restic hang forever
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2021-12-28 13:14:46 +08:00
rawtaz
e6e51b84ac Merge pull request #3605 from gum3ng/issue_3463
Improve clarity of error message in restic ls command
2021-12-27 21:45:23 +01:00
MichaelEischer
c5c3dfe10f Merge pull request #3590 from metalsp0rk/documentation-enhance
Document Safe Passwords and Clarify B2 App Key information
2021-12-27 20:21:43 +01:00
Kyle Brennan
19ec4d8f17 Document safe passwords. Fix #2238 2021-12-27 10:43:18 -08:00
Kyle Brennan
47ecd950b8 Enhance details about user application keys. Fix #2672 2021-12-27 10:43:15 -08:00
MichaelEischer
051cc7ce71 Merge pull request #3589 from metalsp0rk/copy-no-lock
Make Copy respect no lock
2021-12-27 19:11:02 +01:00
Michael Eischer
64e733f3d6 tweak copy --no-lock changelog 2021-12-27 18:22:25 +01:00
Gautam Menghani
017614c41a [#issue 3463] Improve clarity of error message in restic ls command 2021-12-27 22:42:27 +05:30
Michael Eischer
0cfdb82ea4 prune: Handle --max-repack-size=0 as expected
Previously the flag was ignored and `--max-repack-size=1` had to be
used.
2021-12-27 15:48:56 +01:00
Kyle Brennan
d5ed5da85c update changelog for copy --no-lock 2021-12-03 12:16:40 -08:00
Kyle Brennan
8eb83029a8 Make copy honor --no-lock 2021-12-03 09:50:28 -08:00
MichaelEischer
882d58abce Merge pull request #3163 from palbr/patch-1
Add PGP fingerprint to 020_installation.rst
2021-11-19 23:58:50 +01:00
Peter Albrecht
8de4401bb5 Changed URL for key-file
The keyfile provided by restic's own webserver (https://restic.net) should be
more stable than relying on public keyservers. So I changed the URL to the
GPG keyfile, as recommended by MichaelEischer.
2021-11-19 15:47:59 +01:00
MichaelEischer
f7a9b90eb9 Merge pull request #3573 from magandrez/feat/ls-report-symbolic-notation
Provide mode in symbolic notation for `restic ls --json`
2021-11-18 21:29:04 +01:00
MichaelEischer
aa214f99b4 Merge pull request #3565 from mathstuf/doc-missing-metadata
doc: mention metadata that is not backed up currently
2021-11-18 21:28:55 +01:00
MichaelEischer
4a25bbaed3 Merge pull request #3578 from adsultana/patch-1
docs: Fix link to "help wanted" issues
2021-11-18 21:20:30 +01:00
Michael Eischer
583edc39b8 doc: reorder backup metadata exceptions 2021-11-18 21:17:38 +01:00
Manuel González
212b2f651f Add file mode in symbolic notation to ls --json
This aligns `restic ls --json` with `restic find --json`, utilizing the same
naming.
2021-11-16 19:45:50 +02:00
Andrew Sultana
15ab96ecd6 docs: Fix link to "help wanted" issues 2021-11-16 00:03:34 +00:00
Ben Boeckel
d71afb3d32 doc: mention metadata that is not backed up currently
See: #3497
See: #1622
See: #2075
2021-11-13 18:40:26 -05:00
MichaelEischer
4bf05d91a1 Merge pull request #3571 from garrmcnu/blazer-unknown-authority
Update Backblaze B2 blazer module
2021-11-13 22:46:49 +01:00
MichaelEischer
de3afc1005 Merge pull request #3574 from gurjeet/rename_aws_s3_to_amazon_s3
Use S3's proper product name, Amazon S3
2021-11-13 22:30:32 +01:00
Michael Eischer
2ea998f70e Add PR link to changelog 2021-11-13 22:25:54 +01:00
Garry McNulty
e8fa3855e7 Update blazer
If a request fails with "x509: certificate signed by unknown authority",
the B2 backend now returns the error without retrying the request.

Closes #3556
Closes #2355
2021-11-13 22:25:54 +01:00
Gurjeet Singh
34a6a24544 Use S3's proper product name, Amazon S3
Per Amazon's product page [1], S3 is officially called "Amazon S3". The
restic project uses the phrase "AWS S3" in some places. This patch
corrects the product name.

[1]:https://aws.amazon.com/s3/
2021-11-13 22:21:06 +01:00
MichaelEischer
1d8a0b06cb Merge pull request #3575 from MichaelEischer/adjust-http2-canary-test
rest: Adjust http2-missing-eof test to golang >= 1.17.3
2021-11-13 22:13:24 +01:00
Michael Eischer
50053a85d3 rest: Adjust http2 missing eof test to golang >= 1.17.3, >= 1.16.10
The missing eof with http2 when a response included a content-length
header but no data, has been fixed in golang 1.17.3/1.16.10. Therefore
just drop the canary test and schedule it for removal once go 1.18 is
required as minimum version by restic.
2021-11-13 21:57:30 +01:00
MichaelEischer
f1cfb97237 Merge pull request #3514 from phcreery/rclone_timeout
rclone: extend timeout from 60s to 240s
2021-11-07 18:10:34 +01:00
MichaelEischer
cb81ee9396 Merge pull request #3474 from kitone/fix-issue-3382
Honor RESTIC_CACHE_DIR environment variable
2021-11-07 17:57:54 +01:00
Michael Eischer
b0e64deb27 rclone: Fix timeout calculation 2021-11-07 17:49:33 +01:00
phcreery
43d173b042 rclone: add timeout option and documentation 2021-11-07 17:49:21 +01:00
rawtaz
1b152a2c4d Merge pull request #3568 from MichaelEischer/fix-local-fd-leak
local: Fix fd leak when encountering files directly inside data/
2021-11-07 01:36:48 +01:00
MichaelEischer
15cc3c0e23 Merge pull request #3566 from mathstuf/check-progress-output-stomping
check: wait for progress bar output
2021-11-06 23:34:49 +01:00
kitone
5904f80cfa restic cache should display the name of the cache without shortening it in the case of the restic check 2021-11-06 20:18:51 +01:00
Ben Boeckel
4d579c4387 check: wait for progress bar output
Further code will also output to the terminal and the bar's cursor
positioning causes its output to overlap with the remaining output in a
racy way.

Fixes: #3344
2021-11-06 15:05:09 -04:00
Michael Eischer
15d7313387 local: Fix fd leak when encountering files directly inside data/ 2021-11-06 19:44:57 +01:00
MichaelEischer
6c84ea1412 Merge pull request #3548 from gum3ng/issue_3490
Support for specifying amount of data in read-data-subset
2021-11-05 23:28:11 +01:00
MichaelEischer
78c7dd53ef Merge pull request #3526 from greatroar/dump-refactor
Refactor internal/dump + concurrent load/write
2021-11-05 22:38:39 +01:00
MichaelEischer
a34bfa8269 Merge pull request #3562 from restic/rawtaz-mount-usage
mount: Improve usage information when mounted
2021-11-05 22:10:32 +01:00
kitone
0425a30420 add changelog issue-3382 2021-11-04 15:13:48 +01:00
kitone
1b23675f21 cache --cleanup should handle directories created by restic check.
Because there is no guarantee that a cleanup of these directories will occur
after the "restic check", we extend the behavior to detect and manage these
specific cache directories and allow their cleanup too.
2021-11-04 15:10:38 +01:00
Gautam Menghani
836fbb9133 [#issue 3490] Support for specifying file size in read-data-subset 2021-11-02 15:25:46 +05:30
greatroar
c71729dfc4 Refactor internal/dump + concurrent load/write
Package internal/dump has been reworked so its API consists of a single
type Dumper that handles tar and zip formats. Tree loading and node
writing happen concurrently.
2021-11-01 23:01:55 +01:00
Leo R. Lundgren
711ceb0109 mount: Improve usage information when mounted 2021-11-01 20:59:20 +01:00
MichaelEischer
829c0a67af Merge pull request #3520 from nxt-engineering/Docker
Docker Multistage build
2021-10-24 17:37:54 +02:00
Alexander Neumann
fb5d9345a7 Merge pull request #3510 from MichaelEischer/fix-archiver-early-on-abort
archiver: Fix TestArchiverAbortEarlyOnError test
2021-10-16 15:37:41 +02:00
kitone
95eb859b54 Honor RESTIC_CACHE_DIR environment variable
Fix #3382: restic check doesn't obey the RESTIC_CACHE_DIR environment variable
2021-10-10 16:00:02 +02:00
Michael Eischer
257740b0cc b2: Successful delete if file does not exist
When deleting a file, B2 sometimes returns a "500 Service Unavailable"
error but nevertheless correctly deletes the file. Due to retries in
the B2 library blazer, we sometimes also see a "400 File not present"
error. The retries of restic for the delete request then fail with
"404 File with such name does not exist.".

As we have to rely on request retries in a distributed system to handle
temporary errors, also consider a delete request to be successful if the
file is reported as not existing. This should be safe as B2 claims to
provide a strongly consistent bucket listing and thus a missing file
shouldn't mysteriously show up again later on.
2021-10-09 23:51:12 +02:00
MichaelEischer
46d08d9404 Merge pull request #3535 from jtagcat/writingverbose
Change: selfupdate: 'writing restic to' as verbose
2021-10-09 22:41:51 +02:00
jtagcat
a7853057ab Change: selfupdate: 'writing restic to' as verbose
Running restic self-update --quiet no longer
prints "writing restic to /usr/local/bin/restic".

The only output printed with -q is failures or
"successfully updated restic to version 0.12.1"

https://github.com/restic/restic/pull/3535

fix test fail: changelog title can't end with `.`

shorten changelog title
2021-10-09 23:24:18 +03:00
MichaelEischer
eb282532dc Merge pull request #3534 from jtagcat/patch-1
docs/example: useradd: restic now system account
2021-10-09 20:42:43 +02:00
MichaelEischer
f2a3b3b4a1 Merge pull request #3537 from greatroar/dump-lru
Check cap instead of len in bloblru
2021-10-09 20:00:27 +02:00
MichaelEischer
58e8b34633 Merge pull request #3532 from garrmcnu/s3-credentials-config
s3: Add warning if key ID or secret is empty
2021-10-09 19:32:32 +02:00
MichaelEischer
a02cea6e83 Merge pull request #3539 from jnoxon/fix-ec2-metadata-v2
get iam roles working with ec2 metadata v2
2021-10-07 23:00:27 +02:00
Garry McNulty
708d7a2574 s3: Add warning if key ID or secret is empty
Also add debug message if no credential types are available.

Closes #2388
2021-10-06 23:13:40 +01:00
Jeff Noxon
6f4b5ab8d1 get iam roles working with ec2 metadata v2 2021-10-03 20:06:55 -05:00
greatroar
634a9c162d Check cap instead of len in bloblru
restic dump uses bloblru.Cache to keep buffers alive, but also reuses
evicted buffers. That means large buffers may be used to store small
blobs, causing the cache to think it's using less memory than it
actually does.
2021-10-03 09:34:17 +02:00
jtagcat
632ca2ef52 docs/example: useradd: restic now system account
and: use /sbin/nologin
and: use long flags in the useradd command
and: download v0.12.1 instead of v0.9.6
2021-10-03 00:51:06 +03:00
MichaelEischer
24088f8307 Merge pull request #3528 from greatroar/tree-insert
internal/restic: Don't allocate in Tree.Insert
2021-09-27 22:22:03 +02:00
greatroar
c892c0bab9 internal/restic: Don't allocate in Tree.Insert
name         old time/op    new time/op    delta
BuildTree-8    34.6µs ± 4%     7.0µs ± 3%  -79.68%  (p=0.000 n=18+19)

name         old alloc/op   new alloc/op   delta
BuildTree-8    34.0kB ± 0%     0.9kB ± 0%  -97.37%  (p=0.000 n=20+20)

name         old allocs/op  new allocs/op  delta
BuildTree-8       108 ± 0%         1 ± 0%  -99.07%  (p=0.000 n=20+20)
2021-09-26 18:08:48 +02:00
MichaelEischer
78dac2fd48 Merge pull request #3523 from greatroar/swift-v2
Upgrade ncw/swift to v2
2021-09-24 21:41:36 +02:00
Michael Eischer
5ea8bba1a1 swift: restore context err check for list() 2021-09-24 21:19:46 +02:00
MichaelEischer
a5e103a212 Merge pull request #3522 from greatroar/dump-lru
Use LRU cache in restic dump
2021-09-24 20:33:58 +02:00
greatroar
e7ec0453b1 Upgrade ncw/swift to v2 2021-09-24 19:08:37 +02:00
Uli Martens
1ebcb1d097 Add changelog entry for PR #3508 2021-09-24 15:45:09 +02:00
greatroar
fe04d024c7 Use LRU cache in restic dump 2021-09-24 15:45:08 +02:00
Uli Martens
718966a81a Move Blobcache into dedicated internal package 2021-09-24 15:45:00 +02:00
greatroar
4f33eca634 Remove unused Writer arg to internal/dump.writeDump 2021-09-24 15:40:42 +02:00
MichaelEischer
cc110c42e6 Merge pull request #2657 from mansam/add-skip-tls-verification-flag
Add --insecure-tls flag to disable SSL cert verification
2021-09-22 21:40:01 +02:00
Sam Lucidi
897d8e662c Add --insecure-tls flag to disable SSL cert verification
Signed-off-by: Sam Lucidi <slucidi@redhat.com>
2021-09-21 10:52:40 -04:00
cimnine
4a95af5290 GitHub action for Docker build 2021-09-21 08:23:33 +02:00
cimnine
f28c8bc1c2 Multistage Docker build 2021-09-21 08:23:33 +02:00
MichaelEischer
1827b16ade Merge pull request #3519 from greatroar/maphash
Replace siphash by hash/maphash
2021-09-19 19:46:03 +02:00
greatroar
8b758c78a3 Require Go 1.14 to build 2021-09-19 16:18:19 +02:00
greatroar
8d2996eaaa Replace siphash by hash/maphash
In Go 1.17.1, maphash has become quite a bit faster than siphash, so we
can drop one third-party dependency. maphash is just an interface to the
standard Go map's hash function, which we already trust for other use
cases.

Benchmark results on linux/amd64, -benchtime=3s:

name                                             old time/op    new time/op    delta
IndexHasUnknown-8                                  50.6ns ±10%    41.0ns ±19%  -18.92%  (p=0.000 n=9+10)
IndexHasKnown-8                                    52.6ns ±12%    41.5ns ±12%  -21.13%  (p=0.000 n=9+10)
IndexMapHash-8                                     3.64µs ± 1%    2.00µs ± 0%  -45.09%  (p=0.000 n=10+9)
IndexAlloc-8                                        700ms ± 1%     601ms ± 6%  -14.18%  (p=0.000 n=8+10)
IndexAllocParallel-8                                205ms ± 5%     192ms ± 8%   -6.18%  (p=0.043 n=10+10)
MasterIndexAlloc-8                                  319ms ± 1%     279ms ± 5%  -12.58%  (p=0.000 n=10+10)
MasterIndexLookupSingleIndex-8                      156ns ± 8%     147ns ± 6%   -5.46%  (p=0.023 n=10+10)
MasterIndexLookupMultipleIndex-8                    150ns ± 7%     142ns ± 8%   -5.69%  (p=0.007 n=10+10)
MasterIndexLookupSingleIndexUnknown-8              74.4ns ± 6%    72.0ns ± 9%     ~     (p=0.175 n=10+9)
MasterIndexLookupMultipleIndexUnknown-8            67.4ns ± 9%    65.5ns ± 7%     ~     (p=0.340 n=9+9)
MasterIndexLookupParallel/known,indices=25-8        461ns ± 2%     445ns ± 2%   -3.49%  (p=0.000 n=10+10)
MasterIndexLookupParallel/unknown,indices=25-8      408ns ±11%     378ns ± 5%   -7.22%  (p=0.035 n=10+9)
MasterIndexLookupParallel/known,indices=50-8        479ns ± 1%     437ns ± 4%   -8.82%  (p=0.000 n=10+10)
MasterIndexLookupParallel/unknown,indices=50-8      406ns ± 8%     343ns ±15%  -15.44%  (p=0.001 n=10+10)
MasterIndexLookupParallel/known,indices=100-8       480ns ± 1%     455ns ± 5%   -5.15%  (p=0.000 n=8+10)
MasterIndexLookupParallel/unknown,indices=100-8     391ns ±18%     382ns ± 8%     ~     (p=0.315 n=10+10)
MasterIndexLookupBlobSize-8                        71.0ns ± 8%    57.2ns ±11%  -19.36%  (p=0.000 n=9+10)
PackerManager-8                                     254ms ± 1%     254ms ± 1%     ~     (p=0.285 n=15+15)

name                                             old speed      new speed      delta
IndexMapHash-8                                   1.12GB/s ± 1%  2.05GB/s ± 0%  +82.13%  (p=0.000 n=10+9)
PackerManager-8                                   208MB/s ± 1%   207MB/s ± 1%     ~     (p=0.281 n=15+15)

name                                             old alloc/op   new alloc/op   delta
IndexMapHash-8                                      0.00B          0.00B          ~     (all equal)
IndexAlloc-8                                        400MB ± 0%     400MB ± 0%     ~     (p=1.000 n=9+10)
IndexAllocParallel-8                                401MB ± 0%     401MB ± 0%   +0.00%  (p=0.000 n=10+10)
MasterIndexAlloc-8                                  258MB ± 0%     262MB ± 0%   +1.42%  (p=0.000 n=9+10)
PackerManager-8                                    73.1kB ± 0%    73.1kB ± 0%     ~     (p=0.382 n=13+13)

name                                             old allocs/op  new allocs/op  delta
IndexMapHash-8                                       0.00           0.00          ~     (all equal)
IndexAlloc-8                                         907k ± 0%      907k ± 0%   -0.00%  (p=0.000 n=10+10)
IndexAllocParallel-8                                 907k ± 0%      907k ± 0%   +0.00%  (p=0.009 n=10+10)
MasterIndexAlloc-8                                   327k ± 0%      317k ± 0%   -3.06%  (p=0.000 n=10+10)
PackerManager-8                                       744 ± 0%       744 ± 0%     ~     (all equal)
2021-09-19 16:05:18 +02:00
MichaelEischer
58efe21eca Merge pull request #3264 from amozoss/upstream-master
Refactor backup progress
2021-09-19 14:54:42 +02:00
MichaelEischer
71fcf48533 Merge pull request #2594 from greatroar/concurrent-restore-verify
Concurrent restore --verify
2021-09-19 14:52:31 +02:00
Michael Eischer
921e328b56 restore: Fix linting error 2021-09-19 14:41:07 +02:00
Michael Eischer
e62d4f622f Reword changelog 2021-09-19 14:41:07 +02:00
Michael Eischer
2cdc0719af restorer: Sanitize verify errors 2021-09-19 14:01:26 +02:00
greatroar
bdcdfaf6b4 restore --verify: buffer reuse consistency and comment 2021-09-19 13:11:27 +02:00
greatroar
2b94742ca5 Replace no-op closures in restorer by nil check 2021-09-19 13:11:16 +02:00
greatroar
d357744104 Handle canceled context in restore --verify properly 2021-09-19 13:11:05 +02:00
greatroar
d4225ec803 Simplify buffer growing in Restorer.verifyFile
Suggested-by: Igor Fedorenko <igor@ifedorenko.com>
2021-09-19 13:10:55 +02:00
greatroar
de8521ae56 Report number of successes from Restorer.VerifyFiles
Previously, the number of attempts was reported.
2021-09-19 13:10:44 +02:00
greatroar
bb066cf7d3 Concurrent Restorer.VerifyFiles
Time to verify a 2GB snapshot down from 9.726s to 4.645s (-52%).
2021-09-19 13:10:41 +02:00
greatroar
556424d61b Reuse buffer in Restorer.VerifyFiles
Time to verify a 2GB snapshot down from 11.568s to 9.726s (-16%).
2021-09-19 12:55:31 +02:00
greatroar
92ae951ffa Report timing from restic restore --verify 2021-09-19 12:53:09 +02:00
greatroar
973fa921cb Test and document Restorer.VerifyFiles 2021-09-19 12:52:11 +02:00
Michael Eischer
e0d615c264 archiver: Fix TestArchiverAbortEarlyOnError test
This can be caused when the test has uploaded four blobs, then queues
two blobs for upload which are delayed. Then a seventh file can be
opened which lead to a test failure.
2021-09-12 22:17:17 +02:00
MichaelEischer
ef5672a902 Merge pull request #3509 from ajaspers/patch-1
Update instructions for reproducing build
2021-09-12 21:40:37 +02:00
ajaspers
c0eddc9969 Update instructions for reproducing build
Dependencies are fetched at build time and stored in the GOPATH. These paths end up being in the final binary.

Bump restic version to latest and go version to the 1.16.6, which was used to build restic 0.12.1.
2021-09-12 09:02:57 -07:00
Michael Eischer
fbb0e6499a ui: consolidate backup ui in ui/backup package 2021-09-12 16:20:15 +02:00
Michael Eischer
503d4c3e2f Add changelog 2021-09-12 16:00:49 +02:00
Michael Eischer
cccb0d4064 ui: assert that ProgressPrinter interface is implemented 2021-09-12 15:26:40 +02:00
Michael Eischer
a144c986f2 backup: Reenable JSON status updates with redirected output
After the refactoring status updates were no longer printed in quiet
mode or when the output is not an interactive terminal. However, the
JSON output is often piped to e.g. another program. Thus, don't set the
update frequency to 0 in that case. The status updates are still
disabled for backup --quiet.

This also reduces the status update frequency to 60fps compared to a
potentially much higher value before the refactoring.
2021-09-12 15:26:40 +02:00
Michael Eischer
d62bfed65d ui: move SetDryRun to ProgressReporter 2021-09-12 15:25:58 +02:00
Michael Eischer
77b129ec74 ui: remove unused Summary field 2021-09-12 15:25:58 +02:00
Michael Eischer
3024239e40 ui/json: restore summary output 2021-09-12 15:25:58 +02:00
Michael Eischer
5ccf583b8a ui: restore a few comments 2021-09-12 15:25:58 +02:00
Michael Eischer
80cbaf6d38 ui: Simplify ReportTotal methods 2021-09-12 15:25:58 +02:00
Dan Willoughby
448419990c Refactor backup progress
Move the shared logic into the progress

Allows logic to be shared with forth coming restore status
2021-09-12 15:25:58 +02:00
Andy
7baa9a570d document AWS_PROFILE support
Since restic (or whatever library it is using) seems to respect/use AWS_PROFILE, it's worth documenting this.
2021-09-10 14:06:46 +01:00
Alexander Neumann
bf9c8771a4 Merge pull request #3482 from MichaelEischer/changelog-3429
Add changelog for #3246 and #3429
2021-09-08 09:15:40 +02:00
Michael Eischer
5e84f38f31 Add changelog for #3246 2021-09-07 21:18:11 +02:00
Alexander Neumann
8fe122d675 Merge pull request #3488 from MichaelEischer/rebuild-broken-index
Fix `rebuild-index` for damaged index
2021-09-07 17:00:23 +02:00
Alexander Neumann
74c47f1f12 Merge pull request #3502 from restic/rawtaz-issue-519
doc: Add note about scheduling to backup section
2021-09-07 16:58:17 +02:00
MichaelEischer
fa5ca8af81 Merge pull request #2856 from aawsome/remove-readahead
Simplify cache logic
2021-09-04 20:25:49 +02:00
MichaelEischer
b45d88e124 Merge pull request #3496 from juergenhoetzel/imporove-snapshot-filter-warning-message
Improve snapshot filter warning message
2021-09-03 23:42:27 +02:00
MichaelEischer
bc4cbd775b Merge pull request #2880 from aawsome/enhance-recover
Improve recover command
2021-09-03 23:40:43 +02:00
MichaelEischer
a29777f467 Merge pull request #3501 from greatroar/printprogress
Streamline progress printing in cmd/restic
2021-09-03 23:34:36 +02:00
Alexander Weiss
bce87922c0 Improve recover
- only save directories not referenced by any snapshot
- dont't write empty snapshot
- use progress bar
2021-09-03 21:36:57 +02:00
Alexander Weiss
81876d5c1b Simplify cache logic 2021-09-03 21:01:00 +02:00
greatroar
7f0aa49f45 cmd/restic: Streamline progress printing
* PrintProgress no longer does unnecessary Sprintf calls, and performs
  fewer allocations in general
* newProgressMax's callback checks whether the terminal supports
  line updates once instead of once per call
* the callback looks up the terminal width once per call instead of
  twice (on Windows)
* the status shortening now uses the Unicode-aware version from
  internal/ui/termstatus (future-proofing)
2021-09-03 11:48:22 +02:00
greatroar
5aaa3e93c1 internal/ui/termstatus: Optimize and publish Truncate
name               old time/op  new time/op  delta
TruncateASCII-8     347ns ± 1%    69ns ± 1%  -80.02%  (p=0.000 n=9+10)
TruncateUnicode-8   447ns ± 3%   348ns ± 1%  -22.04%  (p=0.000 n=10+10)
2021-09-03 11:48:22 +02:00
MichaelEischer
ec2e3b260e Merge pull request #3499 from greatroar/wrappedconn-pointer
rclone: Return one fewer value from run
2021-08-31 21:48:36 +02:00
rawtaz
26914abe62 doc: Add note about scheduling to backup section
Explains that restic doesn't have built-in scheduling
and mentions a few keywords one can search for.
2021-08-29 22:03:22 +02:00
greatroar
950b818274 rclone: Return one fewer value from run 2021-08-26 18:12:08 +02:00
Juergen Hoetzel
defe19fdf6 Quote snapshot prefix in error string 2021-08-25 16:11:28 +02:00
Juergen Hoetzel
409e4936af Improve snapshot filter warning message
Include the root-cause why the snapshot prefix is ignored.
2021-08-25 15:46:21 +02:00
MichaelEischer
10b39d7591 Merge pull request #3487 from andreaso/rest-proto-readme-link
Fix REST protocol link in README file
2021-08-22 18:47:32 +02:00
Michael Eischer
194ed19557 Add changelog 2021-08-22 18:29:58 +02:00
Michael Eischer
877fc9f352 rebuild-index: test that invalid indexes are skipped and removed 2021-08-22 18:24:19 +02:00
Michael Eischer
64258a2c2a rebuild-index: Ignore broken index files
Previously, an e.g. truncated index could prevent rebuild-index from
working.
2021-08-22 18:23:47 +02:00
Andreas Olsson
c520672982 Fix REST protocol link in README file
1. All other document links goes to the _Read the Docs_ site
2. The GitHub reStructuredText renderer doesn't appear to do includes,
   making for a rather empty read.
2021-08-22 13:16:51 +02:00
Alexander Neumann
9374c3ce81 Merge pull request #3485 from lbausch/fix_typo
Fix typo in changelog
2021-08-21 13:16:20 +02:00
Lorenz Bausch
4d56b34096 Fix typo in changelog 2021-08-21 12:16:04 +02:00
Alexander Neumann
66382b2861 Update Go to 1.17 2021-08-17 21:38:34 +02:00
Michael Eischer
1fab5892b5 Add changelog for #3429 2021-08-16 17:42:16 +02:00
MichaelEischer
c898f7a6bf Merge pull request #3479 from BUFU1610/patch-1
fixed --keep-within-X options in example
2021-08-15 20:06:26 +02:00
BUFU
7659790923 fixed --keep-within-X options in example
The order of the words was wrong in the example, i.e. fixed to --keep-within-hourly from --keep-hourly-within.
2021-08-15 19:56:22 +02:00
MichaelEischer
ecf34783ef Merge pull request #3480 from MichaelEischer/fix-rest-tests
Fix rest backend tests
2021-08-15 18:59:02 +02:00
Michael Eischer
68370feeee backends: Remove TestSaveFilenames test
Filenames are expected to match the sha256 sum of the file content. This
rule is now enforced by the rest server thus making this test useless.
2021-08-15 18:24:16 +02:00
Michael Eischer
574c83e47f rest: Fix test to use paths which are the sha256 sum of the data 2021-08-15 18:19:43 +02:00
Michael Eischer
e6a5801155 rest: Fix test backend url
The rest config normally uses prepareURL to sanitize URLs and ensures
that the URL ends with a slash. However, the test used an URL without a
trailing slash, which after the rest server changes causes test
failures.
2021-08-15 18:16:17 +02:00
Alexander Neumann
d90efd7704 Fix test 2021-08-09 10:30:10 +02:00
Alexander Neumann
9fe5a87785 Merge pull request #3429 from MichaelEischer/safe-key-change
key: Check that new key works before deleting the old one
2021-08-09 10:07:15 +02:00
Alexander Neumann
7f1608dc77 Merge pull request #3246 from restic/content-hash-for-upload
Calculate content hashes during upload
2021-08-08 17:24:08 +02:00
Michael Eischer
f4c5dec05d backend: test that a wrong hash fails an upload 2021-08-04 22:17:46 +02:00
Michael Eischer
7c1903e1ee panic if hash returns an error
Add a sanity check that the interface contract is honoured.
2021-08-04 22:17:46 +02:00
Michael Eischer
51b7e3119b mem: calculate md5 content hash for uploads
The mem backend is primarily used for testing. This ensures that the
upload hash calculation gets appropriate test coverage.
2021-08-04 22:17:46 +02:00
Michael Eischer
a009b39e4c gs/swift: calculate md5 content hash for upload 2021-08-04 22:17:46 +02:00
Michael Eischer
1d3e99f475 azure: check upload using md5 content hash
For files below 256MB this uses the md5 hash calculated while assembling
the pack file. For larger files the hash for each 100MB part is
calculated on the fly. That hash is also reused as temporary filename.
As restic only uploads encrypted data which includes among others a
random initialization vector, the file hash shouldn't be susceptible to
md5 collision attacks (even though the algorithm is broken).
2021-08-04 22:17:46 +02:00
Michael Eischer
9aa2eff384 Add plumbing to calculate backend specific file hash for upload
This enables the backends to request the calculation of a
backend-specific hash. For the currently supported backends this will
always be MD5. The hash calculation happens as early as possible, for
pack files this is during assembly of the pack file. That way the hash
would even capture corruptions of the temporary pack file on disk.
2021-08-04 22:17:46 +02:00
Michael Eischer
ee2f14eaf0 s3: enable content hash calculation for uploads 2021-08-04 22:12:12 +02:00
MichaelEischer
553ea36ca6 Merge pull request #2838 from greatroar/cache-conflicts
Make cache concurrency-safe
2021-08-04 22:11:50 +02:00
greatroar
6586e90acf Modernize internal/cache error handling 2021-08-04 22:02:42 +02:00
greatroar
ea04f40eb3 Save cached files to a temporary location first 2021-08-04 22:02:42 +02:00
greatroar
f9b6f8fd45 Replace duplicate type checking in cache with a function 2021-08-04 22:02:42 +02:00
MichaelEischer
1b1a2115fa Merge pull request #3436 from greatroar/local-save-tmp
Save files under temporary name in local backend
2021-08-04 22:01:53 +02:00
Michael Eischer
65908647e3 Add changelog 2021-08-04 21:51:53 +02:00
greatroar
81e2499d19 Sync directory to get durable writes in local backend 2021-08-04 21:51:53 +02:00
greatroar
195a5cf996 Save files under temporary name in local backend
Fixes #3435.
2021-08-04 21:51:53 +02:00
MichaelEischer
bc97a3d1f9 Merge pull request #3300 from aawsome/backup-dryrun
backup: add --dry-run/-n flag
2021-08-04 21:50:38 +02:00
Michael Eischer
702cff636f Add use case to changelog 2021-08-04 21:19:29 +02:00
Alexander Weiss
780e11b7e2 Adapt changelog 2021-08-04 21:19:29 +02:00
erin
4126435663 resolve rawtaz's review comments
make majority of suggestions from review by @rawtaz verbatim, with one clarification on my part in changelog
2021-08-04 21:19:29 +02:00
Alexander Weiss
d107a2cfdf Separate dry run tests 2021-08-04 21:19:29 +02:00
Alexander Weiss
38a8a48a25 Simplify dry run backend 2021-08-04 21:19:29 +02:00
Ryan Hitchman
77bf148460 backup: add --dry-run/-n flag to show what would happen.
This can be used to check how large a backup is or validate exclusions.
It does not actually write any data to the underlying backend. This is
implemented as a simple overlay backend that accepts writes without
forwarding them, passes through reads, and generally does the minimal
necessary to pretend that progress is actually happening.

Fixes #1542

Example usage:

$ restic -vv --dry-run . | grep add
new       /changelog/unreleased/issue-1542, saved in 0.000s (350 B added)
modified  /cmd/restic/cmd_backup.go, saved in 0.000s (16.543 KiB added)
modified  /cmd/restic/global.go, saved in 0.000s (0 B added)
new       /internal/backend/dry/dry_backend_test.go, saved in 0.000s (3.866 KiB added)
new       /internal/backend/dry/dry_backend.go, saved in 0.000s (3.744 KiB added)
modified  /internal/backend/test/tests.go, saved in 0.000s (0 B added)
modified  /internal/repository/repository.go, saved in 0.000s (20.707 KiB added)
modified  /internal/ui/backup.go, saved in 0.000s (9.110 KiB added)
modified  /internal/ui/jsonstatus/status.go, saved in 0.001s (11.055 KiB added)
modified  /restic, saved in 0.131s (25.542 MiB added)
Would add to the repo: 25.892 MiB
2021-08-04 21:19:29 +02:00
MichaelEischer
533ac4fd95 Merge pull request #3467 from greatroar/wrappedconn-pointer
Use rclone.wrappedConn by pointer
2021-08-04 21:18:53 +02:00
Alexander Neumann
7049f1cbfc Set development version for 0.12.1 2021-08-03 11:45:39 +02:00
Alexander Neumann
dc7a8aab24 Add version for 0.12.1 2021-08-03 11:45:36 +02:00
Alexander Neumann
94983a1f36 Update manpages and auto-completion 2021-08-03 11:45:36 +02:00
Alexander Neumann
a92faca10e Generate CHANGELOG.md for 0.12.1 2021-08-03 11:45:19 +02:00
Alexander Neumann
b19cd8c50f Prepare changelog for 0.12.1 2021-08-03 11:45:19 +02:00
Alexander Neumann
b862732318 Merge pull request #3468 from restic/rework-changelog
Reword changelog entries
2021-08-03 10:06:56 +02:00
Leo R. Lundgren
cb844e7136 Polish changelog entries 2021-08-03 00:01:09 +02:00
Alexander Neumann
b7fe1fe6b4 Reword changelog entries 2021-08-03 00:01:09 +02:00
MichaelEischer
c98bbdcdbe Merge pull request #3457 from systemmonkey42/feature/untagged
Feature to match untagged snapshots only when listing or forgetting
2021-08-02 23:19:39 +02:00
David le Blanc
326fefcd80 Allow --tag and --keep-tag to match untagged snapshots 2021-08-02 23:06:20 +02:00
greatroar
fa3eed1998 Use rclone.wrappedConn by pointer
This shaves a kilobyte off the Linux binary by not generating a
non-pointer interface implementation.
2021-08-01 09:11:50 +02:00
MichaelEischer
5571c3f7fd Merge pull request #3453 from MichaelEischer/http2-zero-length-workaround
rest: Workaround Http2 zero-length reply bug
2021-07-31 20:30:06 +02:00
Alexander Neumann
d8ea10db8c rest: Rework handling HTTP2 zero-length replies bug
Add comment that the check is based on the stdlib HTTP2 client. Refactor
the checks into a function. Return an error if the value in the
Content-Length header cannot be parsed.
2021-07-31 17:12:24 +02:00
Alexander Neumann
77551597b2 Merge pull request #3416 from torfason/keep-hourly-within
Keep hourly within
2021-07-30 10:36:41 +02:00
Alexander Neumann
92f293cd0b Merge pull request #3427 from MichaelEischer/find-packs-from-index
find: List missing pack files based on the index
2021-07-30 10:33:02 +02:00
Magnus Thor Torfason
2081bd12fb forget: Ensure future snapshots do not affect --keep-within-*
Ensure that only snapshots made in the past are taken into account when running restic forget with the within switches (--keep-within, --keep-within- hourly, and friends)
2021-07-24 16:14:43 +00:00
Magnus Thor Torfason
74ebc650ab forget: Add --keep-within-hourly (and friends)
Allow keeping hourly/daily/weekly/monthly/yearly snapshots for a given time period.

This adds the following flags/parameters to restic forget:
  --keep-within-hourly duration
  --keep-within-daily duration
  --keep-within-weekly duration
  --keep-within-monthly duration
  --keep-within-yearly duration

Includes following changes:
  - Add tests for --keep-within-hourly (and friends)
  - Add documentation for --keep-within-hourly (and friends)
  - Add changelog for --keep-within-hourly (and friends)
2021-07-24 16:14:43 +00:00
Alexander Neumann
c707d71b72 Merge pull request #3401 from MichaelEischer/goroutine-shutdown-cleanups
Goroutine shutdown cleanups
2021-07-11 16:32:28 +02:00
Alexander Neumann
691866ce43 Merge pull request #3452 from MichaelEischer/add-s390x-releases
Add release binaries for linux/s390x
2021-07-11 16:30:25 +02:00
Alexander Neumann
efd918c59e Merge pull request #3454 from MichaelEischer/update-dependencies
Update dependencies
2021-07-11 16:29:31 +02:00
Michael Eischer
7d28006e2e Add changelog 2021-07-10 22:39:01 +02:00
Michael Eischer
0880afe67b Use our generate command instead of cobra's complete command 2021-07-10 19:44:18 +02:00
Michael Eischer
100baf74c0 Update cobra
The most noteworthy change seems to be

https://github.com/spf13/cobra/pull/1192
Have Cobra create 'completion' command automatically
2021-07-10 19:43:14 +02:00
Michael Eischer
c733ae6b16 Readd and update indirect dependencies
The azure-sdk-for-go is the only remaining module without a go.mod file.
Thus we only need indirect dependencies for it. Remove all other
indirect dependencies.
2021-07-10 19:21:43 +02:00
Michael Eischer
989b398fee Misc library updates
The most noteworthy change is the xattr update which includes
https://github.com/pkg/xattr/pull/54 that adds xattr support for solaris
and illumos
2021-07-10 19:04:47 +02:00
Michael Eischer
bbc8146934 Update minio/sha256-simd
Apparently the standard Go sha256 implementation is now faster than the
assembly implementation. The library now only adds support for SHA
extensions available in some processors.

See https://github.com/minio/sha256-simd/pull/57 for more details.
2021-07-10 18:34:16 +02:00
Michael Eischer
aa22ebac69 Update backend dependencies
Possibly interesting changes:

https://github.com/Azure/azure-sdk-for-go/pull/14521
Retry on HTTP client errors

https://github.com/minio/minio-go/pull/1452
fix: make sure getObject returns error on truncated responses

f5854403a9
http2: close Transport connection on write errors
2021-07-10 18:27:13 +02:00
Michael Eischer
097ed659b2 rest: test that zero-length replies over HTTP2 work correctly
The first test function ensures that the workaround works as expected.
And the second test function is intended to fail as soon as the issue
has been fixed in golang to allow us to eventually remove the
workaround.
2021-07-10 17:22:42 +02:00
Michael Eischer
185a55026b rest: workaround for HTTP2 zero-length replies bug
The golang http client does not return an error when a HTTP2 reply
includes a non-zero content length but does not return any data at all.
This scenario can occur e.g. when using rclone when a file stored in a
backend seems to be accessible but then fails to download.
2021-07-10 16:59:01 +02:00
Michael Eischer
495831d53c add release binaries for linux/s390x 2021-07-10 00:10:24 +02:00
Michael Eischer
3442dc87fb find: Address review comments 2021-07-06 21:04:34 +02:00
Michael Eischer
a81f34ae47 Add changelog 2021-07-06 21:04:34 +02:00
Michael Eischer
95b44490a0 find: search blob ids for pack in index if pack is missing
If a pack file is missing try to determine the contained pack ids based
on the repository index. This helps with assessing the damage to a
repository before running `rebuild-index`.
2021-07-06 21:04:34 +02:00
Michael Eischer
3caab3c7ac find: Print not found pack files 2021-07-06 21:04:34 +02:00
Michael Eischer
40745b4f82 find: stop file listing after resolving all pack files 2021-07-06 21:04:34 +02:00
Michael Eischer
6c01078f3d find: support resolving multiple pack ids to blobs
Just passing the list of blobs to packsToBlobs would also work in most
cases, however, it could cause unexpected results when multiple pack
files have the same prefix. Forget found prefixes to prevent this.
2021-07-06 21:04:34 +02:00
Leo R. Lundgren
790294dc26 contributing: Change freenode to libera 2021-07-04 11:05:36 +02:00
Alexander Neumann
30d968b0e4 Merge pull request #3449 from MichaelEischer/fix-restore-retries
restore: Correctly handle partial pack download errors
2021-07-04 10:55:09 +02:00
Michael Eischer
43b82d69b4 Add changelog 2021-06-29 21:27:00 +02:00
Michael Eischer
bd316d3893 restore: Test partial pack downloads in filerestorer 2021-06-29 21:11:30 +02:00
Michael Eischer
e8bbb05328 restore: Correctly handle partial pack download errors
Failed pack/blob downloads should be retried. For blobs that fail
decryption assume that the pack file is really damaged and try to
restore the remaining blobs.
2021-06-29 20:54:16 +02:00
rawtaz
58be5172ff Merge pull request #3437 from MichaelEischer/fix-prune-output
prune: Add missing newlines in error descriptions
2021-06-28 00:56:51 +02:00
MichaelEischer
cb6fd281a0 Merge pull request #3442 from restic/rawtaz-doc-verbose
doc: Correct position of --verbose in backup docs
2021-06-27 18:19:37 +02:00
rawtaz
eb61de7b3a Merge pull request #3287 from tjrana/clean-up-pr-template
Clean up PR template
2021-06-25 21:06:50 +02:00
rawtaz
98a88b483d doc: Correct position of --verbose in backup docs 2021-06-25 19:26:46 +02:00
MichaelEischer
6a4c1ed50d Merge pull request #3438 from MichaelEischer/fast-fuse-changelog
Add changelog for #3426
2021-06-20 15:05:19 +02:00
Michael Eischer
409306db2b Add changelog for #3426 2021-06-20 14:46:37 +02:00
Michael Eischer
aad8864835 prune: Add missing newlines in error descriptions 2021-06-20 14:25:40 +02:00
MichaelEischer
c1eb7ac1a1 Merge pull request #3420 from greatroar/local-errs
Modernize error handling in local backend
2021-06-20 14:20:40 +02:00
greatroar
e5f0f67ba0 Modernize error handling in local backend
* Stop prepending the operation name: it's already part of os.PathError,
  leading to repetitive errors like "Chmod: chmod /foo/bar: operation not
  permitted".

* Use errors.Is to check for specific errors.
2021-06-18 11:13:27 +02:00
Alexander Neumann
45eb30388f Merge pull request #3426 from MichaelEischer/fast-fuse
mount: enable fuse readahead
2021-06-14 10:27:41 +02:00
Michael Eischer
454b6d608e key: Add test that failed key saves don't break the repository 2021-06-13 13:46:48 +02:00
TJ Rana
a61a0255a8 Modify pull request template 2021-06-12 19:34:46 -04:00
Michael Eischer
6add186867 key: Check that a new key file actually works 2021-06-12 23:09:19 +02:00
MichaelEischer
a476752962 Merge pull request #3421 from greatroar/s3-fileinfo
Return s3.fileInfos by pointer
2021-06-12 18:55:18 +02:00
MichaelEischer
e8d20ea32c Merge pull request #3409 from greatroar/lchown-mknod
Make restic.{lchown,mknod} regular functions
2021-06-12 18:22:38 +02:00
Michael Eischer
fe43f53528 mount: enable fuse readahead
Apparently readahead was disabled by default. Enable readahead with the
Linux default size of 128kB. Larger values seem to have no effect.
This can speed up reading from the fuse mount by at least factor 5.

Speedup for a 1G random file stored in a local repository:
(Only one result shown, but times were quite stable, restarted restic
after each command)
$ dd if=/dev/urandom bs=1M count=1024 of=rand
$ shasum -a 256 tmp/rand
75dd9b374e712577d64672a05b8ceee40dfc45dce6321082d2c2fd51d60c6c2d  tmp/rand

before: $ time shasum -a 256 fuse/snapshots/latest/tmp/rand
75dd9b374e712577d64672a05b8ceee40dfc45dce6321082d2c2fd51d60c6c2d  fuse/snapshots/latest/tmp/rand

real    0m18.294s
user    0m4.522s
sys     0m3.305s

before: $ time cat fuse/snapshots/latest/tmp/rand > /dev/null
real    0m14.924s
user    0m0.000s
sys     0m4.625s

after:  $ time shasum -a 256 fuse/snapshots/latest/tmp/rand
75dd9b374e712577d64672a05b8ceee40dfc45dce6321082d2c2fd51d60c6c2d  fuse/snapshots/latest/tmp/rand

real    0m6.106s
user    0m3.115s
sys     0m0.182s

after:  $ time cat fuse/snapshots/latest/tmp/rand > /dev/null
real    0m3.096s
user    0m0.017s
sys     0m0.241s
2021-06-12 17:07:30 +02:00
greatroar
0d4f16b6ba Return s3.fileInfos by pointer
Since the fileInfos are returned in a []interface, they're already
allocated on the heap. Making them pointers explicitly means the
compiler doesn't need to generate fileInfo and *fileInfo versions of the
methods on this type. The binary becomes about 7KiB smaller on
Linux/amd64.
2021-06-07 19:48:43 +02:00
greatroar
0666c4d244 Make restic.{lchown,mknod} regular functions
This way, they can be inlined and dead code can be removed on Windows.
Also fixed some comments.
2021-05-27 22:51:40 +02:00
rawtaz
fdbd65485e Merge pull request #3402 from MichaelEischer/misc-fixes
Various small code cleanups
2021-05-24 11:30:31 +02:00
rawtaz
2daf033156 Merge pull request #3403 from MichaelEischer/fast-cat
cat: only load index if really necessary
2021-05-24 11:02:10 +02:00
MichaelEischer
5dad45f005 Merge pull request #3312 from soraxas/master
Bump cobra and add completions for fish
2021-05-23 13:58:27 +02:00
MichaelEischer
7eb6372123 Merge pull request #3257 from greatroar/ls-json-empty
ls: print "size":0 for empty files in JSON
2021-05-23 13:49:56 +02:00
Michael Eischer
61b368ddea cat: only load index if really necessary 2021-05-23 13:11:55 +02:00
Michael Eischer
fd8bce8184 backup: cleanly shutdown goroutines on error 2021-05-23 13:02:44 +02:00
greatroar
d7322a5f36 restic ls --json: print "size":0 for empty files 2021-05-21 21:06:00 +02:00
Tin Lai
9cc1ecdd45 bump cobra and add completions for fish
Signed-off-by: Tin Lai <oscar@tinyiu.com>
2021-05-21 13:47:52 +10:00
Alexander Neumann
af3de702c7 Merge pull request #3332 from restic/debug-1999
Merge `debug examine` to salvage damaged pack files
2021-05-18 09:38:40 +02:00
Alexander Neumann
226cd8d4d1 Merge pull request #3310 from MichaelEischer/copy-unstable-tree
`copy` raw tree blobs
2021-05-18 09:36:51 +02:00
Alexander Neumann
4cabad8c34 Merge pull request #3325 from MichaelEischer/fix-mintty-output
Fix windows terminal output for mintty
2021-05-18 09:29:24 +02:00
Michael Eischer
cf92c58460 Properly wrap errors in readerat helper 2021-05-17 21:08:23 +02:00
Michael Eischer
5767c65c62 delete: properly close fileChan if context is canceled 2021-05-17 21:05:54 +02:00
Michael Eischer
75c990504d azure/gs: Fix default value in connections help text 2021-05-17 20:56:51 +02:00
Michael Eischer
5a87a0ba0a find: use Str() to format short ids 2021-05-17 20:56:49 +02:00
Alexander Neumann
38ccddc84f Merge pull request #3399 from oli-h/oli-h-patch-060_forget.rst
Sudden find: fix within first "forget" example
2021-05-17 11:39:36 +02:00
Alexander Neumann
d7b5061aa5 Merge pull request #3394 from MichaelEischer/apple-m1-build
Build darwin/arm64 binaries for Apple M1
2021-05-17 09:56:44 +02:00
rawtaz
27141ae87f Merge pull request #2999 from restic/issue-2109
backup: Improve wording for --one-file-system description
2021-05-15 00:23:36 +02:00
Leo R. Lundgren
90d75651e6 backup: Improve wording for --one-file-system description 2021-05-15 00:06:27 +02:00
rawtaz
2a915069a8 Merge pull request #3393 from MichaelEischer/fix-filter-crash
filter: Fix crash for '**' pattern
2021-05-14 23:58:44 +02:00
Michael Eischer
55bea6e7a6 filter: Fix crash for '**' pattern 2021-05-14 23:50:31 +02:00
Michael Eischer
af6f6fba15 Build darwin/arm64 binaries for Apple M1 2021-05-14 23:04:45 +02:00
MichaelEischer
6d8ceefd67 Merge pull request #3373 from greatroar/simplify-limiter
Simplify internal/limiter
2021-05-14 20:27:31 +02:00
MichaelEischer
7349c6d338 Merge pull request #3167 from renard/limit-snapshots-list
Add option to limit snapshots list
2021-05-13 20:26:33 +02:00
Sébastien Gross
2a92b68e65 cmd/snapshots: Add option to limit snapshots list
This patch adds a `--latest` option to limit snapshots list to the n
last snapshots. It is very similar to the `--last` one but does not
limit to one entry. It also deprecates the `--last` flag usage in
favor of `--latest 1`

Output example:

    $ restic snapshots --latest 2
    repository 0d3eb989 opened successfully, password is correct
    ID        Time                 Host        Tags        Paths
    ------------------------------------------------------------
    5a33bdcc  2020-12-14 12:30:00  local                   /home
    73887d8e  2020-12-15 12:30:00  local                   /home
    ------------------------------------------------------------
    2 snapshots

Signed-off-by: Sébastien Gross <seb•ɑƬ•chezwam•ɖɵʈ•org>
2021-05-13 20:18:23 +02:00
MichaelEischer
64b00d28b1 Merge pull request #3345 from greatroar/sftp-enospc
Check for ENOSPC and remove broken files in SFTP
2021-05-13 20:09:38 +02:00
oli-h
23f9cb838d Sudden find: fix within first "forget" example
I think this was a typo or copy/paste problem when created the doc.
Feel free to merge. Cheers
2021-05-10 08:55:42 +02:00
rawtaz
01261770bb Merge pull request #3386 from adrian5/doc-change
doc: Add direct link to GitHub issue
2021-05-09 00:12:21 +02:00
adrian5
a0f1c74000 doc: Add direct link to GitHub issue 2021-05-08 21:13:11 +00:00
Alexander Neumann
be6fc02c04 Merge pull request #3376 from restic/rawtaz-doc-exclude
doc: Polish exclude file documentation
2021-04-27 19:25:00 +02:00
Alexander Neumann
3ce5544796 Merge pull request #3321 from restic/doc-files-from
doc: Improve docs for --files-from et al
2021-04-27 19:15:31 +02:00
rawtaz
556caa326f doc: Polish exclude file documentation 2021-04-25 20:36:11 +02:00
greatroar
ae170e2b38 Simplify internal/limiter 2021-04-24 11:54:43 +02:00
rawtaz
f7316cea07 Merge pull request #3371 from cyounkins/max_unused_default
doc: default for --max-unused
2021-04-23 03:18:05 +02:00
Craig Younkins
32a84ab3e4 doc: default for --max-unused 2021-04-22 21:12:37 -04:00
rawtaz
6c7eabf08c Merge pull request #3363 from davegallant/design-doc-grammar-fix
doc: Fix grammar in design.rst
2021-04-15 11:01:38 +02:00
Dave Gallant
4aaf10c356 doc: Fix grammar in design.rst 2021-04-14 17:25:53 -04:00
Michael Eischer
f65e1211d9 Add changelog entry 2021-04-11 20:02:09 +02:00
Michael Eischer
7cb8ea69ba Add test to mintty pipe detection 2021-04-11 20:02:09 +02:00
Michael Eischer
80564a9bc9 Properly detect mintty output redirection
mintty on windows always uses pipes to connect stdout between processes
and for the terminal output. The previous implementation always assumed
that stdout connected to a pipe means that stdout is displayed on a
mintty terminal. However, this detection breaks when using pipes to
connect processes and for powershell which uses pipes when redirecting
to a file.

Now the pipe filename is queried and matched against the pattern used by
msys / cygwin when connected to the terminal. In all other cases assume
that a pipe is just a regular pipe.
2021-04-11 20:02:09 +02:00
Michael Eischer
5e6af77b7a Unify interactive terminal detection code
Previously the progress bar / status update interval used
stdoutIsTerminal to determine whether it is possible to update the
progress bar or not. However, its implementation differed from the
detection within the backup command which included additional checks to
detect the presence of mintty on Windows. mintty behaves like a terminal
but uses pipes for communication.

This adds stdoutCanUpdateStatus() which calls the same terminal detection
code used by backup. This ensures that all commands consistently switch
between interactive and non-interactive terminal mode.

stdoutIsTerminal() now also returns true whenever stdoutCanUpdateStatus()
does so. This is required to properly handle the special case of mintty.
2021-04-11 20:02:09 +02:00
MichaelEischer
cc254dfefe Merge pull request #3362 from greatroar/darwin-preallocate
Use FcntlFstore to preallocate on Mac
2021-04-10 22:47:56 +02:00
greatroar
23531be272 Use FcntlFstore to preallocate on Mac 2021-04-10 16:54:07 +02:00
Alexander Neumann
b922fc851b Merge pull request #3356 from restic/rawtaz-doc-dollar-sign
doc: Clarify dollar sign expansion in exclude files
2021-04-09 12:30:54 +02:00
rawtaz
ccfd5f1d4a Merge pull request #3298 from jniggemann/patch-1
doc: Add note about bash completion path
2021-04-09 00:17:17 +02:00
Jan Niggemann
9a1f685179 doc: Add note about bash completion path 2021-04-08 23:53:38 +02:00
rawtaz
b5e40b370c doc: Clarify dollar sign expansion in exclude files 2021-04-08 23:46:53 +02:00
rawtaz
74c0607c92 Merge pull request #3319 from MichaelEischer/skip-prealloc-test
restorer: Skip preallocate test if not supported by the filesystem
2021-04-07 18:59:06 +02:00
Leo R. Lundgren
5861bb031c doc: Improve docs for --files-from et al 2021-04-07 18:31:46 +02:00
rawtaz
c2569ff923 Merge pull request #3354 from greatroar/filechange-docs
docs: on Windows, the filesize must match for "unchanged"
2021-04-07 15:29:46 +02:00
greatroar
ecbe7f3a99 Backup docs: on Windows, the filesize must match for "unchanged" 2021-04-04 17:01:48 +02:00
Michael Eischer
88731d8c28 Add changelog 2021-03-27 17:08:35 +01:00
greatroar
dc88ca79b6 Handle lack of space and remove broken files in SFTP backend 2021-03-27 15:02:14 +01:00
MichaelEischer
efb10b3c40 Merge pull request #3347 from jazcap53/fix_ambiguous_warning
Change ambiguous Warning message
2021-03-27 12:54:07 +01:00
MichaelEischer
d456437ad1 Merge pull request #3343 from hluup/3334
Display 'created new cache in ' message only if output is terminal
2021-03-27 12:43:37 +01:00
MichaelEischer
4c61825249 Merge pull request #3335 from cyounkins/patch-1
prune --max-unused does not limit metadata repacking
2021-03-27 12:40:35 +01:00
Andrew Jarcho
9f44129c2f Change ambiguous Warning message
Fixes #3346

On branch fix_ambiguous_warning
Changes to be committed:
    modified:   cmd_backup.go
2021-03-24 10:44:47 -04:00
Hendrik Luup
5592c17e4a Display 'created new cache in ' message only if output is terminal 2021-03-22 11:31:08 +02:00
Craig Younkins
aa0a7b78a8 doc: prune --max-unused unlimited will still repack metadata
Adding minor clarification to documentation of `prune --max-unused unlimited` to indicate metadata will still be repacked. The referenced PR indicates `max-repack-size` CAN limit metadata repacking.

Ref: https://forum.restic.net/t/max-unused-unlimited-still-repacks/3661
Ref: https://github.com/restic/restic/pull/3228
2021-03-21 13:19:09 -04:00
Alexander Neumann
88a23521dd Merge pull request #3327 from MichaelEischer/fix-s3-sanity-check
s3: Fix sanity check
2021-03-11 13:13:46 +01:00
MichaelEischer
e678acafcf Merge pull request #3331 from greatroar/upgrade-sftp
Upgrade pkg/sftp to 1.13
2021-03-10 22:27:32 +01:00
Michael Eischer
54d58edacc debug: fix usage for examine command 2021-03-10 22:22:33 +01:00
Michael Eischer
5975ed61f3 debug: fix linter warning 2021-03-10 21:22:53 +01:00
Michael Eischer
dc62ec5933 debug: use Printf/Warnf for output 2021-03-10 21:20:21 +01:00
Michael Eischer
547d9b384d debug: cleanup repair code 2021-03-10 21:15:38 +01:00
greatroar
187a77fb27 Upgrade pkg/sftp to 1.13 and simplify SFTP.IsNotExist 2021-03-10 21:05:24 +01:00
Michael Eischer
fa7b9d5dfe debug: Cleanup pack size checks 2021-03-10 20:57:14 +01:00
Michael Eischer
6774fc6454 debug: check arguments and cleanup error handling 2021-03-10 20:43:22 +01:00
Michael Eischer
096f15db5c debug: extract examinePack function 2021-03-10 20:21:05 +01:00
Michael Eischer
84491ff40b debug: store repaired and correct blobs 2021-03-10 20:03:44 +01:00
Alexander Neumann
b3c3121622 debug: Save raw decrypt (disregarding signature) 2021-03-10 20:03:44 +01:00
Alexander Neumann
ce4b6d0874 examine: add byte repair mode 2021-03-10 20:03:44 +01:00
Alexander Neumann
52061e817c debug: fix percentage 2021-03-10 20:03:44 +01:00
Alexander Neumann
133ac42a0b debug: check file content hash 2021-03-10 20:03:44 +01:00
Alexander Neumann
90f975fa1c debug: make output less verbose 2021-03-10 20:03:44 +01:00
Alexander Neumann
086993bae1 debug: check packs not in index, implement repair 2021-03-10 20:03:44 +01:00
Alexander Neumann
d6f78163d4 Add 'debug examine' command to debug #1999 2021-03-10 20:03:44 +01:00
MichaelEischer
c9b4fadd91 Merge pull request #3294 from Achilleshiel/fix-copy-repofile
Add RepositoryFile2 Option for secondary repository
2021-03-08 22:50:31 +01:00
Michael Eischer
da458a55db Cleanup comments in secondary repo test 2021-03-08 22:41:13 +01:00
Wouter Horlings
9ccdba9df6 Add repositoryFile2 option
The `init` and `copy` commands can now use `--repository-file2` flag and
the `$RESTIC_REPOSITORY_FILE2` environment variable.

This also fixes the conflict with the `--repository-file` and `--repo2`
flag.

Tests are added for the initSecondaryGlobalOpts function.

This adds a NOK function to the test helper functions. This NOK tests if
err is not nil, and otherwise fail the test.

With the NOK function a couple of sad paths are tested in the
initSecondaryGlobalOpts function.

In total the tests checks wether the following are passed correct:
   - Password
   - PasswordFile
   - Repo
   - RepositoryFile

The following situation must return an error to pass the test:
   - no Repo or RepositoryFile defined
   - Repo and RepositoryFile defined both
2021-03-08 22:41:13 +01:00
MichaelEischer
a0f9d73d44 Merge pull request #3305 from juergenhoetzel/terminal-race-condition
backup: In case of error also Wait() for terminal goroutine
2021-03-08 22:20:18 +01:00
Michael Eischer
427781928f copy: test that trees with unstable json encoding are properly copied 2021-03-08 22:16:48 +01:00
Michael Eischer
2fc7abac35 copy: copy raw bytes of tree blobs
This avoids problems when for some reason the JSON encoding changes.
This also ensures forward compatibility with future restic versions
which might e.g. add new fields to the tree metadata.
2021-03-08 22:16:48 +01:00
Michael Eischer
f9c581f219 Tweak changelog 2021-03-08 22:11:34 +01:00
Juergen Hoetzel
18fccb5995 backup: In case of error also Wait() for terminal goroutine
This prevents a race condition where the final summary message can get lost.
2021-03-08 22:11:34 +01:00
Michael Eischer
2a9f0f19b6 s3: Fix sanity check
The sanity check shouldn't replace the error message if there is already
one.
2021-03-08 20:23:57 +01:00
rawtaz
d686fa25de Merge pull request #3323 from restic/doc-setcap
doc: Clarify setcap applying only to current binary
2021-03-08 19:32:11 +01:00
rawtaz
f000f41c91 doc: Clarify setcap applying only to current binary 2021-03-08 19:21:06 +01:00
MichaelEischer
64fe733fa0 Merge pull request #3320 from MichaelEischer/fix-ci
Update to Go 1.16 and pin go version used for golangcil-lint
2021-03-08 19:18:10 +01:00
Michael Eischer
781378a65e Bump go version to 1.16 2021-03-04 23:24:26 +01:00
Michael Eischer
64a9272c9a go mod tidy for go 1.16 2021-03-04 23:18:53 +01:00
Michael Eischer
ee89e33f12 CI: Pin go version used for golint to 1.15.x
This prevents unexpected lint failures when a new go version is
released.
2021-03-04 23:12:48 +01:00
Michael Eischer
f0e9068ef2 go mod tidy for go 1.16 2021-03-04 22:56:08 +01:00
Michael Eischer
f70aca6f6f restorer: Skip preallocate test if not supported by the filesystem 2021-03-04 20:33:46 +01:00
MichaelEischer
814a399e4c Merge pull request #3283 from dennypage/master
Treat an empty password as a fatal error for repository init.
2021-02-28 00:49:21 +01:00
Michael Eischer
13730e3844 Add link to pr to changelog 2021-02-28 00:40:52 +01:00
Denny Page
0f41e99ea7 Treat an empty password as a fatal error for repository init. 2021-02-28 00:40:00 +01:00
MichaelEischer
6712c6de73 Merge pull request #3309 from MichaelEischer/fix-random-check-crash
check: Fix crash of --read-data-subset=x% on empty repository
2021-02-27 16:24:56 +01:00
Michael Eischer
9e852af5be check: Fix crash of --read-data-subset=x% on empty repository
Rounding up to at least one pack file should only be done if at least a
pack file exists.
2021-02-27 16:05:36 +01:00
MichaelEischer
4baebdc6f5 Merge pull request #3308 from MichaelEischer/fix-not-a-directory
local: Ignore files in intermediate folders
2021-02-27 15:34:36 +01:00
Michael Eischer
a293fd9aef local: Ignore files in intermediate folders
For example the data folder of a repository might contain hidden files
which caused list operations to fail.
2021-02-27 13:47:55 +01:00
TJ Rana
91e8f0e486 Clean up writing style 2021-02-16 22:19:11 -05:00
TJ Rana
d7d562b287 Clarify checklist items 2021-02-16 22:11:22 -05:00
TJ Rana
76e1ba6fd0 Clean up pull request template 2021-02-16 22:03:33 -05:00
rawtaz
8eb6a5805b Merge pull request #3286 from MichaelEischer/fix-quiet-backup
backup: Correctly handle --quiet flag
2021-02-15 22:35:04 +01:00
Michael Eischer
65bd2a9a49 backup: Correctly handle --quiet flag
The quiet flag changes the backup output to assume a non-interactive
terminal. However, the output progress interval was not set to 0 by
default.
2021-02-15 22:14:58 +01:00
Alexander Neumann
12f0ccc237 helpers: Also push versioned image to Docker 2021-02-15 20:08:40 +01:00
Alexander Neumann
bb53fcfc0d Set development version for 0.12.0 2021-02-14 11:44:24 +01:00
Alexander Neumann
27f241334e Add version for 0.12.0 2021-02-14 11:44:20 +01:00
Alexander Neumann
4e99a3d650 Update manpages and auto-completion 2021-02-14 11:44:20 +01:00
Alexander Neumann
1cb1cd6f44 Generate CHANGELOG.md for 0.12.0 2021-02-14 11:44:00 +01:00
Alexander Neumann
1a34260cf0 Prepare changelog for 0.12.0 2021-02-14 11:44:00 +01:00
Alexander Neumann
13d52c88fb Merge pull request #3282 from restic/changelog-unreleased
changelog: Correct and slightly polish some unreleased changelog files
2021-02-14 10:49:33 +01:00
Leo R. Lundgren
4b5ca1e914 changelog: Correct and slightly polish some unreleased changelog files 2021-02-14 00:57:54 +01:00
MichaelEischer
917f5b910a Merge pull request #3270 from restic/fix-ls-json
ls: Check for non-nil error before calling panic()
2021-02-09 22:05:04 +01:00
Alexander Neumann
c0f2c1d871 ls: Check for non-nil error before calling panic() 2021-02-07 21:12:54 +01:00
Alexander Neumann
9985368d46 Merge pull request #3255 from MichaelEischer/restorer-check-error
restorer: Check dropped error
2021-02-03 16:10:46 +01:00
Alexander Neumann
2dd592a06c Merge pull request #3254 from lorenz/allow-http2
Allow HTTP/2
2021-02-02 21:08:20 +01:00
Lorenz Brun
362338dd60 Add changelog entry 2021-02-01 20:20:34 +01:00
MichaelEischer
6ac032be64 Merge pull request #3256 from aawsome/rebuild-index-bar-done
rebuild-index: Add missing bar.Done()
2021-01-31 18:48:14 +01:00
MichaelEischer
0ce05d5725 Merge pull request #3236 from Wouter0100/patch-1
doc: Note only path-style URL support for S3
2021-01-31 18:44:58 +01:00
Michael Eischer
0aed8d47d7 docs: Fix typo in S3 URL support 2021-01-31 18:33:15 +01:00
Alexander Weiss
39a26066f7 rebuild-index: add missing bar.Done() 2021-01-31 18:28:02 +01:00
Michael Eischer
47faf69230 restorer: Check dropped error 2021-01-31 18:06:28 +01:00
MichaelEischer
b3dc127af5 Merge pull request #3207 from aawsome/filerestorer-fix-errorhandling
Fix error handling in filerestorer and fix #3166
2021-01-31 17:47:46 +01:00
Wouter van Os
8442c43209 doc: Update wording for S3 path-only requirement 2021-01-31 15:10:03 +01:00
Alexander Weiss
6e942693ba Fix #3166 2021-01-31 14:22:57 +01:00
Alexander Weiss
5e22ae10f1 Add error handling for fileRestorer 2021-01-31 14:22:57 +01:00
Alexander Weiss
573221aa40 Add failing test for fileRestorer 2021-01-31 13:40:42 +01:00
MichaelEischer
b8550a21f2 Merge pull request #3253 from restic/improve-prune-error
prune: Improve error message for missing files
2021-01-31 12:03:20 +01:00
Alexander Neumann
027a51529d prune: Improve error message for missing files
This commit changes the error message so that a list of file names is
printed. Before, just the raw map was printed, which is not a great user
interface.
2021-01-31 11:31:27 +01:00
Lorenz Brun
5427119205 Allow HTTP/2 2021-01-31 02:44:30 +01:00
Alexander Neumann
f647614e24 Merge pull request #3248 from MichaelEischer/backend-cleanups
Backend code and test cleanups
2021-01-30 21:37:33 +01:00
Michael Eischer
e0867c9682 backend: try to cleanup test leftovers 2021-01-30 21:23:20 +01:00
Michael Eischer
f740b2fb23 mem: check upload length before storing upload 2021-01-30 21:23:20 +01:00
Alexander Neumann
0e5f2fff71 Merge pull request #3243 from restic/fix-scanner-overlap
backup: Fix total size for overlapping targets
2021-01-30 21:17:21 +01:00
MichaelEischer
99228be623 Merge pull request #3250 from restic/add-golangci-lint-config
Add golangci lint config, add many error checks
2021-01-30 21:09:38 +01:00
Alexander Neumann
04ca69cc78 Address issues reported by golint 2021-01-30 20:45:57 +01:00
Alexander Neumann
f867e65bcd Fix issues reported by staticcheck 2021-01-30 20:43:53 +01:00
Alexander Neumann
a00e27adf6 Add entry to changelog 2021-01-30 20:22:25 +01:00
Alexander Neumann
0858fbf6aa Add more error handling 2021-01-30 20:19:47 +01:00
Alexander Neumann
aef3658a5f Address review comments 2021-01-30 20:02:37 +01:00
Alexander Neumann
200f09522d Add more error checks 2021-01-30 20:02:37 +01:00
Alexander Neumann
cbd88c457a backup: Improve error handling 2021-01-30 20:02:37 +01:00
Alexander Neumann
1a0eb05bfa errcheck: Add more error checks 2021-01-30 20:02:37 +01:00
Alexander Neumann
3c753c071c errcheck: More error handling 2021-01-30 20:02:37 +01:00
Alexander Neumann
16313bfcc9 errcheck: Add error check for MergeFinalIndexes() 2021-01-30 20:02:37 +01:00
Alexander Neumann
75f53955ee errcheck: Add error checks
Most added checks are straight forward.
2021-01-30 20:02:37 +01:00
Alexander Neumann
1632a84e7b Add a section explaining golangci-lint 2021-01-30 20:02:37 +01:00
Alexander Neumann
b3d5bf7c99 Update golangci-lint version 2021-01-30 20:02:37 +01:00
Alexander Neumann
57627a307f Add config for golangci-lint 2021-01-30 20:02:37 +01:00
Alexander Neumann
6ab7d49a03 Merge pull request #3251 from MichaelEischer/rest-dropped-error
rest: handle dropped error in save operation
2021-01-30 19:44:50 +01:00
Michael Eischer
a53778cd83 rest: handle dropped error in save operation 2021-01-30 19:25:04 +01:00
Alexander Neumann
dd94efb307 Merge pull request #3249 from MichaelEischer/fix-dropped-gs-error
gs: Don't drop error when finishing upload
2021-01-30 16:24:57 +01:00
Michael Eischer
8a486eafed gs: Don't drop error when finishing upload
The error returned when finishing the upload of an object was dropped.
This could cause silent upload failures and thus data loss in certain
cases. When a MD5 hash for the uploaded blob is specified, a wrong
hash/damaged upload would return its error via the Close() whose error
was dropped.
2021-01-30 13:31:32 +01:00
MichaelEischer
4d576c2f79 Merge pull request #3217 from M4a1x/read-data-subset-fix
Fix missing rand seed for restic check --read-data-subset=x%
2021-01-29 23:34:13 +01:00
Max Stabel
f9e1fa26ff Fix missing rand seed for restic check --read-data-subset=x% 2021-01-29 23:18:35 +01:00
MichaelEischer
fb3cf3f885 Merge pull request #3245 from aawsome/prune-fix-statistics-cacheable
prune: Fix statistics for --repack-cacheable-only
2021-01-29 23:14:49 +01:00
Alexander Weiss
e08e65dc30 prune: Simplify logic selecting packs to repack 2021-01-29 22:27:22 +01:00
Alexander Weiss
daeb4cdf8f prune: Fix statistics for --repack-cacheable-only 2021-01-29 22:27:22 +01:00
Alexander Neumann
cdd704920d azure: Pass data length to Azure libray
The azureAdapter was used directly without a pointer, but the Len()
method was only defined with a pointer receiver (which means Len() is
not present on a azureAdapter{}, only on a pointer to it).
2021-01-29 21:08:41 +01:00
Alexander Neumann
bbdf18c4a2 Merge pull request #3176 from MichaelEischer/backend-content-length
Pass upload size to backends and sanity check it
2021-01-29 20:33:44 +01:00
Michael Eischer
1f583b3d8e backend: test that incomplete uploads fail 2021-01-29 13:51:53 +01:00
Michael Eischer
c73316a111 backends: add sanity check for the uploaded file size
Bugs in the error handling while uploading a file to the backend could
cause incomplete files, e.g. https://github.com/golang/go/issues/42400
which could affect the local backend.

Proactively add sanity checks which will treat an upload as failed if
the reported upload size does not match the actual file size.
2021-01-29 13:51:51 +01:00
Michael Eischer
4526d5d197 swift: explicitly pass upload size to library
This allows properly setting the content-length which could help the
server-side to detect incomplete uploads.
2021-01-29 13:50:46 +01:00
Michael Eischer
dca9b6f5db azure: explicitly pass upload size
Previously the fallback from the azure library was to read the whole
blob into memory and use that to determine the upload size.
2021-01-29 13:50:46 +01:00
Alexander Neumann
a16ce65295 Merge pull request #3244 from MichaelEischer/better-damage-reports
Print more details about possible repository damages
2021-01-29 11:45:45 +01:00
Alexander Neumann
5c41120c70 Add entry to changelog 2021-01-29 11:31:36 +01:00
Alexander Neumann
5c617859ab backup/scanner: Fix total size for overlapping targets
Before, the scanner would could files twice if they were included in the
list of backup targets twice, e.g. `restic backup foo foo/bar` would
could the file `foo/bar` twice.

This commit uses the tree structure from the archiver to run the
scanner, so both parts see the same files.
2021-01-29 11:31:36 +01:00
Alexander Neumann
81211750ba archiver/tree: Introduce functions Leaf() and NodeNames() 2021-01-29 11:11:28 +01:00
rawtaz
de7e3a0648 Merge pull request #2823 from greatroar/trust-mtime
Add --ignore-ctime flag to backup and document change detection
2021-01-29 00:02:19 +01:00
greatroar
6bd8a2faaa backup: Add --ignore-ctime option and document change detection 2021-01-28 23:42:10 +01:00
Michael Eischer
58b5679f14 prune: reword missing blobs error
The previous wording could be understood such that the prune run did
damage the repository.
2021-01-28 21:48:24 +01:00
Michael Eischer
7b8886c052 prune: report missing but unneeded pack files
This indicates a damaged repository so add some output to help with
debugging.
2021-01-28 21:46:01 +01:00
Michael Eischer
ff95999246 rebuild-index: report added/removed/reindexed files
This should help with investigating missing pack files.
2021-01-28 21:46:01 +01:00
Michael Eischer
b71c52797a find: correctly expand multiple blob ids
For example `restic find --show-pack-id --blob f78dc991 5b9e4366 ddd8c7d4`
would previously only expand one blob if all of them belong to the same
file.
2021-01-28 21:21:54 +01:00
MichaelEischer
82140967d3 Merge pull request #3228 from aawsome/prune-all-trees
prune: Remove all unused trees
2021-01-28 21:04:27 +01:00
MichaelEischer
43cb26010a Merge pull request #3242 from greatroar/fprintln
internal/ui/termstatus: Use Fprintln to get a newline
2021-01-28 20:34:06 +01:00
MichaelEischer
35033d9b79 Merge pull request #3177 from MichaelEischer/fix-2759
prune: don't print stacktrace on console
2021-01-28 20:28:45 +01:00
Alexander Neumann
84822d44d4 Merge pull request #2536 from MatthewVance/threat-model
Update threat model
2021-01-28 14:26:03 +01:00
Matthew Vance
58c7f4694d Update threat model 2021-01-28 14:20:42 +01:00
Alexander Neumann
4d40c70214 Merge pull request #3211 from MichaelEischer/sftp-speedup
Speed-up caching/pack download via SFTP
2021-01-28 14:16:55 +01:00
Alexander Neumann
44169d0dc4 Merge pull request #3205 from MichaelEischer/fix-quiet-verbose
Properly check that --quiet and --verbose are not combined
2021-01-28 13:53:31 +01:00
Alexander Neumann
6aa7e9f9c6 Merge pull request #3174 from MichaelEischer/parallelize-lock-loading
Parallelize lock file loading
2021-01-28 13:52:12 +01:00
Alexander Neumann
bdfedf1f5b Merge pull request #3173 from MichaelEischer/unify-index-loading
Unify index loading
2021-01-28 13:50:42 +01:00
greatroar
b9cfe6f68a internal/ui/termstatus: Use Fprintln to get a newline 2021-01-28 13:30:10 +01:00
Alexander Neumann
72eec8c0c4 Merge pull request #3106 from MichaelEischer/parallel-tree-walk
Parallelize tree walk in prune and copy and add progress bar to check
2021-01-28 12:06:42 +01:00
Michael Eischer
68608a89ad restic: add comment about StreamTrees shutdown 2021-01-28 11:10:50 +01:00
Michael Eischer
1e306be000 Add changelog entry 2021-01-28 11:10:50 +01:00
Michael Eischer
ddb7697d29 restic: Test progress reporting of StreamTrees 2021-01-28 11:10:50 +01:00
Michael Eischer
313ad0e32f progress/counter: Fix test for final report call 2021-01-28 11:10:50 +01:00
Michael Eischer
e2b0072441 check: add progress bar to the tree structure check 2021-01-28 11:10:50 +01:00
Michael Eischer
505f8a2229 progress/counter: Support updating the progress bar maximum 2021-01-28 11:10:47 +01:00
Michael Eischer
eda8c67616 restic: let FindUsedBlobs handle multiple snapshots at once 2021-01-28 11:08:43 +01:00
Michael Eischer
258ce0c1e5 parallel: report progress for StreamTrees
This assigns an id to each tree root and then keeps track of how many
tree loads (i.e. trees referenced for the first time) are pending per
tree root. Once a tree root and its subtrees were fully processed there
are no more pending tree loads and the tree root is reported as
processed.
2021-01-28 11:08:43 +01:00
Michael Eischer
3d6a3e2555 copy: Remove treeCloner struct 2021-01-28 11:08:43 +01:00
Michael Eischer
0caad1e890 copy: parallelize tree walk 2021-01-28 11:08:43 +01:00
Michael Eischer
f2a1b125cb restic: Actually parallelize FindUsedBlobs 2021-01-28 11:08:43 +01:00
Michael Eischer
6e03f80ca2 check: Split the parallelized tree loader into a reusable component
The actual code change is minimal
2021-01-28 11:08:43 +01:00
Michael Eischer
1d7bb01a6b check: Cleanup tree loading and switch to use errgroup
The helper methods are now wired up in the Structure method.
2021-01-28 11:08:43 +01:00
Alexander Neumann
a4689eb3b9 Merge pull request #3199 from MichaelEischer/non-interactive-counter
Don't print progress on non-interactive terminals
2021-01-28 10:53:38 +01:00
Alexander Neumann
c5a66e9181 ui: Simlify channel receive 2021-01-28 10:42:02 +01:00
Wouter van Os
b5972f184c doc: Note only path-style URL support for S3
This adds a node to mention that currently only path-style URL's are supported for S3, as per code at:
- aa0faa8c7d/internal/backend/s3/config.go (L42-L45)
- aa0faa8c7d/internal/backend/s3/config.go (L48-L62)
2021-01-23 16:54:08 +01:00
Alexander Weiss
d7dc19a496 prune: Always repack packs containing tree blobs 2021-01-15 16:42:04 +01:00
Michael Eischer
f3442ce8a5 Test that WriteTo of a backend's Load remains accessible 2021-01-03 22:23:53 +01:00
Michael Eischer
678e75e1c2 sftp: enforce use of optimized upload method
ReadFrom was already used by Save before, this just ensures that this
won't accidentally change in the future.
2021-01-03 22:23:53 +01:00
Michael Eischer
6b5b29dbee limiter: add unit tests 2021-01-03 22:23:53 +01:00
Michael Eischer
f35f2c48cd limiter: support WriteTo in LimitBackend for read rate limiting 2021-01-03 22:23:53 +01:00
Michael Eischer
bcb852a8d0 hashing: support WriteTo in the reader 2021-01-03 22:23:53 +01:00
MichaelEischer
aa0faa8c7d Merge pull request #3208 from restic/add-mips
Add mips* architectures to CI and release
2021-01-03 16:06:10 +01:00
MichaelEischer
f7ec263a22 Merge pull request #3109 from aawsome/optimize-filerestorer
restore: Don't save pack content in memory
2021-01-03 14:53:41 +01:00
Alexander Neumann
7d665fa1f4 Add entry to changelog 2021-01-03 14:41:11 +01:00
Michael Eischer
69d5b4c36b restorer: lower-case variable name 2021-01-03 13:55:59 +01:00
Alexander Neumann
36db248e30 Split cross compilation targets into two jobs 2021-01-03 10:50:54 +01:00
Alexander Neumann
eb72b10f55 Add mips* architectures to CI and release 2021-01-03 10:41:54 +01:00
Alexander Neumann
622f4c7daa Run cloud backends also on pushes 2021-01-03 10:38:10 +01:00
Alexander Neumann
f8c50394d6 Revert "Run cloud tests if secrets are available"
This reverts commit aa648bdcac.
2021-01-03 10:28:01 +01:00
Alexander Neumann
aa648bdcac Run cloud tests if secrets are available 2021-01-03 10:26:55 +01:00
Alexander Neumann
e8abc79ce9 Add hints for keeping the list of architectures in sync 2021-01-01 10:09:04 +01:00
Alexander Weiss
34a33565c8 Fix loadBlob in filerestorer 2021-01-01 08:06:04 +01:00
Alexander Weiss
7409225fa8 Add filerestorer test where only parts of pack are used 2021-01-01 07:24:46 +01:00
Alexander Weiss
07b3f65a6f filesrestorer: Re-use buffer 2021-01-01 07:24:46 +01:00
Alexander Weiss
3e0acf1395 restore: Don't save (part of) pack in memory 2021-01-01 07:24:46 +01:00
Michael Eischer
97388b3504 Properly check that --quiet and --verbose are not combined
If --verbose is specified once, then globalOptions.Verbose == 1.
Previously --quiet --verbose would silently ignore the --verbose flag.
2020-12-30 21:24:18 +01:00
Alexander Neumann
8b84c96d9d Merge pull request #3204 from MichaelEischer/archiver-tomb-race
archiver: fix race condition during worker startup
2020-12-30 20:04:20 +01:00
Michael Eischer
debc4a3a99 archiver: fix race condition during worker startup
When the tomb is created with a canceled context, then the workers
started via `t.Go` exist nearly immediately. Once for the first time all
started goroutines have been stopped, it is not allowed to issue further
calls to `t.Go`. This is a problem when the started goroutines exit
immediately, as for example the first goroutine might already have
stopped before starting the second one, which is not allowed as once the
first goroutines has stopped no goroutines were running.

To fix this race condition the startup and main task of the archiver now
also run within a `t.Go` function. This also allows unifying the error
handling as it is no longer necessary to distinguish between errors
returned by the workers or the saveTree processing. The tomb now just
returns the first error encountered, which should also be the most
descriptive one.
2020-12-30 17:31:22 +01:00
MichaelEischer
e1efc193e1 Merge pull request #3139 from aawsome/prune-healing
prune: Add healing of repository in some situations
2020-12-29 22:17:27 +01:00
Alexander Weiss
f0113139ea prune: Correct error message 2020-12-29 20:20:05 +01:00
Alexander Weiss
f6df94a50e prune: Add self-healing
Allow prune to heal situations where blobs in the index are missing or
the corresponding packfiles are damaged if those blobs are not needed.
2020-12-29 20:20:05 +01:00
MichaelEischer
31e56f1ad5 Merge pull request #3197 from SkYNewZ/fix/3183
Fix tag handling for multiple tag lists
2020-12-29 18:44:38 +01:00
MichaelEischer
7fda2f2ad8 Merge pull request #3134 from greatroar/unlock-warn
Warn when unlock fails instead of returning an error
2020-12-29 18:30:08 +01:00
greatroar
dec5008369 Warn when unlock fails instead of returning an error
Only one caller was checking the error.
2020-12-29 17:48:20 +01:00
Quentin Lemaire
873505ed3b Update related changelog 2020-12-29 17:12:46 +01:00
Quentin Lemaire
25ecf9eafb fix(cmd_tag): Use restic.TagLists 2020-12-29 17:12:46 +01:00
Quentin Lemaire
e88f3fb80c fix(cmd_backup): Use restic.TagLists 2020-12-29 17:12:46 +01:00
Alexander Neumann
b2efa0af39 Merge pull request #3164 from MichaelEischer/improve-context-cancel
Improve context cancel handling in archiver and backends
2020-12-29 17:03:42 +01:00
Michael Eischer
25f4acdaa8 Add changelog entry 2020-12-29 16:32:18 +01:00
Michael Eischer
cff4955a48 ui: remove dead struct member 2020-12-29 16:32:18 +01:00
Michael Eischer
05a987b07c Support values less than 1 for RESTIC_PROGRESS_FPS
For example set the variable to 0.016666 to print the progress once per
minute.
2020-12-29 16:32:18 +01:00
Michael Eischer
92da5168e1 ui: force backup progress update on signal 2020-12-29 16:32:18 +01:00
Michael Eischer
34afc93ddc ui/progress: extract signal handling into own package 2020-12-29 16:32:18 +01:00
Michael Eischer
023eea6463 ui: don't shorten non-interactive progress output 2020-12-29 16:32:18 +01:00
Michael Eischer
684600cf42 ui: update status for the backup command on non-interactive terminals
Allow the backup command to print status on non-interactive terminals.
The output is disabled by setting a MinUpdatePause == 0.
2020-12-29 16:03:43 +01:00
Michael Eischer
85fe5feadb Unify progress report frequency calculation 2020-12-29 16:03:43 +01:00
Michael Eischer
969141b5e9 Honor RESTIC_PROGRESS_FPS env variable on non-interactive terminals
This makes it possible to use the environment variable to also get
regular progress updates on non-interactive terminals.
2020-12-29 16:03:43 +01:00
Michael Eischer
13ce981794 ui: cleanup backup status shutdown 2020-12-29 16:03:43 +01:00
Michael Eischer
c2ef049f1b ui/progress: don't print progress on non-interactive terminals
This reverts to the old behavior of not printing progress updates on
non-interactive terminals. It was accidentally changed in #3058.
2020-12-29 16:03:43 +01:00
MichaelEischer
a488d4c847 Merge pull request #2833 from greatroar/aix
AIX port
2020-12-29 12:32:50 +01:00
Alexander Neumann
4133b1ea65 Synchronize OS and architectures for testing 2020-12-29 11:11:50 +01:00
Alexander Neumann
46d2ca5095 Remove old test script 2020-12-29 11:02:48 +01:00
Quentin Lemaire
334d8ce724 feat(tags): Create Flatten() method 2020-12-29 10:59:46 +01:00
Alexander Neumann
c661518df9 Merge pull request #2793 from greatroar/rclone-doc
Clarify rclone-over-SSH docs
2020-12-29 10:46:34 +01:00
Michael Eischer
0d81f16343 Add AIX as cross-compile target to CI 2020-12-29 01:35:01 +01:00
greatroar
3b09ae9074 AIX port 2020-12-29 01:35:01 +01:00
greatroar
18531e3d6f Portability fixes to internal/restic
syscall.Mknod is not available on AIX.
2020-12-29 01:35:01 +01:00
Michael Eischer
ca07317815 add changelog 2020-12-28 21:06:47 +01:00
Michael Eischer
d0ca8fb0b8 backend: test that a canceled context prevents RetryBackend operations 2020-12-28 21:06:47 +01:00
Michael Eischer
08b7f2b58d archiver: test that context canceled error is not dropped 2020-12-28 21:06:47 +01:00
Michael Eischer
e483b63c40 retrybackend: Fail operations when context is already canceled
Depending on the used backend, operations started with a canceled
context may fail or not. For example the local backend still works in
large parts when called with a canceled context. Backends transfering
data via http don't work. It is also not possible to retry failed
operations in that state as the RetryBackend will abort with a 'context
canceled' error.

Ensure uniform behavior of all backends by checking for a canceled
context by checking for a canceled context as a first step in the
RetryBackend. This ensures uniform behavior across all backends, as
backends are always wrapped in a RetryBackend.
2020-12-28 21:06:47 +01:00
Michael Eischer
fc60b560ba archiver: Let saveTree report a canceled context as an error
If the context was canceled then saveTree might receive a treeID or not
depending on the timing. This could cause saveTree to incorrectly return
a nil treeID as valid. Fix this always returning an error when the
context was canceled in the meantime.
2020-12-28 21:06:47 +01:00
Michael Eischer
736e964317 archiver: Don't loose error if background context is canceled
A canceled background context lets the blob/tree/fileSavers exit
without reporting an error. The error handling previously replaced
a 'context canceled' error received by the main backup method with
the error reported by the savers. However, in case of a canceled
background context that error is nil, causing restic to loose the
error and save a snapshot with a nil tree.
2020-12-28 21:06:47 +01:00
MichaelEischer
9c41e4a343 Merge pull request #3192 from DRON-666/fix-options
Fix formatting of `restic options` command
2020-12-23 22:53:40 +01:00
DRON-666
332b1896d1 Some options fixes
Add tests for bool type.
Fix subtle bug in TestOptionsApplyInvalid.
Fix options list formatting.
2020-12-23 23:26:04 +03:00
MichaelEischer
cb6b0f6255 Merge pull request #3181 from tWido/feature-no-retry-permission
Don't retry when "Permission denied" occurs in local backend
2020-12-23 20:24:29 +01:00
MichaelEischer
1e73aac610 Merge pull request #3179 from aawsome/check-filesize
check: Remove filesize counter
2020-12-23 20:22:02 +01:00
Alexander Weiss
2a1add7538 check: remove file size counter 2020-12-23 02:34:31 +01:00
tWido
7dab113035 Don't retry when "Permission denied" occurs in local backend 2020-12-22 23:41:12 +01:00
MichaelEischer
8efb874f48 Merge pull request #3148 from aawsome/simplify-index-rebuild
rebuild-index: code simplification
2020-12-22 23:27:53 +01:00
Michael Eischer
de99207046 repository: tweak comment for packs method 2020-12-22 23:01:58 +01:00
Alexander Weiss
d8d2cc6dd9 Simplify rebuild-index code 2020-12-22 23:01:58 +01:00
Alexander Weiss
68b74e359e Count packs directly in RebuildIndexFiles 2020-12-22 23:01:58 +01:00
Michael Eischer
b9f5d3fe13 repository: Add test for ForAllIndexes 2020-12-22 22:36:18 +01:00
Michael Eischer
a12c5f1d37 repository: move otherwise unused LoadIndex to tests 2020-12-22 22:36:18 +01:00
Michael Eischer
24474a36f4 repository: deduplicate index loading implementation 2020-12-22 22:36:18 +01:00
Michael Eischer
ccc84af73d debug/list: parallelize index loading 2020-12-22 22:36:18 +01:00
Michael Eischer
96904f8972 check: extract parallel index loading 2020-12-22 22:36:18 +01:00
MichaelEischer
69f9d269eb Merge pull request #3189 from restic/fix-counter-segfault
ui/progress: Use mutex instead of atomic
2020-12-22 22:14:08 +01:00
rawtaz
ec59c73489 Merge pull request #3188 from restic/forget-prune-dry-run
forget: Enable --dry-run together with --prune
2020-12-22 21:09:15 +01:00
Alexander Neumann
6c514adb8a ui/progress: Use mutex instead of atomic
The counter value needs to be aligned to 64 bit in memory for the
atomic functions to work on some platform (such as 32 bit ARM).

The atomic package says in its documentation:

> These functions require great care to be used correctly. Except for
> special, low-level applications, synchronization is better done with
> channels or the facilities of the sync package.

This commit replaces the atomic functions with a simple sync.Mutex, so
we don't have to care about alignment.
2020-12-22 21:03:27 +01:00
Alexander Neumann
edf89e1c74 forget: Enable --dry-run together with --prune 2020-12-22 20:58:02 +01:00
Alexander Neumann
f7c7c2f730 Merge pull request #3175 from MichaelEischer/fix-3099-changelog
Fix changelog for #3099
2020-12-20 11:29:57 +01:00
Michael Eischer
cfea79d0c5 prune: don't print stacktrace on console 2020-12-19 14:28:48 +01:00
Michael Eischer
5cd40f8b58 Add changelog 2020-12-19 11:38:38 +01:00
MichaelEischer
d32949ee54 Merge pull request #3081 from DRON-666/dump-zip
Add zip support to the `dump` command
2020-12-19 11:33:33 +01:00
DRON-666
83b10dbb12 Deduplicate dumper closing logic 2020-12-19 02:42:46 +03:00
DRON-666
e136dd8696 Deduplicate test code 2020-12-19 02:06:54 +03:00
DRON-666
33adb58817 Minor fixes and linter suggestions 2020-12-19 02:04:17 +03:00
DRON-666
da9053b184 Some gramma fixes in documentation 2020-12-19 01:16:15 +03:00
DRON-666
ef1aeb8724 dump: Update docs and changelog 2020-12-19 01:09:47 +03:00
DRON-666
2ca76afc2b dump: Add new option --archive 2020-12-19 01:09:47 +03:00
DRON-666
89ab6d557e dump: Add zip dumper 2020-12-19 01:09:47 +03:00
DRON-666
0256f95994 dump: Split tar and walk logic 2020-12-19 01:09:47 +03:00
Michael Eischer
bfadc82a20 Fix changelog for #3099 2020-12-18 20:58:24 +01:00
Michael Eischer
34b6130a0e restic: parallelize lock file loading 2020-12-18 20:46:16 +01:00
MichaelEischer
22260d130d Merge pull request #3170 from greatroar/dont-retry
Don't retry permanent errors in backends
2020-12-17 23:29:56 +01:00
greatroar
9341a83b05 Changelog entry for #2453 and #3170 2020-12-17 23:15:48 +01:00
greatroar
66d904c905 Make invalid handles permanent errors 2020-12-17 12:47:53 +01:00
greatroar
746dbda413 Mark "ssh exited" errors in SFTP as permanent 2020-12-17 12:43:09 +01:00
greatroar
f7784bddb3 Don't retry when "no space left on device" in local backend
Also adds relevant documentation to the restic.Backend interface.
2020-12-17 12:43:09 +01:00
Alexander Neumann
1cdd38d9e0 Merge pull request #3165 from restic/2747-doc-forget
doc: Clarify calendar boundaries for --keep-* options
2020-12-16 08:38:03 +01:00
Leo R. Lundgren
b3c0d2f45b doc: Clarify calendar boundaries for --keep-* options 2020-12-15 12:43:06 +01:00
Peter Albrecht
cd25e36811 Add PGP fingerprint to 020_installation.rst
I like the idea of verifying the integrity of applications, I download from the internet. So I was very happy to see that restic does provide SHA256-checksums which are signed with the maintainers PGP key.

The only thing I miss: I could not find a direct way to download the used PGP key and verify the keys fingerprint.

Doing some searches, I found:
* https://github.com/restic/rest-server/issues/121
* https://restic.net/blog/2015-09-16/verifying-code-archive-integrity/

To help other restic users, I think you should add information about your PGP key/fingerprint to this installation doc, too. To save you some precious time, I created a draft, how this doc might be expanded, in this pull-request. You are free to accept it or change the text to your liking.

I copied the key/fingerprint text from: ``restic/restic/master/doc/090_participating.rst``

Thank you for your work in restic!
2020-12-13 17:00:00 +01:00
Alexander Neumann
e96677cafb Merge pull request #3158 from MichaelEischer/support-swift-auth-id-variables
swift: Add support for id based keystone v3 auth parameters
2020-12-12 16:27:38 +01:00
Michael Eischer
1d69341e88 swift: Add support for id based keystone v3 auth parameters
This adds support for the following environment variables, which were
previously missing:

OS_USER_ID            User ID for keystone v3 authentication
OS_USER_DOMAIN_ID     User domain ID for keystone v3 authentication
OS_PROJECT_DOMAIN_ID  Project domain ID for keystone v3 authentication
OS_TRUST_ID           Trust ID for keystone v3 authentication
2020-12-11 19:22:34 +01:00
Alexander Neumann
36c5d39c2c Fix issues reported by semgrep 2020-12-11 09:41:59 +01:00
Alexander Neumann
7facc8ccc1 Merge pull request #2505 from aawsome/fix-repo-configfile
Fix repo configfile
2020-12-07 07:52:37 +01:00
Alexander Neumann
ba31c6fdaa Merge pull request #3150 from MichaelEischer/fix-windows-redir-output
termstatus: Fix canUpdateStatus detection for redirected output on windows
2020-12-06 21:17:27 +01:00
Alexander Neumann
b58799d83a Merge pull request #3152 from MichaelEischer/fix-backup-background-hang
backup: Fix shutdown hang when running in the background on linux
2020-12-06 21:16:00 +01:00
Alexander Neumann
0d5b764f90 Merge pull request #3130 from aawsome/snapshots-parallel
Make loading snapshots parallel
2020-12-06 21:08:18 +01:00
Alexander Weiss
d6b3859e48 Add changelog 2020-12-06 19:29:18 +01:00
Michael Eischer
b48f579530 termstatus: Fix canUpdateStatus detection for redirected output
The canUpdateStatus check was simplified in #2608, but it accidentally flipped
the condition. The correct check is as follows: If the output is a pipe then
restic probably runs in mintty/cygwin. In that case it's possible to
update the output status. In all other cases it isn't.

This commit inverts to condition again to offer the previous and correct
behavior.
2020-12-06 19:02:42 +01:00
Michael Eischer
401ef92c5f backup: Fix shutdown hang when running in the background
On shutdown the backup commands waits for the terminal output goroutine
to stop. However while running in the background the goroutine ignored
the canceled context.
2020-12-06 18:53:41 +01:00
Alexander Weiss
e329623771 Remove LoadAllSnapshots 2020-12-06 05:22:27 +01:00
Alexander Weiss
26f85779be Parallelize ForAllSnapshots 2020-12-06 05:09:58 +01:00
Alexander Weiss
5b9ee56335 Add ForAllSnapshots 2020-12-06 05:04:21 +01:00
MichaelEischer
3264eae9f6 Merge pull request #3149 from aawsome/fix-rebuild-index
Bugfix for rebuild-index
2020-12-05 22:18:22 +01:00
Alexander Weiss
83c8a9b058 Bugfix: packSizeFromList should save size from List() 2020-12-05 20:58:36 +01:00
Alexander Neumann
43cf301450 Merge pull request #3141 from MichaelEischer/fix-fuse-dir-owner
fuse: Properly set uid/gid for directories
2020-12-01 16:06:24 +01:00
Michael Eischer
d05c88a5d6 fuse: Properly set uid/gid for directories
In #2584 this was changed to use the uid/gid of the root node. This
would be okay for the top-level directory of a snapshot, however, this
change also applied to normal directories within a snapshot. This
change reverts the problematic part and adds a test that directory
attributes are represented correctly.
2020-11-30 23:41:49 +01:00
Alexander Neumann
058b102db0 Merge pull request #3138 from MichaelEischer/fix-debug-build 2020-11-30 12:27:51 +01:00
rawtaz
fcebc7d250 Merge pull request #3135 from restic/manual
doc: Fix misc missing/incorrect text in manual
2020-11-29 19:20:51 +01:00
MichaelEischer
f2959127b6 Merge pull request #3065 from greatroar/local-subdirs
Don't recurse in local backend's List if not required
2020-11-29 19:03:59 +01:00
Leo R. Lundgren
61460dee52 doc: Fix misc missing/incorrect text in manual 2020-11-29 18:59:24 +01:00
Michael Eischer
54a6d98945 Enable debug builds for CI 2020-11-29 18:47:00 +01:00
Michael Eischer
f72f6c9c80 Fix debug build 2020-11-29 18:44:36 +01:00
MichaelEischer
52b98f7f95 Merge pull request #3017 from greatroar/files-from0
Add backup options --files-from-verbatim and --files-from-raw
2020-11-29 18:15:21 +01:00
Alexander Neumann
04d856e601 Merge pull request #3136 from restic/rawtaz-copy-doc
doc: Emphasize double transfer and duplication in copy command
2020-11-29 13:57:10 +01:00
Alexander Neumann
a7b49c4889 Merge pull request #3119 from restic/keep-mountpoints
Keep mountpoints as empty directories for --one-file-system
2020-11-29 11:22:12 +01:00
rawtaz
a568211b98 Merge pull request #3128 from vrenaville/tzdata
[FIX] Timezone in docker image
2020-11-28 22:43:18 +01:00
Leo R. Lundgren
f70b10d0ee doc: Emphasize double transfer and duplication in copy command 2020-11-28 19:49:34 +01:00
greatroar
55bf76ba0c backup: Add --files-from-{verbatim,raw} options 2020-11-28 18:22:31 +01:00
Alexander Neumann
162117c42c Add changelog 2020-11-28 17:00:31 +01:00
Alexander Neumann
82ae942965 backup: Keep mountpoints for --one-file-system
When a file system is mounted at a directory, lstat() returns attributes
of the root node of the mounted file system, including the device ID of
the other file system. The previous code used when --one-file-system is
specified excluded the directory itself because of that.

This commit changes the code so that mountpoints are kept as empty
directories, its attributes set to the root note of the mounted file
system. The behavior mimics `tar`, which does the same.
2020-11-28 17:00:31 +01:00
Alexander Neumann
f576d3d826 Add tests 2020-11-28 17:00:31 +01:00
Alexander Neumann
037f0a4c91 Refactor device ID checking 2020-11-28 17:00:31 +01:00
Alexander Neumann
2f9346a5af Merge pull request #3125 from metalsp0rk/cat-respect-no-lock
Make restic cat respect --no-lock
2020-11-28 12:45:59 +01:00
Alexander Neumann
a3105799c9 Print warning when unlocking the repo fails 2020-11-28 11:31:25 +01:00
Kyle Brennan
666768cd17 cat: Respect --no-lock flag 2020-11-28 11:30:02 +01:00
vrenaville
adc7a6555f [FIX] Timezone in docker image 2020-11-27 08:05:19 +01:00
Alexander Neumann
9a97095a4c Merge pull request #3120 from aawsome/blob-implementation
Blob implementation
2020-11-22 20:59:30 +01:00
Alexander Weiss
aa7a5f19c2 Use BlobHandle in index methods 2020-11-22 20:41:12 +01:00
Alexander Weiss
e3013271a6 Harmonize naming 2020-11-22 20:41:12 +01:00
Alexander Weiss
92bd448691 Make BlobHandle substruct of Blob 2020-11-22 20:41:10 +01:00
Alexander Neumann
c844580e0f Merge pull request #3101 from aawsome/packsizes
Compute packsizes in MasterIndex
2020-11-22 15:49:19 +01:00
Alexander Weiss
67c938f232 Use PackSize in prune 2020-11-21 22:13:54 +01:00
Alexander Weiss
a851c53cbe Use PackSize in checker 2020-11-21 22:13:54 +01:00
Alexander Weiss
4960b841e6 Use PackSize in rebuild-index 2020-11-21 22:13:54 +01:00
Alexander Weiss
ce5d630681 Add MasterIndex.PackSize() 2020-11-21 22:13:54 +01:00
Alexander Weiss
c3ddde9e7d Return hdrSize in ListPack 2020-11-21 22:13:54 +01:00
rawtaz
cac481634c Merge pull request #3115 from johanbove/patch-1
Docs: Update 04_backup.rst
2020-11-20 11:23:03 +01:00
Johan Bové
c23b1a4cba Update 04_backup.rst
Fixed typo - _files_ are included from _folders_, not other _files_.
2020-11-20 07:52:23 +01:00
Alexander Neumann
110a32a08b Merge pull request #3113 from aawsome/fix-prune-stats-duplicates
Fix statistics in prune for duplicates
2020-11-19 20:34:11 +01:00
greatroar
8e213e82fc backend/local: replace fs.Walk with custom walker
This code is more strict in what it expects to find in the backend:
depending on the layout, either a directory full of files or a directory
full of such directories.
2020-11-19 16:46:42 +01:00
Alexander Neumann
8a150ee91f Merge pull request #3112 from MichaelEischer/reduce-progress-log-spam
Limit progress bar updates to once per second on non-terminal outputs
2020-11-19 07:54:36 +01:00
Alexander Weiss
cb5ec7ea6b Fix statistics in prune for duplicates
Note that this fix only solves the statistics problem, if
all duplicates are marked for repacking.

If not all duplicates are marked for repacking, we lack the
information which

The situation that not all duplicates are marked for repacking can occur
when using the `max-repack-size` option
2020-11-18 22:30:22 +01:00
Michael Eischer
625410f003 Limit progress bar updates to once per second on non-terminal outputs
The code accidentally checked whether stdin is a terminal instead of
stdout, the former is not relevant here as the output is printed on
stdout.
2020-11-18 22:12:07 +01:00
Alexander Neumann
75eff92b56 Merge pull request #3107 from eleith/do-not-require-bucket-permissions-for-init
do not require gs bucket permissions to init repository
2020-11-18 16:53:45 +01:00
eleith
a24e986b2b do not require gs bucket permissions to init repository
a gs service account may only have object permissions on an existing
bucket but no bucket create/get permissions.

these service accounts currently are blocked from initialization a
restic repository because restic can not determine if the bucket exists.

this PR updates the logic to assume the bucket exists when the bucket
attribute request results in a permissions denied error.

this way, restic can still initialize a repository if the service
account does have object permissions

fixes: https://github.com/restic/restic/issues/3100
2020-11-18 06:14:11 -08:00
rawtaz
6822ce8479 Merge pull request #3105 from restic/fix-init-repo-file
Allow using --repository-file in init
2020-11-17 21:54:44 +01:00
Alexander Neumann
d857fb6e59 Allow using --repository-file in init 2020-11-17 20:43:46 +01:00
rawtaz
342520b648 Merge pull request #3103 from greatroar/rephrase-pr3102
Rephrase #3095/#3102 changelog entry
2020-11-17 12:52:12 +01:00
greatroar
028f2b8c0e Rephrase #3095/#3102 changelog entry 2020-11-17 12:37:06 +01:00
rawtaz
1b6e8c888f Merge pull request #3102 from tofran/remove-rclone-drive-use-tras-default-param
Remove `--drive-use-trash=false` from rclone param
2020-11-17 11:21:20 +01:00
Alexander Neumann
5f3b802ee7 Merge pull request #3099 from MichaelEischer/check-less-memory 2020-11-16 10:11:25 +01:00
Michael Eischer
022dc35be9 Add changelog 2020-11-15 19:02:51 +01:00
Michael Eischer
1f43cac12d check: Only track data blobs when unused blobs should be reported
This improves the memory usage of check a lot as it now only has to
track tree blobs when run using the default parameters.
2020-11-15 18:43:07 +01:00
Michael Eischer
6da66c15d8 check: Simplify referenced blob tracking
The result is identical as long as the context in not canceled. However,
in that case the result is incomplete anyways.
2020-11-15 18:42:55 +01:00
Michael Eischer
3500f9490c check: Simplify blob status tracking
UnusedBlobs now directly reads the list of existing blobs from the
repository index. This removes the need for the blobStatusExists flag,
which in turn allows converting the blobRefs map into a BlobSet.
2020-11-15 18:42:42 +01:00
Michael Eischer
b8c7543a55 check: Merge 'size could not be found' and 'not found in index' errors
By construction these two errors always show up in pairs: 'size could
not be found' is printed when the blob is not found in the repository
index. That blob is also part of the `blobs` array. Later on, check
iterates over that array and checks whether the blob is marked as
existing. Which cannot be the case as that mark is generated by
iterating over the repository index.

The merged warning no longer reports the blob index within a file. That
information could also be derived by printing the affected tree using
`cat` and searching for the blob.
2020-11-15 18:41:50 +01:00
MichaelEischer
45ba456291 Merge pull request #3038 from fgma/check-data-subset-percentage
Check data subset: check random percentage subset
2020-11-15 18:24:13 +01:00
Michael Eischer
caac38ed27 check: Tweak subset percentage over 100% error message 2020-11-15 18:13:50 +01:00
Michael Eischer
15c537f9db Rename changelog entry to contain issue id 2020-11-15 18:13:50 +01:00
fgma
8f9cea8cc0 Check data subset: check random percentage subset 2020-11-15 18:13:50 +01:00
Alexander Neumann
3c0c0c132b Merge pull request #3006 from aawsome/new-rebuild-index
Reimplement rebuild-index and remove /internal/index
2020-11-15 17:48:43 +01:00
Alexander Neumann
9968220652 Merge pull request #2850 from greatroar/packer-malloc
Decrease allocation rate in internal/pack
2020-11-15 17:19:49 +01:00
kitone
0649828555 Add changelog 2020-11-15 17:09:30 +01:00
kitone
3c03b35212 Fix #1681 should not try to create the mount point if it doesn't exist, rather return an error 2020-11-15 17:09:30 +01:00
greatroar
ab2b7d7f9a Decrease allocation rate in internal/pack
internal/repository benchmark results:

name             old time/op    new time/op    delta
PackerManager-8     179ms ± 1%     181ms ± 1%   +0.78%  (p=0.009 n=10+10)

name             old speed      new speed      delta
PackerManager-8   294MB/s ± 1%   292MB/s ± 1%   -0.77%  (p=0.009 n=10+10)

name             old alloc/op   new alloc/op   delta
PackerManager-8    91.3kB ± 0%    72.2kB ± 0%  -20.92%  (p=0.000 n=9+7)

name             old allocs/op  new allocs/op  delta
PackerManager-8     1.38k ± 0%     0.76k ± 0%  -45.20%  (p=0.000 n=10+7)
2020-11-15 16:51:47 +01:00
greatroar
9a8a2cae4c Move pack testing logic to test file 2020-11-15 16:45:49 +01:00
Alexander Weiss
9607cad267 Remove internal/index 2020-11-15 07:05:09 +01:00
Alexander Weiss
3d1d5295cc Add changelog 2020-11-15 07:05:09 +01:00
Alexander Weiss
30b6a0878a Reimplement rebuild-index 2020-11-15 07:05:09 +01:00
Alexander Weiss
187c8fb259 Parallelize MasterIndex.Save() 2020-11-15 07:05:09 +01:00
Alexander Weiss
1ec628ddf5 Add extraObsolete to MasterIndex.Save 2020-11-15 07:05:09 +01:00
Alexander Weiss
5898cb341f Use CreateIndexFromPacks() in test 2020-11-15 07:05:05 +01:00
Alexander Weiss
43732bb885 Add CreateIndexFromPacks() 2020-11-15 07:04:51 +01:00
MichaelEischer
6feaf6bd1f Merge pull request #3030 from greatroar/concrete-statt
Replace restic.statT interface by concrete types
2020-11-14 23:32:17 +01:00
greatroar
c45f8ee075 Replace restic.statT interface by concrete types
name                old time/op    new time/op    delta
NodeFillUser-8        1.81µs ± 9%    1.50µs ± 5%  -17.07%  (p=0.000 n=19+20)
NodeFromFileInfo-8    1.76µs ± 4%    1.49µs ± 6%  -15.63%  (p=0.000 n=20+19)

name                old alloc/op   new alloc/op   delta
NodeFillUser-8          496B ± 0%      352B ± 0%  -29.03%  (p=0.000 n=20+20)
NodeFromFileInfo-8      496B ± 0%      352B ± 0%  -29.03%  (p=0.000 n=20+20)

name                old allocs/op  new allocs/op  delta
NodeFillUser-8          3.00 ± 0%      2.00 ± 0%  -33.33%  (p=0.000 n=20+20)
NodeFromFileInfo-8      3.00 ± 0%      2.00 ± 0%  -33.33%  (p=0.000 n=20+20)
2020-11-14 23:23:26 +01:00
MichaelEischer
3601a9b6cd Merge pull request #2690 from SkYNewZ/master
Fix #2688: Handle comma-separated list tags when using backup command
2020-11-14 23:19:42 +01:00
Michael Eischer
fdec8051ab tags: Tweak description of --add/set/remove options 2020-11-14 22:55:30 +01:00
MichaelEischer
333c5a19d4 Merge pull request #3082 from aawsome/check-sizes
Check: check sizes of packs from index and packheader
2020-11-14 22:37:42 +01:00
Quentin Lemaire
a8ad6b9a4b fix(tags): Change tags list type according to restic.TagList 2020-11-14 16:24:58 +00:00
Quentin Lemaire
b0882b3f3c fix(snapshots): Update help message to better match the 'forget' command one 2020-11-14 15:48:57 +00:00
Quentin Lemaire
e74110a833 docs: Write new entry to changelog/unreleased 2020-11-14 15:48:56 +00:00
Quentin Lemaire
ae441d3134 fix(backup): Switch tags cobra type to handle comma-separated list 2020-11-14 15:48:56 +00:00
Alexander Neumann
913a34f568 Merge pull request #3094 from restic/fix-ci-tests
Fix ci tests
2020-11-14 11:24:28 +01:00
Alexander Neumann
468612b108 Disable fuse test for macOS 2020-11-14 11:02:49 +01:00
Alexander Weiss
7eabcabf68 Adjust changelog 2020-11-14 00:42:49 +01:00
Alexander Weiss
17bb77b1f9 check: Also check blob length and offset 2020-11-14 00:42:49 +01:00
Alexander Weiss
80dcfca191 check: Check sizes computed from index and pack header 2020-11-14 00:42:49 +01:00
tofran
94a154c7ca Remove --drive-use-trash=false from rclone param
Google drive trash retention policy changed making this
no longer a good default
a go
Issue #3095
2020-11-13 22:58:48 +00:00
Alexander Neumann
219d9a62f2 Download assets from repo 2020-11-13 21:56:14 +01:00
rawtaz
e8713bc209 Merge pull request #3093 from restic/fix-diff-metadata
diff: Correctly count top-level blobs
2020-11-13 21:50:29 +01:00
Alexander Neumann
04d1983800 Merge pull request #3090 from fgma/vss-fix-386
vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386
2020-11-13 21:30:34 +01:00
fgma
88208c3db2 vss: improved changelog entry 2020-11-13 21:13:17 +01:00
Alexander Neumann
59ea5a4208 diff: Correctly count top-level blobs 2020-11-13 21:11:21 +01:00
Alexander Neumann
145830005b Use old versions, otherwise tar doesn't work 2020-11-12 21:35:01 +01:00
Alexander Neumann
8ad9f88993 helpers: Improve error message 2020-11-12 20:38:31 +01:00
fgma
859d89b032 vss: add changelog file for issue 3090 2020-11-12 19:38:22 +01:00
fgma
f9223cd827 vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386 2020-11-12 19:31:00 +01:00
Alexander Neumann
0be906a92f CI: Use netcologne mirror 2020-11-11 20:59:23 +01:00
rawtaz
dfb9326b1b Merge pull request #3085 from LordGaav/s3-list-objects-v1-flag
Extended option to select V1 API for ListObjects on S3 backend
2020-11-11 20:52:44 +01:00
Alexander Neumann
e4e0ce09ad CI: Update links for Windows binaries 2020-11-11 20:51:01 +01:00
Alexander Neumann
0334114865 CI: Hardcode mirror URL 2020-11-11 20:42:47 +01:00
Alexander Neumann
b1bbdcb637 Improve changelog 2020-11-11 20:30:30 +01:00
Alexander Neumann
4a0b7328ec s3: Remove dots for config description 2020-11-11 20:20:35 +01:00
Alexander Neumann
3e0456d88b Highlight that s3.list-objects-v1 is a temporary 2020-11-11 20:11:35 +01:00
Alexander Neumann
dd94174379 Merge pull request #3086 from greatroar/diff-errors
Improve error reporting from restic diff
2020-11-11 19:48:03 +01:00
greatroar
63e32c44c0 Improve error reporting from restic diff
Instead of a stacktrace, restic diff 111 222 now reports:

	Fatal: no matching ID found for prefix "111"
2020-11-11 16:40:40 +01:00
Nick Douma
f013662e3f Remove separate section on Ceph, and move s3.list-objects-v1 note to S3 section 2020-11-11 15:11:14 +01:00
Nick Douma
4320ff2bbf Add changelog entry for s3.list-objects-v1 2020-11-11 12:33:22 +01:00
Nick Douma
354b7e89cc Document the extended s3.list-objects-v1 flag in a new Ceph section 2020-11-11 12:32:46 +01:00
Nick Douma
829959390a Provide UseV1 parameter to minio.ListObjectsOptions based on s3.list-objects-v1 2020-11-11 11:54:38 +01:00
Nick Douma
ccd55d529d Add s3.list-objects-v1 extended option and default to false 2020-11-11 11:54:36 +01:00
Nick Douma
4ddcc17135 Add support for boolean extended options 2020-11-11 11:54:27 +01:00
MichaelEischer
407843c5f9 Merge pull request #3034 from hoyho/gh_master
bugfix: omit ENOTDATA for extended attributes
2020-11-09 22:38:49 +01:00
MichaelEischer
46d31ab86d Merge pull request #3058 from greatroar/counter
Replace restic.Progress with new progress.Counter (fixes two race conditions)
2020-11-09 22:19:09 +01:00
Alexander Neumann
c986823d3f Merge pull request #3048 from aawsome/check-pack-index
check: check index for packs that are read
2020-11-09 20:12:32 +01:00
Alexander Weiss
239931578c check: check index for packs that are read 2020-11-09 17:28:14 +01:00
hoyho
9df52327cc bugfix: omit ENOTDATA for extended attributes
Signed-off-by: hoyho <luohaihao@gmail.com>
2020-11-10 00:20:34 +08:00
greatroar
21b787a4d1 Stop Counters where they're constructed and started 2020-11-09 13:03:31 +01:00
greatroar
ddca699cd2 Replace restic.Progress with new progress.Counter
This fixes two race conditions while cleaning up the code.
2020-11-09 12:12:35 +01:00
rawtaz
605db3b389 Merge pull request #3077 from MichaelEischer/tame-golangci-lint
Switch to the default set of golangci-lint linters
2020-11-08 22:34:27 +01:00
Michael Eischer
8de129e12f Switch to the default set of golangci-lint linters 2020-11-08 22:17:50 +01:00
Alexander Neumann
2072f0a481 Merge pull request #3076 from restic/check-go-mod-tidy
Check that go.mod/go.sum are up to date
2020-11-08 21:37:22 +01:00
Alexander Neumann
5731e391f8 Merge pull request #3075 from restic/warn-prune-no-cache
UI: Add several hints
2020-11-08 20:37:10 +01:00
Alexander Neumann
6a0a1d1f1c Check that go.mod/go.sum are up to date 2020-11-08 20:32:25 +01:00
Alexander Neumann
a8d21b5dcf prune: Warn if no cache is present 2020-11-08 20:25:35 +01:00
Alexander Neumann
823d0afd6e backup: Always print parent snapshot info 2020-11-08 20:25:35 +01:00
Alexander Neumann
a5989707ac Move test for golang-ci 2020-11-08 19:58:16 +01:00
Alexander Neumann
3a0cfafeb5 Fix quotes 2020-11-08 19:52:37 +01:00
Alexander Neumann
c923bd957d Fix test for pull request 2020-11-08 19:51:11 +01:00
Alexander Neumann
1a3f885d3d Only run golangci-lint for PRs 2020-11-08 17:34:07 +01:00
Alexander Neumann
3bf43d7951 Merge pull request #2475 from restic/use-github-actions
Use GitHub Actions for CI
2020-11-08 17:30:12 +01:00
Alexander Neumann
561da92396 Replace badges in README 2020-11-08 17:26:59 +01:00
Alexander Neumann
5cf42884c8 Run tests on GitHub Actions 2020-11-08 16:04:57 +01:00
MichaelEischer
9e4e0077fb Merge pull request #3069 from greatroar/xattr-eintr
Update pkg/xattr to handle EINTR on Linux
2020-11-08 12:52:23 +01:00
MichaelEischer
1758da855f Merge pull request #3068 from aawsome/context-mem-backend
Return context error in mem backend
2020-11-08 12:24:12 +01:00
greatroar
15ea90feed Update pkg/xattr to handle EINTR on Linux
Updates #2659. This is a case where the stdlib will not handle EINTR for
us, even with Go 1.16. That xattr calls are directly affected can be
seen in the report for issue #2968.
2020-11-08 09:34:24 +01:00
Alexander Weiss
826cfa0533 fix context in archiver tests 2020-11-08 08:24:24 +01:00
Alexander Weiss
fef408a8bd Return context error in mem backend 2020-11-08 00:05:53 +01:00
greatroar
a2d4209322 Don't recurse in local backend's List if not required
Due to the return if !isFile, the IsDir branch in List was never taken
and subdirectories were traversed recursively.

Also replaced isFile by an IsRegular check, which has been equivalent
since Go 1.12 (golang/go@a2a3dd00c9).
2020-11-07 08:54:13 +01:00
Alexander Neumann
275f713211 Remove Travis and AppVeyor 2020-11-06 21:49:52 +01:00
MichaelEischer
4707bdb204 Merge pull request #2842 from aawsome/rebuild-index-inmem
Rebuild index in prune by using in-memory index
2020-11-06 20:51:20 +01:00
Alexander Neumann
47277c4b4c Add comments, clarify computation 2020-11-06 20:23:30 +01:00
Alexander Weiss
d2e53730d6 Add test that repo.List is only called once 2020-11-06 20:23:30 +01:00
Alexander Weiss
fd33030556 Use in-memory index to rebuild index in prune 2020-11-06 20:23:30 +01:00
Alexander Weiss
38cc4393f6 Add Masterindex.Save(); Add Index.Packs() 2020-11-06 20:23:30 +01:00
Alexander Neumann
7f6f31c34b Remove Hound 2020-11-06 11:43:58 +01:00
Alexander Neumann
164b4cb2f6 Add changelog for #3014 2020-11-06 10:05:42 +01:00
Alexander Neumann
4a9b05aff1 Merge pull request #3063 from aawsome/fix-3062
Fix #3062
2020-11-05 19:36:40 +01:00
Alexander Weiss
aaf1c44362 Fix #3062 2020-11-05 17:05:42 +01:00
Alexander Neumann
a5592e83f7 Merge pull request #3014 from ivandeex/pr-stream-reset
Fix sporadic stream reset between rclone and restic
2020-11-05 16:11:50 +01:00
Ivan Andreev
ab2790d9de Fix http2 stream reset between restic and rest backends #3014 2020-11-05 15:57:40 +03:00
Alexander Neumann
8a2a326189 Merge pull request #2535 from ncw/fix-2528
s3: add bucket-lookup parameter to select path or dns style bucket lookup
2020-11-05 12:39:34 +01:00
Nick Craig-Wood
86b5d8ffaa s3: add bucket-lookup parameter to select path or dns style bucket lookup
This is to enable restic working with Alibaba cloud

Fixes #2528
2020-11-05 12:20:10 +01:00
Alexander Neumann
636b2f2e94 Merge pull request #2941 from MichaelEischer/parallel-repack
prune: Parallelize repack step
2020-11-05 11:00:41 +01:00
Alexander Neumann
ae5302c7a8 Add comment that keepBlobs is modified 2020-11-05 10:33:38 +01:00
Alexander Neumann
866a52ad4e Remove unneeded seek
The file returned from DownloadAndHash() is already seeked to the start
of the file.
2020-11-05 10:31:49 +01:00
Alexander Neumann
a4507610a0 Fix typo 2020-11-05 10:31:49 +01:00
Alexander Neumann
7def2d8ea7 Use a non-constant seed 2020-11-05 10:31:49 +01:00
Alexander Neumann
ee0112ab3b Clarify message about expected error 2020-11-05 10:31:49 +01:00
Michael Eischer
b373f164fe prune: Parallelize repack command 2020-11-05 10:31:49 +01:00
Alexander Neumann
8a0dbe7c1a helpers: Remove old changelog files for release 2020-11-05 10:26:00 +01:00
Alexander Neumann
4e3ad8b3b1 Remove changelog files from 0.11.0 release 2020-11-05 10:22:39 +01:00
Alexander Neumann
5144141321 Merge pull request #2718 from aawsome/new-cleanup-command
Reimplementation of prune
2020-11-05 10:12:19 +01:00
Alexander Neumann
d35d279455 Set development version for 0.11.0 2020-11-05 09:41:40 +01:00
Alexander Neumann
b7e1ece1e0 Add version for 0.11.0 2020-11-05 09:41:35 +01:00
Alexander Neumann
c5300a2c56 Update manpages and auto-completion 2020-11-05 09:41:35 +01:00
Alexander Neumann
1bd92896d7 Generate CHANGELOG.md for 0.11.0 2020-11-05 09:41:22 +01:00
Alexander Neumann
9dba01021e Prepare changelog for 0.11.0 2020-11-05 09:41:22 +01:00
Alexander Neumann
23f6b8c3fd Add .vscode to gitignore 2020-11-05 09:40:56 +01:00
Alexander Neumann
79a50e3b1f helpers: Create changelog subdir for release 2020-11-05 09:39:06 +01:00
Alexander Neumann
b8a5ca2d10 Add gitignore to changelog/unreleased 2020-11-05 09:01:32 +01:00
fgma
916b2d303b vss: fix potential crash (not reachable in restic) (#3045)
HasSufficientPrivilegesForVSS() now returns an error
2020-11-04 22:14:18 +01:00
Alexander Neumann
a06f5c28c0 Merge pull request #3055 from greatroar/cleanup
Don't separately allocate sync.{Mutex,Once} if not necessary
2020-11-04 11:11:21 +01:00
greatroar
c2f3eee5af Don't separately allocate sync.{Mutex,Once} if not necessary
Separate allocation of synchronization devices suggests they're shared
between objects, but they're not.
2020-11-04 09:55:43 +01:00
rawtaz
62345abe4a Merge pull request #3053 from maikell/patch-1
Update README.md
2020-11-03 19:06:59 +01:00
Maikel van Leeuwen
e024fc6d4d Update README.md
Typo in link. 
Links to  https://forum.restic.net%3E/ should be https://forum.restic.net
2020-11-03 18:51:11 +01:00
Alexander Neumann
1ca60bccfb Refactor condition for MaxRepackBytes
Don't depend on the string (opts.MaxRepackSize) for the condition,
instead check if there's a (positive) limit configured.
2020-11-03 16:42:21 +01:00
Alexander Neumann
7f86eb4ec0 Move helper function 2020-11-03 16:42:21 +01:00
Alexander Neumann
c1a3de4a6e Refactor max-unused calculation, add unlimited option
Add a callback to the PruneOptions struct which calculates the number of
bytes allowed to be unused after prune is done. This way, the logic is
closer to the option parsing code.

Also, add an explicit option `unlimited` for the use case when storage
does not matter but bandwidth and time do. Internally, this sets the
maximum number of unused bytes to MaxUint64.

Rework the documentation slightly so that no more "packs" are
mentioned and it talks about "files" instead.

Make it clear in the documentation that the percentage given to
`--max-unused` is relative to the whole repository size after pruning is
done. If specified, it must be below 100%, otherwise the repository
would contain 100% of unused data, which is pointless.

I had a hard time coming up with the correct formula to calculate the
maximum number of unused bytes based on the number of used bytes. For a
fraction `p` (0 ≤ p < 1), a repo with `u` bytes used, and the number of
unused bytes `x` the following holds:

      x ≤ p * (u+x)
    ⇔ x ≤ p*u + p*x
    ⇔ x - p*x ≤ p*u
    ⇔ x * (1-p) ≤ p*u
    ⇔ x ≤ p/(1-p) * u
2020-11-03 16:42:21 +01:00
Alexander Neumann
f8c4dd7b1a Split packe rewrite logic into two case branches
The comma is too sublte, let's split this into two separate branches.
2020-11-03 16:42:21 +01:00
Alexander Neumann
a5b80452fe Add comment that usedBlobs is modified 2020-11-03 16:42:21 +01:00
Alexander Neumann
aff1e220f5 Split struct members, add comments 2020-11-03 16:42:21 +01:00
Alexander Neumann
095155d9ce Remove RepackSmall 2020-11-03 16:42:21 +01:00
Alexander Neumann
1dd9fdce74 Reword changelog slightly 2020-11-03 16:42:21 +01:00
Alexander Weiss
b2f5381737 Make realistic forget --prune --dryrun 2020-11-03 16:42:21 +01:00
Alexander Weiss
7f9a0a5907 Reimplementation of prune 2020-11-03 16:42:21 +01:00
Alexander Weiss
3b591ed987 Add Verboseff 2020-11-03 16:42:21 +01:00
Alexander Weiss
ce7d613749 Add Blob.Handle() 2020-11-03 16:42:21 +01:00
Alexander Weiss
581d90cf91 Make some pack parameters public 2020-11-03 16:42:21 +01:00
Alexander Neumann
0db9024aad Merge pull request #3052 from restic/error-on-invalid-size
Return an error for invalid sizes
2020-11-03 12:13:49 +01:00
Alexander Neumann
21ba15577e Return an error for invalid sizes 2020-11-03 11:53:49 +01:00
Alexander Neumann
2fb1957ca4 Rewrite README as Markdown 2020-11-03 09:20:09 +01:00
Alexander Neumann
9a88fb253b Merge pull request #3051 from greatroar/sanitize-env
Sanitize environment before starting backend processes (rclone, ssh)
2020-11-02 21:18:57 +01:00
Alexander Neumann
f14436953a Merge pull request #3050 from greatroar/widechars
Fix string truncation in ui/termstatus
2020-11-02 16:58:51 +01:00
greatroar
11fbaaae9a Sanitize environment before starting backend processes (rclone, ssh)
The restic security model includes full trust of the local machine, so
this should not fix any actual security problems, but it's better to be
safe than sorry.

Fixes #2192.
2020-11-02 16:41:23 +01:00
greatroar
3ed84ff0c6 Fix string truncation in ui/termstatus
Fixes #3046.
2020-11-02 12:50:49 +01:00
Alexander Neumann
8e965ed4eb Merge pull request #3039 from MichaelEischer/tar-proper-dirs
Properly dump directories to tar
2020-11-02 12:30:26 +01:00
Alexander Neumann
5f0fa2129e Improve readability
It's time to use a switch statement.
2020-11-02 11:24:26 +01:00
Alexander Neumann
04dfa19c7e Improve changelog 2020-11-02 11:23:09 +01:00
Alexander Neumann
6509c207f4 Merge pull request #2997 from MichaelEischer/faster-excludes
Speedup exclude/include checking
2020-11-02 11:07:21 +01:00
Alexander Neumann
445b845267 Merge pull request #2978 from MichaelEischer/warn-tree-error
Warn if backup failed to read tree blob
2020-11-02 10:14:12 +01:00
Alexander Neumann
3ff37215df Merge pull request #2935 from MichaelEischer/upgrade-minio
Upgrade minio SDK to version 7
2020-11-02 09:09:10 +01:00
MichaelEischer
5d379b5359 Merge pull request #3042 from rubiojr/remove-unused-function
Remove unused function
2020-11-01 21:55:22 +01:00
Sergio Rubio
e708628cfd Remove unused function
Not currently used, and it'd need to be added to the MasterIndex interface first.
2020-10-28 13:24:49 +01:00
MichaelEischer
bb4b3481a6 Merge pull request #3031 from greatroar/debug-no-seek
Allow debug logging to pipes and terminals
2020-10-25 17:59:06 +01:00
MichaelEischer
ad3a52e6f0 Merge pull request #3026 from greatroar/refactor-ui
internal/ui refactoring
2020-10-25 17:51:39 +01:00
rawtaz
e8b4d8d8bc Merge pull request #2998 from greatroar/env-docs
Systematize documentation of environment variables
2020-10-24 22:49:31 +02:00
Michael Eischer
1aa61e6def Add changelog 2020-10-24 22:49:29 +02:00
Michael Eischer
8d7d6ad2d5 dump: include username in tar 2020-10-24 22:42:01 +02:00
Michael Eischer
fe09e6f865 dump: test proper permissions and directory name 2020-10-24 22:42:01 +02:00
Michael Eischer
1e3c9a2c11 dump: Fix file permission to tar mapping
The file permissions included a go specific directory bit which
accidentially forced the usage of the GNU header format. This leads
to problems with 7zip on Windows or when extended attributes are
used.
2020-10-24 22:42:01 +02:00
Michael Eischer
e21dcb0eea dump: Additional ACL tests 2020-10-24 22:41:01 +02:00
MichaelEischer
31b8d7a639 Merge pull request #2274 from fgma/master
Support for Volume Shadow Copy Service (VSS) on windows
2020-10-24 15:43:34 +02:00
fgma
5695f9ebd2 vss: Implement VSS support for Windows
The VSS support works for 32 and 64-bit windows, this includes a check that
the restic version matches the OS architecture as required by VSS. The backup
operation will fail the user has not sufficient permissions to use VSS.

Snapshotting volumes also covers mountpoints but skips UNC paths.
2020-10-24 11:35:57 +02:00
greatroar
8091151638 doc: Update restic mount availability and requirements
NetBSD doesn't support restic mount either, so it's easier to list the
positive cases. Also noted that FUSE for macOS is required on the Mac.
2020-10-22 09:57:30 +02:00
greatroar
ae179ee63e Systematize documentation of environment variables
Cache locations were documented inconsistently in three places.

The backup docs mentioned PATH being used to find fusermount, which is
never run by restic backup. It now mentions ssh and rclone, which are
used by backends.

The notion of a "system-wide" environment variable makes no sense.
TMPDIR is now mentioned because it allows for optimization and may
have security implications.
2020-10-22 09:57:30 +02:00
MichaelEischer
0590e3e12d Merge pull request #3035 from greatroar/no-unmount-on-failure
restic mount: don't umount unless we actually tried to mount
2020-10-21 22:10:42 +02:00
greatroar
3807d13bdc restic mount: don't umount unless we actually tried to mount
Previously, failure to open a repository would result in an "unable to
umount" message.
2020-10-21 12:15:05 +02:00
greatroar
63be3704d9 Allow debug logging to pipes and terminals 2020-10-19 22:30:30 +02:00
greatroar
35419de232 Simplify ui.StdioWrapper.Write
Instead of looping to find line breaks, make it look for the last one.
2020-10-17 20:30:46 +02:00
greatroar
863a590a81 Refactor termstatus.Term.{Print,Error} methods 2020-10-17 20:30:46 +02:00
greatroar
7c0b6a82db Remove unused public method ui.linesWriter.Flush 2020-10-17 20:30:24 +02:00
MichaelEischer
96a912b65a Merge pull request #3025 from aawsome/fix-index-ids
Fix setting of ID in DecodeIndex
2020-10-17 09:54:45 +02:00
Alexander Weiss
b44ecde8b0 Fix setting of ID in DecodeIndex 2020-10-17 09:12:58 +02:00
MichaelEischer
39fe1e96fe Merge pull request #3018 from greatroar/background-check
Fix IsProcessBackground on Linux with stdin redirection
2020-10-16 22:50:19 +02:00
MichaelEischer
4ba237bb93 Merge pull request #3019 from greatroar/refactor-decodeindex
Refactor index decoding
2020-10-15 23:22:33 +02:00
MichaelEischer
ce87fbd7dc Merge pull request #3022 from greatroar/refactor-runworkers
Defer channel closing outside repository.RunWorkers
2020-10-14 22:44:42 +02:00
greatroar
b27375f5ce defer close(ch) outside repository.RunWorkers 2020-10-14 15:50:16 +02:00
greatroar
720e0ee0c7 if cond { return true }; return false => return cond 2020-10-13 20:56:43 +02:00
greatroar
27db3ec262 Refactor index decoding
Decoding old-format indices no longer requires loading and decrypting
twice.
2020-10-13 20:47:50 +02:00
greatroar
f80b07b2c8 Fix IsProcessBackground on Linux with stdin redirection
The previous implementation assumed that stdin was a terminal.
It now checks the terminal's fd.
2020-10-13 13:12:02 +02:00
MichaelEischer
6003dada14 Merge pull request #3011 from restic/doc-alpine
doc: Add restic install instructions for Alpine Linux
2020-10-12 23:29:40 +02:00
MichaelEischer
41a45ae908 Merge pull request #3016 from greatroar/uid-gid-lookup
Simplify/optimize cached UID/GID lookups
2020-10-12 23:24:55 +02:00
MichaelEischer
8299c5559c Merge pull request #3015 from greatroar/changelog-template
Improved changelog template
2020-10-12 23:12:29 +02:00
Alexander Neumann
56883817d8 Merge pull request #2990 from MichaelEischer/fix-goreport-warnings
Fix some goreport warnings
2020-10-12 20:44:56 +02:00
greatroar
f0cd16e5ea Cache uint32-typed ids in lookup{Username,Group}
NodeFillUser-8        1.92µs ± 5%    1.75µs ± 3%   -8.89%  (p=0.000 n=10+10)
NodeFromFileInfo-8    1.89µs ± 7%    1.76µs ± 4%   -6.69%  (p=0.001 n=10+10)

name                old alloc/op   new alloc/op   delta
NodeFillUser-8          504B ± 0%      496B ± 0%   -1.59%  (p=0.000 n=10+10)
NodeFromFileInfo-8      504B ± 0%      496B ± 0%   -1.59%  (p=0.000 n=10+10)

name                old allocs/op  new allocs/op  delta
NodeFillUser-8          5.00 ± 0%      3.00 ± 0%  -40.00%  (p=0.000 n=10+10)
NodeFromFileInfo-8      5.00 ± 0%      3.00 ± 0%  -40.00%  (p=0.000 n=10+10)
2020-10-12 15:08:15 +02:00
greatroar
a03fe4562e Remove unused error return in lookup{Username,Group} 2020-10-12 14:48:23 +02:00
greatroar
efc075df87 Improved changelog template 2020-10-12 12:15:15 +02:00
Leo R. Lundgren
f500b0d90e doc: Add restic install instructions for Alpine Linux 2020-10-11 18:31:07 +02:00
Alexander Neumann
c193cea119 Pass Context to NewRestorer() 2020-10-10 15:24:26 +02:00
Alexander Neumann
f0d49ca600 Merge pull request #2933 from MichaelEischer/less-context-todo
Replace most usages of context.TODO()
2020-10-10 15:03:44 +02:00
MichaelEischer
164d8af3dd Merge pull request #2906 from kitone/fix-inconsistent-timestamps-permissions
Restore inconsistent timestamps permissions
2020-10-10 14:56:43 +02:00
kitone
052564007a Add changelog 2020-10-10 14:37:51 +02:00
kitone
6099f81692 Fix #1212 restore code produces inconsistent timestamps/permissions.
Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.
2020-10-10 13:46:44 +02:00
kitone
9d7f616190 Improve restorer debug log information 2020-10-10 13:46:43 +02:00
kitone
295ddb9e57 Add test case for inconsistent timestamps and permissions restoration
Reproduce from https://github.com/restic/restic/issues/1212
2020-10-10 13:46:43 +02:00
Michael Eischer
e638b46a13 Embed context into ReaderAt
The io.Reader interface does not support contexts, such that it is
necessary to embed the context into the backendReaderAt struct. This has
the problem that a reader might suddenly stop working when it's
contained context is canceled. However, this is now problem here as the
reader instances never escape the calling function.
2020-10-09 22:39:07 +02:00
Michael Eischer
d6cfe857b7 pass proper context into MasterIndex.RebuildIndex 2020-10-09 22:39:07 +02:00
Michael Eischer
2964d2ad15 Skip unlocking for nil locks
Now that lockRepo receives a context, it is possible that it is canceled
before a lock was created. Thus `unlockRepo` must be able to handle this
case.
2020-10-09 22:39:06 +02:00
Michael Eischer
0c9efa9c2a Pass context to lockRepo 2020-10-09 22:39:06 +02:00
Michael Eischer
37a5e2d681 rest: use global context on repository creation 2020-10-09 22:39:06 +02:00
Michael Eischer
4b0fcaed45 unlock: use proper context for locks cleanup
The list operation used by RemoveStaleLocks or RemoveAllLocks will
already be canceled by the passed in context. Therefore we can also just
cancel the remove operation as the unlock command won't process all lock
files anyways.
2020-10-09 22:37:56 +02:00
Michael Eischer
645a6efaf2 restorer: remove redundant type specification 2020-10-09 22:37:56 +02:00
Michael Eischer
dc2e664209 integration_fuse_test: use global context
No need to use the TODO context.
2020-10-09 22:37:56 +02:00
Michael Eischer
a449450021 init: pass proper context to master key generation
This is no change in behavior as a canceled context did later on cause
the config file creation to fail. Therefore this change just lets the
repository initialization fail a bit earlier.
2020-10-09 22:37:56 +02:00
Michael Eischer
603bb0e309 restore: Use proper context while loading snapshot 2020-10-09 22:37:56 +02:00
Michael Eischer
27456f6545 debug: use proper context
This allows the debug commands to be properly interrupted.
2020-10-09 22:37:56 +02:00
Michael Eischer
c458e114d4 pass context to Find / FindSnapshot
This allows proper interruption of restic while it searches for
snapshots or key files.
2020-10-09 22:37:56 +02:00
Michael Eischer
45e9a55c62 Wire context into backend layout detection 2020-10-09 22:37:24 +02:00
Michael Eischer
307a6ba3a3 Upgrade minio sdk to v7
This changes are primarily straightforward modifications to pass the
parameters in the now expected way.
2020-10-09 22:37:24 +02:00
Michael Eischer
50da20d93d Warn if backup failed to read tree blob 2020-10-09 22:36:27 +02:00
Alexander Neumann
5fd3dbccb7 Merge pull request #2963 from MichaelEischer/fix-status-deadlock
backup: Fix possible deadlock of scanner goroutine
2020-10-09 21:35:50 +02:00
Alexander Neumann
35655a481b Merge pull request #2931 from MichaelEischer/reduce-debug-overhead
Reduce debug log overhead
2020-10-09 21:35:11 +02:00
Alexander Neumann
30cb553c8d Merge pull request #2932 from MichaelEischer/proper-rclone-create
Call rclone.Create to create a new repository for the rclone backend
2020-10-09 21:29:15 +02:00
Alexander Neumann
2f3eeff2e7 Merge pull request #2964 from MichaelEischer/fix-repro-instructions
Fix instructions to reproduce the release binaries
2020-10-09 21:28:24 +02:00
MichaelEischer
356ac404cd Merge pull request #2996 from greatroar/setconsolecursorposition
Use windows.SetConsoleCursorPosition in ui/termstatus
2020-10-09 18:19:02 +02:00
rawtaz
ab769abeaf Merge pull request #3002 from MichaelEischer/update-contributing
Update links to help wanted labels in CONTRIBUTING.md and link to forum
2020-10-09 17:24:25 +02:00
Michael Eischer
4036991c91 Update links to help wanted labels in CONTRIBUTING.md and link to forum 2020-10-09 16:15:43 +02:00
Michael Eischer
88c8e903d2 filter: Fix glob matching on absolute path marker on windows
A pattern part containing "/" is used to mark a path or a pattern as
absolute. However, on Windows the path separator is "\" such that glob
patterns like "?" could match the marker. The code now explicitly skips
the marker when the pattern does not represent an absolute path.
2020-10-09 16:11:05 +02:00
greatroar
740758a5fa Optimize filter pattern matching
By replacing "**" with "", checking for this special path component can
be reduced to a length-zero check.

name                          old time/op    new time/op    delta
FilterLines-8                   44.7ms ± 5%    44.9ms ± 5%     ~     (p=0.631 n=10+10)
FilterPatterns/Relative-8       13.6ms ± 4%    13.4ms ± 5%     ~     (p=0.165 n=10+10)
FilterPatterns/Absolute-8       10.9ms ± 5%    10.7ms ± 4%     ~     (p=0.052 n=10+10)
FilterPatterns/Wildcard-8       53.7ms ± 5%    50.4ms ± 5%   -6.00%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-8     128ms ± 2%      95ms ± 1%  -25.54%  (p=0.000 n=10+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-8       3.57MB ± 0%    3.57MB ± 0%     ~     (p=1.000 n=9+8)
FilterPatterns/Absolute-8       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.903 n=9+8)
FilterPatterns/Wildcard-8       19.7MB ± 0%    19.7MB ± 0%   -0.00%  (p=0.022 n=10+9)
FilterPatterns/ManyNoMatch-8    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-8        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-8        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-8        88.7k ± 0%     88.7k ± 0%     ~     (all equal)
FilterPatterns/ManyNoMatch-8     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
2020-10-09 15:51:14 +02:00
MichaelEischer
862ee4b2c9 Merge pull request #2970 from labkode/fopenskip
Skip fopen for file change check to avoid network penalty
2020-10-09 11:43:54 +02:00
Hugo Gonzalez Labrador
958dc6aafc Skip fopen for file change check to avoid network penalty 2020-10-09 10:53:28 +02:00
MichaelEischer
88cc444779 Merge pull request #2934 from MichaelEischer/upgrade-backoff
Upgrade github.com/cenkalti/backoff module
2020-10-08 20:33:30 +02:00
MichaelEischer
4a424af1d5 Merge pull request #2989 from MichaelEischer/remove-local-chmod
local: mark repository files as read-only and handle chmod errors
2020-10-08 19:04:13 +02:00
Michael Eischer
bc65da2baf Add changelog 2020-10-08 14:45:26 +02:00
Michael Eischer
b79f18209f Upgrade github.com/cenkalti/backoff module
We now use v4 of the module. `backoff.WithMaxRetries` no longer repeats
an operation endlessly when a retry count of 0 is specified. This
required a few fixes for the tests.
2020-10-07 22:04:59 +02:00
Michael Eischer
8388e67c4b filter: cleanup path separator conversion 2020-10-07 21:14:07 +02:00
Michael Eischer
0acc3c5923 filter: special case patterns without globbing characters
In case a part of a path is a simple string, we can just check for
equality without complex parsing in filepath.Match.

name                          old time/op    new time/op    delta
FilterLines-4                   34.8ms ±17%    41.2ms ±23%  +18.36%  (p=0.000 n=10+10)
FilterPatterns/Relative-4       21.7ms ± 6%    12.1ms ±23%  -44.46%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4       10.0ms ± 5%     9.1ms ±11%   -9.80%  (p=0.006 n=10+9)
FilterPatterns/Wildcard-4       47.0ms ± 7%    42.2ms ± 5%  -10.19%  (p=0.000 n=9+10)
FilterPatterns/ManyNoMatch-4     190ms ± 1%     131ms ±20%  -31.47%  (p=0.000 n=8+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.870 n=9+9)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.145 n=10+10)
FilterPatterns/Wildcard-4       14.3MB ± 0%    19.7MB ± 0%  +37.91%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.421 n=10+9)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        88.7k ± 0%     88.7k ± 0%     ~     (all equal)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
2020-10-07 20:55:43 +02:00
Michael Eischer
54a124de3b filter: explicitly test separate ListWithChild function 2020-10-07 20:47:52 +02:00
Michael Eischer
bcc3bddcf4 filter: only check whether a child path could match when necessary
When checking excludes there is no need to test whether a child path
could also match the pattern, as it is by definition excluded.
Previously childMayMatch was calculated but then discarded. For simple
absolute paths this can account for half the time spent for checking
pattern matches.

name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       23.3ms ± 9%    21.7ms ± 6%   -6.68%  (p=0.004 n=10+10)
FilterPatterns/Absolute-4       13.9ms ± 7%    10.0ms ± 5%  -27.61%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4       51.4ms ± 7%    47.0ms ± 7%   -8.51%  (p=0.001 n=9+9)
FilterPatterns/ManyNoMatch-4     551ms ± 9%     190ms ± 1%  -65.41%  (p=0.000 n=10+8)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.665 n=10+9)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.480 n=9+10)
FilterPatterns/Wildcard-4       14.3MB ± 0%    14.3MB ± 0%     ~     (p=0.431 n=9+10)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        88.7k ± 0%     88.7k ± 0%     ~     (all equal)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
2020-10-07 20:47:52 +02:00
Michael Eischer
17c53efb0d filter: Optimize double wildcard expansion
This only allocates a single slice to expand the double wildcard and
only copies the pattern prefix once.

name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       22.7ms ± 5%    23.3ms ± 9%     ~     (p=0.353 n=10+10)
FilterPatterns/Absolute-4       14.2ms ±13%    13.9ms ± 7%     ~     (p=0.853 n=10+10)
FilterPatterns/Wildcard-4        266ms ±16%      51ms ± 7%  -80.67%  (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4     554ms ± 6%     551ms ± 9%     ~     (p=0.436 n=10+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.349 n=10+10)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.073 n=10+9)
FilterPatterns/Wildcard-4        141MB ± 0%      14MB ± 0%  -89.89%  (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        1.63M ± 0%     0.09M ± 0%  -94.56%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
2020-10-07 20:47:48 +02:00
Michael Eischer
7959796269 filter: Special case for absolute paths
name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       23.6ms ±20%    22.7ms ± 5%     ~     (p=0.684 n=10+10)
FilterPatterns/Absolute-4       32.3ms ± 8%    14.2ms ±13%  -56.01%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        334ms ±17%     266ms ±16%  -20.56%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     709ms ± 7%     554ms ± 6%  -21.89%  (p=0.000 n=10+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%   +0.00%  (p=0.046 n=9+10)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.464 n=10+10)
FilterPatterns/Wildcard-4        141MB ± 0%     141MB ± 0%     ~     (p=0.163 n=9+10)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        1.63M ± 0%     1.63M ± 0%     ~     (p=0.072 n=10+10)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
2020-10-07 20:47:29 +02:00
Michael Eischer
375c2a56de filter: Parse filter patterns only once
name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       30.3ms ±10%    23.6ms ±20%  -22.12%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4       49.0ms ± 3%    32.3ms ± 8%  -33.94%  (p=0.000 n=8+10)
FilterPatterns/Wildcard-4        345ms ± 9%     334ms ±17%     ~     (p=0.315 n=10+10)
FilterPatterns/ManyNoMatch-4     3.93s ± 2%     0.71s ± 7%  -81.98%  (p=0.000 n=9+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       4.63MB ± 0%    3.57MB ± 0%  -22.98%  (p=0.000 n=9+9)
FilterPatterns/Absolute-4       8.54MB ± 0%    3.57MB ± 0%  -58.20%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        146MB ± 0%     141MB ± 0%   -2.93%  (p=0.000 n=9+9)
FilterPatterns/ManyNoMatch-4     907MB ± 0%       4MB ± 0%  -99.61%  (p=0.000 n=9+9)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        66.6k ± 0%     22.2k ± 0%  -66.67%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4        88.7k ± 0%     22.2k ± 0%  -75.00%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        1.70M ± 0%     1.63M ± 0%   -3.92%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     4.46M ± 0%     0.02M ± 0%  -99.50%  (p=0.000 n=10+10)
2020-10-07 20:47:27 +02:00
Michael Eischer
b8eacd1364 filter: Reduce redundant path and pattern splitting
A single call to filter.List will split the path only once and also
split each search pattern only once and use it for both match and
childMatch.

name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       62.1ms ±15%    30.3ms ±10%  -51.22%  (p=0.000 n=9+10)
FilterPatterns/Absolute-4        111ms ±10%      49ms ± 3%  -56.08%  (p=0.000 n=10+8)
FilterPatterns/Wildcard-4        393ms ±15%     345ms ± 9%  -12.30%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     10.0s ± 3%      3.9s ± 2%  -60.53%  (p=0.000 n=10+9)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       16.4MB ± 0%     4.6MB ± 0%  -71.76%  (p=0.000 n=10+9)
FilterPatterns/Absolute-4       31.4MB ± 0%     8.5MB ± 0%  -72.77%  (p=0.000 n=9+10)
FilterPatterns/Wildcard-4        168MB ± 0%     146MB ± 0%  -13.19%  (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4    3.23GB ± 0%    0.91GB ± 0%  -71.96%  (p=0.000 n=10+9)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4         178k ± 0%       67k ± 0%  -62.50%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4         266k ± 0%       89k ± 0%  -66.67%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        1.87M ± 0%     1.70M ± 0%   -9.47%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     17.7M ± 0%      4.5M ± 0%  -74.87%  (p=0.000 n=9+10)
2020-10-07 18:13:19 +02:00
Michael Eischer
e73c281142 filter: Benchmark absolute paths, wildcards and long filter lists 2020-10-07 17:54:36 +02:00
Michael Eischer
fdd3b14db3 filter: test some corner cases 2020-10-07 17:09:44 +02:00
Michael Eischer
f4282aa6fd local: mark repository files as read-only
This is intended to prevent accidental modifications of data files.
Marking the files as read-only was accidentally removed in #1258.
2020-10-07 12:29:37 +02:00
Michael Eischer
40ee17167e local: Ignore permission errors on chmod call in Save/Remove operation
The file is already created with the proper permissions, thus the chmod
call is not critical. However, some file systems have don't allow
modifications of the file permissions. Similarly the chmod call in the Remove
operation should not prevent it from working.
2020-10-07 12:29:37 +02:00
greatroar
f3e933f0c1 Use windows.SetConsoleCursorPosition in ui/termstatus 2020-10-07 11:18:54 +02:00
MichaelEischer
0ae02f3030 Merge pull request #2844 from aawsome/prune-integration-tests
Add prune integration tests for many edge cases
2020-10-06 23:16:41 +02:00
rawtaz
e401859afb Merge pull request #2992 from lbausch/repo_wording
Replace repo/repos with repository/repositories in `--no-lock` help text
2020-10-06 23:03:55 +02:00
Lorenz Bausch
af4100e07d Replace repo/repos with repository/repositories in --no-lock help text 2020-10-06 22:49:48 +02:00
Alexander Neumann
ea5bbe0857 Remove dysfunctional link to saythanks 2020-10-06 20:43:44 +02:00
Alexander Weiss
6822a58413 Add prune integration tests for many edge cases 2020-10-06 20:20:05 +02:00
MichaelEischer
09d39e260d Merge pull request #2980 from greatroar/error-wrapping
Remove repetitive error wrapping from internal/cache
2020-10-06 15:19:00 +02:00
Michael Eischer
1579d2a8ec Remove some unused assignments 2020-10-06 14:55:13 +02:00
Michael Eischer
eba5dd831f Fix typos reported by misspell 2020-10-06 14:55:13 +02:00
Michael Eischer
9ffb698c8d Modernize context import in internal/fuse
Migrated using `go fix`.
2020-10-06 14:55:13 +02:00
Michael Eischer
efbb850d92 Remove a few redundant type specifiers
This is the result of running `gofmt -s -w **/*.go`
2020-10-06 14:55:13 +02:00
MichaelEischer
cfd57c480a Merge pull request #2991 from restic/nonsense
cache/check: Clarify error message when given arguments
2020-10-06 00:40:12 +02:00
Leo R. Lundgren
e105a3f391 cache/check: Clarify error message when given arguments 2020-10-06 00:08:59 +02:00
rawtaz
5d8cfff3f2 Merge pull request #2988 from greatroar/snapshots-json
Regression test for #2979
2020-10-05 21:02:52 +02:00
greatroar
673dda77c0 Less repetitive error messages in internal/cache
Many instances of errors.Wrap in this package would produce messages
like "Open: open <filename>: no such file or directory"; those now omit
the first "Open:" (or "Stat:", or "MkdirAll"). The function readVersion
now appends its own name to the error message, rather than the function
that failed, to make it easier to spot. Other function names (e.g.,
Load) are already added further up in the call chain.
2020-10-05 20:28:54 +02:00
MichaelEischer
1ab4c710e1 Merge pull request #2987 from greatroar/no-password-for-help
Don't read password for generate, help or self-update
2020-10-05 20:16:19 +02:00
greatroar
feedf0ebce Don't read password for generate, help or self-update
Fixes #2951.
2020-10-05 19:54:07 +02:00
MichaelEischer
1a490acd67 Merge pull request #2982 from greatroar/archiver-error-handling
Check error in archiver before calling Select
2020-10-05 19:19:47 +02:00
Michael Eischer
187518a8a3 docs: Properly reproduce zip archives for windows binaries 2020-10-05 18:24:46 +02:00
Michael Eischer
a232c833dc docs: Update binary reproduction instructions and refer to older instructions 2020-10-05 18:24:46 +02:00
MichaelEischer
c84643c6a9 Merge pull request #2981 from greatroar/mkdircachedir
Simplify cache directory creation
2020-10-05 18:15:23 +02:00
MichaelEischer
8390a8aaf3 Merge pull request #2986 from greatroar/upgrade-xxhash
Upgrade cespare/xxhash to 2.1.1
2020-10-05 17:49:08 +02:00
greatroar
1e1a1f3078 Upgrade cespare/xxhash to 2.1.1 2020-10-05 17:25:39 +02:00
MichaelEischer
abe9fa261f Merge pull request #2983 from greatroar/upgrade-siphash
Upgrade siphash to 1.2.2
2020-10-05 17:18:07 +02:00
MichaelEischer
6001b45bf7 Merge pull request #2984 from restic/snapshots-json
snapshots: Make --json output [] instead of null when no snapshots
2020-10-05 17:14:23 +02:00
greatroar
6bee0aafc2 Regression test for #2979 2020-10-05 15:08:09 +02:00
greatroar
8252ea8e3d Upgrade siphash to 1.2.2
New version is potentially faster on 32-bit ARM.
2020-10-05 14:56:13 +02:00
Leo R. Lundgren
137d20a06a snapshots: Make --json output [] instead of null when no snapshots 2020-10-05 13:14:10 +02:00
greatroar
c4e2203e45 Check error in archiver before calling Select
The archiver first called the Select function for a path before checking
whether the Lstat on that path actually worked. The RejectFuncs in
exclude.go worked around this by checking whether they received a nil
os.FileInfo. Checking first is more obvious and requires less code.
2020-10-05 11:11:04 +02:00
greatroar
7d0fa1a686 Simplify cache directory creation 2020-10-05 10:46:45 +02:00
rawtaz
a23d90d270 Merge pull request #2975 from restic/rawtaz-doc-resume
Add FAQ on whether restic can resume backups
2020-10-03 23:12:34 +02:00
MichaelEischer
2630411530 Merge pull request #2974 from gottwald/swap-gcs-lib
Swap deprecated GCS lib with replacement
2020-10-03 21:58:11 +02:00
rawtaz
b9b82d878d Add FAQ on whether restic can resume backups
This new FAQ entry explains that restic can resume interrupted backups.
2020-10-03 21:39:45 +02:00
Ingo Gottwald
8b8e230771 Swap deprecated GCS lib with replacement 2020-10-03 18:55:56 +02:00
rawtaz
7c3c6fa431 Merge pull request #2977 from MichaelEischer/downgrade-cobra
Downgrade cobra to v0.0.5
2020-10-03 18:23:13 +02:00
Michael Eischer
29908906b7 Downgrade cobra to v0.0.5 2020-10-03 18:13:07 +02:00
rawtaz
bbeb439f41 Merge pull request #2937 from andreaso/self-update-output-path-fix
Don't require `self-update --output` placeholder file
2020-10-03 13:35:20 +02:00
Andreas Olsson
ce14df303b Don't require self-update --output placeholder file
This removes the requirement on `restic self-update --output` to point
to a path of an existing file, to overwrite. In case the specified
path does exist we still want to verify that it's a regular file,
rather than a directory or a device, which gets overwritten.

We also want to verify that a path to a new file exists within an
existing directory. The alternative being running into that issue
after the actual download, etc has completed.

While at it I also replace `errors.Errorf` with the more appropriately
verbose `errors.Fatalf`.

Resolves #2491
2020-10-03 07:02:23 +02:00
MichaelEischer
3c6671b18b Merge pull request #2973 from restic/rawtaz-repo2
Correct data type for --repo2 option
2020-10-02 22:22:41 +02:00
MichaelEischer
86ff1f2bf6 Merge pull request #2972 from restic/rawtaz-verbose
Clarify max verbose level in option description
2020-10-02 22:21:45 +02:00
rawtaz
0cce6dc31c Correct data type for --repo2 option 2020-10-02 21:10:52 +02:00
rawtaz
6253ff0187 Clarify max verbose level in option description
Clarifies that the max verbosity level is 3, in the description of the --verbose option.
2020-10-02 20:25:34 +02:00
rawtaz
e9943e864f Merge pull request #2955 from jtagcat/clarify--compact
help: --compact: clarify description
2020-10-02 15:59:08 +02:00
jtagcat
a687261804 --help: --compact: clarify description 2020-10-02 16:55:56 +03:00
MichaelEischer
4df8861e09 Merge pull request #2967 from gottwald/fix-deprctd-gs-constructor
Replace deprecated method in gs backend
2020-10-01 11:01:49 +02:00
Ingo Gottwald
00cedd22aa Replace deprecated method in gs backend 2020-10-01 10:02:42 +02:00
MichaelEischer
f361ed66de Merge pull request #2910 from mtdcr/repository-file
Add new option --repository-file (default: $RESTIC_REPOSITORY_FILE)
2020-10-01 01:04:23 +02:00
Michael Eischer
7b50a65492 Update backup help output in documentation 2020-10-01 00:50:27 +02:00
Michael Eischer
c18b119a9b Document new option --repository-file 2020-10-01 00:50:27 +02:00
Michael Eischer
61035d68bc Add test for --repository-file 2020-10-01 00:50:27 +02:00
Andreas Oberritter
97f7855de3 Add new option --repository-file (default: $RESTIC_REPOSITORY_FILE)
As an alternative to -r, this allows to read the repository URL
from a file in order to prevent certain types of information leaks,
especially for URLs containing credentials.

Fixes #1458, fixes #2900.
2020-10-01 00:50:26 +02:00
Michael Eischer
d44df9d00d backup: Always remove the status lines once a backup ends 2020-09-30 23:13:10 +02:00
Michael Eischer
8d0ba55ecd backup: Fix possible deadlock of scanner goroutine
When the backup is interrupted for some reason while the scanner is
still active this could lead to a deadlock. Interruptions are triggered
by canceling the context object used by both the backup progress UI and
the scanner. It is possible that a context is canceled between the
respective check in the scanner and it calling the `ReportTotal` method
of the UI. The latter method sends a message to the UI goroutine.
However, a canceled context will also stop that goroutine, which can
cause the channel send operation to block indefinitely.

This is resolved by adding a `closed` channel which is closed once the
UI goroutine is stopped and serves as an escape hatch for reported UI
updates.

This change covers not just the ReportTotal method but all potentially
affected methods of the progress UI implementation.
2020-09-30 23:13:10 +02:00
rawtaz
34ea960559 Merge pull request #2966 from MichaelEischer/recover-help-typo
recover: Fix typo in the command help
2020-09-30 18:16:26 +02:00
rawtaz
5ea01b00df Merge pull request #2965 from MichaelEischer/rework-password-prompt
Clarify verbose password prompt
2020-09-30 18:15:05 +02:00
MichaelEischer
48d0ab5276 Merge pull request #2959 from restic/keep-text
backup: Correct keep policy text
2020-09-30 17:57:00 +02:00
Michael Eischer
2a79c1a44d recover: Fix typo in the command help
Reported-by: Steve Divskinsy <stevesbrain@users.noreply.github.com>
2020-09-30 17:44:34 +02:00
MichaelEischer
fd02407863 Merge pull request #2849 from classmarkets/gcs-access-token
gs: support authentication with access token
2020-09-30 17:42:56 +02:00
Michael Eischer
aea9f7d286 clarify verbose password prompt 2020-09-30 17:25:54 +02:00
Leo R. Lundgren
028c9a5343 backup: Correct keep policy text
Makes the following corrections to the "Applying Policy:" output:

- keep the last 1 snapshots snapshots => keep 1 latest snapshots
- keep the last 1 snapshots, 3 hourly, 5 yearly snapshots => keep 1 latest, 3 hourly, 5 yearly snapshots
2020-09-28 14:26:53 +02:00
MichaelEischer
94136132e3 Merge pull request #2957 from plumbeo/patch-3
Don’t print excessively detailed debug messages on object deletion when —verbose is used
2020-09-27 22:08:07 +02:00
plumbeo
009bd907f2 Don’t print too many messages when deleting files
Print detailed debug messages on file deletions only when --verbose=2 is used
2020-09-27 14:24:04 +02:00
MichaelEischer
14b312f00d Merge pull request #2658 from creativeprojects/issue-2241
Don't echo authentication passwords (rest backend)
2020-09-22 22:17:53 +02:00
Fred
206cadfab4 Hide password from repository URLs 2020-09-22 22:00:51 +02:00
MichaelEischer
4875f7b659 Merge pull request #2614 from greatroar/simplify-fs
Simplify internal/fs
2020-09-21 22:05:10 +02:00
MichaelEischer
6f2093e491 Merge pull request #2940 from andreaso/appveyor-tar-https
Use https:// to download appveyor tar
2020-09-21 21:50:26 +02:00
MichaelEischer
16f31b2f73 Merge pull request #2939 from J0WI/patch-1
Update Go version to 1.15 in Docker build script
2020-09-21 21:46:33 +02:00
greatroar
0d65b78168 Simplify os.ModeType|os.ModeCharDevice => os.ModeType
Since Go 1.12, ModeCharDevice is included in ModeType:
golang/go@a2a3dd00c9
2020-09-21 14:21:32 +02:00
greatroar
95ebba85ff Remove stray Printf from internal/fs 2020-09-21 14:21:32 +02:00
greatroar
59b343a9bf Remove OS-specific versions of fs.MkdirAll
Go has supported Windows paths correctly since 1.11, see
https://github.com/golang/go/issues/10900 and the commit referenced
there.
2020-09-21 14:21:32 +02:00
greatroar
1557c58eef Fix and simplify fs.Reader
fakeDir.{Readdir,Readdirnames} weren't handling the case n == 0
correctly. fakeFileInfo.sys is always nil, so omit the field.
2020-09-21 14:21:32 +02:00
greatroar
c504aa505c Remove unused fs.Rename 2020-09-21 14:21:32 +02:00
greatroar
1b20f6beec Remove io.Writer from fs.File
It was only used in a single test, which now uses plain *os.File instead.
2020-09-21 14:21:32 +02:00
rawtaz
10e3340863 Merge pull request #2945 from YoshieraHuang/max-file-size
Fix nil check in rejectBySize
2020-09-21 14:18:45 +02:00
yoshiera
3cf29a777d Fix nil check in rejectBySize 2020-09-21 19:20:24 +08:00
Andreas Olsson
fd1f7b7268 Use https:// to download appveyor tar
When supported one might as well use https://.
2020-09-20 18:49:46 +02:00
J0WI
090a73f7c5 Update Go version to 1.15 in Docker build script 2020-09-20 15:42:12 +00:00
rawtaz
1cfb01a8a6 Merge pull request #2936 from andreaso/contrib-update-version
Update versions in contribution documentation
2020-09-20 13:27:13 +02:00
Andreas Olsson
ce62d3d689 Update versions in contribution documentation
Reflects 3c44598 and 429f97b.
2020-09-20 10:32:27 +02:00
Michael Eischer
8c36317b71 rclone: use configured number of connections during create 2020-09-19 19:11:43 +02:00
Michael Eischer
60a5c35de9 rclone: close connection to rclone if open fails 2020-09-19 19:11:43 +02:00
Michael Eischer
9333f707fa init: use Create method for rclone backend
This properly issues the initial repository creation command

Fixes #1896
2020-09-19 19:11:43 +02:00
Alexander Neumann
429f97b887 Set development version for 0.10.0 2020-09-19 17:38:47 +02:00
Alexander Neumann
40832b2927 Add version for 0.10.0 2020-09-19 17:38:26 +02:00
Alexander Neumann
c8a94eced7 Update manpages and auto-completion 2020-09-19 17:38:26 +02:00
Alexander Neumann
ee6e981b4e Generate CHANGELOG.md for 0.10.0 2020-09-19 17:38:10 +02:00
Alexander Neumann
96fd982f6a Prepare changelog for 0.10.0 2020-09-19 17:37:59 +02:00
Alexander Neumann
6ff0082c02 Merge pull request #2928 from MichaelEischer/init-copy-chunker
init: Add `--copy-chunker-parameters` option
2020-09-19 17:32:13 +02:00
Alexander Neumann
95c1d7d959 Merge pull request #2927 from restic/update-deps
Update dependencies
2020-09-19 17:29:35 +02:00
Michael Eischer
07f4e7d10b Only log HTTP requests when a debug log is configured
The logged HTTP requests are only visible when a debug log is
configured.
2020-09-19 17:07:53 +02:00
Michael Eischer
f003410402 init: Add --copy-chunker-params option
This allows creating multiple repositories with identical chunker
parameters which is required for working deduplication when copying
snapshots between different repositories.
2020-09-19 16:53:05 +02:00
Michael Eischer
9ad8250a78 Reduce overhead of debug calls if no log is enabled
In case no debug log is configured, then calls to debug.Log only incur
the costs to check a single boolean flag making the call really cheap.
2020-09-19 16:45:01 +02:00
Michael Eischer
655430550b Extract parameters for second repository from copy command 2020-09-19 16:07:55 +02:00
MichaelEischer
1823b8195c Merge pull request #2930 from rawtaz/password-file-text
Improve wording for --password-file and related options
2020-09-19 16:06:15 +02:00
Leo R. Lundgren
311ad2d2d0 Improve wording for --password-file and related options 2020-09-19 15:47:32 +02:00
Alexander Neumann
a10b44a265 Add note about fuse 2020-09-19 14:35:55 +02:00
Alexander Neumann
baf3a9aa3b Merge pull request #2929 from rawtaz/password-command-text
Improve wording for --password-command and --password-command2 options
2020-09-19 14:30:39 +02:00
Leo R. Lundgren
ffe6dce7e7 Improve wording for --password-command and --password-command2 options 2020-09-19 14:23:25 +02:00
Alexander Neumann
8ce0ce387f Add changelog 2020-09-19 14:22:31 +02:00
Alexander Neumann
3c44598bf6 Drop Go 1.12, require at least 1.13 2020-09-19 14:20:04 +02:00
Alexander Neumann
3bb55fd6bf Merge pull request #2914 from YoshieraHuang/max-file-size
Support excluding files by size
2020-09-19 14:12:15 +02:00
Alexander Neumann
36efefa7bd Update dependencies, pin fuse library
This updates all dependencies which causes the following changes:
 * The fuse library is set to the last version supporting macOS
 * The minimal version of Go required to build restic is now 1.13
2020-09-19 14:06:37 +02:00
yoshiera
ac4b8c98ac Support excluding files by size 2020-09-19 19:52:09 +08:00
MichaelEischer
4dcd6abf37 Merge pull request #2616 from greatroar/no-cache-interface
Remove practically unused cache interface
2020-09-18 19:50:47 +02:00
greatroar
cb3f531050 Make Cache.Path private
It's only used inside the package.
2020-09-18 11:17:29 +02:00
greatroar
23fcbb275a Remove restic.Cache interface
It was used in one code path, which then asserted its concrete type as
*cache.Cache.

Privatised some of the interface methods.
2020-09-18 10:48:13 +02:00
greatroar
6e3215a80d Remove unused error type from internal/cache 2020-09-18 10:38:14 +02:00
greatroar
9abef3bf1a Move internal/fs.TestChdir to internal/test.Chdir 2020-09-17 10:43:33 +02:00
Alexander Neumann
b10dce541e Merge pull request #2598 from MichaelEischer/diff-speedup
Diff speedup
2020-09-13 16:42:05 +02:00
Michael Eischer
4f221c4022 Add changelog entry 2020-09-13 16:28:46 +02:00
Michael Eischer
f5c448aa65 diff: Optimize diff calculation for shared subtrees
When the diff calculation compares two trees with identical id then no
differences between them can ever show up. Optimize for that case by
simply traversing the tree only once to collect all referenced blobs for
a proper calculation of added and removed blobs.

Just skipping the common subtrees is not possible as this would skew the
results if the added or removed blobs are shared with one of the
subtrees.
2020-09-13 16:28:46 +02:00
Michael Eischer
c0fc85d303 diff: Add integration test 2020-09-13 16:28:46 +02:00
Alexander Neumann
0c48e515f0 Merge pull request #2630 from MichaelEischer/fix-staticcheck
Fix lots of small issues reported by staticcheck
2020-09-13 16:19:22 +02:00
Alexander Neumann
97950ab81a options: Fix test for Go >= 1.15 2020-09-12 17:36:44 +02:00
Alexander Neumann
59fca85844 Remove darwin/386 from integration tests 2020-09-12 17:09:33 +02:00
Alexander Neumann
e207257714 Fix Appveyor 2020-09-12 16:59:03 +02:00
Alexander Neumann
82e1cbed4f Update Go versions 2020-09-12 16:52:21 +02:00
Alexander Neumann
8903b6c88a helpers: Rework list of architectures
This also removes darwin/386 which is not supported on Go 1.15 any more.
2020-09-12 16:39:41 +02:00
Alexander Neumann
93583c01b1 helpers: Fetch modules for release binaries 2020-09-12 16:25:36 +02:00
MichaelEischer
88664ba222 Merge pull request #2893 from MichaelEischer/restore-preallocate
restorer: Preallocate files
2020-09-08 22:43:05 +02:00
Michael Eischer
121233e1b3 Add preallocate tests 2020-09-07 21:41:47 +02:00
Michael Eischer
8cc9514879 restorer: pre-allocate files before loading chunks 2020-09-07 21:41:47 +02:00
Michael Eischer
2e7d475029 Process packs in order of first appearance 2020-09-06 21:25:35 +02:00
rawtaz
d3a286928a Merge pull request #2905 from MichaelEischer/fix-ls-help
ls: Explicitly enforce that the user specifies a snapshot ID
2020-09-05 10:44:01 +02:00
Michael Eischer
c46edcd9d6 error strings should not end with punctuation 2020-09-05 10:07:17 +02:00
Michael Eischer
1ede018ea6 error variable names should start with 'Err' 2020-09-05 10:07:17 +02:00
Michael Eischer
b77e933d80 Convert underscore variable names to camelCase 2020-09-05 10:07:16 +02:00
Michael Eischer
d0329cf3eb Adjust comments to match name of exported methods 2020-09-05 10:07:16 +02:00
Michael Eischer
dc31529fc3 Unindent else block after if block ending with a return statement 2020-09-05 10:07:16 +02:00
Michael Eischer
4784540f04 repository: Simplify worker group code 2020-09-05 10:07:16 +02:00
Michael Eischer
84ea2389ae archiver_test: Only skip symlinks on windows 2020-09-05 10:07:16 +02:00
Michael Eischer
b4a7ce86cf uint cannot be less than zero 2020-09-05 10:07:16 +02:00
Michael Eischer
7ee0964880 crypto: Remove unused error 2020-09-05 10:07:16 +02:00
Michael Eischer
f6f11400c2 restic: Remove unused variable value 2020-09-05 10:07:16 +02:00
Michael Eischer
e2dc5034d3 restic/file: Use FileType for all File constants
The data type of a variable is only carried on when using iota.
2020-09-05 10:07:16 +02:00
Michael Eischer
9a1b3cb5d9 restorer: Remove unused treeID struct member from test 2020-09-05 10:07:16 +02:00
Michael Eischer
b22655367c integration_test: Replace fprintf without format string 2020-09-05 10:07:16 +02:00
Michael Eischer
068a3ce23f Remove redundant return 2020-09-05 10:07:16 +02:00
Michael Eischer
ee05501ce7 archiver: Remove unused chmod function 2020-09-05 10:07:16 +02:00
Michael Eischer
014600bee6 archiver: Remove unused variable from test 2020-09-05 10:07:16 +02:00
Michael Eischer
d9a80e07b9 repository: Simplify index age calculation 2020-09-05 10:07:16 +02:00
Michael Eischer
d19f05c960 Use buf.String() instead of string(buf.Bytes()) 2020-09-05 10:07:16 +02:00
Michael Eischer
460e2ffbf6 Collapse a few boolean operations 2020-09-05 10:07:14 +02:00
Michael Eischer
49b6aac3fa Use !bytes.Equal instead of bytes.Compare !=0 to check for inequality 2020-09-05 10:06:23 +02:00
Michael Eischer
2f8335554c Remove a few unused variables 2020-09-05 10:06:23 +02:00
Michael Eischer
37113282ca Merge a few variable declaration and initializations 2020-09-05 10:05:34 +02:00
Michael Eischer
337725c354 Use sort.Strings 2020-09-05 10:05:34 +02:00
MichaelEischer
2ddb7ffb7e Merge pull request #2255 from Kidswiss/tar
Fix dumping issues with / and the first sub level
2020-09-01 21:52:17 +02:00
Michael Eischer
81dcfea11a dump: Mention tar output in command help text 2020-08-31 22:43:10 +02:00
MichaelEischer
55071ee367 Merge pull request #2859 from buschjost/stats-filter-by-tag-and-path
Add filter by tag and path to stats command
2020-08-31 22:11:01 +02:00
Michael Eischer
dcf9ded977 dump: Use slashes as path separators in tar on windows
This conversion is necessary as paths returned by filepath.Rel use the
operating-system native path separator
2020-08-30 18:26:50 +02:00
Michael Eischer
bcf44a9c3f dump: Properly handle file paths without / prefix
filepath.Rel failed if the requested path did not start with a `/` e.g.
`restic` instead of `/restic`.
2020-08-30 18:25:42 +02:00
Oliver Buschjost
a7b4c19abf Fix display of warning about ignored filters 2020-08-30 17:14:56 +02:00
Oliver Buschjost
d3fcfeba3a Add filter by tag and path to stats command 2020-08-30 17:14:48 +02:00
Michael Eischer
e69449bf2c dump: Add test for splitPath 2020-08-30 16:00:48 +02:00
Simon Beck
da4193c3ef Add changelog 2020-08-30 16:00:48 +02:00
Simon Beck
fe6445e0f4 dump: Added unit tests for tar output 2020-08-30 16:00:48 +02:00
MichaelEischer
ea81a0e282 Merge pull request #2606 from middelink/fix-323
Add copy functionality.
2020-08-30 10:18:24 +02:00
Simon Beck
80a11960dd dump: Always dump relative paths into tarballs
Tarballs should only contain relative paths.
2020-08-29 21:14:34 +02:00
Simon Beck
d6f739ec22 dump: Extract tar code to internal/dump 2020-08-29 21:14:34 +02:00
Simon Beck
b98598e55f dump: Fix dumping issues with / and the first sub level
There was an issue that prevented the dump command from working
correctly when either:

* `/` contained multiple nodes (e.g. `restic backup /`)
* dumping a file in the first sublevel was attempted (e.g. `/foo`)
2020-08-29 21:14:29 +02:00
Michael Eischer
d5f86effa1 ls: Explicitly enforce that the user specifies a snapshot ID
The help messages suggested that the `ls` command work without
explicitly passing a snapshot ID. However, this was never the case:
without a snapshot ID the command just failed with the error
`Ignoring "", it is not a snapshot id`.

Fixes #2299
2020-08-29 14:28:53 +02:00
MichaelEischer
c34c731698 Merge pull request #2702 from MichaelEischer/fix-backup-dir-count
backup: Fix reporting of directory count in summary
2020-08-29 11:29:21 +02:00
Michael Eischer
412623b848 copy: Reuse buffer for downloaded blobs 2020-08-29 10:48:44 +02:00
Michael Eischer
bbe8b73f03 Update help text of backup command in docs 2020-08-29 10:48:44 +02:00
Michael Eischer
91e8d998cd Add documentation for copy command 2020-08-29 10:48:44 +02:00
Michael Eischer
9a4796594a integration tests: Fix checking of wrong snapshot 2020-08-29 10:48:44 +02:00
Michael Eischer
15374d22e9 integration tests: Add basic tests for copy command 2020-08-29 10:48:44 +02:00
Michael Eischer
88ad58d6cd integration tests: Redirect directory diff into intermediate buffer 2020-08-29 10:48:44 +02:00
Michael Eischer
591a8c4cdf integration tests: Deduplicate backup test-data setup code 2020-08-29 10:48:44 +02:00
Michael Eischer
ec9a53b7e8 copy: Mark and skip previously copied snapshots
Use the `Original` field of the copied snapshot to store a persistent
snapshot ID. This can either be the ID of the source snapshot if
`Original` was not yet set or the previous value stored in the
`Original` field. In order to still copy snapshots modified using the
tags command the source snapshot is compared to all snapshots in the
destination repository which have the same persistent ID. Snapshots are
only considered equal if all fields except `Original` and `Parent`
match. That way modified snapshots are still copied while avoiding
duplicate copies at the same time.
2020-08-29 10:48:44 +02:00
MichaelEischer
34a3adfd8d Merge pull request #2898 from aawsome/add-iexclude-file
backup: Add --iexclude-file option
2020-08-28 21:59:16 +02:00
Alexander Weiss
9867c4bbb4 Add flag --iexclude-file to backup 2020-08-28 21:04:41 +02:00
MichaelEischer
efb4a981cf Merge pull request #2904 from jasonccox/consistent-usage-strings
Update usage strings to put flags before args
2020-08-28 20:39:14 +02:00
Jason Cox
2447f3f110 Update usage strings to put flags before args
The standard UNIX-style ordering of command-line arguments places
optional flags before other positional arguments. All of restic's
commands support this ordering, but some of the usage strings showed the
flags after the positional arguments (which restic also parses just
fine). This change updates the doc strings to reflect the standard
ordering.

Because the `restic help` command comes directly from Cobra, there does
not appear to be a way to update the argument ordering in its usage
string, so it maintains the non-standard ordering (positional arguments
before optional flags).
2020-08-28 02:16:22 +00:00
Michael Eischer
b25978a53c backup: Fix reporting of directory count in summary
Previously the directory stats were reported immediately after calling
`SaveDir`. However, as the latter method saves the tree asynchronously
the stats were still initialized to their nil value. The stats are now
reported via a callback similar to the one used for the fileSaver.
2020-08-27 22:43:51 +02:00
Michael Eischer
b0a8c4ad6c copy: Only process each tree once
This speeds up copying multiple overlapping snapshots from one
repository to another, as we only have to copy the changed parts of
later snapshots.
2020-08-26 22:17:52 +02:00
Michael Eischer
908b23fda0 copy: Update for modernized repository interface 2020-08-26 22:17:52 +02:00
greatroar
4508d406ef copy: Remove separate SaveIndex in restic copy
Flush does this now.
2020-08-26 22:17:52 +02:00
Pauline Middelink
7048cc3e58 Add copy functionality
Add a copy command to copy snapshots between repositories. It allows the user
to specify a destination repository, password, password-file, password-command
or key-hint to supply the necessary details to open the destination repository.

You need to supply a list of snapshots to copy, snapshots which already exist
in the destination repository will be skipped.

Note, when using the network this becomes rather slow, as it needs to read the
blocks, decrypt them using the source key, then encrypt them again using the
destination key before finally writing them out to the destination repository.
2020-08-26 22:17:52 +02:00
MichaelEischer
eb7c00387c Merge pull request #2899 from MichaelEischer/fix-check-progress-crash
Unify progress bar of check and prune commands
2020-08-26 00:00:50 +02:00
Michael Eischer
bc0501d72c Add changelog 2020-08-25 23:33:10 +02:00
Michael Eischer
17995dec7a Unify progress bar of check and prune commands 2020-08-25 22:47:38 +02:00
MichaelEischer
e915cedc3d Merge pull request #2897 from restic/rawtaz-github-issue-feature
github: Add question to feature issue template
2020-08-24 23:52:38 +02:00
MichaelEischer
cdcaecd27d Merge pull request #2896 from restic/rawtaz-key-hint
doc: Add missing environment variables
2020-08-24 23:51:05 +02:00
rawtaz
b43ab67a22 doc: Add missing environment variables 2020-08-24 00:49:34 +02:00
rawtaz
7ddfd6cabe github: Add question to feature issue template 2020-08-24 00:22:44 +02:00
Alexander Neumann
b1b3f1ecb6 Merge pull request #2674 from MichaelEischer/prune-strict-checks
prune: Stricter error checks
2020-08-23 10:30:41 +02:00
rawtaz
fa135f72bf Merge pull request #2439 from htrendev/ignore-sync-on-macos
Ignore not supported error on sync() when using local backend
2020-08-22 01:38:49 +02:00
Hristo Trendev
51c22f4223 local backend: ignore not supported error on sync()
Closes #2395
2020-08-22 01:27:07 +02:00
rawtaz
1a5b66f33b Merge pull request #2879 from 0xMH/master
Added more explantation for --one-file-system option
2020-08-21 19:21:01 +02:00
NoNE
da6a34e044 doc: Add more explanation for --one-file-system option 2020-08-21 19:19:13 +02:00
Bruce Dillahunty
fe69b83074 Update key password prompt (#2847) 2020-08-19 21:42:08 +02:00
Michael Eischer
08d24ff99e prune: Include ID of all missing blobs in error message 2020-08-16 11:36:14 +02:00
Michael Eischer
d8b80e9862 Add changelog 2020-08-16 11:36:14 +02:00
Michael Eischer
1c84aceb39 prune: Test for abort on damaged repositories 2020-08-16 11:36:12 +02:00
Michael Eischer
575ed9a47e Test that rebuild-index errors when old index cannot be removed 2020-08-16 11:34:01 +02:00
Michael Eischer
8f811642c3 Add support for integration tests to wrap the backend 2020-08-16 11:34:01 +02:00
Michael Eischer
f4b9544ab2 prune: Add test that repack aborts on wrong blob 2020-08-16 11:34:01 +02:00
Michael Eischer
367449dede prune: Reduce memory allocations while repacking
The slicing operator `slice[low:high]` default to 0 for the lower bound and
len(slice) for the upper bound when either or both are not specified.
Fix the code to use `cap(slice)` to check for the slice capacity.
2020-08-16 11:34:01 +02:00
Michael Eischer
7042bafea5 prune: Abort repacking when a pack contains a wrong blob
If a blob in a pack file can be decrypted successfully but contains data
that results in a different hash than stated in the header pack, then
abort repacking. As both the pack header and the blob are
cryptographically verified this either means than a malicious entity
tampered with the backup or indicates hardware problems on the client.
prune should fail with an error in both cases.
2020-08-16 11:34:01 +02:00
Michael Eischer
744a15247d prune/rebuild-index: Fail if an old index cannot be removed
The old behavior was problematic in the context of rebuild-index as it
could leave old, possibly invalid index files behind without returning a
fatal error.

Prune calls rebuildIndex before removing any data from the repository.
For this use case failing to delete an old index MUST be treated as a
fatal error. Otherwise the index could still contain an old index file
that refers to blobs/packs that were later on deleted by prune. Later
backup runs will assume that the affected blobs already exist in the
repository which results in a backup which misses data.
2020-08-16 11:34:01 +02:00
Michael Eischer
3ba19869be prune: Abort if any used blobs are missing
The previous check only approximately verified whether all required
blobs were found. However, after forgetting a few snapshots the
repository contains lots of unused blobs whose number can be sufficient
to make up for missing packs.

When coupled with a malfunctioning backend that temporarily returns broken
data this could cause restic to regard the corresponding packs as
invalid and thereby delete data that's still in use. This change lets
restic play it safe and refuse to delete anything if data is missing.
2020-08-16 11:34:01 +02:00
aawsome
0fed6a8dfc Use "pack file" instead of "data file" (#2885)
- changed variable names, especially changed DataFile into PackFile
- changed in some comments
- always use "pack file" in docu
2020-08-16 11:16:38 +02:00
rawtaz
643bbbe156 doc: Correct wording for repository (#2884)
Replaces "backend" with "repository" to match restic output.
2020-08-11 22:30:17 +02:00
MichaelEischer
eca0f0ad24 Merge pull request #2863 from aawsome/index-no-duplicates
Don't save exact duplicates in merged index
2020-08-08 18:24:14 +02:00
MichaelEischer
08dee8a52b Merge pull request #2865 from greatroar/unused
Dead code removal
2020-08-08 16:15:28 +02:00
Alexander Weiss
b112533812 Don't save exact duplicates when merging indexes 2020-08-05 06:32:02 +02:00
Alexander Weiss
5e63294355 Add benchmark MasterIndexAlloc 2020-08-05 06:32:02 +02:00
MichaelEischer
84b6f1ec53 Merge pull request #2874 from MichaelEischer/fix-verbose
Adjust description of --verbose=n parameter
2020-08-04 23:21:05 +02:00
MichaelEischer
06fb4ea3f0 Merge pull request #2805 from jwilk-forks/verbose
Fix examples of --verbose with argument
2020-08-04 23:11:35 +02:00
Michael Eischer
e38d415173 Adjust description of --verbose=n parameter 2020-08-04 23:07:53 +02:00
greatroar
d81a396944 Dead code removal
Found by running golangci-lint on the entire code base.
2020-08-04 08:38:57 +02:00
rawtaz
0b21ec44b7 Merge pull request #2869 from josephrocca/patch-1
doc: --verbose 2 to --verbose=2
2020-08-04 01:06:58 +02:00
josephrocca
bd36731119 --verbose 2 to --verbose=2
`--verbose 2` seems to be incorrect here (gives an error/warning that "the `2` directory does not exist, skipping")
2020-08-04 08:39:43 +10:00
MichaelEischer
38a2f9c07b Merge pull request #2864 from greatroar/dump-error
Fix error handling in dump (err != err)
2020-08-03 20:09:42 +02:00
rawtaz
5af2815627 Merge pull request #2821 from renard/mount-nolock
cmd/mount: honur --no-lock flag
2020-08-03 20:09:26 +02:00
MichaelEischer
b55de2260d Merge pull request #2868 from MichaelEischer/fix-getusedblobs
prune: Stop progress bar after searching used blobs
2020-08-03 19:58:59 +02:00
Sébastien Gross
9be4fe3e84 cmd/mount: honour --no-lock flag
Do not lock the repository if --no-lock global flag is set. This allows
to mount repositories which are archived on a read only system.

Signed-off-by: Sébastien Gross <seb•ɑƬ•chezwam•ɖɵʈ•org>
2020-08-03 19:45:39 +02:00
Michael Eischer
05116e4787 prune: Cleanup progress bar handling while repacking 2020-08-03 19:32:46 +02:00
Michael Eischer
04f79b9642 prune: Stop progress bar after searching used blobs 2020-08-03 19:31:49 +02:00
greatroar
8b358935a0 Fix error handling in dump (err != err) 2020-08-03 09:43:44 +02:00
MichaelEischer
66d089e239 Merge pull request #2841 from aawsome/optimize-getUsedBlobs
Extract get used blobs in prune as separate function
2020-08-01 22:40:33 +02:00
MichaelEischer
49d3efe547 Merge pull request #2840 from aawsome/delete-parallel
Make delete parallel
2020-08-01 22:26:15 +02:00
Alexander Weiss
9762bec091 Use optimized getUsedBlobs in prune 2020-08-01 21:07:31 +02:00
Alexander Weiss
0eb8553c87 add changelog for #2840 2020-08-01 20:43:18 +02:00
Alexander Weiss
d3692f5b81 Delete files in parallel in rebuild-index 2020-08-01 20:43:18 +02:00
Alexander Weiss
1c0b61204b Delete files in parallel in forget 2020-08-01 20:43:18 +02:00
Alexander Weiss
2ee654763b Delete files in parallel 2020-08-01 20:39:24 +02:00
Alexander Neumann
b7b479b668 Merge pull request #2599 from MichaelEischer/tweak-mem-usage
Reduce memory usage when searching for used blobs
2020-08-01 13:49:02 +02:00
Alexander Neumann
4cf9656f12 Merge pull request #2861 from MichaelEischer/fix-rclone-crash
rclone: Don't panic after unexpected subprocess exit
2020-08-01 13:08:50 +02:00
Alexander Neumann
2580eef2aa Merge pull request #2318 from classmarkets/2175-named-keys
Allow specifying user and host when adding keys
2020-08-01 13:06:31 +02:00
Michael Eischer
2d7ab9115f Add changelog entry 2020-08-01 12:29:16 +02:00
Michael Eischer
a178e5628e Remove duplicate TreeLoader interface 2020-08-01 12:29:16 +02:00
Michael Eischer
af66a62c04 FindUsedBlobs: Test that seen blobs are skipped
This also copies the TreeLoader interface from internal/walker to allow
stubbing the repository in the call to `FindUsedBlobs`.
2020-08-01 12:29:16 +02:00
Michael Eischer
9ea1a78bd4 FindUsedBlobs: Check for seen blobs before loading trees
The only effective change in behavior is that that toplevel nodes can
also be skipped.
2020-08-01 12:29:16 +02:00
Michael Eischer
184103647a FindUsedBlobs: merge seen into blobs BlobSet
The seen BlobSet always contained a subset of the entries in blobs.
Thus use blobs instead and avoid the memory overhead of the second set.

Suggested-by: Alexander Weiss <alex@weissfam.de>
2020-08-01 12:29:16 +02:00
Michael Eischer
c81b122374 rclone: Don't panic after unexpected subprocess exit
As the connection to the rclone child process is now closed after an
unexpected subprocess exit, later requests will cause the http2
transport to try to reestablish a new connection. As previously this never
should have happened, the connection called panic in that case. This
panic is now replaced with a simple error message, as it no longer
indicates an internal problem.
2020-08-01 12:17:40 +02:00
rawtaz
48f97f3567 Merge pull request #2857 from MichaelEischer/incomplete-backup-error
Don't print a stacktrace if some files could not be read
2020-07-29 00:06:25 +02:00
Michael Eischer
3ce9893e0b Don't print a stacktrace if some files could not be read 2020-07-28 23:52:48 +02:00
MichaelEischer
248c7c3828 Merge pull request #2587 from aawsome/optimize-fuse
Make `restic mount` faster and consume less memory
2020-07-28 23:19:17 +02:00
Alexander Weiss
f8316948d1 Optimize FUSE - make command restic mount faster and consume less memory
- Add Open() functionality to dir
- only access index for blobs when file is read
- Implement NodeOpener and put one-time file stuff there
- Add comment about locking as suggested by bazil.org/fuse

=> Thanks at Michael Eischer for suggesting the last two improvements
2020-07-28 23:01:18 +02:00
MichaelEischer
be54ceff66 Merge pull request #2855 from MichaelEischer/rclone-exit
Fix rclone subprocess handling
2020-07-26 22:55:25 +02:00
Michael Eischer
ea97ff1ba4 rclone: Skip crash test when rclone is not found 2020-07-26 12:06:18 +02:00
Michael Eischer
01b9581453 rclone: Better field names for stdio conn 2020-07-26 00:29:25 +02:00
Michael Eischer
3cd927d180 rclone: Give rclone time to finish before closing stdin pipe
Calling `Close()` on the rclone backend sometimes failed during test
execution with 'signal: Broken pipe'. The stdio connection closed both
the stdin and stdout file descriptors at the same moment, therefore
giving rclone no chance to properly send any final http2 data frames.

Now the stdin connection to rclone is closed first and will only be
forcefully closed after a timeout. In case rclone exits before the
timeout then the stdio connection will be closed normally.
2020-07-26 00:29:25 +02:00
greatroar
bf7b1f12ea rclone: Add test for pipe handling when rclone exits 2020-07-26 00:29:25 +02:00
Michael Eischer
8554332894 rclone: Close rclone side of stdio_conn pipes
restic did not notice when the rclone subprocess exited unexpectedly.

restic manually created pipes for stdin and stdout and used these for the
connection to the rclone subprocess. The process creating a pipe gets
file descriptors for the sender and receiver side of a pipe and passes
them on to the subprocess. The expected behavior would be that reads or
writes in the parent process fail / return once the child process dies
as a pipe would now just have a reader or writer but not both.

However, this never happened as restic kept the reader and writer
file descriptors of the pipes. `cmd.StdinPipe` and `cmd.StdoutPipe`
close the subprocess side of pipes once the child process was started
and close the parent process side of pipes once wait has finished. We
can't use these functions as we need access to the raw `os.File` so just
replicate that behavior.
2020-07-26 00:29:25 +02:00
greatroar
3e93b36ca4 Make rclone.New private 2020-07-26 00:28:45 +02:00
MichaelEischer
573a2fb240 Merge pull request #2789 from aawsome/fix-lookup
make Lookup() return all blobs
2020-07-25 21:32:23 +02:00
Michael Eischer
c847aace35 Rename Index interface to MasterIndex
The interface is now only implemented by repository.MasterIndex.
2020-07-25 21:19:46 +02:00
Alexander Weiss
9d1fb94c6c make Lookup() return all blobs
+ simplify syntax
2020-07-25 21:18:34 +02:00
MichaelEischer
020cab8e08 Merge pull request #2787 from greatroar/no-blobsize-cache
Remove blob size cache from restic mount
2020-07-25 20:46:35 +02:00
greatroar
07da61baee Remove blob size cache from restic mount 2020-07-25 19:39:14 +02:00
MichaelEischer
37c95bf5da Merge pull request #2835 from J0WI/patch-1
Update Go version to 1.14.6 in Docker build script
2020-07-25 17:23:54 +02:00
rawtaz
c86d2f23aa Merge pull request #2854 from restic/rawtaz-patch-1
Clarify questions and text in PR template
2020-07-25 15:41:27 +02:00
rawtaz
96ec04d74d Clarify questions and text in PR template 2020-07-25 15:38:00 +02:00
J0WI
9c3414374a Update Go version to 1.14.6 in Docker build script 2020-07-25 14:45:36 +02:00
MichaelEischer
3d530dfc91 Merge pull request #2827 from aawsome/archiver-test-contents
Make self-healing work when backing up with parent snapshot
2020-07-25 13:13:18 +02:00
MichaelEischer
c43f5b2664 Merge pull request #2733 from hibbert/support_irsa
Enable support for EKS IRSA
2020-07-25 12:58:43 +02:00
Andy Hibbert
38087e40d9 Update minio-go version so it supports EKS IRSA 2020-07-25 12:41:18 +02:00
MichaelEischer
bbc960f957 Merge pull request #2635 from greatroar/optimize-sortbycached
Optimize sorting blobs by cache status
2020-07-25 12:35:42 +02:00
greatroar
309598c237 Simplify sortCachedPacksFirst test in internal/repository
The test now uses the fact that the sort is stable. It's not guaranteed
to be, but the test is cleaner and more exhaustive. sortCachedPacksFirst
no longer needs a return value.
2020-07-25 12:12:59 +02:00
greatroar
03d23e6faa Speed up blob sorting in internal/repository
name                    old time/op    new time/op    delta
SortCachedPacksFirst-8     208µs ± 3%     186µs ± 3%  -10.74%  (p=0.000 n=10+8)

name                    old alloc/op   new alloc/op   delta
SortCachedPacksFirst-8     213kB ± 0%     139kB ± 0%  -34.62%  (p=0.000 n=10+10)

name                    old allocs/op  new allocs/op  delta
SortCachedPacksFirst-8     1.03k ± 0%     1.03k ± 0%   -0.19%  (p=0.000 n=10+10)
2020-07-25 12:12:59 +02:00
greatroar
b10acd2af7 Test and benchmark blob sorting in internal/repository 2020-07-25 12:12:58 +02:00
Alexander Weiss
9175795fdb Check contents in archiver
When backing up with a parent snapshot and the file is not changed, also
check if contents are still available in index.
2020-07-25 08:18:28 +02:00
rawtaz
5d8d70542f Merge pull request #2852 from MichaelEischer/drop-go-1.11
Drop support for Go version 1.11
2020-07-25 01:17:36 +02:00
Michael Eischer
7c23381a2b Drop support for Go version 1.11 2020-07-24 18:52:39 +02:00
MichaelEischer
34181b13a2 Merge pull request #2328 from MichaelEischer/no-repeated-checks
Fix duplicate tree checks within `restic check`
2020-07-22 22:08:02 +02:00
MichaelEischer
bcd47ec3a2 Merge pull request #2818 from aawsome/merge-index-chaining
Merge index (based on chaining index implementation)
2020-07-22 22:05:36 +02:00
Alexander Weiss
a666a6d576 Add tests and merge indexes in index benchmarks 2020-07-22 21:54:02 +02:00
Alexander Weiss
e388d962a5 Merge final indexes together for faster index access 2020-07-22 21:54:02 +02:00
Peter Schultz
758b44b9c0 gs: support authentication with access token
In the Google Cloud Storage backend, support specifying access tokens
directly, as an alternative to a credentials file. This is useful when
restic is used non-interactively by some other program that is already
authenticated and eliminates the need to store long lived credentials.

The access token is specified in the GOOGLE_ACCESS_TOKEN environment
variable and takes precedence over GOOGLE_APPLICATION_CREDENTIALS.
2020-07-22 16:23:03 +02:00
Alexander Weiss
3b7a3711e6 Add more realistic index benchmarks
+ reduce test size of BenchmarkMasterIndexLookupParallel
2020-07-21 07:18:20 +02:00
Michael Eischer
9b0e718852 checker: Test that blob types are not confused 2020-07-20 23:43:47 +02:00
MichaelEischer
82c908871d Merge pull request #2812 from greatroar/chaining
Chaining hash table for repository.Index
2020-07-20 23:29:36 +02:00
Michael Eischer
ddf0b8cd0b checker: Properly distinguish between data and tree blobs
If a data blob and a tree blob with the same ID (= same content) exist,
then the checker did not report a data or tree blob as unused when the
blob of the other type was still in use.
2020-07-20 22:58:39 +02:00
Michael Eischer
2d0c138c9b checker: Test that check only decodes trees once
The `DuplicateTree` flag is necessary to ensure that failures cannot be
swallowed. The old checker implementation ignores errors from LoadTree
if the corresponding tree was already checked.
2020-07-20 22:51:53 +02:00
Michael Eischer
ef325ffc02 checker: Cleanup error handling code
This change only moves code around but does not result in any change in
behavior.
2020-07-20 22:51:53 +02:00
Michael Eischer
0f67ae813a Add changelog 2020-07-20 22:51:53 +02:00
Michael Eischer
7a165f32a9 checker: Traverse trees in depth-first order
Backups traverse the file tree in depth-first order and saves trees on
the way back up. This results in tree packs filled in a way comparable
to the reverse Polish notation.  In order to check tree blobs in that
order, the treeFilter would have to delay the forwarding of tree nodes
until all children of it are processed which would complicate the
implementation.

Therefore do the next similar thing and traverse the tree in depth-first
order, but process trees already on the way down. The tree blob ids are
added in reverse order to the backlog, which is once again reverted when
removing the ids from the back of the backlog.
2020-07-20 22:51:53 +02:00
Michael Eischer
36c69e3ca7 checker: Unify blobs, processed trees and referenced blobs map
The blobRefs map and the processedTrees IDSet are merged to reduce the
memory usage. The blobRefs map now uses separate flags to track blob
usage as data or tree blob. This prevents skipping of trees whose
content is identical to an already processed data blob. A third flag
tracks whether a blob exists or not, which removes the need for the
blobs IDSet.
2020-07-20 22:51:47 +02:00
Michael Eischer
35d8413639 checker: Remove dead index map 2020-07-20 22:37:31 +02:00
Michael Eischer
c66a0e408c checker: Reduce cost of debug log
Avoid duplicate allocation of the Subtree list.
2020-07-20 22:37:31 +02:00
Michael Eischer
70f4c014ef checker: Decode identical tree nodes only once
Even though the checkTreeWorker skips already processed chunks,
filterTrees did queue the same tree blob on every occurence. This
becomes a serious performance bottleneck for larger number of snapshots
that cover mostly the same directories. Therefore decode a tree blob
exactly once.
2020-07-20 22:37:31 +02:00
Michael Eischer
f0d8710611 Add benchmark for checker scaling with snapshot count 2020-07-20 22:37:31 +02:00
MichaelEischer
bd3e280f6d Merge pull request #2546 from Summerdave/fix-backup-error-code
Return an error when errors occured during backup
2020-07-20 22:31:46 +02:00
MichaelEischer
2746dcdb5f Merge pull request #2845 from restic/update-chunker
Update chunker
2020-07-20 22:20:28 +02:00
David Sommer
5729d967f5 backup: Return exit status code 3 when failing to read source data
The backup command used to return a zero exit code as long as a snapshot
could be created successfully, even if some of the source files could not
be read (in which case the snapshot would contain the rest of the files).

This made it hard for automation/scripts to detect failures/incomplete
backups by looking at the exit code. Restic now returns the following exit
codes for the backup command:

 - 0 when the command was successful
 - 1 when there was a fatal error (no snapshot created)
 - 3 when some source data could not be read (incomplete snapshot created)
2020-07-20 22:19:01 +02:00
Alexander Neumann
f9f6124558 Update chunker 2020-07-20 22:07:02 +02:00
Alexander Neumann
8074879c5f Remove 'go generate' 2020-07-19 17:28:42 +02:00
greatroar
7bda28f31f Chaining hash table for repository.Index
These are faster to construct but slower to access. The allocation rate
is halved, the peak memory usage almost halved compared to standard map.

Benchmark results on linux/amd64, -benchtime=3s -count=20:

name                                            old time/op    new time/op    delta
PackerManager-8                                    178ms ± 0%     178ms ± 0%     ~     (p=0.231 n=20+20)
DecodeIndex-8                                      4.54s ± 0%     4.30s ± 0%   -5.20%  (p=0.000 n=18+17)
DecodeIndexParallel-8                              4.54s ± 0%     4.30s ± 0%   -5.22%  (p=0.000 n=19+18)
IndexHasUnknown-8                                 44.4ns ± 5%    50.5ns ±11%  +13.82%  (p=0.000 n=19+17)
IndexHasKnown-8                                   48.3ns ± 0%    51.5ns ±12%   +6.68%  (p=0.001 n=16+20)
IndexAlloc-8                                       758ms ± 1%     616ms ± 1%  -18.69%  (p=0.000 n=19+19)
IndexAllocParallel-8                               234ms ± 3%     204ms ± 2%  -12.60%  (p=0.000 n=20+18)
MasterIndexLookupSingleIndex-8                     122ns ± 0%     145ns ± 9%  +18.44%  (p=0.000 n=14+20)
MasterIndexLookupMultipleIndex-8                   369ns ± 2%     429ns ± 8%  +16.27%  (p=0.000 n=20+20)
MasterIndexLookupSingleIndexUnknown-8             68.4ns ± 5%    74.9ns ±13%   +9.47%  (p=0.000 n=20+20)
MasterIndexLookupMultipleIndexUnknown-8            315ns ± 3%     369ns ±11%  +17.14%  (p=0.000 n=20+20)
MasterIndexLookupParallel/known,indices=5-8        743ns ± 1%     816ns ± 2%   +9.87%  (p=0.000 n=17+17)
MasterIndexLookupParallel/unknown,indices=5-8      238ns ± 1%     260ns ± 2%   +9.14%  (p=0.000 n=19+20)
MasterIndexLookupParallel/known,indices=10-8      1.01µs ± 3%    1.11µs ± 2%   +9.79%  (p=0.000 n=19+20)
MasterIndexLookupParallel/unknown,indices=10-8     222ns ± 0%     269ns ± 2%  +20.83%  (p=0.000 n=16+20)
MasterIndexLookupParallel/known,indices=20-8      1.06µs ± 2%    1.19µs ± 2%  +12.95%  (p=0.000 n=19+18)
MasterIndexLookupParallel/unknown,indices=20-8     413ns ± 1%     530ns ± 1%  +28.19%  (p=0.000 n=18+20)
SaveAndEncrypt-8                                  30.2ms ± 1%    30.4ms ± 0%   +0.71%  (p=0.000 n=19+19)
LoadTree-8                                         540µs ± 1%     576µs ± 1%   +6.73%  (p=0.000 n=20+20)
LoadBlob-8                                        5.64ms ± 0%    5.64ms ± 0%     ~     (p=0.883 n=18+17)
LoadAndDecrypt-8                                  5.93ms ± 0%    5.95ms ± 1%     ~     (p=0.247 n=20+19)
LoadIndex-8                                       25.1ms ± 0%    24.5ms ± 1%   -2.54%  (p=0.000 n=18+17)

name                                            old speed      new speed      delta
PackerManager-8                                  296MB/s ± 0%   296MB/s ± 0%     ~     (p=0.229 n=20+20)
SaveAndEncrypt-8                                 139MB/s ± 1%   138MB/s ± 0%   -0.71%  (p=0.000 n=19+19)
LoadBlob-8                                       177MB/s ± 0%   177MB/s ± 0%     ~     (p=0.890 n=18+17)
LoadAndDecrypt-8                                 169MB/s ± 0%   168MB/s ± 1%     ~     (p=0.227 n=20+19)

name                                            old alloc/op   new alloc/op   delta
PackerManager-8                                   91.8kB ± 0%    91.8kB ± 0%     ~     (p=0.772 n=12+19)
IndexAlloc-8                                       786MB ± 0%     400MB ± 0%  -49.04%  (p=0.000 n=20+18)
IndexAllocParallel-8                               786MB ± 0%     401MB ± 0%  -49.04%  (p=0.000 n=19+15)
SaveAndEncrypt-8                                  21.0MB ± 0%    21.0MB ± 0%   +0.00%  (p=0.000 n=19+19)

name                                            old allocs/op  new allocs/op  delta
PackerManager-8                                    1.41k ± 0%     1.41k ± 0%     ~     (all equal)
IndexAlloc-8                                        977k ± 0%      907k ± 0%   -7.18%  (p=0.000 n=20+20)
IndexAllocParallel-8                                977k ± 0%      907k ± 0%   -7.17%  (p=0.000 n=19+15)
SaveAndEncrypt-8                                    73.0 ± 0%      73.0 ± 0%     ~     (all equal)
2020-07-19 13:58:22 +02:00
greatroar
255ba83c4b Parallel index benchmarks + benchmark optimizations
createRandomIndex was using the global RNG, which locks on every call
It was also using twice as many random numbers as necessary and doing
a float division in every iteration of the inner loop.

BenchmarkDecodeIndex was using too short an input, especially for a
parallel version. (It may now be using one that is a bit large.)

Results on linux/amd64, -benchtime=3s -count=20:

name                                     old time/op    new time/op       delta
PackerManager-8                             178ms ± 0%        178ms ± 0%           ~     (p=0.165 n=20+20)
DecodeIndex-8                              13.6µs ± 2%  4539886.8µs ± 0%  +33293901.38%  (p=0.000 n=20+18)
IndexHasUnknown-8                          44.4ns ± 7%       44.4ns ± 5%           ~     (p=0.873 n=20+19)
IndexHasKnown-8                            49.2ns ± 3%       48.3ns ± 0%         -1.86%  (p=0.000 n=20+16)
IndexAlloc-8                                802ms ± 1%        758ms ± 1%         -5.51%  (p=0.000 n=20+19)
MasterIndexLookupSingleIndex-8              124ns ± 1%        122ns ± 0%         -1.41%  (p=0.000 n=20+14)
MasterIndexLookupMultipleIndex-8            373ns ± 2%        369ns ± 2%         -1.13%  (p=0.001 n=20+20)
MasterIndexLookupSingleIndexUnknown-8      67.8ns ± 3%       68.4ns ± 5%           ~     (p=0.753 n=20+20)
MasterIndexLookupMultipleIndexUnknown-8     316ns ± 3%        315ns ± 3%           ~     (p=0.846 n=20+20)
SaveAndEncrypt-8                           30.5ms ± 1%       30.2ms ± 1%         -1.09%  (p=0.000 n=19+19)
LoadTree-8                                  527µs ± 1%        540µs ± 1%         +2.37%  (p=0.000 n=19+20)
LoadBlob-8                                 5.65ms ± 0%       5.64ms ± 0%         -0.21%  (p=0.000 n=19+18)
LoadAndDecrypt-8                           7.07ms ± 2%       5.93ms ± 0%        -16.15%  (p=0.000 n=19+20)
LoadIndex-8                                32.1ms ± 2%       25.1ms ± 0%        -21.64%  (p=0.000 n=20+18)

name                                     old speed      new speed         delta
PackerManager-8                           296MB/s ± 0%      296MB/s ± 0%           ~     (p=0.159 n=20+20)
SaveAndEncrypt-8                          138MB/s ± 1%      139MB/s ± 1%         +1.10%  (p=0.000 n=19+19)
LoadBlob-8                                177MB/s ± 0%      177MB/s ± 0%         +0.21%  (p=0.000 n=19+18)
LoadAndDecrypt-8                          141MB/s ± 2%      169MB/s ± 0%        +19.24%  (p=0.000 n=19+20)

name                                     old alloc/op   new alloc/op      delta
PackerManager-8                            91.8kB ± 0%       91.8kB ± 0%           ~     (p=0.826 n=19+12)
IndexAlloc-8                                786MB ± 0%        786MB ± 0%         +0.01%  (p=0.000 n=20+20)
SaveAndEncrypt-8                           21.0MB ± 0%       21.0MB ± 0%         -0.00%  (p=0.012 n=20+19)

name                                     old allocs/op  new allocs/op     delta
PackerManager-8                             1.41k ± 0%        1.41k ± 0%           ~     (all equal)
IndexAlloc-8                                 977k ± 0%         977k ± 0%         +0.01%  (p=0.022 n=20+20)
SaveAndEncrypt-8                             73.0 ± 0%         73.0 ± 0%           ~     (all equal)
2020-07-19 13:58:05 +02:00
MichaelEischer
7dc200c593 Merge pull request #2832 from alrs/fix-fuse-test-err
internal/fuse: fix dropped test error
2020-07-16 22:28:20 +02:00
Lars Lehtonen
9ac90cf5cd internal/fuse: fix dropped test error 2020-07-12 21:42:31 -07:00
MichaelEischer
b84f5177cb Merge pull request #2790 from greatroar/fix-quadratic-read
Fix quadratic file reading in restic mount
2020-07-12 18:42:14 +02:00
greatroar
4cf1c8e8da Changelog entry for subquadratic reading in mount 2020-07-12 18:27:16 +02:00
greatroar
58719e1f47 Replace mount's per-file cache by a global LRU cache 2020-07-12 18:27:16 +02:00
greatroar
d42c169458 Fix quadratic file reading in restic mount 2020-07-12 18:27:16 +02:00
rawtaz
8598bb042b Merge pull request #2769 from darkdragon-001/patch-1
Update documentation for environment variables
2020-07-12 00:40:21 +02:00
darkdragon-001
c6b74962df Update 040_backup.rst
Changes proposed in #2763:

- Adding `RESTIC_CACHE_DIR` environment variables (introduced in #2425 for Unix and #2607 for Mac, Win).
- Adding used system-wide environment variables with links to the corresponding section.
2020-07-12 00:31:56 +02:00
MichaelEischer
2c72924ffb Merge pull request #2813 from greatroar/encrypt-benchmark
Fix repository_test.BenchmarkSaveAndEncrypt
2020-07-05 17:52:10 +02:00
greatroar
02bec13ef2 Fix repository_test.BenchmarkSaveAndEncrypt
The benchmark was actually testing the speed of index lookups.

name              old time/op    new time/op        delta
SaveAndEncrypt-8     101ns ± 2%    31505824ns ± 1%    +31311591.31%  (p=0.000 n=10+10)

name              old speed      new speed          delta
SaveAndEncrypt-8  41.7TB/s ± 2%       0.0TB/s ± 1%         -100.00%  (p=0.000 n=10+10)

name              old alloc/op   new alloc/op       delta
SaveAndEncrypt-8     1.00B ± 0%  20989508.40B ± 0%  +2098950740.00%  (p=0.000 n=10+10)

name              old allocs/op  new allocs/op      delta
SaveAndEncrypt-8      0.00             123.00 ± 0%            +Inf%  (p=0.000 n=10+9)

(The actual speed is ca. 131MiB/s.)
2020-07-05 17:41:42 +02:00
MichaelEischer
64976b1a4d Merge pull request #2815 from scelfo/patch-1
--hostname flag is deprecated
2020-06-29 22:04:56 +02:00
Tony Scelfo
6a607d6ded --hostname flag is deprecated
Update documentation to follow the deprecation warning "Flag --hostname has been deprecated, use --host" that is given on the latest build(s).
2020-06-29 12:53:57 -06:00
rawtaz
6fedf1a7f4 Merge pull request #2809 from lessfoobar/patch-2
Update 080_examples.rst
2020-06-26 00:12:05 +02:00
@lessfoobar
df946fd9f8 Update 080_examples.rst 2020-06-26 00:01:30 +02:00
Jakub Wilk
4e6a9767de Fix examples of --verbose with argument 2020-06-23 20:22:14 +02:00
rawtaz
1bc80c3c8d Merge pull request #2802 from andreaso/cachedir-https-links
Use https:// for linking CACHEDIR.TAG specs
2020-06-21 20:01:42 +02:00
Andreas Olsson
0fcef2ec23 Use https:// for linking CACHEDIR.TAG specs
That site might not have supported https:// when those links were
originally added. It does now.

Also dropping the _spec.html_ ending of the url, there being a `<link
rel="canonical" ...>` tag suggesting that that no longer being the
preferred address.
2020-06-21 19:36:26 +02:00
MichaelEischer
212607dc8a Merge pull request #2760 from greatroar/backend-benchmark
Fix backend benchmarks + a micro-optimization
2020-06-17 23:17:05 +02:00
greatroar
55c3a90a0d Clarify rclone-over-SSH docs
Also added a link to S. Ruderich's blog post explaining append-only
repos using rclone and SSH.
2020-06-17 15:22:20 +02:00
greatroar
190d8e2f51 Flatten backend.LimitedReadCloser structure
This inlines the io.LimitedReader into the LimitedReadCloser body to
achieve fewer allocations. Results on linux/amd64:

name                                      old time/op    new time/op    delta
Backend/BenchmarkLoadPartialFile-8           412µs ± 4%     413µs ± 4%    ~     (p=0.634 n=17+17)
Backend/BenchmarkLoadPartialFileOffset-8     455µs ±13%     441µs ±10%    ~     (p=0.072 n=20+18)

name                                      old speed      new speed      delta
Backend/BenchmarkLoadPartialFile-8        10.2GB/s ± 3%  10.2GB/s ± 4%    ~     (p=0.817 n=16+17)
Backend/BenchmarkLoadPartialFileOffset-8  9.25GB/s ±12%  9.54GB/s ± 9%    ~     (p=0.072 n=20+18)

name                                      old alloc/op   new alloc/op   delta
Backend/BenchmarkLoadPartialFile-8            888B ± 0%      872B ± 0%  -1.80%  (p=0.000 n=15+15)
Backend/BenchmarkLoadPartialFileOffset-8      888B ± 0%      872B ± 0%  -1.80%  (p=0.000 n=15+15)

name                                      old allocs/op  new allocs/op  delta
Backend/BenchmarkLoadPartialFile-8            18.0 ± 0%      17.0 ± 0%  -5.56%  (p=0.000 n=15+15)
Backend/BenchmarkLoadPartialFileOffset-8      18.0 ± 0%      17.0 ± 0%  -5.56%  (p=0.000 n=15+15)
2020-06-17 13:11:45 +02:00
greatroar
f4cd2a7120 Make backend benchmarks fairer by removing checks
Checking whether the right data is returned takes up half the time in
some benchmarks. Results for local backend benchmarks on linux/amd64:

name                                      old time/op    new time/op    delta
Backend/BenchmarkLoadFile-8                 4.89ms ± 0%    2.72ms ± 1%   -44.26%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFile-8           936µs ± 6%     439µs ±15%   -53.07%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFileOffset-8     940µs ± 1%     456µs ±10%   -51.50%  (p=0.008 n=5+5)
Backend/BenchmarkSave-8                     23.9ms ±14%    24.8ms ±41%      ~     (p=0.690 n=5+5)

name                                      old speed      new speed      delta
Backend/BenchmarkLoadFile-8               3.43GB/s ± 0%  6.16GB/s ± 1%   +79.40%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFile-8        4.48GB/s ± 6%  9.63GB/s ±14%  +114.78%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFileOffset-8  4.46GB/s ± 1%  9.22GB/s ±10%  +106.74%  (p=0.008 n=5+5)
Backend/BenchmarkSave-8                    706MB/s ±13%   698MB/s ±31%      ~     (p=0.690 n=5+5)
2020-06-17 13:11:45 +02:00
MichaelEischer
aba270df7e Merge pull request #2786 from aawsome/memory-list-blobs
Loop over index files for 'list blobs'
2020-06-15 22:27:29 +02:00
Alexander Weiss
b5543cff5d Loop over index files for 'list blobs'
=> reduces memory consumption a lot!
2020-06-14 17:25:34 +02:00
Alexander Neumann
285b5236c2 Merge pull request #2684 from MichaelEischer/cleanup-output
Cleanup command output code
2020-06-14 11:25:40 +02:00
Michael Eischer
bb1e258bb7 cat/stats: Get stdout from globalOptions 2020-06-14 11:08:11 +02:00
Michael Eischer
182655bc88 Replace fmt.Printf/Println/Fprintf with wrapper functions
cmd/restic/globals.go already provides Printf, Println and Warnf wrapper
which get their output streams from the globalOptions object. This
allows for stream replacements when testing.
2020-06-14 11:08:11 +02:00
MichaelEischer
74bc7141c1 Merge pull request #2781 from aawsome/reduce-index-memory
Reduce index memory
2020-06-14 10:58:28 +02:00
Alexander Weiss
1361341c58 don't save duplicate packIDs when using internal/repository/Index.Store 2020-06-14 07:56:24 +02:00
Alexander Weiss
ce4a2f4ca6 save packIDs and duplicates separately
A side remark to the definition of Index.blob:

Another possibility would have been to use:
blob       map[restic.BlobHandle]*indexEntry

This would have led to the following sizes:
key: 32 + 1 = 33 bytes
value: 8 bytes
indexEntry:  8 + 4 + 4 = 16 bytes
each packID: 32 bytes

To save N index entries, we would therefore have needed:
N * OF * (33 + 8) bytes + N * 16 + N * 32 bytes / BP = N * 82 bytes

More precicely, using a pointer instead of a direct entry is the better memory choice if:
OF * 8 bytes + entrysize < OF * entrysize <=> entrysize > 8 bytes * OF/(OF-1)
Under the assumption of OF=1.5, this means using pointers would have been the better choice
if sizeof(indexEntry) > 24 bytes.
2020-06-14 07:56:21 +02:00
Alexander Weiss
cf979e2b81 make offset and length uint32 2020-06-14 07:50:19 +02:00
Michael Eischer
d92e2c5769 simplify index code 2020-06-14 07:50:19 +02:00
Alexander Weiss
7419844885 add changelog, benchmark, memory calculation 2020-06-14 07:50:15 +02:00
rawtaz
1d66bb9e62 Merge pull request #2784 from MichaelEischer/minio-sha256-changelog
Add changelog for #2709
2020-06-13 18:33:54 +02:00
Michael Eischer
0b2c31b05b Add changelog for #2709 2020-06-13 18:13:36 +02:00
Alexander Weiss
d3c59d18e5 Fix inconsistency of saving/loading config file
Fix saving/loading config file: Always set ID to a zero ID.
2020-06-13 16:30:23 +02:00
MichaelEischer
dd7b4f54f5 Merge pull request #2709 from greatroar/minio-sha256
Use Minio's optimized SHA-256
2020-06-12 23:32:58 +02:00
MichaelEischer
6896c6449b Merge pull request #2779 from greatroar/archiver-comment
Fix up comment on archiver.BlobSaver.Save
2020-06-12 23:04:21 +02:00
MichaelEischer
735a8074d5 Merge pull request #2773 from aawsome/index-uploads+knownblobs
Fix non-intuitive repo behavior
2020-06-12 22:41:04 +02:00
Alexander Weiss
70347e95d5 disable index uploads for prune command
+ modifications of changelog
2020-06-12 09:24:38 +02:00
greatroar
0fa3091c78 Fix up comment on archiver.BlobSaver.Save 2020-06-11 13:40:30 +02:00
Alexander Weiss
91906911b0 Fix non-intuitive repository behavior
- The SaveBlob method now checks for duplicates.
- Moves handling of pending blobs to MasterIndex.
  -> also cleans up pending index entries when they are saved in the index
  -> when using SaveBlob no need to care about index any longer
- Always check for full index and save it when storing packs.
  -> removes the need of an index uploader
  -> also removes the verbose "uploaded intermediate index" messages
- The Flush method now also saves the index
- Fix race condition when checking and saving full/non-finalized indexes
2020-06-11 13:05:23 +02:00
MichaelEischer
fae7f78057 Merge pull request #2755 from greatroar/solaris-workaround
Remove Solaris build workaround in integration tests
2020-06-10 23:30:51 +02:00
MichaelEischer
ac9ec4b990 Merge pull request #2778 from MichaelEischer/fix-unlock-hint
Revive hint to the unlock command if a repository is locked
2020-06-10 21:26:38 +02:00
Michael Eischer
087c770161 Revive hint to the unlock command if a repository is locked
errors.Fatalf wraps a error and just keeps an error message as a string.
This prevents the `restic.IsAlreadyLocked(err)` check from working as
the error is no longer an ErrAlreadyLocked.

Just add an additional remark to the error using `errors.WithMessage`.
2020-06-10 20:58:56 +02:00
MichaelEischer
6856d1e422 Merge pull request #2749 from aawsome/fix-fullindex
Change condition for full index
2020-06-10 20:40:19 +02:00
Alexander Weiss
8c1261ff02 changed condition for full index 2020-06-07 22:00:49 +02:00
rawtaz
26704be17f Merge pull request #2776 from nunoperalta/patch-1
Fix #2774 - make unable to umount message clearer
2020-06-07 17:19:50 +02:00
nunoperalta
2c3360db98 Fix #2774 - make unable to umount message clearer 2020-06-07 15:57:39 +01:00
MichaelEischer
cba6ad8d8e Merge pull request #2711 from greatroar/update-x-net
Update golang.org/x/net for fewer allocations in http2
2020-05-25 20:43:29 +02:00
greatroar
2a3312ac35 Remove Solaris build workaround in integration tests
This was introduced in #1821 because of issues with Go 1.9, but that
compiler version is no longer supported.
2020-05-25 10:33:36 +02:00
greatroar
c35c4e0cbf Update golang.org/x/net for fewer allocations in http2
name                                          old allocs/op  new allocs/op  delta
BackendREST/BenchmarkLoadFile-8                  2.15k ± 0%     2.14k ± 0%  -0.27%  (p=0.002 n=20+20)
BackendREST/BenchmarkLoadPartialFile-8             587 ± 0%       582 ± 0%  -0.88%  (p=0.000 n=20+20)
BackendREST/BenchmarkLoadPartialFileOffset-8       584 ± 0%       579 ± 0%  -0.90%  (p=0.000 n=19+16)

The throughput isn't significantly higher or lower.
2020-05-24 16:39:17 +02:00
MichaelEischer
84475aa3a8 Merge pull request #2730 from greatroar/mount-build-tags
Simplify build tags for restic mount
2020-05-23 20:33:43 +02:00
MichaelEischer
f12f9ae240 Merge pull request #2741 from tbm/typos
Fix typos
2020-05-21 23:40:12 +02:00
Martin Michlmayr
5cc1760fdf Fix typos 2020-05-16 14:05:26 +08:00
MichaelEischer
32ac5486e9 Merge pull request #2732 from restic/rawtaz-update-contributing
Reword parts of Providing Patches in CONTRIBUTING.md
2020-05-12 21:37:08 +02:00
rawtaz
c4336978eb Reword parts of Providing Patches in CONTRIBUTION.md
Primarily makes the request that contributors create a new branch for their changes stand out more.
2020-05-12 14:03:57 +02:00
greatroar
649cbec6c5 Simplify build tags for restic mount
This command can only be built on Darwin, FreeBSD and Linux
(and if we upgrade bazil.org/fuse, only FreeBSD and Linux:
https://github.com/bazil/fuse/issues/224).

Listing the few supported operating systems explicitly here makes
porting restic to new platforms easier.
2020-05-12 11:30:41 +02:00
MichaelEischer
b17bd7f860 Merge pull request #2719 from greatroar/update-x-crypto
Update x/crypto to stop poly1305 segfaulting on ARM
2020-05-09 00:02:18 +02:00
greatroar
68f1e9c524 Update x/crypto to fix poly1305 on ARM
The broken poly1305 implementation for arm64 was removed, fixing the
segfault issue on arm64 (gh-2618). The version for amd64 has improved
performance, which shows up in the internal/repository benchmark:

name              old speed      new speed      delta
SaveAndEncrypt-8   110MB/s ± 2%   113MB/s ± 1%  +2.24%  (p=0.000 n=20+20)
2020-05-05 11:05:35 +02:00
MichaelEischer
1ee2306033 Merge pull request #2717 from MichaelEischer/fix-flaky-password-prompt
backup: Avoid race between password prompt and open repository message
2020-05-01 23:58:55 +02:00
Michael Eischer
c882a92cd6 backup: Avoid race between password prompt and open repository message
`term.Print` sends the output via a channel to a goroutine which
actually prints the message. This may race with the password prompt
printed by `OpenRepository` resulting in a missing prompt.
2020-05-01 23:42:16 +02:00
MichaelEischer
f54db5d796 Merge pull request #2713 from greatroar/unused
Move Index.FindBlob to tests
2020-05-01 20:41:59 +02:00
MichaelEischer
843e7f404e Merge pull request #2716 from MichaelEischer/revert-apfs-test-workaround
Revert "Darwin test fix: allow 1μs timestamp difference"
2020-05-01 18:13:38 +02:00
Michael Eischer
d465b5b9ad Revert "Darwin test fix: allow 1μs timestamp difference"
This reverts commit f3016a9096.
2020-05-01 17:11:30 +02:00
greatroar
9f7cd69f13 Move Index.FindBlob to tests 2020-04-29 10:57:01 +02:00
greatroar
f97a680887 Fix repository benchmarks
BenchmarkLoad{AndDecrypt,Blob} were spending between 38% and 50% of
their time measuring SHA-256 performance in their checks.
2020-04-28 07:57:29 +02:00
greatroar
42a3db05b0 Use Minio's optimized SHA-256
internal/repository benchmarks on an Intel i7-3770k:

name               old speed      new speed       delta
PackerManager-8     209MB/s ± 1%    291MB/s ± 1%  +38.94%  (p=0.008 n=5+5)
SaveAndEncrypt-8    112MB/s ± 1%    135MB/s ± 1%  +20.25%  (p=0.008 n=5+5)
2020-04-28 07:57:18 +02:00
MichaelEischer
070d43e290 Merge pull request #2704 from azak-azkaran/patch-1
Update 030_preparing_a_new_repo.rst
2020-04-23 20:48:34 +02:00
MichaelEischer
d4bd32a37e Merge pull request #2640 from greatroar/simplify-loadblob-usage
Simplify Repository.LoadBlob usage
2020-04-23 20:23:35 +02:00
greatroar
e7d7b85d59 Merge Repository.{LoadBlob,loadBlob}
Pushing the allocation logic down into the former loadBlob body means
that fewer allocations have to be performed:

name              old time/op    new time/op    delta
LoadTree-8           478µs ± 1%     481µs ± 2%    ~     (p=0.315 n=9+10)
LoadBlob-8          11.6ms ± 1%    11.6ms ± 2%    ~     (p=0.393 n=10+10)
LoadAndDecrypt-8    13.3ms ± 3%    13.3ms ± 3%    ~     (p=0.905 n=10+9)
LoadIndex-8         33.6ms ± 2%    33.2ms ± 1%  -1.15%  (p=0.028 n=10+9)

name              old alloc/op   new alloc/op   delta
LoadTree-8          41.2kB ± 0%    41.1kB ± 0%  -0.23%  (p=0.000 n=10+10)
LoadBlob-8          2.28kB ± 0%    2.18kB ± 0%  -4.21%  (p=0.000 n=10+10)
LoadAndDecrypt-8    2.10MB ± 0%    2.10MB ± 0%    ~     (all equal)
LoadIndex-8         5.22MB ± 0%    5.22MB ± 0%    ~     (p=0.631 n=10+10)

name              old allocs/op  new allocs/op  delta
LoadTree-8             652 ± 0%       651 ± 0%  -0.15%  (p=0.000 n=10+10)
LoadBlob-8            24.0 ± 0%      23.0 ± 0%  -4.17%  (p=0.000 n=10+10)
LoadAndDecrypt-8      30.0 ± 0%      30.0 ± 0%    ~     (all equal)
LoadIndex-8          30.2k ± 0%     30.2k ± 0%    ~     (p=0.610 n=10+10)

name              old speed      new speed      delta
LoadBlob-8        86.4MB/s ± 1%  85.9MB/s ± 2%    ~     (p=0.393 n=10+10)
LoadAndDecrypt-8  75.4MB/s ± 3%  75.4MB/s ± 3%    ~     (p=0.858 n=10+9)
2020-04-23 10:04:20 +02:00
greatroar
be5a0ff59f Centralize buffer allocation and size checking in Repository.LoadBlob
Benchmark results for internal/repository:

name              old time/op    new time/op    delta
LoadTree-8           479µs ± 2%     478µs ± 1%   ~     (p=0.780 n=10+9)
LoadBlob-8          11.6ms ± 2%    11.6ms ± 1%   ~     (p=0.631 n=10+10)
LoadAndDecrypt-8    13.2ms ± 2%    13.3ms ± 3%   ~     (p=0.631 n=10+10)

name              old alloc/op   new alloc/op   delta
LoadTree-8          41.2kB ± 0%    41.2kB ± 0%   ~     (all equal)
LoadBlob-8          2.28kB ± 0%    2.28kB ± 0%   ~     (all equal)
LoadAndDecrypt-8    2.10MB ± 0%    2.10MB ± 0%   ~     (all equal)

name              old allocs/op  new allocs/op  delta
LoadTree-8             652 ± 0%       652 ± 0%   ~     (all equal)
LoadBlob-8            24.0 ± 0%      24.0 ± 0%   ~     (all equal)
LoadAndDecrypt-8      30.0 ± 0%      30.0 ± 0%   ~     (all equal)

name              old speed      new speed      delta
LoadBlob-8        86.2MB/s ± 2%  86.4MB/s ± 1%   ~     (p=0.594 n=10+10)
LoadAndDecrypt-8  75.7MB/s ± 2%  75.4MB/s ± 3%   ~     (p=0.617 n=10+10)
2020-04-23 10:04:20 +02:00
azak-azkaran
c5100d5632 Update 030_preparing_a_new_repo.rst
fixed markdown for wasabi region link
2020-04-23 09:24:19 +02:00
MichaelEischer
956a1b0f96 Merge pull request #2294 from BenWiederhake/debugsyms
build: Preserve debug symbols in debug build
2020-04-19 17:26:03 +02:00
Ben Wiederhake
fab626a3df build: Preserve debug symbols in debug and profile build
Signed-off-by: Ben Wiederhake <BenWiederhake.GitHub@gmx.de>
2020-04-19 14:48:40 +02:00
MichaelEischer
4f00564574 Merge pull request #2621 from greatroar/fixes
Some small fixes
2020-04-18 18:07:08 +02:00
MichaelEischer
f77477129f Merge pull request #2584 from greatroar/mount-cache-uid-gid
Cache uid and gid for top directories in internal/fuse
2020-04-18 17:45:14 +02:00
greatroar
2e31120f89 Remove unused argument to restic.fakeFile 2020-04-18 17:40:13 +02:00
greatroar
8fb2c0d3c1 Typo in crypto test name 2020-04-18 17:39:06 +02:00
greatroar
072cf7b02d Fix debug messages in internal/fuse 2020-04-18 17:39:06 +02:00
greatroar
df66daa5c9 Fix context usage in backend tests
Found by go vet. This is also the only complaint is has.
2020-04-18 17:39:06 +02:00
MichaelEischer
9790d8ce1c Merge pull request #2668 from MichaelEischer/fix-stats-blobs-crash
stats: Fix crash in blobs-per-file mode on missing blob
2020-04-18 17:21:26 +02:00
MichaelEischer
16710454f4 Merge pull request #2628 from MichaelEischer/one-element-pack-lists
cache: Don't sort one element pack lists
2020-04-18 17:09:06 +02:00
MichaelEischer
08ec6c9f17 Merge pull request #2677 from MichaelEischer/complain-about-invalid-indexes
rebuild_index: Report invalid packs that were ignored
2020-04-18 16:57:50 +02:00
MichaelEischer
7910ff4c0e Merge pull request #2648 from nairb774/iowritestring
termstatus: Use io.WriteString to output messages.
2020-04-18 13:48:42 +02:00
MichaelEischer
c4da9d1e90 Merge pull request #2638 from greatroar/no-close-in-packer
Don't Close in Packer.Finalize
2020-04-18 13:07:21 +02:00
rawtaz
3ed61987a2 Merge pull request #2695 from restic/rawtaz-doc-exclude-file-tilde
doc: Add note on tilde expansion in exclude files
2020-04-18 13:01:17 +02:00
rawtaz
9dba7a2577 doc: Add note on tilde expansion in exclude files
Explains to the reader that tilde expansion does not work in exclude files, and that they should instead use the $HOME variable.
2020-04-18 12:47:47 +02:00
MichaelEischer
a1352906e2 Merge pull request #2622 from greatroar/optimize-packer-manager
Fix PackerManager benchmark and optimize hashing.Writer
2020-04-18 12:46:34 +02:00
MichaelEischer
b7c0d4d8bf Merge pull request #2644 from greatroar/signal-notify-buffered
Make all signal.Notify channels buffered
2020-04-18 11:31:02 +02:00
MichaelEischer
f033850aa0 Merge pull request #2692 from MichaelEischer/fix-archiveraborttest
archiver: Fix race condition triggered by TestArchiverAbortEarlyOnError
2020-04-13 18:36:48 +02:00
Michael Eischer
bdf7ba20cb archiver: Fix race condition triggered by TestArchiverAbortEarlyOnError
The Save methods of the BlobSaver, FileSaver and TreeSaver return early
on when the archiver is stopped due to an error. For that they select on
both the tomb.Dying() and context.Done() channels, which can lead to a
race condition when the tomb is killed due to an error: The tomb first
closes its Dying channel before canceling all child contexts.
Archiver.SaveDir only aborts its execution once the context was
canceled. When the tomb killing is paused between closing its Dying
channel and canceling the child contexts, this lets the
FileSaver/TreeSaver.Save methods return immediately, however, ScanDir
still reads further files causing the test case to fail.

As a killed tomb always cancels all child contexts and as the Savers
always use a context bound to the tomb, it is sufficient to just use
context.Done() as escape hatch in the Save functions. This fixes the
mismatch between SaveDir and Save.

Adjust the tests to use contexts bound to the tomb for all interactions
with the Savers.
2020-04-13 18:23:17 +02:00
rawtaz
3ee6b8ec63 Merge pull request #2689 from MichaelEischer/fix-background-hang
Fix shutdown hang when restic is started as background job
2020-04-12 22:39:44 +02:00
Michael Eischer
4a400f94bb Add changelog 2020-04-12 22:27:09 +02:00
Michael Eischer
1a1c572bac Fix shutdown hang when restic is started as background job
restic uses a cleanup hook to ensure that it restores the terminal
configuration to a sane state, when restic is interrupted while reading
a password from the terminal. However, this causes a problem, when
restic runs in a background job, as reconfiguring a terminal will cause
a SIGTTOU to be sent to restic pausing it. Therefore, restic seems to
hang on shutdown when it was running in the background.

This commit changes the behavior to only restore the terminal
configuration if restic was interrupted while reading a password from
the terminal. As reading a password from the terminal requires that
restic is in the foreground, this should avoid restic getting stopped.

Fixes #2298
Issue introduced in #402
2020-04-12 22:27:09 +02:00
rawtaz
5a7c27ddb6 Merge pull request #2681 from MichaelEischer/optimize-debug
Reduce memory usage and startup time of debug command
2020-04-04 00:08:14 +02:00
Michael Eischer
fb842759fc debug: don't load the repository index
The existing commands don't need a loaded repository index which can
take several minutes to load on larger repositories.
2020-04-04 00:01:01 +02:00
Michael Eischer
7aa2f8a61e debug: get stdout/stderr from gopts/globalOptions 2020-04-04 00:01:01 +02:00
Michael Eischer
08bf3bae79 debug: explicitly pass stdout to dump functions 2020-04-03 23:32:44 +02:00
rawtaz
e7b741b2d7 Merge pull request #2682 from MichaelEischer/cleanup-cli-paramter-names
Cleanup CLI parameter names for backup / global flags
2020-04-03 21:35:34 +02:00
Michael Eischer
6bee62e346 Update doc excerpts for --help
This adds some previously missing changes and the new paramters names
from the previous commit.
2020-04-03 19:49:06 +02:00
Michael Eischer
bc74cd3ae5 backup/global: Use proper name for command line argument parameters
Several paramters printed a generic "string" or "stringArray" name.
2020-04-03 19:49:04 +02:00
Michael Eischer
90243ed1c4 rebuild_index: Report invalid packs that were ignored 2020-04-02 22:38:31 +02:00
Michael Eischer
0ce81d88b6 stats: Fix crash in blobs-per-file mode on missing blob
In a damaged repository with a missing blob, the error message tried to
dereference the subtreeID field of the current node, which is a file
however. Said field is set to nil for a file thus causing a segfault
when dereferenced.

Fix this by using the actual parentTreeID.
2020-03-27 22:17:54 +01:00
rawtaz
c03bc88b29 Merge pull request #2669 from MichaelEischer/doc-cifs-linux-bug
doc: Warn about compatibility issues with CIFS and restic
2020-03-26 23:01:06 +01:00
Brian Atkinson
b8da7b1f4d termstatus: Use io.WriteString to output messages.
The previous implementation was repeating the implementation that is
found inside of io.WriteString. Simplify by making use of the stdlib's
implementation.
2020-03-26 14:55:00 -07:00
Michael Eischer
f1b4d97945 doc: Warn about compatibility issues with CIFS and restic
On Linux CIFS (SMB) seems to be incompatible with the async preemption
implementation of Go 1.14. CIFS seems not to restart syscalls (open,
read, chmod, readdir, ...) as expected by Go, which sets SA_RESTART for
its signal handler to have syscalls restarted automatically. This leads
to Go passing up lots of EINTR return codes to restic.

See https://github.com/restic/restic/issues/2659 for a detailed explanation.
2020-03-26 21:52:37 +01:00
Peter Schultz
90fc639a67 Allow specifying user and host when adding keys
The username and hostname for new keys can be specified with the new
--user and --host flags, respectively. The flags are used only by the
`key add` command and are otherwise ignored.

This allows adding keys with for a desired user and host without having
to run restic as that particular user on that particular host, making
automated key management easier.

Co-authored-by: James TD Smith <ahktenzero@mohorovi.cc>
2020-03-23 13:11:10 +01:00
rawtaz
2b5a6d255a Merge pull request #2660 from restic/rawtaz-delete-old-issue-template
Delete ISSUE_TEMPLATE.md (not used anymore)
2020-03-21 20:48:39 +01:00
rawtaz
f004dbe605 Delete ISSUE_TEMPLATE.md (not used anymore)
Nowadays the ISSUE_TEMPLATE/ directory and its files are used for this feature.
2020-03-21 20:30:19 +01:00
rawtaz
9efbe98879 Merge pull request #2623 from alrs/internal-restic-err-before-close
internal/restic: close os.File after checking for error
2020-03-18 22:23:20 +01:00
rawtaz
7d9300efca Merge pull request #2637 from greatroar/unused
Remove Go 1.5 compatibility code
2020-03-18 22:22:15 +01:00
rawtaz
c38aaaa768 Merge pull request #2652 from greatroar/upgrade-fuse
Upgrade bazil.org/fuse and remove restic mount --allow-root
2020-03-18 22:21:50 +01:00
greatroar
8941041355 Upgrade bazil.org/fuse to version that fixes FreeBSD opBmap bug 2020-03-18 10:00:39 +01:00
greatroar
18fee4806f Remove broken --allow-root from restic mount 2020-03-17 23:35:06 +01:00
greatroar
47d4d5bf1b Make all signal.Notify channels buffered 2020-03-12 20:59:39 +01:00
greatroar
74a64c47e4 Move testing logic to test file in internal/pack 2020-03-09 14:32:28 +01:00
greatroar
a23e9c86ba Remove closing logic from Packer.Finalize
The method only ever receives *hashing.Writers, which don't implement
io.Closer. These come from packerManager.findPacker and have their
actual writers closed in Repository.savePacker. Moving the closing logic
to hashing.Writer results in "file already closed" errors.
2020-03-09 14:31:45 +01:00
greatroar
4de12bf593 Remove restic.RandReader
math/rand.Rand has implemented Reader since Go 1.6. The repacking tests
are not deterministic, but they weren't before, either.
2020-03-09 10:00:28 +01:00
rawtaz
c542a509f0 Merge pull request #2633 from greatroar/fix-ssh-commandline
Revert "Put host last in SSH command line"
2020-03-08 17:29:41 +01:00
greatroar
8cf3bb8737 Revert "Put host last in SSH command line"
This reverts commit e1969d1e33.
2020-03-08 16:45:33 +01:00
Michael Eischer
b46cc6d57e repository: Don't sort one element pack lists
When loading a blob, restic first looks up pack files containing the
blob. To avoid unnecessary work an already cached pack file is preferred.
However, if there is only a single pack file to choose from (which is
the normal case) sorting the one-element list won't change anything.
Therefore avoid the unnecessary cache check in that case.
2020-03-07 10:26:06 +01:00
Lars Lehtonen
4a2156d3f0 internal/restic: close os.File after checking for error 2020-03-05 16:22:46 -08:00
greatroar
41fee11f66 Micro-optimization for hashing.Writer/PackerManager
name             old time/op    new time/op    delta
PackerManager-8     247ms ± 1%     246ms ± 1%  -0.43%  (p=0.001 n=18+18)

name             old speed      new speed      delta
PackerManager-8   213MB/s ± 1%   214MB/s ± 1%  +0.43%  (p=0.001 n=18+18)

name             old alloc/op   new alloc/op   delta
PackerManager-8    92.2kB ± 0%    91.5kB ± 0%  -0.82%  (p=0.000 n=19+20)

name             old allocs/op  new allocs/op  delta
PackerManager-8     1.43k ± 0%     1.41k ± 0%  -1.67%  (p=0.000 n=20+20)
2020-03-05 22:30:04 +01:00
greatroar
b592614061 Improve PackerManager benchmark
The previous benchmark spent much of its time allocating RNGs and
generating too many random numbers. It now spends 90% of its time
hashing and half of the rest writing to files.

name             old time/op    new time/op    delta
PackerManager-8     319ms ± 1%     247ms ± 1%  -22.48%  (p=0.000 n=20+18)

name             old speed      new speed      delta
PackerManager-8   143MB/s ± 1%   213MB/s ± 1%  +48.63%  (p=0.000 n=10+18)

name             old alloc/op   new alloc/op   delta
PackerManager-8     635kB ± 0%      92kB ± 0%  -85.48%  (p=0.000 n=10+19)

name             old allocs/op  new allocs/op  delta
PackerManager-8     1.64k ± 0%     1.43k ± 0%  -12.76%  (p=0.000 n=10+20)
2020-03-05 22:30:03 +01:00
greatroar
b7c3039eb2 Remove Go 1.5 compatibility code from PackerManager benchmark
This alone is enough to speed up the benchmark by ~10%.
2020-03-05 22:29:06 +01:00
Alexander Neumann
a307797c11 Merge pull request #2612 from restic/rawtaz-maintainer-pr
Add notes about maintainer edit access in PRs
2020-03-01 21:24:34 +01:00
rawtaz
a03f107144 Add note about allowing maintainers edits in PRs. 2020-03-01 21:12:11 +01:00
rawtaz
a141ab1bda Add maintaner edit checkbox to PR template 2020-03-01 21:05:52 +01:00
Alexander Neumann
52abec967f Merge pull request #2605 from middelink/fix-2604
Fix running tests on a SELinux enabled system
2020-03-01 20:25:28 +01:00
Pauline Middelink
2828a9c2b0 Fix running tests on a SELinux enabled system
Archivers TestMetadataChanged incorrectly clears the Extended Attributes
from the expected metadata of the temporary file. This is incorrect as on
SELinux enabled filesystem, as the kernel will automaticly add a SElinux
label. However, since ExtendedAttributes{} != ExtendedAttributes{nil} we
still need to clear them if there are no attributes found.
2020-03-01 20:23:22 +01:00
rawtaz
cec7d581f3 Merge pull request #2610 from restic/remove-vendor
Remove vendored dependencies
2020-03-01 20:22:45 +01:00
Alexander Neumann
12aa1e61da CI: Enable Go Module Proxy 2020-03-01 19:58:20 +01:00
Alexander Neumann
95da6c1c1d Merge pull request #2589 from greatroar/no-stable-sort
Replace sort.Stable by sort.Strings
2020-03-01 19:40:28 +01:00
Alexander Neumann
0c03a80fc4 Merge pull request #2592 from greatroar/sftp-ipv6
Support IPv6 in SFTP backend
2020-03-01 19:38:44 +01:00
Alexander Neumann
b1c77172c2 Add entry to changelog 2020-03-01 19:32:35 +01:00
Alexander Neumann
c0373cd307 Remove -mod=vendor from all documentation and code 2020-03-01 19:32:35 +01:00
Alexander Neumann
28121090c2 Remove vendor from CI tests 2020-03-01 19:32:35 +01:00
Alexander Neumann
44a57d66c3 Remove vendor from build scripts 2020-03-01 11:30:02 +01:00
Alexander Neumann
266f9dbe16 Remove vendor dir 2020-03-01 11:20:42 +01:00
greatroar
60e4a88f17 Changelog entry for SFTP w/ IPv6 addresses 2020-03-01 11:04:06 +01:00
Alexander Neumann
c50f91b7f9 Merge pull request #2602 from greatroar/no-bufpool
Remove sync.Pool from internal/repository
2020-03-01 10:50:57 +01:00
Alexander Neumann
e851d29565 Merge pull request #2608 from greatroar/simplify-termstatus
Simplify termstatus
2020-03-01 10:48:29 +01:00
Alexander Neumann
b67b7ebfe6 Merge pull request #2583 from greatroar/unused
Remove some unused or duplicated code
2020-03-01 10:46:17 +01:00
greatroar
751eba0e68 Remove unnecessary pipe checks in termstatus
canUpdateStatus has already determined that the file descriptor is not a
pipe.
2020-02-29 18:03:49 +01:00
greatroar
7447c44484 Use golang.org/x/sys/windows in termstatus
Some functionality is missing, but at least the types are all defined.
Replaced short, word, dword by their Go names to match the x/sys
convention.
2020-02-29 18:03:49 +01:00
greatroar
c8a672fa29 Remove code copy-pasted from x/crypto/ssh/terminal 2020-02-29 18:03:49 +01:00
greatroar
863ba76494 Drop mattn/go-isatty in favor of crypto/ssh/terminal 2020-02-29 18:03:47 +01:00
greatroar
8526cc6647 Remove sync.Pool from internal/repository
The pool was used improperly, causing more allocations to be
performed than without it.

name              old time/op    new time/op    delta
SaveAndEncrypt-8    36.8ms ± 2%    36.9ms ± 2%    ~     (p=0.218 n=10+10)

name              old speed      new speed      delta
SaveAndEncrypt-8   114MB/s ± 2%   114MB/s ± 2%    ~     (p=0.218 n=10+10)

name              old alloc/op   new alloc/op   delta
SaveAndEncrypt-8    21.1MB ± 0%    21.0MB ± 0%  -0.44%  (p=0.000 n=10+10)

name              old allocs/op  new allocs/op  delta
SaveAndEncrypt-8      79.0 ± 0%      77.0 ± 0%  -2.53%  (p=0.000 n=10+10)
2020-02-29 17:54:46 +01:00
rawtaz
694b7a17e7 Merge pull request #2607 from greatroar/cachedir
Honor RESTIC_CACHE_DIR on Mac and Windows
2020-02-28 20:41:32 +01:00
greatroar
5cd0bce452 Honor RESTIC_CACHE_DIR on Mac and Windows 2020-02-28 15:44:32 +01:00
rawtaz
58bd165253 Merge pull request #2581 from aawsome/multiple-hostnames
Allow multiple hostnames tags
2020-02-27 08:35:23 +01:00
rawtaz
65d3fb6b33 Merge pull request #2603 from greatroar/restorer-waitgroup
Fix unsafe sync.WaitGroup usage in restorer.fileRestorer
2020-02-27 00:30:59 +01:00
rawtaz
f165048172 Merge pull request #2582 from greatroar/no-go1.9
Remove Go 1.9 compatibility code
2020-02-27 00:15:06 +01:00
greatroar
de5516a90e Fix sync.WaitGroup usage in restorer.fileRestorer 2020-02-27 00:07:49 +01:00
greatroar
4f6fd9fb98 Remove remnant of Go 1.9 compatibility code from tests 2020-02-26 22:23:38 +01:00
Alexander Weiss
9a9101d144 Support specifying multiple host flags for various commands
The `dump`, `find`, `forget`, `ls`, `mount`, `restore`, `snapshots`,
`stats` and `tag` commands will now take into account multiple
`--host` and `-H` flags.
2020-02-26 22:17:59 +01:00
rawtaz
616f9499ae Merge pull request #2600 from restic/update-go
Update Go version to >= 1.11, add Go 1.14
2020-02-26 21:29:30 +01:00
Alexander Neumann
a40ac37550 Add changelog file 2020-02-26 20:55:12 +01:00
Alexander Neumann
99fd80a585 Remove all workarounds for Go < 1.11 2020-02-26 20:35:13 +01:00
Alexander Neumann
2464f7c4d1 Update Go versions, drop Go 1.10 2020-02-26 20:35:13 +01:00
rawtaz
b5c7778428 Merge pull request #2195 from ifedorenko/out-of-order-restore-no-progress
restorer: allow writing target file blobs out of order
2020-02-26 20:07:51 +01:00
Igor Fedorenko
c52198d12c restorer: go mod vendor; go mod tidy
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2020-02-26 16:15:04 +01:00
Igor Fedorenko
f17ffa0283 restorer: Allow writing target file blobs out of order
Much simpler implementation that guarantees each required pack
is downloaded only once (and hence does not need to manage
pack cache). Also improves large file restore performance.

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2020-02-26 16:14:45 +01:00
greatroar
5e2afd91e7 Assert that archiver.Tree implements fmt.Stringer 2020-02-26 11:05:38 +01:00
greatroar
79b882e901 Merge duplicated readdir functionality
internal/archiver.readdir and internal/fs.ReadDir were unused.

internal/fs.ReadDirNames and internal/archiver.readdirnames were doing
nearly the same thing, except one sorted its output and opened with
fs.O_NOFOLLOW. Both were only used in internal/archiver.
2020-02-26 11:05:38 +01:00
greatroar
1b502fa9ef Cache uid and gid for top directories in internal/fuse 2020-02-24 10:46:09 +01:00
greatroar
e1969d1e33 Put host last in SSH command line
This is how the SSH manpage says the command line should look, and the
"--" prevents mistakes in hostnames from being interpreted as options.
2020-02-19 15:53:20 +01:00
greatroar
6ac6bca7a1 Support IPv6 in SFTP backend
The previous code was doing its own hostname:port splitting, which
caused IPv6 addresses to be misinterpreted.
2020-02-19 15:42:12 +01:00
greatroar
3a6feb0596 Replace sort.Stable by sort.Strings
Calling the slow, O(n lg² n) sort.Stable is equivalent to sort.Strings
for a slice of unique strings.
2020-02-18 19:41:06 +01:00
greatroar
2f8aa2ce30 Remove unused fs.FS from archiver.FileSaver 2020-02-18 10:39:14 +01:00
rawtaz
f2bf06a419 Merge pull request #2540 from dp-github/issue-2531
Count hard-linked files correctly in stats command
2020-02-15 21:06:46 +01:00
David Potter
71900248ab Correct #2531 (stats restore-size calculates hard links incorrectly) 2020-02-15 19:34:22 +01:00
David Potter
23055aaadf Correct #2537 (cmd_stats file counting issue) 2020-02-15 19:34:16 +01:00
rawtaz
8bf6a3af97 Merge pull request #2285 from curiousleo/bugfix/2281-json-output
Fix JSON Printf issues with format verbs
2020-02-15 02:00:57 +01:00
curiousleo
27d6e5e186 Add changelog file. 2020-02-15 01:32:43 +01:00
curiousleo
4214b1746e Ensure Print{,f,ln} use global stdout 2020-02-15 01:32:43 +01:00
curiousleo
f6f240573a Don't Printf already formatted output
Fixes #2281.
2020-02-15 01:32:43 +01:00
rawtaz
ecdf49679e Merge pull request #2579 from MichaelEischer/fix-flaky-archiver-test
Fix flaky TestArchiverAbortEarlyOnError
2020-02-15 01:29:37 +01:00
Michael Eischer
e1f722d266 archiver: Fix flaky TestArchiverAbortEarlyOnError
Each of the random test files was split into the same five blobs. The
test fails once the fifth blob is passed on to `SaveBlob`. That is for
certain interleavings of goroutine execution it would be possible for
the test to trigger the testErr just after storing the first file.

The fixed test uses a different file content for each of the nine files
and fails after writing the fourth blob. The file content is also small
enough to ensure that for each file only a single blob is saved. This
guarantees that the test cannot fail before reading the first four
files. FileReadConcurrency = 2 allows up to two files queued for
processing. Therefore the test can at most open the sixth file before it
has to save the fourth file / blob which triggers the testErr.
2020-02-14 23:16:13 +01:00
rawtaz
2d47381b1d Merge pull request #2545 from MichaelEischer/fix-racy-backup-json
Fix racy json output for backup command
2020-02-13 22:33:17 +01:00
Michael Eischer
294ca55967 Add changelog entry 2020-02-13 21:14:20 +01:00
Michael Eischer
78c518ccac Print backup summary after status output is shutdown 2020-02-13 21:14:20 +01:00
Michael Eischer
42a3292bcf Better name for jsonstatus package
internal/ui/jsonstatus and termstatus sound similar but are not related
in any way. Instead `internal/ui/backup` and `internal/ui/jsonstatus/status`
are the counterparts. Rename the latter to `internal/ui/json/backup` to
make this clear.
2020-02-13 21:14:20 +01:00
Michael Eischer
ef70a2fcb3 Fix mangled JSON output by backup command
jsonstatus wrote the JSON output without synchronization to the
stdio_wrapper which caused mangling between different status lines.

Use the Print and Error methods of termstatus instead which use a
central goroutine to synchronize output.
2020-02-13 21:14:20 +01:00
rawtaz
af20015725 Merge pull request #2287 from benchti/patch-1
doc: Correct parameter order in dump example
2020-02-13 01:54:10 +01:00
rawtaz
680a14afa1 Merge pull request #2530 from restic/fix-sftp-mkdirall
sftp: Use MkdirAll provided by the client
2020-02-13 01:12:59 +01:00
rawtaz
bf199e5ca7 Minor speling corection. 2020-02-13 00:47:09 +01:00
rawtaz
ae127a412d Merge pull request #2337 from rfjakob/favicon
docs: switch to gopher favicon
2020-02-13 00:14:33 +01:00
Jakob Unterwurzacher
5ae7e6f945 docs: switch to gopher favicon
The favicon on restic.readthedocs.io still contained the old
superman-style logo.

This changes it to the 32x32 gopher icon also used on https://forum.restic.net/ ,
just converted to .ico using Gimp.

Tested by building the documentation and opening index.html in Chrome.
The new favicon looks fine.
2020-02-13 00:09:37 +01:00
rawtaz
d8da9c4401 Merge pull request #2577 from alrs/fix-internal-errs
internal: Fix code and test dropped errors
2020-02-12 23:41:54 +01:00
rawtaz
daf3bfc882 Merge pull request #2461 from rigtorp/man-exit-status
Add documentation on exit status codes to man pages
2020-02-12 23:17:49 +01:00
Erik Rigtorp
94f4f13388 Add documentation on exit status codes to man pages
This is step one to start defining useful exit codes for all the commands.
2020-02-12 23:09:26 +01:00
rawtaz
4615bdfd70 Merge pull request #2576 from restic/update-chunker
Update the chunker
2020-02-12 22:41:36 +01:00
rawtaz
299f5971f2 Merge pull request #2560 from brualan/master
Two small improvements to code quality
2020-02-12 22:40:20 +01:00
rawtaz
7a1352ae87 Merge pull request #2570 from MichaelEischer/close-file-on-type-change
Close file if file type has changed after initial stat
2020-02-12 22:39:22 +01:00
Lars Lehtonen
72734d59b5 internal/archiver: fix dropped error 2020-02-12 13:37:37 -08:00
rawtaz
3679669aae Merge pull request #2574 from alrs/fix-dropped-cmd-err
cmd/restic: fix dropped error
2020-02-12 22:36:45 +01:00
Lars Lehtonen
3ed54e762e internal/backend/sftp: fix dropped test error 2020-02-12 13:36:21 -08:00
Lars Lehtonen
fea835b4e2 internal/repository: fix dropped test error 2020-02-12 13:33:54 -08:00
Lars Lehtonen
16b321b140 internal/restic: fix dropped test error 2020-02-12 13:32:45 -08:00
Alexander Neumann
b58dfda212 Add entry to changelog 2020-02-12 21:29:26 +01:00
Alexander Neumann
b229aa5cf1 Update the chunker 2020-02-12 21:29:26 +01:00
Alexander Neumann
81c0b031f9 Add link to the forum in 'new issue' dialog 2020-02-12 21:25:48 +01:00
Michael Eischer
760863e7f9 archiver: Fix TestRacyFileSwap on windows 2020-02-11 21:09:47 +01:00
Michael Eischer
a135699397 Close file if file type has changed after initial stat 2020-02-11 21:09:47 +01:00
Lars Lehtonen
94a4d45dfb cmd/restic: fix a dropped error 2020-02-11 09:19:03 -08:00
rawtaz
4de384087d Merge pull request #2575 from alrs/fix-travis
restic/internal/ui: fix gofmt nit that popped up in Go 1.13
2020-02-11 13:49:13 +01:00
Lars Lehtonen
57815d9cd6 restic/internal/ui: fix gofmt nit that popped up in Go 1.13 2020-02-10 11:06:47 -08:00
rawtaz
644673bcf2 Merge pull request #2573 from dhoffend/fix-processed-bytes
fix backup --json total_bytes_processed output
2020-02-10 13:35:55 +01:00
Daniel Hoffend
e7cdf2acbb fix backup --json total_bytes_processed output
Closes #2429
2020-02-09 02:02:45 +01:00
rawtaz
0f22f008ed Merge pull request #2552 from iambryancs/update-to-go1.13
Update Go version to 1.13.6 in Docker build script
2020-02-07 15:24:37 +01:00
rawtaz
c184da21d0 Merge pull request #2568 from flofeld/patch-1
corrected option s3.storage-class in documentation
2020-02-06 23:34:36 +01:00
Florian Feldmann
ef5efc6a82 corrected option s3.storage-class
Took me some time to figure it out
2020-02-06 22:53:47 +01:00
Alexander Bruyako
688014487b avoid "index out of range" error
in case of len(format) == 0, we will get an
"index out of range" error
Usage of strings.HasSuffix is allowing to avoid that
2020-01-27 19:24:42 +03:00
Alexander Bruyako
38ddfbc4d3 simpler error return 2020-01-27 18:41:46 +03:00
Alexander Bruyako
da48b925ff remove unnecessary error return
I was running "golangci-lint" and found this two warnings

internal/checker/checker.go:135:18: (*Checker).LoadIndex$3 - result 0 (error) is always nil (unparam)
        final := func() error {
                        ^
internal/repository/repository.go:457:18: (*Repository).LoadIndex$3 - result 0 (error) is always nil (unparam)
        final := func() error {
                        ^

It turns out that these functions are used only in "RunWorkers(...)",
which is used only two times in whole project right after this "final"
functions.
And because these "final" functions always return "nil", I've
descided, that it would be better to remove requriments for "final" func
to return error to avoid magick "return nil" at their end.
2020-01-27 18:28:21 +03:00
iambryancs
4bcd56fbae Update Go version to 1.13.6 in Docker build script 2020-01-17 02:13:11 +08:00
Alexander Neumann
6e9778ae0e Add changelog file 2020-01-06 21:46:22 +01:00
Alexander Neumann
63c67be908 Add hint abouth absolute sftp paths for Synology NAS 2020-01-06 21:37:57 +01:00
rawtaz
d70a4a9350 Merge pull request #2373 from vrusinov/issue-2372
Ignore username difference in TestMetadataChanged
2020-01-05 21:07:34 +01:00
Alexander Neumann
2cd9c7ef16 sftp: Use MkdirAll provided by the client
Closes #2518
2020-01-01 17:26:38 +01:00
Alexander Neumann
6ec5dc8016 Fix template for new version of calens
The `wrap` function has been renamed to `wrapIndent`
2020-01-01 11:58:21 +01:00
rawtaz
fe430a680a Merge pull request #2524 from cure/patch-1
Fix typos in documentation of FixTime
2019-12-22 16:36:23 +01:00
Ward Vandewege
69a0d0ee90 fix typos 2019-12-21 21:00:28 -05:00
rawtaz
4557881066 Merge pull request #2423 from streambinder/master
sftp: support user@domain parsing as user
2019-12-19 22:39:36 +01:00
streambinder
9b5d069ade doc: 030: sfpt: hint for domain-confined users 2019-12-19 13:15:37 +01:00
streambinder
c56cbfe95b changelog: add pull-2423 2019-12-19 13:15:37 +01:00
streambinder
97e5ce4344 internal: backend: sftp: support user@domain parsing as user 2019-12-19 13:15:37 +01:00
rawtaz
72bd467e23 Merge pull request #2271 from lorenzbausch/feature/find_timestamp
Display snapshot date when using `restic find`
2019-12-17 10:35:59 +01:00
Lorenz Bausch
d818b7618b Display respective snapshot date when using restic find 2019-12-17 07:51:12 +01:00
Matt Holt
1c3812a6f6 Merge pull request #2342 from rpsene/master
Add support for ppc64le
2019-12-16 11:50:52 -07:00
Rafael Peria de Sene
ec91b80f09 Add support for ppc64le
This commit adds support for cross-compiling Restic for ppc64le
as part of the release process.

Add a new file:   changelog/unreleased/issue-2277
Modifies:   helpers/build-release-binaries/main.go
Modifies:   run_integration_tests.go
2019-12-15 22:52:13 -03:00
rawtaz
18a336c154 Merge pull request #2510 from rawtaz/kastenhq-changelog-2390
Add changelog for #2390.
2019-12-10 23:31:20 +01:00
Eric Hamilton
d21a13f1ee Create issue-2390 2019-12-10 23:20:18 +01:00
rawtaz
e0eac30ee5 Merge pull request #2509 from restic/improve-check-docs
doc: Improve check documentation
2019-12-10 23:07:23 +01:00
rawtaz
fccb579471 doc: Improve check documentation
Make it clearer what the difference is between regular check and check with --read-data.
2019-12-10 21:15:59 +01:00
Alexander Neumann
6c700f95b5 Merge pull request #2358 from filippobottega/master 2019-11-29 20:27:30 +01:00
Filippo Bottega
95d070c147 FAQ: Add hint for antivirus setup
Add a tip to configure antivirus to exclude restic process.
2019-11-29 20:26:20 +01:00
rawtaz
da4473aba6 Merge pull request #2391 from kastenhq/refresh-lock-time
Update Lock.Time in lock.Refresh()
2019-11-26 21:44:29 +01:00
rawtaz
6e85a58045 Merge pull request #2465 from whs-dot-hk/fix-test-metadata-changed
Fix test metadata changed
2019-11-26 21:42:42 +01:00
rawtaz
476e2e8762 Merge pull request #2354 from HugoReeves/master
Doc Addition: Wasabi repo setup instructions
2019-11-25 13:56:49 +01:00
Alexander Neumann
246bb46e82 Correct CHANEGLOG.md 2019-11-24 15:18:25 +01:00
Alexander Neumann
fe0361ffcb Move changelog entry for #2179
For background on how this happened see
https://forum.restic.net/t/psa-attention-anyone-not-using-a-recent-master-ie-0-9-5-version-restic-will-not-detect-changed-files-if-mtime-is-reset-by-the-application-eg-ms-excel-and-xls-xlsx-files-on-macos/1772/8?u=fd0
2019-11-24 15:15:19 +01:00
rawtaz
952469473f Merge pull request #2486 from rawtaz/changelog-gitkeep
changelog: Add .gitkeep to persist unreleased/ folder
2019-11-24 14:59:39 +01:00
rawtaz
02108f202e Merge pull request #2487 from rawtaz/2469-fix-diff-stats
diff: Fix wrong bytes reported in diff stats (#2469)
2019-11-23 12:15:28 +01:00
Alexander Neumann
b02357f542 Fix message 2019-11-23 10:31:40 +01:00
Alexander Neumann
f2aeaef8f1 Improve questions 2019-11-23 10:19:08 +01:00
Alexander Neumann
547702d285 Remove coverage badge 2019-11-23 10:19:08 +01:00
rawtaz
599831f874 Merge pull request #2489 from restic/improve-forget-docs
Improve example for forget --keep-daily
2019-11-22 21:20:35 +01:00
Alexander Neumann
14c90d9e85 Improve example for forget --keep-daily
Following up on https://github.com/restic/restic/pull/2406
2019-11-22 20:44:50 +01:00
Alexander Neumann
c41bbb3761 Re-formulate the question in the issue template
Closes #2410
2019-11-22 20:39:47 +01:00
rawtaz
8a54a0f5ac Merge pull request #2488 from rawtaz/changelog-template
changelog: Fix typo in TEMPLATE
2019-11-22 18:23:02 +01:00
Leo R. Lundgren
c0a5530950 changelog: Fix typo in TEMPLATE. 2019-11-22 17:32:56 +01:00
Leo R. Lundgren
77ef92b95c diff: Fix wrong bytes reported in diff stats (#2469) 2019-11-22 17:26:01 +01:00
Leo R. Lundgren
69f75bbf70 changelog: Add .gitkeep to persist the unreleased/ folder. 2019-11-22 17:14:17 +01:00
Alexander Neumann
30519f01ff Set development version for 0.9.6 2019-11-22 16:19:03 +01:00
Alexander Neumann
b723ca3de5 Add version for 0.9.6 2019-11-22 16:18:59 +01:00
Alexander Neumann
f5084d70d7 Generate CHANGELOG.md for 0.9.6 2019-11-22 16:18:55 +01:00
Alexander Neumann
29b7b17491 Prepare changelog files for 0.9.6 2019-11-22 16:18:33 +01:00
rawtaz
e14c4b1737 Merge pull request #2484 from restic/add-s3-region
s3: Allow specifying region
2019-11-22 15:51:17 +01:00
Alexander Neumann
745d79fe5f Merge pull request #2483 from restic/update-minio-go
Update minio-go
2019-11-22 15:31:12 +01:00
Alexander Neumann
fb95426f64 Rename environment variable to AWS_DEFAULT_REGION
This seems to be the correct name, at least the AWS cli uses it:
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html
2019-11-22 15:24:42 +01:00
Alexander Neumann
4cadc89ad3 Add documentation and changelog 2019-11-22 15:16:59 +01:00
Alexander Neumann
409909a7f5 Add option description for Region 2019-11-22 15:09:09 +01:00
mdauphin
df500a372d Add AWS_REGION env var to specify s3 region 2019-11-22 15:04:48 +01:00
Alexander Neumann
a444731dc0 Update vendored dependencies 2019-11-22 14:57:56 +01:00
Alexander Neumann
a6e8af7e0f Update minio-go 2019-11-22 14:50:46 +01:00
Alexander Neumann
aa5af8af0e Merge pull request #2478 from restic/update-ci
Update Go version for Appveyor/Travis
2019-11-22 14:36:38 +01:00
Alexander Neumann
4e3353109d Reset changes by 'go get' 2019-11-22 13:17:52 +01:00
Alexander Neumann
02c8d38095 Fix error check for findGoFiles 2019-11-22 12:20:27 +01:00
Alexander Neumann
fd6211653c More debug for findImports 2019-11-22 12:16:46 +01:00
rawtaz
3d4f2dd6b4 Merge pull request #2368 from rlue/doc/backblaze
Clarify docs for Backblaze B2 support
2019-11-22 11:10:22 +01:00
Ryan Lue
c1ddc0c18b Clarify docs for Backblaze B2 support 2019-11-22 18:07:29 +08:00
rawtaz
c95f032a9c Merge pull request #2310 from vincentbernat/fix/doc-contributing
Update documentation for contributing
2019-11-21 20:30:49 +01:00
Vincent Bernat
3087776135 Update documentation for contributing
- No need to checkout into `GOPATH` anymore
 - `CHANGELOG.md` shouldn't be updated directly
2019-11-21 20:27:48 +01:00
rawtaz
b6f01ffbe6 Merge pull request #2394 from bugvillage/doc-backup-passwd
add description env var RESTIC_PASSWORD_COMMAND
2019-11-20 23:26:55 +01:00
rawtaz
41fe9318b1 Merge pull request #2425 from thiell/restic_cache_dir_env
Add support for $RESTIC_CACHE_DIR
2019-11-20 21:18:57 +01:00
Alexander Neumann
8387d18d4d Update Go version used for CI tests 2019-11-20 21:13:56 +01:00
rawtaz
929d2b8df3 Merge pull request #2479 from rawtaz/sing-pulitzer-group
Accept both singular and plural for --group-by (#2330)
2019-11-20 21:03:05 +01:00
Leo R. Lundgren
4f0682d730 Correct name of changelog file for #2321 2019-11-20 21:01:10 +01:00
Alexander Neumann
967d1bbf0c Run 'go mod tidy' 2019-11-20 20:54:43 +01:00
Alexander Neumann
2f80b37b93 Update go-autorest so restic builds with Go 1.13 2019-11-20 20:53:57 +01:00
Leo R. Lundgren
4d2aa18273 Accept both singular and plural for --group-by (#2330) 2019-11-20 19:25:48 +01:00
Alexander Neumann
6b1e5d4e18 Update Go version for Appveyor/Travis 2019-11-20 18:32:58 +01:00
rawtaz
26d1f9f4ba Merge pull request #2307 from gary-kim/fix-2306-password-retries
Allow multiple retries for interactive password input
2019-11-20 18:30:20 +01:00
rawtaz
6a89c0f0ef Merge pull request #2476 from rawtaz/update-doc
doc: Simplify wording for check's --read-data-subset parameter.
2019-11-19 03:38:37 +01:00
rawtaz
b87230b93d Merge pull request #2456 from jenting/installation-opensuse
Docs: add installation guide for openSUSE
2019-11-19 02:15:19 +01:00
JenTing Hsiao
6f2b8d622a Add installation guide for openSUSE
Signed-off-by: JenTing Hsiao <jenting.hsiao@suse.com>
2019-11-19 09:08:18 +08:00
Leo R. Lundgren
90440212f2 doc: Simplify wording for check's --read-data-subset parameter. 2019-11-19 00:43:41 +01:00
rawtaz
3a5c9aadad Merge pull request #2444 from chanibal/patch-2
Docs: fixed escaping of argument in documentation
2019-11-18 21:45:00 +01:00
rawtaz
a78142c1bb Merge pull request #2450 from hallamjeff/patch-1
Update 080_examples.rst
2019-11-18 21:43:19 +01:00
rawtaz
07045c7e23 Merge pull request #2442 from lbausch/patch-1
Docs: Highlight option correctly
2019-11-18 21:39:07 +01:00
rawtaz
0a5d42db3f Merge pull request #2463 from whs-dot-hk/fix-ci-rclone
Update rclone repo
2019-11-18 21:37:02 +01:00
Jeff Hallam
67d99b8cfb Update 080_examples.rst
added backticks around the -o and --option text.
2019-11-18 16:36:20 -04:00
Alexander Neumann
1a0c0dc277 Remove codecov.io
While it was a nice idea, some tests like the backend integration tests
required credentials which were not available to third-party PRs. This
lead to a lot of uncovered code which confused people. So let's disable
codecov.io for now.
2019-11-18 21:21:40 +01:00
rawtaz
e86d9307d0 Merge pull request #2472 from rawtaz/update-backup-doc
doc: Improve exclude/include patterns info
2019-11-18 21:04:18 +01:00
rawtaz
923e681af3 Merge pull request #2471 from 8176135/patch-1
Small fix to the forget --tag command documentation
2019-11-18 20:40:24 +01:00
Leo R. Lundgren
37770b1d82 doc: Improve include patterns info
Clarifies and adds information on how to specify include patterns.
2019-11-18 02:23:01 +01:00
rawtaz
02fea4f76a doc: Improve exclude patterns info
Improves grammar, clarifies and adds information on how to specify exclude patterns.
2019-11-18 01:59:50 +01:00
whs
7cacba0394 Assume WithAtime default to false 2019-11-06 16:38:46 +08:00
whs
e6db3596f1 Update rclone repo 2019-11-06 11:59:48 +08:00
8176135
3acc7af310 Small fix to the forget --tag command
As described in #2460
2019-11-03 11:48:56 +13:00
Jeff Hallam
5c4653f427 Update 080_examples.rst
Added an explanation of how to specify the AWS S3 storage classes and what the default value is.
2019-10-21 19:24:50 -03:00
Krzysztof Bociurko
f7317a9287 Docs: fixed escaping of argument in documentation
Issue was visible as `foo`` in https://restic.readthedocs.io/en/latest/040_backup.html
2019-10-16 19:03:40 +02:00
lbausch
30db8057e4 Docs: Highlight option correctly 2019-10-13 21:19:03 +02:00
Stephane Thiell
0e897ef7b8 Add support for $RESTIC_CACHE_DIR
Add support for restic-specific $RESTIC_CACHE_DIR environment variable
to override the cache directory like --cache-dir would have.
2019-09-26 15:59:56 -07:00
bugvillage
b3e727f40d fixing typo/align around RESTIC_PASSWORD_COMMAND 2019-09-07 15:30:23 +02:00
bugvillage
17feccd998 add description env var RESTIC_PASSWORD_COMMAND
The environment variable RESTIC_PASSWORD_COMMAND works but has
not been documented yet. e.g. it could contain a command that
would fetch the password from a local user keyring

enhances: https://github.com/restic/restic/pull/2094
2019-09-07 14:00:53 +02:00
Eric Hamilton
1596d06f8e Update Lock.Time in lock.Refresh() 2019-09-04 11:38:35 -07:00
Vladimir Rusinov
db20c0b8d0 Ignore username difference in TestMetadataChanged.
In some (rare) cases "fake" UID 51234 may exist in a system running a
test. When this is the case, `cmp.Equal(want, node3)` will fail based on
difference between empty string and an actual username present in a
system.

Fixes github issue #2372
2019-08-13 22:25:00 +01:00
Alexander Neumann
604b18aa74 build: Fix building in ~/go
Before, build.go only unset GO111MODULE and GOPATH, so the Go compiler
did not see either and worked in Module mode. But if the code is checked
out below ~/go (the default GOPATH), it will detect that the source is
within GOPATH and switch to non-Module mode. Now we're setting
GO111MODULE to "on" explicitly.
2019-07-29 09:22:22 +02:00
Alexander Neumann
01c51b3449 Merge pull request #2321 from restic/fix-index-error-handling
Check errors returned by LoadIndex()
2019-07-27 09:37:18 +02:00
Alexander Neumann
de8cf5e345 Merge pull request #2324 from jkahrs/fix/change-cleanup-cache-message
updade message that is shown when a cache cleanup is advised
2019-07-27 09:37:10 +02:00
Alexander Neumann
cfa2ac69e0 Merge pull request #2322 from brualan/master
minor refactoring
2019-07-27 09:37:06 +02:00
Hugo Reeves
3ca306d69a Doc Addition: Added Wasabi repo setup 2019-07-26 22:00:48 +12:00
Alexander Neumann
1e9eefa066 Debian stable has restic 0.9.4 (#2333)
Debian stable has restic 0.9.4
2019-07-11 08:17:40 +02:00
Michael M. Chang
e9af012229 Debian has updated restic
Debian now has restic 0.9.4+ds-2+b1 in stable (buster)
2019-07-10 11:53:17 -04:00
Jean Kahrs
8066e93f47 updade message that is shown when a cache cleanup is advised 2019-07-02 10:49:49 +02:00
Alexandr Bruyako
e19622e4b1 start using fileMode const 2019-07-01 00:26:00 +03:00
Alexandr Bruyako
38ea7ed4f6 remove unused code 2019-07-01 00:24:45 +03:00
Alexandr Bruyako
76d1866444 avoiding unnecessary type conversions 2019-06-30 23:58:00 +03:00
Alexandr Bruyako
8b22fe29cf improved slice copying 2019-06-30 23:56:36 +03:00
Alexandr Bruyako
02014be76c simplified prefix removal, removed unnecessary if-else statements 2019-06-30 23:34:47 +03:00
Alexandr Bruyako
16eeed2ad5 simplified string sorting by using a more suitable function 2019-06-30 23:20:32 +03:00
Alexander Neumann
3f94f63967 Add entry to changelog 2019-06-30 21:38:10 +02:00
Alexander Neumann
88716794e3 Check errors returned by LoadIndex()
Bug was reported in the forum here: https://forum.restic.net/t/check-rebuild-index-prune/1848/13
2019-06-30 21:34:53 +02:00
Fred Akalin
3ca424050f Add upper bound for t in --read-data-subset=n/t (#2304)
* Add upper bound for t in --read-data-subset=n/t

* Add changelog entry
2019-06-29 14:34:53 -04:00
Gary Kim
fea2464d4d Allow multiple retries for interactive password input
Restic used to quit if the repository password was typed incorrectly once.
Restic will now ask the user again for the repository password if typed incorrectly.
The user will now get three tries to input the correct password before restic quits.
2019-06-13 20:11:02 +08:00
Ben
5d272e5c08 Dump : parameter order in sample
Usage:
  restic dump [flags] snapshotID file
2019-05-28 18:20:52 +02:00
Alexander Neumann
5bd5db4294 Merge pull request #2206 from garrmcnu/fs-stdin-directory
fs: Handle absolute pathname for --stdin-filename
2019-05-08 16:05:37 +02:00
Garry McNulty
4429a66b5f backup: Convert relative pathname for --stdin-filename to absolute (#2063) 2019-05-08 15:21:27 +02:00
Garry McNulty
8066195e6e fs: Handle absolute pathname for --stdin-filename
Return valid directory info from Lstat() for parent directories of the
specified filename. Previously only "/" and "." were valid directories.

Also set directory mode as this is checked by archiver.

Closes #2063
2019-05-08 15:21:27 +02:00
Garry McNulty
f7f14cf8c9 fs: Add file info base name check in reader tests (#2063) 2019-05-08 15:21:27 +02:00
Garry McNulty
5096f3b491 fs: Update directory check in reader tests (#2063) 2019-05-08 15:21:27 +02:00
Alexander Neumann
cf3fc2a5b1 Merge pull request #2266 from restic/fix-windows-tests
Fix tests on Windows
2019-05-05 21:34:04 +02:00
Alexander Neumann
920d458a4a archiver: Use untyped constants for testing FileInfo 2019-05-05 14:57:38 +02:00
Alexander Neumann
b016dc2ff0 archiver/Windows: Skip test new-content-same-filestamp 2019-05-05 14:02:11 +02:00
Alexander Neumann
355db0bc29 windows: Use LastWriteTime for ctime and mtime
Windows does not have a concept of a `change time` in the sense as Unix
has it: the field `CreationTime` of the `Win32FileAttributeData` struct
is not updated when attributes or content is changed. So from now on
we're using the `LastWriteTime` as the `change time` on Windows.
2019-05-05 14:02:11 +02:00
Alexander Neumann
6e2fe73189 archiver: Move tests back into the same file
Move all Archiver tests back into `archiver_test.go` and add some tiny
helpers to mock what `lstat` returns (for Windows and Unix separately).
2019-05-05 14:02:11 +02:00
Alexander Neumann
303a5dab6a archiver: Clarify value in test struct
Since I could not remember what the value for `Check` means this commit
renames it to `SameFile`: when set to true, the test should make sure
that `FileChanged` should return false (=file is unmodified).
2019-05-05 12:57:00 +02:00
Alexander Neumann
7dcd2968b6 Merge pull request #2261 from lorenzbausch/patch-1
Fix typo
2019-05-04 10:23:49 +02:00
Lorenz Bausch
298f490195 Fix typo 2019-05-02 11:51:35 +02:00
Alexander Neumann
37cb82b28b Merge pull request #2257 from gliptak/windows1
Don't run TestMetadataChanged test on Windows
2019-04-28 08:39:00 +02:00
Gábor Lipták
bce6438d22 Don't run TestMetadataChanged test on Windows
Signed-off-by: Gábor Lipták <gliptak@gmail.com>
2019-04-27 21:23:47 -04:00
Alexander Neumann
919dd2ac84 Merge pull request #2252 from restic/fix-2249
Read fresh metadata for unmodified files
2019-04-25 09:15:50 +02:00
Alexander Neumann
870bc5108e Merge pull request #2253 from restic/fix-2174
Make sure timestamps are valid
2019-04-25 09:14:43 +02:00
Alexander Neumann
418296c5c9 Add hint for "closes" wording to PR template 2019-04-25 09:00:15 +02:00
Alexander Neumann
a6481b3707 Merge pull request #2212 from cbane/check-ctime
Examine file ctime when checking if files have changed.
2019-04-25 08:58:27 +02:00
Courtney Bane
00b527fb09 Update changelog text, and add pull request link. 2019-04-24 20:54:15 -05:00
Courtney Bane
0ebfc55ee3 Use existing setTimestamp function for ctime test and improve error checking. 2019-04-24 20:53:08 -05:00
Courtney Bane
35b7607802 Don't check ctime when ignoring inode. 2019-04-24 20:53:08 -05:00
Alexander Neumann
fad9f65c65 Merge pull request #2251 from restic/update-deps
Update dependencies
2019-04-24 16:42:57 +02:00
Alexander Neumann
939f3e972c node: Make sure year of all timestamps is valid
Sometimes restic gets bogus timestamps which cannot be converted to
JSON, because the stdlib JSON encoder returns an error if the year is
not within [0, 9999]. We now make sure that we at least record _some_
timestamp and cap the year either to 0000 or 9999. Before, restic would
refuse to save the file at all, so this improves the status quo.

This fixes #2174 and #1173
2019-04-24 16:39:36 +02:00
Alexander Neumann
ca8c3b4fd5 Update dependencies 2019-04-24 15:17:48 +02:00
Alexander Neumann
4f45b14f25 Add changelog file 2019-04-24 15:17:25 +02:00
Alexander Neumann
389067fb8b Only use list of blobs for old node
Closes #2249
2019-04-24 15:07:26 +02:00
Alexander Neumann
4b0ca9ddab Add test for #2249 2019-04-24 15:07:23 +02:00
Courtney Bane
b8c2544dcb Examine file ctime when checking if files have changed. 2019-04-23 21:54:35 -05:00
Alexander Neumann
c7762453cf Set development version for 0.9.5 2019-04-23 13:19:43 +02:00
Alexander Neumann
303210aa08 Add version for 0.9.5 2019-04-23 13:19:39 +02:00
Alexander Neumann
c029881379 Update manpages and auto-completion 2019-04-23 13:19:39 +02:00
Alexander Neumann
6e89963c21 Generate CHANGELOG.md for 0.9.5 2019-04-23 13:19:36 +02:00
Alexander Neumann
1ac560181b Prepare changelog for 0.9.5 2019-04-23 13:19:23 +02:00
Alexander Neumann
18ec27a0da Merge pull request #2243 from treymo/ls-length-2104
backend find: Check prefix length < snapshot IDs when searching
2019-04-23 09:41:19 +02:00
Alexander Neumann
b40dea29ad Merge pull request #2247 from tbm/docs
Minor docs fixes
2019-04-23 09:17:22 +02:00
Martin Michlmayr
0561155963 Split paragraph from list in restic stats --help
The help text for `restic stats` lists a number of modes in a list.
Make sure the "more info" text is a separate paragraph rather than
being part of the list.
2019-04-23 10:36:33 +07:00
Martin Michlmayr
1aafc17212 Avoid error "undefined control sequence" in LaTeX
Converting the changelog to PDF using pandoc leads to:

! Undefined control sequence.
l.1497 ...mple, by creating a file named ``..\test

This is because \t is interpreted as a control sequence.  Use ``
instead of "" to work around this.
2019-04-23 10:36:33 +07:00
Martin Michlmayr
f11789c437 Add missing opening parenthesis 2019-04-23 10:36:26 +07:00
Alexander Neumann
8cab0c121d Merge pull request #2221 from classmarkets/2203-password-from-stdin
Fix reading password from stdin
2019-04-22 20:59:59 +02:00
Alexander Neumann
5979414bcd Merge pull request #2124 from Kidswiss/tar
Ability to dump folders to tar via stdout
2019-04-22 20:58:59 +02:00
Simon Beck
cc8b690b52 Restore whole folder to sdtout as tar
With this change it is possible to dump a folder to stdout as a tar. The

It can be used just like the normal dump command:

`./restic dump fa97e6e1 "/data/test/" > test.tar`

Where `/data/test/` is a a folder instead of a file.
2019-04-22 20:44:33 +02:00
Alexander Neumann
a164dc9391 Merge pull request #2087 from ArcticXWolf/add_group_by_option_for_snapshots
Add GroupBy option to snapshots command
2019-04-22 20:27:24 +02:00
Jan Niklas Richter
9a26be4e5b Add documentation for --group-by flag for snapshots command 2019-04-22 14:57:53 +02:00
Jan Niklas Richter
733519d895 Refactor duplicate code for grouping snapshots
This commit is a followup to the addition of the --group-by flag for the
snapshots command. Adding the grouping code there introduced duplicated
code (the forget command also does grouping). This commit refactors
boths sides to only use shared code.
2019-04-22 14:57:53 +02:00
Jan Niklas Richter
3d5a0c799b Add changelog of group-by option for snapshots command 2019-04-22 14:37:58 +02:00
Jan Niklas Richter
c4475ac58f Move snapshot grouping code into own function to deduplicate code
This commit moves the code which is used to group snapshots in the
snapshots command into an own function to deduplicate code shared by the
snapshots command and forget command.
2019-04-22 14:37:58 +02:00
Jan Niklas Richter
c9fd9b5275 Fix json tags for grouped snapshot output
This commit will add json tags to the structs for json output, so all
json variables of the snapshot command output are lowercase and
snake-case.

Furthermore it adds some internal code changes based on the feedback in
the pull request #2087.
2019-04-22 14:37:58 +02:00
Jan Niklas Richter
cadcab5a19 Add GroupBy option to snapshots command
This commit adds a --group-by option to the snapshots command, which
behaves similar to the --group-by option of forget. Valid option values
are "host, paths, tags". If this option is given, the output of
snapshots will be divided into multiple tables, according to the value
given (i.e. "host" will create a table of snapshots for each host, that
has a snapshot in the list). Also the JSON output will be grouped.

The default behavior (when --group-by is not given) has not changed.

More to this discussion can be found in issue #2037.
2019-04-22 14:37:58 +02:00
Trey Moore
5ac9c1157a Prevent "slice bounds out of range" error if prefix is longer than snapshot IDs.
This includes tests as well as other tests for the backend_find.Find function. Issue #2104.
2019-04-16 20:26:20 -07:00
Peter Schultz
5715517e29 Fix reading password from stdin
Reading the password from non-terminal stdin used io.ReadFull with a
byte slice of length 1000.

We are now using a Scanner to read one line of input, independent of its
length.

Additionally, if stdin is not a terminal, the password is read only
once instead of twice (in an effort to detect typos).

Fixes #2203

Signed-off-by: Peter Schultz <peter.schultz@classmarkets.com>
2019-04-15 10:05:49 +02:00
Alexander Neumann
ecc2458de8 Merge pull request #2230 from cdhowie/issue-2229
Extend find --show-pack-id to work with --tree
2019-04-14 19:38:38 +02:00
Alexander Neumann
2c6ba5d9ac Merge pull request #2231 from cdhowie/issue-2224
find command: don't abort on tree load errors
2019-04-14 19:31:06 +02:00
Alexander Neumann
0cc3647e51 Remove changelog entry, not relevant for end-users 2019-04-14 19:29:12 +02:00
Alexander Neumann
6b700d02f5 Merge pull request #2217 from restic/improve-memory-usage
WIP: improve memory usage
2019-04-13 15:07:07 +02:00
Alexander Neumann
2b09a10234 Merge pull request #2218 from kcthrn/patch-1
Fix minor error in doc for forgetting snapshots
2019-04-13 14:55:40 +02:00
Alexander Neumann
1c87d01bad Merge pull request #2220 from BenoitKnecht/s3-storage-class
Add config option to set S3 storage class
2019-04-13 14:54:32 +02:00
Alexander Neumann
78a3ffcfb9 Merge pull request #2185 from d3zd3z/json-forget
Json forget
2019-04-13 14:23:32 +02:00
Alexander Neumann
4d77c0c21c Merge pull request #2228 from cstegm/patch-1
Update 030_preparing_a_new_repo.rst
2019-04-13 14:20:01 +02:00
Alexander Neumann
fb064afa34 Merge pull request #2232 from jacksgt/document_cachedir-tag
Mention Cache Directoy Tagging Standard in man page and help text
2019-04-13 14:17:38 +02:00
Alexander Neumann
7304738872 check: Reduce default parallelism from 40 to 5 2019-04-13 13:38:39 +02:00
Alexander Neumann
66efa425bf Reuse buffer in worker functions 2019-04-13 13:38:39 +02:00
Alexander Neumann
d51e9d1b98 Add []byte to repo.LoadAndDecrypt and utils.LoadAll
This commit changes the signatures for repository.LoadAndDecrypt and
utils.LoadAll to allow passing in a []byte as the buffer to use. This
buffer is enlarged as needed, and returned back to the caller for
further use.

In later commits, this allows reducing allocations by reusing a buffer
for multiple calls, e.g. in a worker function.
2019-04-13 13:38:39 +02:00
Alexander Neumann
e046428c94 Replace FilesInParallel with an errgroup.Group 2019-04-13 13:38:39 +02:00
Alexander Neumann
75906edef5 mock: Remove unused repository type 2019-04-13 13:38:39 +02:00
Alexander Neumann
203d775190 restic: Make JSON unmarshal for ID more efficient
This commit reduces several allocations in UnmarshalJSON() by decoding
the hex string directly in a single step.
2019-04-13 13:38:39 +02:00
Jack Henschel
ecd7ee85e8 Mention Cache Directoy Tagging Standard in man page and help text
This patch makes it more explicit what is meant by the CACHEDIR.TAG file.
It not only has to have this particular name, but also a specific content
(described at http://bford.info/cachedir/spec.html), which is not immediately
obvious to the user.
2019-04-02 15:53:07 +02:00
Alexander Neumann
2022355800 doc: Fix versions in self-update example 2019-03-30 09:44:59 +01:00
Chris Howie
36f22a0feb find command: don't abort on tree load errors 2019-03-29 22:58:12 -04:00
Chris Howie
f58a44b911 Extend find --show-pack-id to work with --tree 2019-03-29 22:56:36 -04:00
cstegm
fe886a6439 Update 030_preparing_a_new_repo.rst
Helps users having this issue: https://github.com/restic/restic/issues/2216
2019-03-29 18:24:17 +01:00
Benoît Knecht
be23313072 changelog: Add entry for s3.storage-class option 2019-03-26 16:39:27 +01:00
Benoît Knecht
3c112d9cae s3: Add config option to set storage class
The `s3.storage-class` option can be passed to restic (using `-o`) to
specify the storage class to be used for S3 objects created by restic.

The storage class is passed as-is to S3, so it needs to be understood by
the API. On AWS, it can be one of `STANDARD`, `STANDARD_IA`,
`ONEZONE_IA`, `INTELLIGENT_TIERING` and `REDUCED_REDUNDANCY`. If
unspecified, the default storage class is used (`STANDARD` on AWS).

You can mix storage classes in the same bucket, and the setting isn't
stored in the restic repository, so be sure to specify it with each
command that writes to S3.

Closes #706
2019-03-26 16:37:07 +01:00
kcthrn
2970e38d92 Fix minor error in doc for forgetting snapshots 2019-03-24 17:59:19 -04:00
Alexander Neumann
870e7583a1 Merge pull request #2210 from restic/update-deps
Update dependencies
2019-03-17 13:38:34 +01:00
Alexander Neumann
db1c835c37 Update dependencies 2019-03-17 12:13:53 +01:00
Alexander Neumann
190bed9908 Merge pull request #2208 from restic/update-go
Update Go for CI tests
2019-03-17 11:36:06 +01:00
Alexander Neumann
85f4c826db Update Go for CI tests 2019-03-17 11:06:16 +01:00
Alexander Neumann
5da4b0fc7d Merge pull request #2209 from restic/fix-freebsd
Fix build on FreeBSD for Go < 1.12
2019-03-17 11:05:38 +01:00
Alexander Neumann
c1058005c3 Fix build on FreeBSD for Go < 1.12 2019-03-16 20:52:37 +01:00
Alexander Neumann
ca73808649 Merge pull request #2197 from dmgk/master
Fix build on FreeBSD with Go 1.12
2019-03-16 20:23:19 +01:00
Alexander Neumann
f2ea91df38 Merge pull request #2189 from ifedorenko/filewrite-race
Fix rare 'file already closed' during restore
2019-03-16 19:53:18 +01:00
Alexander Neumann
15cc4d74b2 Run 'go mod tidy' and 'go mod vendor' 2019-03-16 17:01:20 +01:00
Igor Fedorenko
bf9a507148 Fix rare 'file already closed' during restore
Fixes #2183

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2019-03-16 17:01:20 +01:00
Alexander Neumann
65b476ead9 Fix gofmt 2019-03-16 13:29:05 +01:00
Alexander Neumann
aaa1cc2c26 Merge pull request 2193 from restic/allow-empty-rclone-args
rclone: Rework backend option parsing
2019-03-16 12:17:38 +01:00
Alexander Neumann
95434cff16 Merge pull request 2156 from kayrus/swift-app-cred
Swift: introduce application credential auth support
2019-03-16 12:17:17 +01:00
Alexander Neumann
1b94ae1c00 Merge pull request 2205 from heikobornholdt/ignore-inode
Add --ignore-inode option to backup cmd
2019-03-16 12:16:31 +01:00
Heiko Bornholdt
d138b38f28 Add pull-2205 to changelog 2019-03-10 21:37:01 +01:00
Heiko Bornholdt
db8f5864fc Add --ignore-inode option to backup cmd
revised version of https://github.com/restic/restic/pull/2047
2019-03-10 21:24:29 +01:00
Dmitri Goutnik
1d8b21cdad Fix build on FreeBSD with Go 1.12 2019-03-06 09:41:49 -05:00
Alexander Neumann
3865b59716 rclone: Rework backend option parsing
This change allows passing no arguments to rclone, using `-o
rclone.args=""`. It is helpful when running rclone remotely via SSH
using a key with a forced command (via `command=` in `authorized_keys`).
2019-03-02 10:36:42 +01:00
Alexander Neumann
7b8d1dc040 Merge pull request 2187 from restic/fix-2181
self-update: Don't cancel download after 30s
2019-02-24 21:21:55 +01:00
David Brown
d19a29f79e Include reasons in json output of forget
This dumps the reasons as well as the list of keeps and removes with the
output from the forget command.
2019-02-23 09:38:33 -07:00
David Brown
449c049ce9 Add changelog for #2184 2019-02-23 09:37:38 -07:00
Alexander Neumann
9f436d80e1 Add changelog file 2019-02-23 11:16:08 +01:00
Alexander Neumann
e277a92a2f self-update: Don't cancel download after 30 seconds 2019-02-23 11:15:18 +01:00
David Brown
d9e22c2df1 Add test for --json support for forget command
This adds a test of the json output of the forget command, by running it
once, asking it to keep one snapshot, and verifying that the output has
the right number of snapshots listed in the Keep and Remove fields of
the result.
2019-02-22 22:00:20 -07:00
David Brown
4b0fb5af36 Add --json support to forget command
Fixes #2184
2019-02-22 22:00:15 -07:00
Alexander Neumann
7519c73987 Clarify documentation on --exclude-if-present 2019-02-22 15:22:29 +01:00
Alexander Neumann
45a48eb4a8 Merge pull request 2171 from tbm/typos
Fix typos
2019-02-21 19:46:10 +01:00
Martin Michlmayr
a2f30cde4c Fix typos 2019-02-17 15:31:07 +00:00
kayrus
6ebcfe7c18 Swift: introduce application credential auth support 2019-02-14 14:19:05 +01:00
Alexander Neumann
0022926eba Merge pull request 2167 from moritzdietz/master
Update dead link in docs
2019-02-11 21:10:44 +01:00
Alexander Neumann
3e3a0220ec Merge pull request 2151 from restic/update-deps
Update dependencies
2019-02-11 20:27:17 +01:00
Moritz Dietz
c125fb763d Update dead link 2019-02-11 20:18:43 +01:00
Alexander Neumann
b9f0f031b6 Update dependencies
Closes #2129
2019-02-10 13:24:37 +01:00
Alexander Neumann
aa7043151a Merge pull request 2147 from kilburn/macports
Document installation with MacPorts
2019-02-10 12:50:52 +01:00
Alexander Neumann
ebf22a35f4 Merge pull request 2154 from ciderpunx/patch-1
Remove stray "in" from b2 instructions
2019-02-10 12:50:11 +01:00
Alexander Neumann
3f069ac404 Merge pull request 2153 from dmd/umount-message
more accurate umount help message, for #815
2019-02-10 12:43:48 +01:00
Alexander Neumann
56e5467096 Merge pull request 2032 from j6s/feature.case-insensitive-exclude
Add options for case insensitive includes & excludes
2019-02-10 12:40:33 +01:00
Alexander Neumann
5ee932a124 Add changelog for #2139 2019-02-10 12:35:41 +01:00
Alexander Neumann
fed25714a4 Merge pull request 2139 from restic/fail-zero-bytes-stdin
Return error when reading zero byte from stdin
2019-02-10 12:34:05 +01:00
Alexander Neumann
8906d85ab8 Merge pull request 2138 from restic/fix-2136
Use processed bytes for summary
2019-02-10 12:32:50 +01:00
Alexander Neumann
97aafc1eec Add changelog file for #1944 2019-02-10 12:31:18 +01:00
Alexander Neumann
6a5c9f57c2 Merge pull request 1944 from mholt/jsonprogress
backup: Support --json flag by streaming JSON to report progress
2019-02-10 12:29:50 +01:00
Matthew Holt
6cf13483b5 Add snapshot ID to summary output 2019-02-10 12:09:09 +01:00
Matthew Holt
f645306a18 Initial implementation of JSON progress reporter 2019-02-10 12:09:09 +01:00
ciderpunx
186e10e0cb Remove stray "in" from b2 instructions 2019-01-29 12:39:46 +00:00
Daniel M. Drucker
29a5bd5b30 more accurate umount help message, for #815 2019-01-28 16:53:35 -05:00
Marc Pujol
06a01bc016 Document installation with MacPorts 2019-01-21 08:13:53 +01:00
Johannes Hertenstein
cdc287a7f6 Store reference to reject function for insensitive pattern rejection 2019-01-19 11:08:13 +00:00
Johannes Hertenstein
deedc38129 Only convert iexclude & iinclude casing once 2019-01-19 11:04:53 +00:00
Alexander Neumann
1107eef215 Merge pull request 2142 from restic/fix-2140
Print message in verbose mode only
2019-01-12 13:30:20 +01:00
Alexander Neumann
60c7020bcb Print message in verbose mode only
Closes #2140
2019-01-10 21:28:22 +01:00
Alexander Neumann
b96ef48562 Set development version for 0.9.4 2019-01-06 21:40:32 +01:00
Alexander Neumann
cd9b2295f1 Add version for 0.9.4 2019-01-06 21:40:32 +01:00
Alexander Neumann
a439cdeb05 Update manpages and auto-completion 2019-01-06 21:40:32 +01:00
Alexander Neumann
827f6d7b24 Generate CHANGELOG.md for 0.9.4 2019-01-06 21:40:31 +01:00
Alexander Neumann
77ab10d401 Create changelog dir for 0.9.4 2019-01-06 21:39:56 +01:00
Alexander Neumann
3b0ad2e368 release: Pull latest alpine image 2019-01-06 21:39:56 +01:00
Alexander Neumann
2996c110f1 Merge pull request 2017 from kylewlacy/fuse_default_permissions_option
mount: Add default-permissions flag to set FUSE option
2019-01-06 21:19:27 +01:00
Alexander Neumann
4609b5c24d Merge pull request 2094 from juergenhoetzel/password-command
Add support for reading password from external command
2019-01-06 21:14:52 +01:00
Alexander Neumann
830511460a mount: Turn on DefaultPermissions for --allow-other`
This commit changes the logic slightly: checking the permissions in the
fuse mount when nobody else besides the current user can access the fuse
mount does not sense. The current user has access to the repo files in
addition to the password, so they can access all data regardless of what
the fuse mount does.

Enabling `--allow-root` allows the root user to access the files in the
fuse mount, for this user no permission checks will be done anyway.

The code now enables `DefaultPermissions` automatically when
`--allow-other` is set, it can be disabled with
`--no-default-permissions` to restore the old behavior.
2019-01-06 20:55:49 +01:00
Alexander Neumann
0dc3648416 Add documentation for --password-command 2019-01-06 20:41:38 +01:00
Alexander Neumann
d71dba3788 Merge pull request 2130 from andreaso/docs-https-links
Prefer https:// links in the documentation
2019-01-06 20:24:42 +01:00
Alexander Neumann
e482633943 Merge pull request 2137 from kevinawoo/patch-1
doc: Add more clarity to B2 application key
2019-01-06 20:24:00 +01:00
Alexander Neumann
900621051a Merge pull request 2120 from turnkey-commerce/fix-doc-typo
Fix a few typos in the FAQ doc.
2019-01-06 15:08:13 +01:00
Alexander Neumann
1f246c5309 Merge pull request 2090 from plumbeo/within-hours
Increase the granularity of the "keep within" snapshot retention policy
2019-01-06 15:04:55 +01:00
Alexander Neumann
e40805b002 Merge pull request 2103 from cdhowie/master
--one-file-system unsupported on Windows
2019-01-06 14:59:46 +01:00
Alexander Neumann
6f69ae1b8d Add docs, changelog file 2019-01-06 14:52:20 +01:00
Alexander Neumann
c4fbf2c779 Return error when reading zero byte from stdin
This commit changes the internal file system implementation for reading
data from stdin, it now returns an error when no bytes could be read. I
think it's worth failing in this case, the user instructed restic to
read some data from stdin, and no data was read at all. Maybe it was in
a pipe and some earlier stage failed.

See #2135 for a short discussion.
2019-01-06 14:52:20 +01:00
Alexander Neumann
7c084014fa Use processed bytes for summary
When restic reads the backup from stdin, the number of bytes processed
was always displayed as zero. The reason is that the UI for the archive
uses the total bytes as returned by the scanner, which is zero for
stdin. So instead we keep track of the real number of bytes processed
and print that at the end.

Closes #2136
2019-01-06 13:15:23 +01:00
Kevin Woo
d65bea1b2a doc: Add more clarity to B2 application key 2019-01-05 20:05:11 -08:00
Andreas Olsson
3b68acf853 Prefer https:// links in the documentation
All the changed urls are available by way of https://. Most of them
already redirect.
2018-12-29 15:57:44 +01:00
James Culbertson
82a70643a2 Fix a few typos in the FAQ doc. 2018-12-18 08:47:52 -06:00
Alexander Neumann
0dd805421e Merge pull request 2108 from mholt/noexitprint
cmd: Don't exit if printing to stdout or stderr fails (closes #2102)
2018-12-15 21:34:18 +01:00
Alexander Neumann
16b82f4b1d Merge pull request 2111 from andreaso/doc-moot-sudo
Remove moot sudo usage from documentation
2018-12-15 21:33:51 +01:00
Andreas Olsson
7a6bfcd58c Remove moot sudo usage from documentation
That usage of sudo only applies to the echo command itself. It has no
effect on the permissions of the stdout redirect.
2018-12-09 17:43:50 +01:00
Matthew Holt
de54618852 cmd: Don't exit if printing to stdout or stderr fails (closes #2102) 2018-12-01 15:11:22 -07:00
Alexander Neumann
98526b8dbe Merge pull request 2100 from restic/fix-1989
gs: Respect bandwidth limiting
2018-11-28 21:39:42 +01:00
Alexander Neumann
0083680d33 Merge pull request 2099 from restic/fix-1229
cache: Don't return an error for truncated files
2018-11-28 21:39:35 +01:00
Chris Howie
05222b7343 --one-file-system unsupported on Windows 2018-11-27 11:14:47 -05:00
Kyle Lacy
d4ff5b6bf4 Add changelog entry about "DefaultPermissions" change 2018-11-26 21:16:01 -08:00
Kyle Lacy
cf0883e16c mount: Add "no-default-permissions" option
This option restores the previous behavior of `mount` by disabling the "DefaultPermissions" FUSE option. This allows any user that can access the mountpoint to read any file from the snapshot. Normal FUSE rules apply, so `allow-root` or `allow-other` can be used to allow users besides the mounting user to access these files.
2018-11-26 21:16:01 -08:00
Kyle Lacy
a35a24b8b4 mount: Enable "DefaultPermissions" FUSE option by default
This enforces the Unix permissions of the snapshot files within the mounted filesystem, which will only allow users to access snapshot files if they had access to the file outside of the snapshot.
2018-11-26 21:16:01 -08:00
Juergen Hoetzel
df7f72cdde Add support for reading password from external command
This allows reading the password from an password manager (like "pass").

Signed-off-by: Juergen Hoetzel <juergen@archlinux.org>
2018-11-26 20:18:31 +01:00
plumbeo
3edc723bf0 Update 'restic forget --keep-within' documentation and add changelog entry 2018-11-26 14:27:48 +01:00
plumbeo
71891b340c Support time ranges expressed in hours in snapshot retention policies
Make restic forget --keep-within accept time ranges measured in hours and choose
accordingly which snapshots to keep and which to forget. Add relative tests.
2018-11-26 14:27:42 +01:00
Alexander Neumann
6f5c3e57f6 Update vendor/modules.txt 2018-11-25 19:22:57 +01:00
Alexander Neumann
56af0ce370 Add entry to changelog 2018-11-25 19:07:42 +01:00
Alexander Neumann
c9745cd47e gs: Respect bandwidth limiting
In 0dfdc11ed9, accidentally we dropped
using the provided http.RoundTripper, this commits adds it back.

Closes #1989
2018-11-25 18:52:32 +01:00
Alexander Neumann
2434ab2106 Merge pull request 2098 from cdhowie/key-hinting
Add key hinting (#2097)
2018-11-25 17:52:54 +01:00
Chris Howie
1688713400 Add key hinting (#2097) 2018-11-25 09:13:18 -05:00
Alexander Neumann
00597284de cache: Don't return an error for truncated files
Closes #1229
Closes #1328
2018-11-25 14:18:02 +01:00
Johannes Hertenstein
879f6e0c81 Add changelog file for insensitive includes & excludes 2018-11-25 13:32:16 +01:00
Johannes Hertenstein
8a97bb8661 Add documentation for insensitive includes & excludes 2018-11-25 13:32:16 +01:00
Johannes Hertenstein
5fe6de219d Add tests for insensitive variants of filter methods 2018-11-25 13:32:16 +01:00
Johannes Hertenstein
c13f79da02 Add iexclude and iinclude options to restore command 2018-11-25 13:32:16 +01:00
Johannes Hertenstein
db82e6b80c Add iexclude option to backup command 2018-11-25 13:32:16 +01:00
Alexander Neumann
6dc7cca597 Merge pull request 2086 from frebib/fix-2085
Allow --files-from multiple times
2018-11-25 13:21:29 +01:00
Alexander Neumann
d32c7c2aba Rework changelog for #2085 2018-11-25 12:55:03 +01:00
Joe Groocock
09e9b74cbd Allow --files-from multiple times
Fixes https://github.com/restic/restic/issues/2085
2018-11-25 12:55:03 +01:00
Alexander Neumann
d53595e43c Merge pull request 2088 from garrmcnu/stats-latest-host
stats: Add shorthand flag `-H` for host name filter
2018-11-24 19:05:53 +01:00
Alexander Neumann
0de19cc87f Merge pull request 2095 from plumbeo/parse-location
Use local time for user-provided time of backup
2018-11-24 18:50:12 +01:00
plumbeo
2c9ec07d0b Add changelog entry 2018-11-20 22:18:12 +01:00
plumbeo
a7971a3ece Use local time for user-provided time of backup 2018-11-20 21:28:19 +01:00
Alexander Neumann
4ab0022da8 Add hint which OS are supported
Closes #2016
2018-11-18 12:33:52 +01:00
Garry McNulty
4b3c054257 stats: Add shorthand flag -H for host name filter
The default value of the `--host` flag was set to 'H' (the shorthand
version of the flag), this caused the snapshot lookup to fail.

Also add shorthand `-H` for `backup` command.

Closes #2040
2018-11-14 21:58:31 +00:00
plumbeo
7486bfea5b Extend custom Duration granularity to hours and add tests 2018-11-14 17:27:02 +01:00
Alexander Neumann
c8fc72364a Merge pull request 2081 from restic/fix-2080
backup: Open repository before async status starts
2018-11-11 14:21:47 +01:00
Alexander Neumann
987ef2f4a9 Merge pull request 2082 from restic/fix-2079
dump: Use path instead of filepath
2018-11-11 14:21:37 +01:00
Alexander Neumann
5b95bb7059 Fix error message on Windows 2018-11-11 14:06:32 +01:00
Alexander Neumann
8471a359ee dump: Use path instead of filepath
Some time ago we changed the paths in the repo to always use a slash for
separation, it seems we missed that the `dump` command still uses the
`filepath` package, so on Windows backslashes are used.

Closes #2079
2018-11-10 12:43:11 +01:00
Alexander Neumann
f9422ff4c7 backup: Open repository before async status starts
Closes #2080
2018-11-10 12:38:29 +01:00
Alexander Neumann
c0572ca15f backup: Add warning when patterns do not match any files 2018-11-03 14:51:30 +01:00
Alexander Neumann
a630d69e0c Merge pull request 2070 from restic/display-local-time
Display local time for all commands
2018-11-02 21:16:29 +01:00
Alexander Neumann
20bcd281a3 Add entry to changelog 2018-11-02 20:40:52 +01:00
Alexander Neumann
c012fccd22 Display local time for all commands 2018-11-02 20:36:15 +01:00
Alexander Neumann
920727dd34 Merge pull request #2068 from restic/prune-check-load-error
prune: Correctly handle errors returned by Load()
2018-10-28 22:15:29 +01:00
Alexander Neumann
157d365894 Add entry to CHANGELOG 2018-10-28 21:38:04 +01:00
Alexander Neumann
bfa18ee8ec DownloadAndHash: Check error returned by Load() 2018-10-28 21:28:56 +01:00
Alexander Neumann
890eebf151 Correct summary for #1978 2018-10-28 20:55:49 +01:00
Alexander Neumann
9310cd0cd6 Merge pull request #2056 from restic/recover-data
Add new command 'recover'
2018-10-28 14:26:48 +01:00
Alexander Neumann
9f7ce7ce5a Merge pull request #2066 from restic/fix-list-error
rebuild-index/prune: Correctly handle error listing files
2018-10-28 14:26:40 +01:00
Alexander Neumann
0b600d6cef index.New: Test various error conditions 2018-10-28 14:02:31 +01:00
Alexander Neumann
3ae2a79bdf restic: Use local rand source instead of global one 2018-10-28 14:01:30 +01:00
Alexander Neumann
f7c0893f76 index: Add tests for error conditions 2018-10-28 13:16:36 +01:00
Alexander Neumann
c3de301fc8 index: Use interfaces instead of *repository.Repository 2018-10-28 12:58:34 +01:00
Alexander Neumann
944b446ac0 index: Cancel context for tests 2018-10-28 11:17:04 +01:00
Alexander Neumann
b096fc7abf index: Correctly process errors listing all files
This also removes the now unused `list` and `worker` packages.
2018-10-28 11:16:29 +01:00
Alexander Neumann
d10754e2b4 Merge pull request 2057 from re1/patch-1
Update install instructions for Arch Linux
2018-10-21 20:04:07 +02:00
Alexander Neumann
7ac683c360 rclone: Inject debug logger for HTTP 2018-10-21 19:58:40 +02:00
Markus
6caa9d38ac Update install instructions for Arch Linux
restic has been added to the official community repositories in august 2018.
2018-10-20 13:09:52 +00:00
Alexander Neumann
19fd0f101f Merge pull request 2053 from andyjack/correct-doc-for-self-update
Fix incorrect command name in long doc for self-update
2018-10-20 11:51:13 +02:00
Alexander Neumann
8c91c51d1b Fix timestamps 2018-10-20 11:08:03 +02:00
Alexander Neumann
7e28bf7e97 Add 'recover' command 2018-10-20 11:08:03 +02:00
Alexander Neumann
43d6e426c8 Merge pull request 2054 from gliptak/patch-3
Correct typo
2018-10-20 11:04:49 +02:00
Alexander Neumann
26fc60e7cb Merge pull request 2055 from gliptak/patch-4
Correct ineffassign
2018-10-20 11:04:32 +02:00
Gábor Lipták
e5d7879622 Correct ineffassign
Signed-off-by: Gábor Lipták <gliptak@gmail.com>
2018-10-19 16:58:14 -04:00
Gábor Lipták
d2ee58f2e9 Correct typo 2018-10-18 21:30:14 -04:00
Andy Jack
3f25537a06 Fix incorrect command name in long doc for self-update 2018-10-18 18:45:31 -04:00
Alexander Neumann
d203ae37f4 Merge pull request 2050 from andreasnuesslein/patch-1
tiny formatting fix
2018-10-18 22:31:15 +02:00
Andreas Nüßlein
6eedd66c1a tiny formatting fix
I noticed this discrepancy when trying to parse the output of restic.
2018-10-17 09:41:11 +02:00
Alexander Neumann
e4b39ae553 Merge pull request 2044 from restic/fix-verify
restore: Close files tested with --verify
2018-10-15 21:16:58 +02:00
Alexander Neumann
7cbcb6d318 Merge pull request 2043 from cgroschupp/patch-1
Add ssh binary to docker image to allow sftp repository
2018-10-14 21:43:04 +02:00
Alexander Neumann
c0fca3f50a restore: Close files tested with --verify
Before, the target files were opened once per blob and never closed,
this commit fixes that.
2018-10-14 21:00:14 +02:00
Alexander Neumann
4c2072d875 Merge pull request 1719 from ifedorenko/mt-restorer
new optimized multithreaded restore implementation
2018-10-14 20:29:16 +02:00
Christian Groschupp
92ecca1808 Add ssh binary to docker image to allow sftp repository 2018-10-14 19:56:02 +02:00
Alexander Neumann
7236635cc1 Merge pull request 2042 from restic/fix-2041
self-update: fix path to output file
2018-10-14 19:48:28 +02:00
Alexander Neumann
21a3486ebb Rework changelog entry 2018-10-14 17:39:42 +02:00
Igor Fedorenko
bda8d7722e restorer: Optimize empty file restore
don't create fileInfo structs for empty files. this saves memory.
this also avoids extra serial scan of all fileInfo, which should
make restore faster and more consistent.

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-10-14 17:39:42 +02:00
Igor Fedorenko
c2bcb764cd restorer: Add a note on hardlink metadata
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-10-14 17:39:42 +02:00
Igor Fedorenko
9e24154ec9 restore: significantly reduce memory footprint
reworked restore error callback to use file location
path instead of much heavier Node. this reduced restore
memory usage by as much as 50% in some of my tests.

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-10-14 17:39:42 +02:00
Igor Fedorenko
9f3ca97ee8 restore: Chang fileInfo to use snapshot location instead of target path
* uses less memory as common prefix is only stored once
* stepping stone for simpler error callback api, which
  will allow further memory footprint reduction

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-10-14 17:39:42 +02:00
Igor Fedorenko
32d5ceba87 restore: Fix packcache capacity math with failed downloads
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-10-14 17:39:42 +02:00
Igor Fedorenko
e010f3b884 restore: Fix small memory leak in filesWriter, add tests
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-10-14 17:39:42 +02:00
Alexander Neumann
941202c119 restore: Rename things, match the rest of the project 2018-10-14 17:39:37 +02:00
Alexander Neumann
c021ad2334 restore: Move documentation to godoc 2018-10-14 17:39:35 +02:00
Igor Fedorenko
2b3420820b go mod vendor
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-10-14 17:37:30 +02:00
Igor Fedorenko
da57302fca restore: Removed legacy restore implementation
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-10-14 17:37:30 +02:00
Igor Fedorenko
1869930d95 restore: New optimized multithreaded implementation
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-10-14 17:37:30 +02:00
Igor Fedorenko
1213d8fef4 Cleanup: more realistic restorer test data setup
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-10-14 17:37:30 +02:00
Alexander Neumann
a432b42c81 self-update: Check current version before download 2018-10-14 17:29:46 +02:00
Alexander Neumann
7d0f2eaf24 self-update: Use correct path to restic binary
Closes #2041
2018-10-14 17:29:19 +02:00
Alexander Neumann
41a4d67d93 Merge pull request 1962 from mholt/lsjson
ls: Stream output when using --json option
2018-10-14 15:51:38 +02:00
Alexander Neumann
afde60e433 Merge pull request #2039 from restic/fix-2038
snapshots: Don't print empty line for --compact
2018-10-13 21:24:26 +02:00
Alexander Neumann
d7baa67acb snapshots: Don't print empty line for --compact
Closes #2038
2018-10-13 20:17:34 +02:00
Alexander Neumann
167397c18c Set development version for 0.9.3 2018-10-13 13:48:31 +02:00
Alexander Neumann
be36c5f150 Add version for 0.9.3 2018-10-13 13:48:31 +02:00
Alexander Neumann
9484a14ab2 Update manpages and auto-completion 2018-10-13 13:48:30 +02:00
Alexander Neumann
0f5fc8fb3d Generate CHANGELOG.md for 0.9.3 2018-10-13 13:48:29 +02:00
Alexander Neumann
a5b40e9372 helpers: Use version string in built binaries 2018-10-13 13:48:03 +02:00
Alexander Neumann
c5ec4efe91 Update prepare-release 2018-10-13 13:48:03 +02:00
Alexander Neumann
e64a0e0454 helpers: Don't run sha256sums in build 2018-10-13 13:48:03 +02:00
Alexander Neumann
8b5b031f90 Prepare changelog for 0.9.3 2018-10-13 13:48:03 +02:00
Alexander Neumann
4a2134bbc5 changelog: Rename issue 2018-10-13 13:48:03 +02:00
Alexander Neumann
484844aa1a Document the build and release processes 2018-10-13 13:48:03 +02:00
Alexander Neumann
4ed10239ad doc: Add developer information 2018-10-11 22:37:25 +02:00
Alexander Neumann
c4896ed642 Add build-release-binaries 2018-10-11 22:37:25 +02:00
Alexander Neumann
29aaec383c Move scripts/ to helpers/ 2018-10-11 22:37:25 +02:00
Alexander Neumann
0cb241b7d3 Remove build tag release 2018-10-11 19:40:38 +02:00
Alexander Neumann
de4750b8e0 Add entry to changelog 2018-10-11 19:37:20 +02:00
Alexander Neumann
7b91c40e21 Merge pull request #2033 from j6s/feature.cache-size
Output directory size in cache command
2018-10-11 19:34:57 +02:00
Alexander Neumann
cc9bf02da1 Merge pull request #2036 from SimJoSt/patch-1
doc: cache: fix typo
2018-10-11 19:34:14 +02:00
Joda Stößer
b7959c44d2 doc: cache: fix typo 2018-10-11 14:00:51 +02:00
Johannes Hertenstein
277cba4b32 Catch errors when walking cache directories 2018-10-08 15:47:34 +02:00
Johannes Hertenstein
ed651df19b Use correct method for joining paths 2018-10-08 15:47:08 +02:00
Johannes Hertenstein
641dc65e6e Output directory size in cache command 2018-10-07 14:37:51 +02:00
Alexander Neumann
de9136b29f Merge pull request #2022 from moritzdietz/update-faq
Add example to spot path expansion errors
2018-10-07 12:11:13 +02:00
Moritz Dietz
b36345fd84 Add example to spot path expansion errros 2018-10-07 12:09:46 +02:00
Alexander Neumann
03402c8a04 Merge pull request #2031 from danielb2/latest
use latest instead of newest in forget language
2018-10-06 21:20:21 +02:00
Daniel Bretoi
966e5a5575 use latest instead of newest in forget language 2018-10-06 12:02:22 -07:00
Alexander Neumann
5aa0deeff9 Merge pull request #2026 from gottwald/update-go4docker
Update Go version to 1.11.1 in Docker build script
2018-10-06 13:11:19 +02:00
Alexander Neumann
af4d822380 Merge pull request #2025 from kurin/fixloop
b2: simplify object iteration
2018-10-06 13:01:27 +02:00
Alexander Neumann
fd95b86894 Merge pull request #2029 from j6s/feature.escape-docker-build
Escape subcommand in docker build script
2018-10-06 12:46:08 +02:00
Alexander Neumann
5dbef3712e Merge pull request #2027 from j6s/feature.cache-dir-information
Print base directory with cache command
2018-10-06 12:40:48 +02:00
Johannes Hertenstein
63647e93e4 Escape subcommand in docker build script 2018-10-05 22:13:49 +02:00
Johannes Hertenstein
9b8deb51ba Print base directory with cache command 2018-10-05 21:23:57 +02:00
Ingo Gottwald
2c4b0d975e Update Go version to 1.11.1 in Docker build script
Restic cannot be built with Go 1.8.3 any more, it requires at least Go
1.9.0.
2018-10-05 21:13:28 +02:00
Toby Burress
8ceda538ef b2: simplify object iteration
Blazer is moving to a simpler object list interface, so I'm changing
this here as well.
2018-10-05 11:39:02 -07:00
Alexander Neumann
233596f4bc Merge pull request #2019 from restic/recheck-cache
cache: Recheck before downloading
2018-10-05 12:26:21 +02:00
Alexander Neumann
6712ee8f92 Merge pull request #2020 from j6s/feature.cache-dir-documentation
Add defaults to command line help of --cache-dir
2018-10-05 12:26:12 +02:00
Johannes Hertenstein
0916ff71bd Add defaults to command line help of --cache-dir 2018-10-04 19:50:05 +02:00
Alexander Neumann
5971650f77 cache: Fix recheck logic, remove channel from inProgress 2018-10-04 17:09:43 +02:00
Alexander Neumann
19725954ee cache: Recheck before downloading 2018-10-04 14:31:18 +02:00
Alexander Neumann
b1e1b71bab docker: Base image on latest alpine release 2018-10-03 21:33:13 +02:00
Alexander Neumann
f1799de309 Merge pull request #2018 from restic/fix-1967
Use `--host` for all commands, deprecate --hostname
2018-10-03 21:19:01 +02:00
Alexander Neumann
585a5e3416 Use --host for all commands, deprecate --hostname 2018-10-03 14:12:35 +02:00
Alexander Neumann
b7eeeedc3f Add minimal docs for the top-level package
Closes #173
2018-10-03 13:43:40 +02:00
Alexander Neumann
a20d4bc6b0 Merge pull request #1892 from Stell0/1891
Expand Glob (wildcards character) in paths in file in --files-from
2018-10-03 12:05:55 +02:00
Alexander Neumann
fb31d66951 backup: Improve error message for invalid pattern 2018-10-03 11:12:51 +02:00
Alexander Neumann
33dfbf5c38 Reword changelog 2018-10-03 10:37:35 +02:00
Stefano Fancello
d1df3718b5 Add changelog unreleased file 2018-10-03 10:37:35 +02:00
Stefano Fancello
e2da0a416c Expand Glob (wildcards character) in paths in file in --files-from 2018-10-03 10:37:35 +02:00
Alexander Neumann
0c0a8e3d2b docs: Explain recognizing modified files 2018-10-02 14:21:08 +02:00
Alexander Neumann
0882aca3a8 Add hint for FUSE on FreeBSD 2018-09-26 21:40:43 +02:00
Alexander Neumann
cd41915e10 Merge pull request #2009 from restic/update-minio
Update github.com/minio/minio-go
2018-09-26 14:33:30 +02:00
Alexander Neumann
2effacd444 Update github.com/minio/minio-go
Closes #1918
2018-09-26 14:12:34 +02:00
Alexander Neumann
c6901ff908 Merge pull request #2002 from ifedorenko/dumb-terminals
termstatus: detect and respect dumb terminals on Unix
2018-09-25 13:53:57 +02:00
Alexander Neumann
2f774acce3 Merge pull request #2006 from Calinou/doc-add-scoop-installation
Add Scoop installation method to the documentation
2018-09-25 13:23:28 +02:00
Alexander Neumann
5f8658238c Merge pull request #2005 from Calinou/doc-osx-to-macos
Rename OS X to macOS in the documentation
2018-09-25 13:21:28 +02:00
Hugo Locurcio
2bb1be4d4e Add Scoop installation method to the documentation 2018-09-23 18:42:50 +02:00
Hugo Locurcio
40e0016403 Rename OS X to macOS in the documentation
macOS is the official name since the release of macOS 10.12 (Sierra).
2018-09-23 16:13:34 +02:00
Igor Fedorenko
541d232f1c termstatus: detect and respect dumb terminals on Unix
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-09-21 07:24:41 -04:00
Alexander Neumann
6bc99ce451 Merge pull request #1993 from rfjakob/lchown
restore: suppress lchown errors when not running as root
2018-09-09 20:10:50 +02:00
Jakob Unterwurzacher
e42d2d1da8 restore: suppress lchown errors when not running as root
Like "cp -a" and "rsync -a" do, only report lchown errors if we run
as root.

Like cp from GNU coreutils does, we check Geteuid() to determine if
we are running as root
( http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/copy.c#n3012 ).

On Windows, lchown errors are always reported.

Fixes https://github.com/restic/restic/issues/1766
2018-09-09 15:39:10 +02:00
Alexander Neumann
bd9022962e Merge pull request #1992 from restic/add-go-mod-tidy
Run go mod tidy during CI
2018-09-09 11:03:16 +02:00
Alexander Neumann
91f1b40206 Run go mod tidy during CI 2018-09-09 10:36:27 +02:00
Alexander Neumann
d9b89eead0 Run 'go mod tidy' 2018-09-08 22:12:23 +02:00
Alexander Neumann
5399297de6 Merge pull request #1980 from restic/update-deps
Update dependencies
2018-09-08 21:57:39 +02:00
Alexander Neumann
96f7be5d9b Update dependencies 2018-09-08 21:17:22 +02:00
Alexander Neumann
0922367308 Merge pull request #1991 from restic/debug-1978
scanner: Use context only for cancellation
2018-09-08 20:22:15 +02:00
Alexander Neumann
e2d9900d82 Add entry to CHANGELOG 2018-09-08 18:53:12 +02:00
Alexander Neumann
1140950d7b scanner: Use context only for cancellation
When the scanner is slower than the actual backup, the tomb cancels the
context passed to Scan(), which then returns ctx.Err(). In the end, the
main function prints an error message that is not helpful ("Context
cancelled") and exits with an error code although no error occurred.

The code now ignores the error in the context and just uses it for
cancellation. The scanner is not supposed to return an error anyway.

Closes #1978
2018-09-08 18:53:12 +02:00
Alexander Neumann
6d9c008900 Add template for "regular" issue 2018-09-08 18:00:29 +02:00
Alexander Neumann
b617444158 Test issue template 2018-09-08 17:58:08 +02:00
Alexander Neumann
e588c42646 Move template 2018-09-08 17:55:44 +02:00
Alexander Neumann
14bb2a9005 Add custom issue template 2018-09-08 17:53:52 +02:00
Alexander Neumann
f04d347e7a Merge pull request #1982 from tyll/installation
Update installation instructions
2018-09-05 21:15:38 +02:00
Alexander Neumann
746182c526 Correct name for Apple's desktop OS 2018-09-05 20:41:37 +02:00
Alexander Neumann
08beb7d84c Add section about FreeBSD, move anchor 2018-09-05 20:40:28 +02:00
Alexander Neumann
9795b00f51 Merge pull request #1983 from armhold/vet-cleanup
gh-1385 clean up some errors from 'go vet ./...'
2018-09-05 20:31:11 +02:00
George Armhold
bfc1bc6ee6 clean up some errors from 'go vet ./...' 2018-09-05 08:04:55 -04:00
Till Maas
e9cdcf131c Update installation instructions
- sort sections for different third-party methods
- Mention that Fedora can now just be installed via dnf
- Fix some typos
2018-09-04 14:39:40 +02:00
Alexander Neumann
35e9885e8b Appveyor: correct build cache location 2018-09-02 13:26:02 +02:00
Alexander Neumann
16885529f7 Merge pull request #1973 from restic/announce-new-cache
cache: Print message when new cache is created
2018-09-02 13:00:55 +02:00
Alexander Neumann
3c02eeb5a8 Merge pull request #1920 from restic/support-go111
Support Go 1.11
2018-09-01 11:38:41 +02:00
Alexander Neumann
9e9bb62ad4 Travis: Don't use cached test results 2018-08-31 22:00:47 +02:00
Alexander Neumann
175e630717 Travis: Also cache $GOPATH/pkg/mod 2018-08-31 22:00:47 +02:00
Alexander Neumann
44f38ad049 Travis: Also cache on Darwin 2018-08-31 22:00:47 +02:00
Alexander Neumann
ca928aeae4 Appveyor: Cache Go build cache 2018-08-31 22:00:47 +02:00
Alexander Neumann
27b60a05b4 Travis: Cache Go build artifacts 2018-08-31 22:00:47 +02:00
Alexander Neumann
8af4b331ef Travis: Remove gotestcover 2018-08-31 22:00:47 +02:00
Alexander Neumann
a5a46e4989 Travis: don't use GOPATH for Go >= 1.11 2018-08-31 22:00:47 +02:00
Alexander Neumann
e4cdb0eab3 Travis: Run with -mod=vendor only for Go >= 1.11 2018-08-31 22:00:47 +02:00
Alexander Neumann
e9a764129f Update docs for Go 1.11 2018-08-31 22:00:47 +02:00
Alexander Neumann
65129bde5e Makefile: Try -mod=vendor first 2018-08-31 22:00:47 +02:00
Alexander Neumann
b4beaf807b Travis: Run build.go with -mod=vendor 2018-08-31 22:00:47 +02:00
Alexander Neumann
4734056583 build.go: Major rework, support Go modules 2018-08-31 22:00:47 +02:00
Alexander Neumann
71e0408390 Add entry to changelog 2018-08-31 21:10:26 +02:00
Alexander Neumann
1352a9d848 run_integration_tests: Replace dep with Go modules 2018-08-31 21:10:26 +02:00
Alexander Neumann
e0f68ec2c0 Vendor dependencies with go mod vendor 2018-08-31 21:10:26 +02:00
Alexander Neumann
9c6e0c6eb9 Document gofmt version 2018-08-31 21:10:26 +02:00
Alexander Neumann
4cbc7c4467 Only run gofmt for latest stable version 2018-08-31 21:10:26 +02:00
Alexander Neumann
aaff8803ef Fix formatting for gofmt with Go 1.11 2018-08-31 21:10:26 +02:00
Alexander Neumann
16e20676b6 build.go: Set GOPROXY=off 2018-08-31 21:10:26 +02:00
Alexander Neumann
6cd5f8b7f5 Set GOXPROXY=off for tests, run with -mod=vendor 2018-08-31 21:10:26 +02:00
Alexander Neumann
10c0b8080e Add Go 1.11 to CI tests 2018-08-31 21:10:26 +02:00
Alexander Neumann
d31666d332 build.go: Don't ignore error 2018-08-31 19:51:48 +02:00
Alexander Neumann
6d53e767d5 cache: Print message when new cache is created
Sometimes, users run restic without retaining the local cache
directories. This was reported several times in the past.

Restic will now print a message whenever a new cache directory is
created from scratch (i.e. it did not exist before), so users have a
chance to recognize when the cache is not kept between different runs of
restic.
2018-08-29 22:01:10 +02:00
Alexander Neumann
f1b0bb33dd Merge pull request #1975 from moritzdietz/fix-env-docu
Update B2 env variable information
2018-08-29 22:00:17 +02:00
Moritz Dietz
99ae913414 Update B2 env variable information
* Fixed the B2 environment variables as they had the same description
* Added the variables for the newly introduced Application Key support #1906
2018-08-29 21:05:57 +02:00
Thomas Damgaard
df78896e59 doc: Add list of environment variables 2018-08-28 22:18:47 +02:00
Alexander Neumann
c896751ce2 Merge pull request #1970 from McKael/pull-1780-changelog
Improve changelog entry (pull-1780)
2018-08-28 20:59:10 +02:00
Alexander Neumann
501189625e Merge pull request #1971 from bobsaintcool/doc/fix_man_output
Doc/fix man output
2018-08-28 20:58:21 +02:00
Quentin Bourgeois
a065ada46a Make generated man pages more easy to read with items
Allow Cobra to output a more list items friendly format, this make
reading of the documentation more easily using information from
manpages.
2018-08-27 17:42:54 +02:00
Mikael Berthe
17d6d537e2 Improve changelog entry (pull-1780) 2018-08-27 15:58:39 +02:00
Quentin Bourgeois
5cc224e44a Merge remote-tracking branch 'upsteam/master' 2018-08-26 20:05:02 +02:00
Alexander Neumann
896089976a Fix comment 2018-08-26 19:54:17 +02:00
Quentin Bourgeois
a563f87818 <docs/manual_rest: Fix broken link to restc Design page> 2018-08-26 01:46:38 +02:00
Alexander Neumann
de307ea2ab Merge pull request #1876 from restic/forget-explain
forget: Add --explain
2018-08-25 21:48:44 +02:00
Alexander Neumann
4bc904a527 Merge pull request #1780 from McKael/what-contains
Add options to 'find' for searching IDs (blobs, ...)
2018-08-25 21:46:25 +02:00
Alexander Neumann
5937b5b355 Add entry to changelog 2018-08-20 22:00:23 +02:00
Alexander Neumann
76387b6cd0 Remove old text table implementation 2018-08-20 22:00:23 +02:00
Alexander Neumann
9aa36a37c7 Vendor cmpopts 2018-08-20 21:47:51 +02:00
Alexander Neumann
9fd3796d93 forget: Display reasons why snapshots are kept
This change displays the reasons for keeping a snapshot in the table,
unless `--compact` is specified.
2018-08-20 21:47:51 +02:00
Mikael Berthe
93fa17b53f Add entry to changelog (new find flags) 2018-08-19 23:28:04 +02:00
Alexander Neumann
15ad0e5bc7 walk: Pass parent tree ID to WalkFunc 2018-08-19 23:28:04 +02:00
Mikael Berthe
1f27d17c0d walker.Walk: Pass parent tree-id to WalkFunc 2018-08-19 23:28:04 +02:00
Mikael Berthe
8af918a1e4 find: Add support for multiple patterns or objects 2018-08-19 23:28:04 +02:00
Mikael Berthe
bb5425a1d8 find: Add support for blobs, IDs and packs lookups
With --blob, --tree and --pack, the find command now lists the snapshots
that contain a specific tree or blob, or the snapshots that contain
blobs belonging to a given pack.
It also displays the pack ID a blob belongs to.

A list of IDs can be given, as long as the IDs are all of the same type.
2018-08-19 22:32:30 +02:00
Alexander Neumann
12246969db ui/table: Add small package for writing tables 2018-08-19 21:39:35 +02:00
Matthew Holt
9151eec24e Add changelog entry 2018-08-19 09:14:58 -06:00
Matthew Holt
22475729ce Select specific Node fields for listing 2018-08-19 09:03:47 -06:00
Matthew Holt
04c67d700d ls: Stream output when using --json option 2018-08-19 00:18:43 -06:00
Alexander Neumann
d708d607fa Merge pull request #1953 from kitone/ls-add-json
ls: Add JSON output support for restic ls cmd
2018-08-18 15:31:15 +02:00
Alexander Neumann
46f71f4c22 Improve changelog entry 2018-08-18 15:14:52 +02:00
kitone
48cc2f2188 fix: switch struct_type value to lower case 2018-08-18 14:57:50 +02:00
kitone
bd6e7c934c add changelog entry 2018-08-18 14:57:50 +02:00
kitone
7925217e25 ls: Add JSON output support for restic ls cmd 2018-08-18 14:57:50 +02:00
Alexander Neumann
401a564486 Merge pull request #1961 from mholt/statsoutput
stats: Show what was scanned and scanning mode used
2018-08-18 14:29:14 +02:00
Alexander Neumann
31176d212b doc: Add explanation for "processed" vs "added" 2018-08-18 14:26:57 +02:00
Matthew Holt
2d89311d49 stats: Show what was scanned and scanning mode used 2018-08-17 17:15:30 -06:00
Alexander Neumann
5a25ad1972 Update version in build.go 2018-08-17 21:17:26 +02:00
Alexander Neumann
79d3a18b31 release.go: Add version to global.go
Closes #1958
2018-08-17 21:17:10 +02:00
Alexander Neumann
89f17847ad Merge pull request #1955 from restic/fix-prune-1954
prune: Fix calculation for removed bytes
2018-08-14 22:30:59 +02:00
Alexander Neumann
1ab5703404 prune: Fix calculation for removed bytes 2018-08-14 22:06:05 +02:00
Alexander Neumann
49d95e9a50 Merge pull request #1949 from restic/add-self-update
Add command self-update
2018-08-13 22:02:34 +02:00
Alexander Neumann
7dff1a08d0 Merge pull request #1950 from ldelouw/master
Add RPM SPEC file to create restic packages on Fedora and RHEL
2018-08-13 22:00:18 +02:00
Alexander Neumann
5fee36fa84 Merge pull request #1941 from mholt/lsfilter
ls: Implement directory filter, optionally subfolders
2018-08-13 21:58:32 +02:00
Luc de Louw
b0211dff49 Small change for RHEL6 which does not know about the %license macro 2018-08-13 21:39:04 +02:00
Luc de Louw
0f6d21cf84 Add the SPEC file restic.spec to build RPMs
Tested on Fedora 28, RHEL 6 and 7
2018-08-13 21:39:04 +02:00
Alexander Neumann
10b5cf8f32 Add self-update to the docs 2018-08-12 23:56:06 +02:00
Alexander Neumann
ad5aec3f3b Add entry to changelog 2018-08-12 23:56:01 +02:00
Alexander Neumann
6e1a3987b7 Add 'self-update' command
This commit adds a command called `self-update` which downloads the
latest released version of restic from GitHub and replacing the current
binary with it. It does not rely on any external program (so it'll work
everywhere), but still verifies the GPG signature using the embedded GPG
public key.

By default, the `self-update` command is hidden behind the `selfupdate`
built tag, which is only set when restic is built using `build.go`. The
reason for this is that downstream distributions will then not include
the command by default, so users are encouraged to use the
platform-specific distribution mechanism.
2018-08-12 23:34:47 +02:00
Alexander Neumann
9630398e3b ls: Rework and simplify logic
This commit introduces two functions: withinDir() and
approachingMatchingTree()

Both bind the list of directories with a closure, so we don't need to
iterate over the list in the function passed to Walk(). This reduces the
indentation level and since we can just use return, we don't need the
breaks any more.

The case that len(dirs) == 0 can also be handled by the functions with a
return, which saves another indentation level.

The main function body of the function passed to Walk() was reduced to
three cases:

 * Within one of the dirs: Print the node, and if recursive operation is
   requested, directly return, so the walker continues recursive
   traversal

 * Approaching one of the dirs: don't print anything, but continue
   recursive traversal.

 * Nothing of the two: abort walking this branch of the tree.
2018-08-12 23:13:34 +02:00
Alexander Neumann
7e34de4c29 ls: Add comments 2018-08-12 22:18:44 +02:00
Alexander Neumann
ace5cc4ed3 ls: Only skip directory nodes
Special case for Walk(): When SkipDir is returned for a non-dir node,
the remaining nodes for the current tree are skipped. We don't want
that.
2018-08-12 22:02:59 +02:00
Alexander Neumann
7f617cfd7f ls: Use nodepath for filter 2018-08-12 22:01:38 +02:00
Alexander Neumann
0deb4e5994 ls: Check dirs before opening the repository
Users get feedback instantly, and before any expensive network calls
have been made.
2018-08-12 21:59:57 +02:00
Alexander Neumann
6b9dde3ce8 Merge pull request #1912 from askielboe/select-funcs
Support for different kinds of select functions
2018-08-12 19:26:36 +02:00
Alexander Neumann
c145b618d4 Add entry to changelog 2018-08-12 17:51:12 +02:00
Andreas Skielboe
b07bb3d8c3 Reject files excluded by name before calling lstat to improve scan speed
Adds a SelectByName method to the archive and scanner which only require
the filename as input, and can thus be run before calling lstat on the
file. Can speed up scanning significantly if a lot of filename excludes
are used.
2018-08-12 17:51:12 +02:00
Alexander Neumann
9b513312e2 Merge pull request #1946 from restic/fix-1945
Remove truncated files from cache
2018-08-12 17:23:56 +02:00
Alexander Neumann
bf26a3ed57 Merge pull request #1948 from restic/update-build-go
Update build.go
2018-08-12 17:18:11 +02:00
Alexander Neumann
77a8d931b8 Update build.go 2018-08-12 15:44:13 +02:00
Matthew Holt
11ce572894 Fix bug where some folder listings were empty 2018-08-11 17:17:43 -06:00
Matthew Holt
7a468d1226 Speed up nonrecursive queries; include exact filter match 2018-08-11 16:18:09 -06:00
Matthew Holt
00e2fd8b5f Apply feedback and use SkipNode 2018-08-11 15:25:22 -06:00
Alexander Neumann
0f83fea007 cache: Fix test for new behavior
Accessing beyond the end of the file now removes the file from the cache
because it is assumed to be truncated. Usually, this means that the data
is fetched directly from the backend instead.
2018-08-11 23:11:51 +02:00
Alexander Neumann
04f7c054cd Add entry to changelog 2018-08-11 22:54:22 +02:00
Alexander Neumann
5dd0df0162 cache: Remove files from cache which are too small 2018-08-11 22:47:01 +02:00
Alexander Neumann
abc923f693 Merge pull request #1942 from mholt/statserr
stats: Improve error message for bad snapshot ID (fixes #1933)
2018-08-11 14:26:24 +02:00
Matthew Holt
ac3bd6b2eb Replace Exitf with errors.Fatalf 2018-08-10 22:15:33 -06:00
Matthew Holt
156d85a29b Minor fixes/tweaks; add docs 2018-08-10 22:10:02 -06:00
Matthew Holt
8c146eac4b ls: Implement directory filter, optionally subfolders 2018-08-10 21:41:38 -06:00
Matthew Holt
6f5b0f3622 stat: Improve error message for bad snapshot ID (fixes #1933) 2018-08-08 16:49:36 -06:00
Alexander Neumann
beb208e159 Add VERSION for 0.9.2 2018-08-06 21:28:59 +02:00
Alexander Neumann
c221d662d0 Update manpages and auto-completion 2018-08-06 21:28:59 +02:00
Alexander Neumann
143597d445 Generate CHANGELOG.md for 0.9.2 2018-08-06 21:28:48 +02:00
Alexander Neumann
16ca837763 Create changelog/ release subdir for 0.9.2 2018-08-06 21:28:16 +02:00
Alexander Neumann
ce7fb166b3 Merge pull request #1927 from prosoitos/minor-fixes_examples
minor-fixes_examples
2018-08-05 12:27:03 +02:00
Marie-Helene Burle
9de51d04ec minor-fixes_examples
- update link to latest restic release to install restic for `restic` user
- correct typo in path to this app
2018-08-05 02:15:17 -07:00
Alexander Neumann
dc39773cd2 Merge pull request #1924 from cuspat96/master
Clarify B2 Application Keys once again
2018-08-03 21:15:16 +02:00
cuspat96
30fa305c07 Clarify B2 Application Keys once again
This commit fixes documentation about Application Keys in Backblaze B2.
2018-08-02 21:29:29 +02:00
Alexander Neumann
686f24b578 doc: Clarify B2 application keys 2018-08-02 21:14:05 +02:00
Alexander Neumann
247d2b7215 Merge pull request #1921 from salbertson/patch-1
Add a "Reviewed by Hound" badge
2018-08-02 20:03:43 +02:00
Alexander Neumann
017cd113d3 Merge pull request #1922 from salbertson/patch-2
Use https when linking to chris.beams.io
2018-08-02 20:03:40 +02:00
Scott Albertson
f744c2553e Use https when linking to chris.beams.io
Why not link to How to [Write a Git Commit Message](https://chris.beams.io/posts/git-commit/) using HTTPS, it's going to redirect anyway.
2018-08-01 14:59:06 -07:00
Alexander Neumann
56cd6bd495 Merge pull request #1919 from restic/update-deps
Update dependencies
2018-08-01 23:56:55 +02:00
Alexander Neumann
bff635bc5f Update dependencies, enable pruning for vendor/
So, `dep` got an nice new feature to remove tests and non-go files from
`vendor/`, and this brings the size of the vendor directory from ~300MiB
down to ~20MiB. We don that now.
2018-08-01 21:32:15 +02:00
Alexander Neumann
3422c1ca83 Merge pull request #1729 from mholt/stats
Implement `restic stats` command to get more info about a repository
2018-07-31 23:24:36 +02:00
Matthew Holt
f6b2731aa5 stats: Add manual doc, improve -h doc
Also rename files-by-content to files-by-contents, once and for all
2018-07-31 22:54:10 +02:00
Scott Albertson
3eb5b45b41 Add a "Reviewed by Hound" badge 2018-07-31 13:53:24 -07:00
Alexander Neumann
01aacf41b5 Merge pull request #1915 from mlissner/patch-2
Adds warning re performance of prune
2018-07-31 22:42:20 +02:00
Mike Lissner
2caf8edc55 Add warning of the performance of prune
I went pretty loud with this, but I think the performance is bad enough
that it's really worth highlighting, especially since it locks the index
during the prune.
2018-07-31 22:41:40 +02:00
Alexander Neumann
3151978f58 Fix changelog type 2018-07-31 21:57:27 +02:00
Alexander Neumann
ab4ef432ff Add entry to changelog 2018-07-31 21:29:47 +02:00
Alexander Neumann
be4f54b603 Merge pull request #1913 from restic/restic-password-stdin-message
Print message for password being read from stdin
2018-07-31 21:28:12 +02:00
Alexander Neumann
7260110c27 Merge pull request #1914 from restic/update-blazer
Add support for B2 application keys
2018-07-31 21:27:50 +02:00
Alexander Neumann
2437f11af7 Update github.com/kurin/blazer to 0.5.1
This adds support for B2 application keys.
2018-07-31 20:51:36 +02:00
Alexander Neumann
57873502f8 Add note about B2 application keys to the docs 2018-07-31 20:49:54 +02:00
Alexander Neumann
3678ec9ad8 Print message for password being read from stdin
Closes #1911
2018-07-31 20:21:18 +02:00
Alexander Neumann
a717e9e6f7 Improve message for number of bytes newly added 2018-07-31 19:08:43 +02:00
Alexander Neumann
12c797700e make statsWalkSnapshot return a function 2018-07-27 21:44:59 +02:00
Matthew Holt
daca9d6815 Consolidate mode flags; use new Walk function 2018-07-27 21:27:40 +02:00
Matthew Holt
930602a444 Update comment now that question was answered 2018-07-27 21:27:39 +02:00
Matthew Holt
acb05e7855 Fix filepath uniqueness bug for blobs-per-file mode 2018-07-27 21:27:39 +02:00
Matthew Holt
a7b95d716a Implement four counting modes 2018-07-27 21:27:39 +02:00
Matthew Holt
925b542eb0 Count unique files by blob sequence rather than tree ID 2018-07-27 21:27:39 +02:00
Matthew Holt
f7659bd8b0 stats: Initial implementation of stats command 2018-07-27 21:27:39 +02:00
Alexander Neumann
8c124a2b75 Merge pull request #1902 from mlissner/patch-1
b2 bucket names need to be unique
2018-07-23 22:58:42 +02:00
Mike Lissner
d3ad63a4ec b2 bucket names need to be unique
Adds a small warning indicating that b2 bucket names need to be unique. It's an easy mistake to make, and it's surprising to get the following error if you're not accustomed to the way B2 works:

    Fatal: create repository at b2:postgres failed: NewBucket: b2_create_bucket: 400: Bucket name is already in use
2018-07-23 11:48:59 -07:00
Alexander Neumann
271c50cf5c Add entry to changelog 2018-07-23 20:15:55 +02:00
Alexander Neumann
1aeb193fd9 Merge pull request #1900 from restic/fix-1870
restorer: Add test for restore with include filter
2018-07-23 20:15:50 +02:00
Alexander Neumann
f715bef82f Merge pull request #1899 from garrmcnu/check-cache-dir
check: Use --cache-dir argument
2018-07-22 21:03:52 +02:00
Alexander Neumann
4fc00d4120 Merge pull request #1901 from restic/update-blazer
Update github.com/kurin/blazer
2018-07-22 20:59:52 +02:00
Garry McNulty
7603ab7ac1 check: Update --cache-dir argument handling based on code review comments
The temporary cache directory is created in the specified directory, or
if not specified in the default temporary directory.
2018-07-22 18:24:11 +01:00
Alexander Neumann
36fa1f8c20 Merge pull request #1894 from restic/fix1893
Return error when exclude file cannot be read
2018-07-22 14:34:27 +02:00
Alexander Neumann
445fb23b6d Rework issue templates for Bug reports and Features 2018-07-22 14:26:23 +02:00
Alexander Neumann
5f79b4cb6c Update issue template again 2018-07-22 14:21:08 +02:00
Alexander Neumann
8e15b59347 Use underline style markup for issue/PR templates 2018-07-22 14:17:53 +02:00
Alexander Neumann
6e2e957332 Add entry to changelog 2018-07-22 14:16:08 +02:00
Alexander Neumann
7ffc03ff8f Update github.com/kurin/blazer to 0.5.0
This includes support for the upcoming B2 application keys feature.
2018-07-22 14:12:02 +02:00
Alexander Neumann
44924ba043 restorer: Fix traverseTree
traverseTree() was meant to call enterDir() whenever a directory is
selected for restore, either explicitly or implicitly (=contains a file
which is to be restored). After restoring a file, leaveDir() is called
in reverse order for all intermediate directories so that the metadata
can be restored.

When a directory is selected implicitly, the metadata for it is
restored. This is different from the previous restorer behavior, which
created implicitly selected intermediate directories with permissions
0700 (only user can read/write it).

This commit changes the behavior back to the old one. Only a directory
is explicitly selected for restore, enterDir()/leaveDir() are called for
it. Otherwise, only visitNode() is called, so visitNode() needs to make
sure the parent directory exists. If the directory is explicitly
included, leaveDir() will then restore the metadata correctly.

When we decide to change the behavior (restore metadata for all
intermediate directories, even if selected implicitly), we should do
that in the selection functions, not here.

This finally resolves #1870
2018-07-21 23:24:40 +02:00
Alexander Neumann
ce19f26948 restorer: Add tests for traverseTree 2018-07-21 23:24:40 +02:00
Alexander Neumann
74016d5981 restorer: Fix return of saveSnapshot 2018-07-21 23:24:40 +02:00
Alexander Neumann
57636a4573 restorer: Run tests in the same package 2018-07-21 23:24:40 +02:00
Alexander Neumann
4f6d2502f7 restorer: Add test for restore with include filter 2018-07-21 23:24:40 +02:00
Garry McNulty
f1f69bc648 check: Use --cache-dir argument
Closes #1880
2018-07-20 20:51:20 +01:00
Alexander Neumann
d7551d7b0c Add entry to changelog 2018-07-18 21:41:20 +02:00
Alexander Neumann
fb74de6360 Return an error when exclude files cannot be read 2018-07-18 21:39:07 +02:00
Alexander Neumann
67535e00a8 Merge pull request #1889 from ProactiveServices/patch-3
doc: Minor grammar, RST syntax fixes
2018-07-18 21:22:10 +02:00
Alexander Neumann
19592285eb Merge pull request #1888 from ProactiveServices/patch-2
doc: Minor grammar fixes
2018-07-18 21:21:52 +02:00
Alexander Neumann
f64862722a Merge pull request #1887 from restic/disable-error-size
checker: Disable size check for now
2018-07-18 21:19:54 +02:00
Adam Piggott
254239c2a9 doc: Minor grammar, RST syntax fixes
Fix unescaped backslash
Fix wording of Windows installation
2018-07-18 02:28:23 +01:00
Adam Piggott
cce1a1f768 doc: Minor grammar fixes 2018-07-18 02:25:31 +01:00
Alexander Neumann
754482fe6c checker: Disable size check for now 2018-07-15 21:52:38 +02:00
Alexander Neumann
73153dbd3f Merge pull request #1885 from restic/create-restore-target
restore: Make sure the target directory exists
2018-07-15 16:28:25 +02:00
Alexander Neumann
92421ec47f restore: Make sure target directory exists 2018-07-15 16:02:04 +02:00
Alexander Neumann
9acc9243ba Add test for not-existing top-level dir and top-level file 2018-07-15 16:00:26 +02:00
Alexander Neumann
df64998649 Merge pull request #1882 from duzvik/aws-credentials-priority
Change AWS credentials priority, to accept AWS_SESSION_TOKEN
2018-07-14 20:48:42 +02:00
Alexander Neumann
64d27eed86 doc: Improve dump to stdout
Closes #1884
2018-07-14 20:45:52 +02:00
Alexander Neumann
abb18a830c Fix test 2018-07-14 11:51:34 +02:00
denis.uzvik
1e42f4f300 S3 backend: accept AWS_SESSION_TOKEN 2018-07-12 16:18:19 +03:00
Alexander Neumann
bd742ddb69 cache: Don't recreate CACHEDIR.TAG 2018-07-08 12:05:12 +02:00
Alexander Neumann
b511f4dce2 Improve help message for check 2018-07-05 22:19:08 +02:00
Alexander Neumann
7961740dcc Fix link 2018-07-05 21:03:40 +02:00
Alexander Neumann
dc3032c360 Mention that AppsCode is sponsoring backend tests 2018-07-05 21:01:57 +02:00
Alexander Neumann
44fb2a860f Merge pull request #1861 from McKael/fix_find_ignore_case
Fix find -i (case-insensitive search)
2018-06-24 21:08:03 +02:00
Mikael Berthe
fbf8073dfc Fix find -i (case-insensitive search) 2018-06-24 19:35:11 +02:00
Alexander Neumann
7ddf91b65c Merge pull request #1772 from ifedorenko/restore-verify
Restore verify
2018-06-23 14:50:29 +02:00
Alexander Neumann
8dae2de2ce Merge pull request #1858 from restic/fix-1857
Allow excluding files with $ via --exclude-file
2018-06-22 21:22:23 +02:00
Alexander Neumann
03a0377410 Merge pull request #1856 from stkw0/master
Ignore ExcludeOtherFs if Stdin is true
2018-06-22 20:49:34 +02:00
Alexander Neumann
025ec9dff5 Allow excluding files with $ via --exclude-file
Previously it wasn't possbile to exclude files with a literal dollar
sign (`$`) via exclude files, now users can write `$$` for that.
2018-06-22 20:46:04 +02:00
David Roman
2384c1cee7 Ignore ExcludeOtherFs if Stdin is true
Closes: #1807
2018-06-21 23:12:38 +02:00
Alexander Neumann
bb2ad76833 Merge pull request #1855 from restic/fix-1854
Allows saving files/dirs on different file systems together with `--one-file-system`.
2018-06-21 20:48:33 +02:00
Alexander Neumann
30cfd13328 Add changelog 2018-06-20 23:05:09 +02:00
Alexander Neumann
9ffc26883a archiver: Unroll tree 2018-06-20 22:56:41 +02:00
Alexander Neumann
83c51db903 fs: Add helper functions ReadDir/ReadDirNames 2018-06-20 22:53:53 +02:00
Alexander Neumann
d30d5d4473 Merge pull request #1853 from skriss/list-keys-json-output
add JSON output support for restic key list cmd
2018-06-19 21:28:45 +02:00
Steve Kriss
5088905502 add JSON output support for restic key list cmd
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-18 15:26:26 -07:00
Alexander Neumann
ae72b438b0 doc: Fix typo 2018-06-18 22:19:01 +02:00
Alexander Neumann
ddf2065ce2 doc: Split references out into smaller files
Closes #1852
2018-06-18 22:17:48 +02:00
Alexander Neumann
228a970540 doc: Add paragraph about finding path in file 2018-06-17 20:49:11 +02:00
Alexander Neumann
c7a8086c19 Merge pull request #1851 from restic/fix-1849
walker: Don't ignore empty trees by default
2018-06-17 12:25:33 +02:00
Alexander Neumann
c2c06ae2c9 walker: Don't ignore empty trees by default
Closes #1849
2018-06-17 09:49:03 +02:00
Alexander Neumann
1824168aa3 Merge pull request #1848 from iquiw/netbsd-support
Support NetBSD without fuse
2018-06-16 17:12:20 +02:00
Iku Iwasa
350761f1ba Add cross-compilation for netbsd/386 and netbsd/amd64 2018-06-16 21:05:26 +09:00
Alexander Neumann
3231945a85 Merge pull request #1846 from cbergmann/patch-1
Clarify live status display
2018-06-16 11:19:58 +02:00
Iku Iwasa
f080142137 Support NetBSD without fuse 2018-06-16 15:55:04 +09:00
Clemens Bergmann
ff785924de Clarify live status display
When I backup one of my filesystems which has a lot of Hard Links (Backup directory of burp) the live status shows me 4.5 TB but it only takes up 1.2 TB of space in the repository. This is confusing because my repo is on S3 and I feared a huge Bill. This change should clarify this.
2018-06-15 12:41:48 +02:00
Alexander Neumann
393a7266c9 Merge pull request #1845 from dimejo/patch-2
doc: enhance FAQ about AV software on Windows
2018-06-14 20:48:15 +02:00
Alex JOST
cb8d2d3df5 doc: enhance FAQ about AV software on Windows
As discussed in #1840, realtime protection of AV software can cause serious performance degradation on Windows OS.
2018-06-14 19:39:04 +02:00
Alexander Neumann
a884ce1566 Merge pull request #1841 from jernst/master
Added documentation about what happens if you run out of space during a backup
2018-06-14 19:13:31 +02:00
Alexander Neumann
5ae8316c24 Merge pull request #1844 from gsanchietti/patch-1
Documentation: minor RST syntax fixes
2018-06-14 19:12:57 +02:00
Giacomo Sanchietti
85eca1b5e9 Documentation: minor RST syntax fixes
- Fix bad link syntax
- Fix compilation warning
2018-06-14 15:01:49 +02:00
Johannes Ernst
a1536f38fa Fixed typo and grammar per feedback on PR. 2018-06-13 00:24:42 +00:00
Johannes Ernst
888f52afd1 Added documentation about what happens if you run out of space
during a backup, based on conversation on the forum:
https://forum.restic.net/t/limited-backup-destination-space/733/6
2018-06-11 21:20:08 +00:00
Igor Fedorenko
e206680947 restore: New --verify flag to verify restored files content
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-06-11 07:57:22 -04:00
Igor Fedorenko
5fa6dc53cb Refactor: introduced restorer tree visitor
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-06-11 07:57:22 -04:00
Igor Fedorenko
26be094f28 Refactor: moved restorer to separate package
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-06-11 07:57:22 -04:00
Alexander Neumann
e4c0d77bdd Add VERSION for 0.9.1 2018-06-10 11:31:03 +02:00
Alexander Neumann
1dd655dad2 Generate CHANGELOG.md for 0.9.1 2018-06-10 11:30:53 +02:00
Alexander Neumann
581d0984fe Fix changelog entry 2018-06-10 11:29:57 +02:00
Alexander Neumann
e62add84bc Move changelog files for 0.9.1 2018-06-10 11:22:32 +02:00
Alexander Neumann
63779c1eb4 Merge pull request #1839 from restic/fix-find
Fix find, do not skip some snapshots
2018-06-10 10:08:47 +02:00
Alexander Neumann
c204382ea9 Revert "Fix integration tests on Windows"
This reverts commit 33dbd0ba5c.
2018-06-10 00:01:28 +02:00
Alexander Neumann
321efec60c Fix integration tests on Windows 2018-06-10 00:00:22 +02:00
Alexander Neumann
33dbd0ba5c Fix integration tests on Windows 2018-06-09 23:58:44 +02:00
Alexander Neumann
9a73869c27 Update docs for RHEL/CentOS 2018-06-09 23:41:40 +02:00
Alexander Neumann
8f26fe271c ls: Use walker for ls 2018-06-09 23:35:20 +02:00
Alexander Neumann
251335f124 Add entry to changelog 2018-06-09 23:35:20 +02:00
Alexander Neumann
081743d0a5 find: Use walker.Walk 2018-06-09 23:35:20 +02:00
Alexander Neumann
3a86f4852b Add walker for trees in the repo 2018-06-09 23:35:20 +02:00
Alexander Neumann
14aead94b3 filter: Allow double wildcard in ChildMatch 2018-06-09 23:18:13 +02:00
Alexander Neumann
ce01ca30d6 find: Correct tree pruning optimization
The `find` command will now take care to only mark trees as "not found"
when the pattern couldn't be found within any subtree.

Closes #1825, #1823
2018-06-09 18:59:13 +02:00
Alexander Neumann
e2d347a698 find: Use OS independent slash-based format 2018-06-09 18:58:13 +02:00
Alexander Neumann
42ebb0a0a6 backup: Parse timestamp earlier 2018-06-09 18:21:12 +02:00
Alexander Neumann
419acad3c3 Merge pull request #1837 from restic/fix-1833
cache: Ensure failed downloads are retried
2018-06-09 18:20:21 +02:00
Alexander Neumann
810b5ea076 Add entry to changelog 2018-06-09 17:55:51 +02:00
Alexander Neumann
fc5439a37a cache: Ensure failed downloads are retried
This fixes #1833, which consists of two different bugs:

 * The `defer` in `cacheFile()` may remove a channel from the
   `inProgress` map although it is not responsible for downloading the
   file

 * If the download fails, goroutines waiting for the file to be cached
   assumed that the file was there, there was no way to signal the
   error.
2018-06-09 17:50:56 +02:00
Alexander Neumann
48aab8bd65 Merge pull request #1836 from restic/update-blazer
Update github.com/kurin/blazer
2018-06-09 14:31:12 +02:00
Alexander Neumann
6fbcd1694b Add entry to changelog 2018-06-09 14:31:02 +02:00
Alexander Neumann
494fe2a8b5 Merge pull request #1835 from restic/fix-1834
backup: Fix deadlock
2018-06-09 14:28:16 +02:00
Alexander Neumann
f761068f4e Update github.com/kurin/blazer 2018-06-09 12:32:18 +02:00
Alexander Neumann
c44e808aa5 backup: Fix deadlock
When the archiver is faster than the scanner, restic deadlocks. This
commit adds a `finished` channel to the struct in `ui/backup.go` so that
scanner results are ignored when the archiver is already finished.

Closes #1834
2018-06-09 12:15:19 +02:00
Alexander Neumann
ab37c6095a Merge pull request #1821 from michaelkoetter/fix-1795
#1795 use unix.IoctlGetWinsize to get terminal size
2018-06-07 20:20:06 +02:00
Michael Kötter
d6fd94e49d Don't run Solaris build for go1.9 2018-06-04 15:04:50 +02:00
Michael Kötter
53040a2e34 add "solaris/amd64" to cross-compile archs 2018-06-04 12:51:34 +02:00
Alexander Neumann
cfc19b4582 Merge pull request #1828 from restic/handle-s3-list-errors
s3: Pass list errors up to the caller
2018-06-02 10:34:49 +02:00
Alexander Neumann
141fabdd09 s3: Pass list errors up to the caller 2018-06-01 22:15:23 +02:00
Alexander Neumann
d49ca42771 Merge pull request #1827 from restic/azure-large-files
azure: Support uploading large files
2018-06-01 18:37:26 +02:00
Alexander Neumann
f6fded729d Add entry to changelog 2018-06-01 14:52:16 +02:00
Alexander Neumann
465700595c azure: Support uploading large files
Closes #1822
2018-06-01 14:52:16 +02:00
Alexander Neumann
0fcd9d6926 Merge pull request #1824 from rfjakob/ssh_command_exited
sftp: persist "ssh command exited" error
2018-05-31 21:26:39 +02:00
Jakob Unterwurzacher
dd3b9910ee sftp: persist "ssh command exited" error
If our ssh process has died, not only the next, but all subsequent
calls to clientError() should indicate the error.

restic output when the ssh process is killed with "kill -9":

  Save(<data/afb68adbf9>) returned error, retrying after 253.661803ms: Write: failed to send packet header: write |1: file already closed
  Save(<data/afb68adbf9>) returned error, retrying after 580.752212ms: ssh command exited: signal: killed
  Save(<data/afb68adbf9>) returned error, retrying after 790.150468ms: ssh command exited: signal: killed
  Save(<data/afb68adbf9>) returned error, retrying after 1.769595051s: ssh command exited: signal: killed
  [...]
  error in cleanup handler: ssh command exited: signal: killed

Before this patch:

  Save(<data/de698d934f>) returned error, retrying after 252.84163ms: Write: failed to send packet header: write |1: file already closed
  Save(<data/de698d934f>) returned error, retrying after 660.236963ms: OpenFile: failed to send packet header: write |1: file already closed
  Save(<data/de698d934f>) returned error, retrying after 568.049909ms: OpenFile: failed to send packet header: write |1: file already closed
  Save(<data/de698d934f>) returned error, retrying after 2.428813824s: OpenFile: failed to send packet header: write |1: file already closed
  [...]
  error in cleanup handler: failed to send packet header: write |1: file already closed
2018-05-30 19:28:14 +02:00
Alexander Neumann
185b60c22b Document project governance 2018-05-28 22:29:06 +02:00
Michael Kötter
589c23dc23 #1795 use unix.IoctlGetWinsize to get terminal size 2018-05-27 23:44:48 +02:00
Alexander Neumann
0183fea926 Merge pull request #1820 from restic/fix-1803
termstatus: Fix panic for non-terminal runs
2018-05-27 13:08:25 +02:00
Alexander Neumann
7d9642523b termstatus: Fix panic for non-terminal runs
Closes #1803
2018-05-27 12:52:01 +02:00
Alexander Neumann
4bf07a74a0 Merge pull request #1806 from mholt/patch-1
doc: Clarify multiple forget policies get ORed
2018-05-26 11:03:22 +02:00
Alexander Neumann
2a976d795f b2: Remove extra error check 2018-05-26 10:12:30 +02:00
Alexander Neumann
1892b314f8 Merge pull request #1815 from restic/update-blazer
Update github.com/kurin/blazer
2018-05-25 20:46:35 +02:00
Alexander Neumann
b7bed406b9 Update github.com/kurin/blazer 2018-05-25 20:26:26 +02:00
Matt Holt
ee4202f7c3 doc: Clarify multiple forget policies get ORed 2018-05-23 17:28:02 -06:00
Alexander Neumann
4cd28713b6 Merge pull request #1802 from restic/rclone-add-limits
rclone: Add limiting bandwidth to the rclone backend
2018-05-22 21:19:52 +02:00
Alexander Neumann
e3fe87f269 Remove superseded feature from the CHANGELOG
For a discussion please see https://github.com/restic/restic/issues/1796
2018-05-22 20:57:07 +02:00
Alexander Neumann
a02698fcdd Add entry to changelog 2018-05-22 20:48:29 +02:00
Alexander Neumann
bfd923e81e rclone: Respect bandwith limits 2018-05-22 20:48:17 +02:00
Alexander Neumann
20bfed5985 Update build.go 2018-05-21 20:31:19 +02:00
Alexander Neumann
e40191942d Add VERSION for 0.9.0 2018-05-21 15:57:32 +02:00
Alexander Neumann
abd34ab76f Update manpages and auto-completion 2018-05-21 15:57:32 +02:00
Alexander Neumann
4b43a269ee Generate CHANGELOG.md for 0.9.0 2018-05-21 15:57:22 +02:00
Alexander Neumann
e2b7dc6528 Move changelog files for the 0.9.0 release 2018-05-21 15:57:02 +02:00
Alexander Neumann
d2431b667f Merge pull request #1794 from restic/fix-1789
Update docs, clarify --cacert
2018-05-21 12:35:06 +02:00
Alexander Neumann
b70fdf61c4 Merge pull request #1791 from restic/fix-1790
archiver: Fix backup from stdin
2018-05-21 12:07:55 +02:00
Alexander Neumann
e6f25c4811 Update docs, clarify --cacert
Closes #1789
2018-05-21 12:06:30 +02:00
Alexander Neumann
adb682bc43 archiver: Don't open files with O_NONBLOCK
This is not necessary any more, we're doing an lstat() before opening
an item, so we already known it's a file and not a pipe.
2018-05-20 16:11:51 +02:00
Alexander Neumann
1e9744c9a4 archiver: Refuse to save an empty snapshot 2018-05-20 16:11:51 +02:00
Alexander Neumann
9a02f17cc2 archiver: Add tests for Save() for fs.Reader 2018-05-20 16:11:36 +02:00
Alexander Neumann
c284712cae Merge pull request #1784 from restic/improve-error-list
list: Improve error message
2018-05-18 21:53:09 +02:00
Alexander Neumann
2dbdf381b2 Merge pull request #1782 from skriss/add-s3-file-creds
Add S3 file creds and reorder creds chain
2018-05-18 21:52:54 +02:00
Alexander Neumann
a1a49ce211 Merge pull request #1787 from tyll/patch-1
Use more descriptive long option
2018-05-18 21:51:24 +02:00
Till Maas
3252e4200c Use more descriptive long option
-r could also mean recursive so use --repo to be clear.
2018-05-18 19:25:56 +02:00
Alexander Neumann
8d9d218d1c list: Improve error message
Before:

    $ restic list
    Fatal: type not specified

After:

    $ restic list
    Fatal: type not specified, usage: list [blobs|packs|index|snapshots|keys|locks]

Closes #1783
2018-05-17 19:41:56 +02:00
Steve Kriss
0785fbd418 changelog entry
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-17 10:09:38 -07:00
Steve Kriss
b358dd369b S3: rearrange credentials chain to be standard
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-16 16:49:33 -07:00
Steve Kriss
d67b9a32c6 S3: add file credentials to chain
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-16 16:35:14 -07:00
Alexander Neumann
ecfe59235e Merge pull request #1781 from McKael/fix_restick_backup_rejectByDevice
Fix restic backup --one-file-system /
2018-05-16 21:58:58 +02:00
Mikael Berthe
a868a30f4d Fix restic backup --one-file-system /
This patch should fix the following panic when trying to backup the
root filesystem with thre --one-file-system flag:

    % restic backup --one-file-system /
    (...)
    panic: item /, device id 2082 not found, allowedDevs: map[/:2082]
2018-05-16 13:44:14 +02:00
Alexander Neumann
347a645450 Fix double error message 2018-05-15 11:03:33 +02:00
Alexander Neumann
9f5565b0fc Merge pull request #1735 from mholt/forget-max-age
forget: Add --max-age policy to set hard cutoff for removing snapshots
2018-05-14 21:18:06 +02:00
Alexander Neumann
fd979ab4c5 Merge pull request #1779 from restic/improve-error
Improve error message for readdir/readdirnames
2018-05-14 21:17:57 +02:00
Alexander Neumann
375868edcf Add documentation 2018-05-13 12:54:23 +02:00
Alexander Neumann
060d8b57e0 Restructure TestApplyPolicy 2018-05-13 12:48:42 +02:00
Alexander Neumann
cc627e832b Add custom Duration type 2018-05-13 12:48:42 +02:00
Alexander Neumann
5a0f0e3faa Add support for keeping a range of snapshots 2018-05-13 12:48:42 +02:00
Matthew Holt
b52f2aa9a4 forget: Add policy to keep snapshots before a date 2018-05-13 12:48:10 +02:00
Alexander Neumann
60ea2435be Improve error message for readdir/readdirnames
As mentioned in the forum[1], restic does not include the dir name when
readdir/readdirnames fails.

[1] https://forum.restic.net/t/readdirnames-readdirent-no-such-file-or-directory/653
2018-05-13 10:34:50 +02:00
Alexander Neumann
159badf5ba Merge pull request #1778 from restic/fix-1771
archiver: Improve error handling
2018-05-13 00:13:54 +02:00
Alexander Neumann
903a3a31dc Merge pull request #1776 from restic/fix-1775
Always use cleaned path for excludes
2018-05-13 00:13:52 +02:00
Alexander Neumann
548227e6df Merge pull request #1773 from ExoUNX/patch-1
Fix small typo
2018-05-13 00:13:49 +02:00
Alexander Neumann
cd03275005 Merge pull request #1774 from ExoUNX/patch-2
Fix formatting
2018-05-13 00:13:46 +02:00
Alexander Neumann
e43c9202a6 archiver: Make sure backend error is passed up 2018-05-12 23:55:59 +02:00
Alexander Neumann
c5e75d1c98 archiver: Add test for early abort on unhandled error 2018-05-12 23:55:59 +02:00
Alexander Neumann
526956af35 archiver: Read files/dirs in order 2018-05-12 23:55:54 +02:00
Alexander Neumann
256104111d archiver: Clarify names 2018-05-12 23:55:54 +02:00
Alexander Neumann
21c83b1725 archiver: Add high-level documentation 2018-05-12 23:55:54 +02:00
Alexander Neumann
581c62ee72 archiver: Improve error handling
This commit changes how the worker goroutines for saving e.g. blobs
interact. Before, it was possible to get stuck sending an instruction to
archive a file or dir when no worker goroutines were available any more.
This commit introduces a `done` channel for each of the worker pools,
which is set to the channel returned by `tomb.Dying()`, so it is closed
when the first worker returned an error.
2018-05-12 23:55:54 +02:00
Alexander Neumann
ef7747313d backup: Use absolute paths for allowed devices 2018-05-11 21:32:44 +02:00
Alexander Neumann
18d4ac2fd9 backup: Always use cleaned path for excludes 2018-05-11 21:32:41 +02:00
Alexander Neumann
9180e2c48a Remove unneeded file excludes 2018-05-11 21:26:10 +02:00
Gaige Lama
a63989afcd Ran gofmt on backend.go
I ran gofmt on backend.go, this appears to valid edit.
2018-05-11 11:07:16 -04:00
Gaige Lama
d3c0bd6d0e Fix small typo
"explicitely" should be "explicitly"
2018-05-11 10:55:12 -04:00
Alexander Neumann
fcfa6f0355 build: Add option to enable PIE build mode 2018-05-11 09:50:10 +02:00
Alexander Neumann
580f90d745 Merge pull request #1770 from restic/handle-blob-save-errors
archiver: Correct error handling
2018-05-10 22:36:00 +02:00
Alexander Neumann
c7b624ba0d Merge pull request #1769 from McKael/cat_snapshot_stacktrace
cat snapshot: Do not display a stack trace with invalid IDs
2018-05-10 21:41:04 +02:00
Alexander Neumann
ca4af43c03 archiver: Return low-level errors
This commit changes the archiver so that low-level errors saving data to
the repo are returned to the caller (instead of being handled by the
error callback function). This correctly bubbles up errors like a full
temp file system and makes restic abort early and makes all other worker
goroutines exit.
2018-05-10 21:30:09 +02:00
Alexander Neumann
1f2463f42e archiver: Return correct error 2018-05-10 20:48:00 +02:00
Alexander Neumann
157c854d04 backup: Remove errored files from status display 2018-05-10 20:48:00 +02:00
Mikael Berthe
ffc276a603 cat: Do not display a stack trace
Don't show a stack trace when "restic cat snapshot" is invoked with
invalid/nonexistent IDs.
2018-05-10 19:22:46 +02:00
Alexander Neumann
e42b7db008 Merge pull request #1767 from restic/update-blazer
Update github.com/kurin/blazer
2018-05-08 22:48:54 +02:00
Alexander Neumann
024148cac9 Update github.com/kurin/blazer 2018-05-08 22:33:55 +02:00
Alexander Neumann
8943741a0b Fix code block 2018-05-08 20:36:58 +02:00
Alexander Neumann
95c5517c35 Merge pull request #1765 from restic/update-deps
Update golang.org/x/{sys,crypto}
2018-05-08 19:34:30 +02:00
Alexander Neumann
06179a7e81 Change URL for Travis 2018-05-08 19:13:39 +02:00
Alexander Neumann
cf1fb50f9c Merge pull request #1764 from restic/fix-display
Fix display
2018-05-07 23:30:50 +02:00
Alexander Neumann
6793300850 Raise minimum Go version to 1.9
The golang.org/sys/crypto library uses math/bits, which was only
introduced in Go 1.9.
2018-05-07 23:28:48 +02:00
Alexander Neumann
2cbdfbf652 Update golang.org/x/{sys,crypto} 2018-05-07 22:47:39 +02:00
Alexander Neumann
b2208bb9c2 Rework termstatus
This now keeps the cursor at the first column of the first status line
so that messages printed to stdout or stderr by some other part of the
progarm will still be visible. The message will overwrite the status
lines, but those are easily reprinted on the next status update.
2018-05-07 21:40:07 +02:00
Alexander Neumann
4c25495d68 backup: Hide percent until total size is known 2018-05-06 20:20:25 +02:00
Alexander Neumann
abdd59ea1b Merge pull request #1757 from restic/fix-debug-1755
Fix panic for nil-pointer dereference
2018-05-03 21:11:37 +02:00
Alexander Neumann
05ca903d48 Mention that first backup will take longer 2018-05-03 20:57:30 +02:00
Alexander Neumann
fd77646f8b Fix panic for nil-pointer dereference
Closes #1755
2018-05-03 20:49:30 +02:00
Alexander Neumann
2a67258867 Merge pull request #1754 from restic/fix-quiet
backup: Disable status output for --quiet
2018-05-02 21:42:17 +02:00
Alexander Neumann
fca4fe4459 backup: Disable status output for --quiet 2018-05-02 21:24:18 +02:00
Alexander Neumann
26757ae2e5 Merge pull request #1750 from restic/archiver-fix-small-things
backup: Tune user interface a bit
2018-05-01 23:15:54 +02:00
Alexander Neumann
9d6890a236 Merge pull request #1751 from restic/fix-archiver
archiver: Use lstat before open/fstat
2018-05-01 23:15:32 +02:00
Alexander Neumann
2218ecd049 archiver: Use lstat before open/fstat
The previous code tried to be as efficient as possible and only do a
single open() on an item to save, and then fstat() on the fd to find out
what the item is (file, dir, other). For normal files, it would then
start reading the data without opening the file again, so it could not
be exchanged for e.g. a symlink.

This behavior starts the watchdog on my machine when /dev is saved
with restic, and after a few seconds, the machine reboots.

This commit reverts the behavior to the strategy the old archiver code
used: run lstat(), then decide what to do. For normal files, open the
file and then run fstat() on the fd to verify it's still a normal file,
then start reading the data.

The downside is that for normal files we now do two stat() calls
(lstat+fstat) instead of only one. On the upside, this does not start
the watchdog. :)
2018-05-01 23:05:50 +02:00
Alexander Neumann
d0974c155d backup: Tune user interface a bit
Make non-verbose mode a bit more verbose.
2018-05-01 22:18:32 +02:00
Alexander Neumann
8026e6fdfb Merge pull request #1749 from restic/add-cache-cmd
Add 'cache' command to list and cleanup cache dirs
2018-05-01 21:25:52 +02:00
Alexander Neumann
01f9662614 Merge pull request #1748 from restic/detect-bom
Respect Encoding and Byte Order Mark when reading text files
2018-05-01 21:25:46 +02:00
Alexander Neumann
f928aeec34 Add 'cache' command 2018-05-01 16:27:44 +02:00
Alexander Neumann
f77bc0fae8 Use textfile.Read() to read files
This converts the text to UTF-8 and removes a Byte Order Mark.
2018-05-01 16:25:53 +02:00
Alexander Neumann
eb6650b201 Add textfile 2018-05-01 14:38:59 +02:00
Alexander Neumann
bc68d55e94 fs: Add TestTempFile 2018-05-01 14:38:41 +02:00
Alexander Neumann
ecbbd851a1 Merge pull request #1744 from restic/fix-parent-detect
backup: Clean target paths before finding parent
2018-05-01 11:13:19 +02:00
Alexander Neumann
336719b058 Merge pull request #1746 from brycied00d/fix_tls-client-cert_argument_parsing
http backend: Parse the correct argument when loading --tls-client-cert
2018-05-01 10:22:10 +02:00
Bryce Chidester
e9f1721678 http backend: Parse the correct argument when loading --tls-client-cert
Previously, the function read from ARGV[1] (hardcoded) rather than the
value passed to it, the command-line argument as it exists in globalOptions.

Resolves #1745
2018-04-30 15:21:09 -07:00
Alexander Neumann
64d98945a6 backup: Clean target paths before finding parent
This resolves an issue described in the forum where restic could not
find a parent snapshot if the target path ends with a slash:
https://forum.restic.net/t/new-archiver-code-please-test/623/23
2018-04-30 22:03:11 +02:00
Alexander Neumann
84f82dae1a Correct typo
Closes #1743
2018-04-30 21:41:30 +02:00
Alexander Neumann
6bfd9f833b Merge pull request #1742 from tbm/typos
Fix typos in section on rclone
2018-04-30 19:21:19 +02:00
Martin Michlmayr
bb1a22d1e6 Fix typos in section on rclone 2018-04-30 18:32:10 +02:00
Alexander Neumann
438719f269 Merge pull request #1741 from restic/improve-archiver
Improve archiver, fix hang on fifos
2018-04-30 16:00:10 +02:00
Alexander Neumann
c83c03ed63 archiver: Fix blocking on pipes 2018-04-30 15:34:58 +02:00
Alexander Neumann
19b9c881ca fs: Add O_NONBLOCK 2018-04-30 15:34:58 +02:00
Alexander Neumann
4e34325035 archiver: Process dirs concurrently 2018-04-30 15:13:28 +02:00
Alexander Neumann
78bd591c7c archiver: Improve buffer pool 2018-04-30 15:13:28 +02:00
Alexander Neumann
39ac12f6ea archiver: Correct comment 2018-04-30 14:19:07 +02:00
Alexander Neumann
400730afca archiver: Improve memory usage, tune buffer pool 2018-04-30 14:19:07 +02:00
Alexander Neumann
d80e108b03 backup: Clear status lines on finish 2018-04-30 14:19:07 +02:00
Alexander Neumann
846c2b6869 backup: Fix ETA calculation for >= 100% 2018-04-30 14:19:07 +02:00
Alexander Neumann
d8bbe5dc84 Print repository ID after opening 2018-04-30 14:19:07 +02:00
Alexander Neumann
d926b9fd80 Add profile build flag 2018-04-30 14:19:04 +02:00
Alexander Neumann
4ba8d40282 Add block profile option 2018-04-30 14:18:45 +02:00
Alexander Neumann
4fb1401266 Fix --cacert help text 2018-04-29 13:17:33 +02:00
Alexander Neumann
6d4c40f8d0 Add section about file read concurrency 2018-04-28 22:39:56 +02:00
Alexander Neumann
56e394ac33 Merge pull request #1494 from restic/new-archiver
New archiver code
2018-04-28 22:24:39 +02:00
Alexander Neumann
c3cc5d7cee Update docs 2018-04-28 22:08:11 +02:00
Alexander Neumann
6b12b92339 Add entry to changelog 2018-04-28 22:08:11 +02:00
Alexander Neumann
16c314ab7f termstatus: Don't print status if in background 2018-04-28 22:08:11 +02:00
Alexander Neumann
1449d7dc29 Remove background checking code 2018-04-28 22:08:11 +02:00
Alexander Neumann
0e78ac92d8 Use new archiver code for backup 2018-04-28 22:08:11 +02:00
Alexander Neumann
c703d21d55 Vendor gopkg.in/tomb.v2 2018-04-27 21:42:15 +02:00
Alexander Neumann
1af96fc6dd Add termstatus 2018-04-27 21:42:15 +02:00
Alexander Neumann
9fac2ca832 Add flags to set verbosity 2018-04-25 14:42:45 +02:00
Alexander Neumann
a5c0cf2324 Add workaround for symlinked temp dir on darwin
Chdir to the tempdir, then use os.Getwd() to get the name that
filepath.Abs() uses (and stores in the Snapshot).
2018-04-25 14:42:45 +02:00
Alexander Neumann
38926d8576 Use new archiver code in tests 2018-04-25 14:42:45 +02:00
Alexander Neumann
f279731168 Add new archiver code 2018-04-25 14:42:45 +02:00
Alexander Neumann
76b616451f Remove unneeded code 2018-04-23 21:40:33 +02:00
Alexander Neumann
fd12a3af20 Remove old archiver code 2018-04-23 21:40:33 +02:00
Alexander Neumann
3cd92efdcf Vendor github.com/mattn/go-isatty 2018-04-22 11:37:05 +02:00
Alexander Neumann
b804279fe8 integration tests: Don't print anything to stdout 2018-04-22 11:37:05 +02:00
Alexander Neumann
a56b8fad87 repository: Improve buffer pooling 2018-04-22 11:37:05 +02:00
Alexander Neumann
4c00efd4bf Vendor go-cmp 2018-04-22 11:37:05 +02:00
Alexander Neumann
b6f98bdb02 node: Fill minimal info 2018-04-22 11:37:05 +02:00
Alexander Neumann
c4b2486b7c fs: Add interface and FS implementations
This adds two implementations of the new `FS` interface: One for the local
file system (`Local`) and one for a single file read from an
`io.Reader` (`Reader`).
2018-04-22 11:37:05 +02:00
Alexander Neumann
83ca08245b checker: Check metadata size and blob sizes 2018-04-22 11:37:05 +02:00
Alexander Neumann
a069467e72 ls: Improve output 2018-04-22 11:37:05 +02:00
Alexander Neumann
6a7c23d2ae tree: Add convenience functions 2018-04-22 11:37:05 +02:00
Alexander Neumann
cc847a3d6d tree: Improve error for pre-existing node 2018-04-22 11:37:05 +02:00
Alexander Neumann
baebf45e2e FindLatestSnapshot: Use absolute paths 2018-04-22 11:37:05 +02:00
Alexander Neumann
fa4f438bc1 snapshot: Do not modify slice of paths 2018-04-22 11:37:05 +02:00
Alexander Neumann
4e0b2a8e3a snapshot: correct error handling for filepath.Abs 2018-04-22 11:37:05 +02:00
Alexander Neumann
0532f08048 Add test.Helper, also works with Go 1.8 2018-04-22 11:37:05 +02:00
Alexander Neumann
a472868e06 fs: Add TestChdir() 2018-04-22 11:37:05 +02:00
Alexander Neumann
e4fdc5eb76 fs: Add IsRegularFile() 2018-04-22 11:37:05 +02:00
Alexander Neumann
09365cc4ea Add --trace-profile 2018-04-22 11:37:05 +02:00
Eri Bastos
2aa6b49651 Return exit code 130 when SIGINT is received 2018-04-20 21:09:50 +02:00
Alexander Neumann
7877797c7e Merge pull request #1720 from mholt/new-password-flag
key: Add --new-password flag for non-interactive password changes
2018-04-20 14:52:41 +02:00
Matthew Holt
1a26355dbe Add changelog file 2018-04-20 14:29:39 +02:00
Matthew Holt
c5829e9ffc key: Add flag for non-interactive password changes 2018-04-20 14:29:39 +02:00
Alexander Neumann
b5b246edd5 Add entry to changelog 2018-04-20 14:23:37 +02:00
Alexander Neumann
ee5e14d536 Merge pull request #1731 from restic/fix-1730
Do not restore sockets, correct error handling
2018-04-20 14:21:01 +02:00
Alexander Neumann
09bd924710 Do not restore sockets, correct error handling
Closes #1730
2018-04-20 13:53:11 +02:00
Alexander Neumann
a9c2e84ccd Merge pull request #1718 from ardichoke/patch-1
Update Autocompletion  Generation Documentation
2018-04-13 22:23:37 +02:00
Ryan DeShone
a144b81c4a Update Autocompletion Generation Documentation
The autocompletion command has been changed to generate. Update documentation to reflect this change.
2018-04-12 11:50:26 -04:00
Alexander Neumann
3c453a4217 Merge pull request #1715 from rawtaz/patch-1
Add "Including and Exluding Files" heading in docs
2018-04-11 21:27:22 +02:00
rawtaz
1e2f23d77a Add "Including and Exluding Files" heading in docs
Adds "Including and Exluding Files" heading in the backup section in the docs.

[ci skip]
2018-04-11 21:14:26 +02:00
Alexander Neumann
2c76e724ab Merge pull request #1712 from restic/list-ignore-non-existing-dirs
Improve handling non-existing dirs in List()
2018-04-10 21:54:00 +02:00
Alexander Neumann
577faa7570 local/sftp: Handling non-existing dirs in List() 2018-04-10 21:35:30 +02:00
Alexander Neumann
6a34e0d10f Merge pull request #1709 from restic/improve-check-errors
Improve error message for orphaned pack files
2018-04-07 13:38:41 +02:00
Alexander Neumann
b08f21cdc6 Add entry to changelog 2018-04-07 13:05:44 +02:00
Alexander Neumann
1c1fede399 Improve error message for orphaned pack files 2018-04-07 10:07:54 +02:00
Alexander Neumann
63a0913e6e Merge pull request #1705 from alirazeen/doc-add-scripting
Add scripting section to docs
2018-04-07 09:47:17 +02:00
Ali Razeen
325957443e Add scripting section to docs 2018-04-04 10:43:20 -04:00
Alexander Neumann
4b2d3b15a2 Add entry to changelog 2018-04-03 20:42:06 +02:00
Alexander Neumann
4e2a87c920 Merge pull request #1703 from ebastos/issue1608
Fixed issue #1608 - Use --time argument properly
2018-04-03 20:40:41 +02:00
Eri Bastos
901e1b129c Fixed issue #1608 - Use --time argument properly
Backups via stdin will now handle --time argument and pass it down as
expected
2018-04-03 14:40:42 -03:00
Alexander Neumann
4478d633e2 Merge pull request #1702 from restic/update-simple-scrypt
Update github.com/elithrar/simple-scrypt
2018-04-02 20:07:18 +02:00
Alexander Neumann
92f516b1d4 Update github.com/elithrar/simple-scrypt
For details see #1697
2018-04-02 19:48:25 +02:00
Alexander Neumann
03193e6d92 Fix changelog entries 2018-04-02 12:42:48 +02:00
Alexander Neumann
01fe719aff check: Make sure temp cache dir is removed 2018-04-01 18:09:53 +02:00
Alexander Neumann
2c964df3e2 Merge pull request #1699 from restic/fix-incremental-backup-test
Slightly increment size for TestIncrementalBackup
2018-04-01 14:33:12 +02:00
Alexander Neumann
8919125b0b Merge pull request #1696 from restic/fix-check-cache
check: Improve cache handling
2018-04-01 14:33:00 +02:00
Alexander Neumann
1f5137aa70 Add entry to CHANGELOG 2018-04-01 13:59:27 +02:00
Alexander Neumann
a95eb33616 check: Use cache in temporary directory if possible
Closes #1694
2018-04-01 13:59:27 +02:00
Alexander Neumann
e68a7fea8a check: Allow filling the cache during check
Closes #1665
2018-04-01 13:59:27 +02:00
Alexander Neumann
2e7ec717c1 repository: Move cache preparation into function 2018-04-01 13:59:27 +02:00
Alexander Neumann
22d5061df2 Merge pull request #1698 from restic/reduce-backend-tests-travis
Travis: Skip cloud backend tests for most Go versions
2018-04-01 13:58:38 +02:00
Alexander Neumann
4544a77172 Slightly increment size for TestIncrementalBackup
This should make the test more reliable, it should hit the accidental
"repo is has grown too much" way less often.
2018-04-01 13:49:42 +02:00
Alexander Neumann
b3a073e066 Travis: Skip cloud backend tests for most Go versions 2018-04-01 13:13:38 +02:00
Alexander Neumann
b077a1227b Merge pull request #1657 from restic/rclone-backend
Rclone backend
2018-04-01 10:56:10 +02:00
Alexander Neumann
3f48e0e0f4 Add extra options to rclone
For details see https://github.com/restic/restic/pull/1657#issuecomment-377707486
2018-04-01 10:34:30 +02:00
Alexander Neumann
86f4b03730 Remove unneeded byte counters 2018-04-01 10:18:38 +02:00
Alexander Neumann
c43c94776b rclone: Make concurrent connections configurable 2018-04-01 10:18:38 +02:00
Alexander Neumann
0b776e63e7 backend/rclone: Request random file name
When `/` is requested, rclone returns the list of all files in the
remote, which is not what we want (and it can take quite some time).
2018-04-01 10:18:38 +02:00
Alexander Neumann
360ff1806a doc: Fix instructions for rclone backend 2018-04-01 10:18:38 +02:00
Alexander Neumann
1beeb7d0dd doc/REST: Make documentation match reality 2018-04-01 10:18:38 +02:00
Alexander Neumann
e978b36713 doc: Add hint how to debug rclone 2018-04-01 10:18:38 +02:00
Alexander Neumann
737d93860a Extend first timeout to 60 seconds. 2018-04-01 10:18:38 +02:00
Alexander Neumann
011217e4bf backend/rclone: Improve documentation and README 2018-04-01 10:18:38 +02:00
Alexander Neumann
362d5afec4 Add entry to changelog 2018-04-01 10:18:38 +02:00
Alexander Neumann
4172fcd167 doc: Add rclone backend 2018-04-01 10:18:38 +02:00
Alexander Neumann
518bf4e5f6 doc: Correct verbatim text in the manual 2018-04-01 10:18:38 +02:00
Alexander Neumann
17312d3a98 backend/rest: Ensure base URL ends with slash
This makes it easier for rclone.
2018-04-01 10:18:38 +02:00
Alexander Neumann
4d5c7a8749 backend/rclone: Make sure rclone terminates 2018-04-01 10:18:38 +02:00
Alexander Neumann
fc0295016a Address code review comments 2018-04-01 10:18:38 +02:00
Alexander Neumann
99b62c11b8 backend/rclone: Stop rclone in case of errors 2018-04-01 10:18:38 +02:00
Alexander Neumann
6d9a029e09 backend/rclone: Prefix all error messages 2018-04-01 10:18:38 +02:00
Alexander Neumann
20352886f3 Update Gopkg.lock 2018-04-01 10:18:38 +02:00
Alexander Neumann
3622b60c13 CI: Check that rclone backend test isn't skipped 2018-04-01 10:16:31 +02:00
Alexander Neumann
065fe1e54f backend/rclone: Skip test if binary is unavailable 2018-04-01 10:16:31 +02:00
Alexander Neumann
4dc0f24b38 backend/tests: Drain reader before returning error 2018-04-01 10:16:31 +02:00
Alexander Neumann
fe99340e40 Add rclone backend 2018-04-01 10:16:31 +02:00
Alexander Neumann
e377759c81 rest: Export Backend struct 2018-04-01 10:16:31 +02:00
Alexander Neumann
61f6db25f4 CI: install rclone 2018-04-01 10:16:31 +02:00
Alexander Neumann
cabbbd2b14 backend/rest: Export Content-Types 2018-04-01 10:16:31 +02:00
Alexander Neumann
cf4cf94418 Move backend/sftp.StartForeground to backend/ 2018-04-01 10:16:31 +02:00
Alexander Neumann
34f27edc03 Refactor SplitShellStrings 2018-04-01 10:16:31 +02:00
Alexander Neumann
345b6c4694 Move backend/sftp.SplitShellArgs to backend/ 2018-04-01 10:16:31 +02:00
Alexander Neumann
e4a39e02d2 Merge pull request #1695 from xulongwu4/patch-2
Add instructions for installation on Solus
2018-03-31 09:14:25 +02:00
xulongwu4
432e167255 Add instructions for installation on Solus 2018-03-30 17:36:29 -04:00
Alexander Neumann
594256bfa4 Merge pull request #1693 from restic/update-deps
Update dependencies
2018-03-30 17:19:08 +02:00
Alexander Neumann
0fcb1e6b7a Merge pull request #1692 from restic/print-forget-policy
forget: Print policy
2018-03-30 17:19:05 +02:00
Alexander Neumann
38795c66c9 Update vendored library gopkg.in/yaml.v2 2018-03-30 12:53:13 +02:00
Alexander Neumann
c0960f538f Update vendored library google.golang.org/api 2018-03-30 12:51:18 +02:00
Alexander Neumann
5b6568875c Update vendored library golang.org/x/text 2018-03-30 12:50:04 +02:00
Alexander Neumann
d8dd79eb0b Update vendored library golang.org/x/sys 2018-03-30 12:48:49 +02:00
Alexander Neumann
2bdeb645b9 Update vendored library golang.org/x/sync 2018-03-30 12:47:30 +02:00
Alexander Neumann
9f2ffa3e50 Update vendored library golang.org/x/oauth2 2018-03-30 12:46:26 +02:00
Alexander Neumann
d4bab5c133 Update vendored library golang.org/x/net 2018-03-30 12:45:07 +02:00
Alexander Neumann
3473d73d0c Update vendored library github.com/spf13/cobra 2018-03-30 12:43:03 +02:00
Alexander Neumann
917cc542c9 Update vendored library github.com/sirupsen/logrus 2018-03-30 12:41:46 +02:00
Alexander Neumann
a9cf5d482a Update vendored library github.com/russross/blackfriday 2018-03-30 12:40:05 +02:00
Alexander Neumann
75946e7c58 Update vendored library github.com/pkg/xattr 2018-03-30 12:38:37 +02:00
Alexander Neumann
19035e977b Update vendored library github.com/pkg/sftp 2018-03-30 12:37:16 +02:00
Alexander Neumann
d9ba9279e0 Update vendored library github.com/ncw/swift 2018-03-30 12:35:13 +02:00
Alexander Neumann
31e156c666 Update vendored library github.com/minio/minio-go 2018-03-30 12:33:40 +02:00
Alexander Neumann
7e6fff324c Update vendored library golang.org/x/crypto 2018-03-30 12:26:26 +02:00
Alexander Neumann
e94d2da890 Update vendored library github.com/golang/protobuf 2018-03-30 11:52:18 +02:00
Alexander Neumann
874b3dbbd9 Update vendored library github.com/go-ini/ini 2018-03-30 11:51:01 +02:00
Alexander Neumann
0d01c27c9e Update vendored library github.com/dgrijalva/jwt-go 2018-03-30 11:49:18 +02:00
Alexander Neumann
30110fcfc2 Update vendored library github.com/cpuguy83/go-md2man 2018-03-30 11:48:16 +02:00
Alexander Neumann
673f0bbd6c Update vendored library github.com/cenkalti/backoff 2018-03-30 11:45:27 +02:00
Alexander Neumann
5a77b2ab49 Update vendored library github.com/Azure/azure-sdk-for-go 2018-03-30 11:42:11 +02:00
Alexander Neumann
a951e7b126 Update vendored library cloud.google.com/go 2018-03-30 11:41:12 +02:00
Alexander Neumann
d3f9c8b362 forget: Print policy 2018-03-30 10:24:26 +02:00
Alexander Neumann
a4ff591165 Update URL to template 2018-03-25 11:55:37 +02:00
Alexander Neumann
49dd70c771 Merge pull request #1686 from gtrafimenkov/minor-spelling-fixes
Correct spelling mistakes
2018-03-25 11:54:53 +02:00
Gennady Trafimenkov
a64f24029b Correct spelling mistakes 2018-03-25 00:42:33 +03:00
Alexander Neumann
0886738d24 Add entry to CHANGELOG 2018-03-24 18:40:49 +01:00
Alexander Neumann
9fc38803e0 Merge pull request #1684 from restic/fix-rest-tests
Fix rest-server tests
2018-03-24 18:39:44 +01:00
Alexander Neumann
e5c929b793 Fix rest-server tests
Since today, the rest-server needs to be explicitly told (via
`--no-auth`) that authentication is not necessary.
2018-03-24 18:06:21 +01:00
Alexander Neumann
0e0fee9c8f Update changelog template 2018-03-24 17:31:21 +01:00
Alexander Neumann
26769a39eb Merge pull request #1679 from rawtaz/version-oneliner
Make version output one line.
2018-03-23 21:14:37 +01:00
Leo R. Lundgren
923be90906 Make version output one line. 2018-03-21 21:49:03 +01:00
Alexander Neumann
84a22eac92 Merge pull request #1675 from oliver/doc-fixes
Minor doc fixes/improvements
2018-03-21 20:54:51 +01:00
Alexander Neumann
6eb1be0be4 Fix changelog files 2018-03-21 20:53:11 +01:00
Alexander Neumann
f31bbcf1a9 CI: Return error when calens fails 2018-03-21 20:53:01 +01:00
Alexander Neumann
5d09fca6a2 Merge pull request #1676 from bowensong/quiet-skip-scan
Skip archiver.Scan before backup when --quiet is set
2018-03-21 20:45:34 +01:00
Bowen Song
34671d7c9b Skip archiver.Scan before backup when --quiet is set 2018-03-20 20:44:10 +00:00
Alexander Neumann
4a524da736 FAQ: Add suggestion on how to solve sftp error 2018-03-18 21:39:11 +01:00
Oliver Gerlich
e361cc3807 040_backup.rst: add note regarding paths in --files-from argument 2018-03-18 20:51:04 +01:00
Alexander Neumann
3cd8a7bc96 Fix small things left open by #1552 2018-03-18 19:54:12 +01:00
Alexander Neumann
8206f85d2e Merge pull request #1552 from lawrencejones/use-auto-auth
Automatically load Google auth
2018-03-18 19:53:30 +01:00
Alexander Neumann
7022144e0f Merge pull request #1673 from rawtaz/doc-precompiled
doc: Add more info about using pre-compiled builds.
2018-03-18 19:50:25 +01:00
Oliver Gerlich
1bee3e01fa 040_backup.rst: fix typo 2018-03-18 17:06:21 +01:00
Oliver Gerlich
624a2d8305 040_backup.rst: quote wildcard characters
This doesn't really matter for --exclude in most cases, but it avoids
spreading bad ideas.
2018-03-18 17:05:53 +01:00
Lawrence Jones
57c6233982 dep ensure 2018-03-16 10:31:30 +00:00
Leo R. Lundgren
c161aba084 doc: Add more info about using pre-compiled builds. 2018-03-15 21:23:07 +01:00
Alexander Neumann
0279fd7212 Merge pull request #1669 from restic/make-tests-faster
Reduce test runtime
2018-03-13 19:50:34 +01:00
Alexander Neumann
dedf17f5e8 Merge pull request #1650 from copart/patch-1
Added copr repositories for RHEL/Centos/Fedora
2018-03-11 21:52:18 +01:00
Alexander Neumann
817890794d Merge pull request #1668 from restic/fix-1663
Return the first password/key which works
2018-03-11 21:51:54 +01:00
Alexander Neumann
b9ada91054 Reduce data for TestCreateSnapshot 2018-03-11 21:42:39 +01:00
Alexander Neumann
dfb6d0fced Reduce data for TestIncrementalBackup 2018-03-11 21:17:27 +01:00
Alexander Neumann
c6c1dccc53 Reduce data set for TestRestore 2018-03-11 21:10:37 +01:00
Alexander Neumann
279566bafe Reduce dataset for integration tests 2018-03-11 21:07:47 +01:00
Alexander Neumann
c67a8452f7 Disable polynomial check for chunker for tests 2018-03-11 20:59:40 +01:00
Alexander Neumann
5253ef218c Remove TestParallelSaveWithDuplication 2018-03-11 19:49:48 +01:00
Alexander Neumann
0923976909 Remove TestArchiverDuplication 2018-03-11 19:44:25 +01:00
Lawrence Jones
492baf991f Update docs and add changelog entry: Google auth
Add documentation around using default Google application credentials,
along with a changelog extra that describes the feature and the
potential impact on existing restic uses (read: none).
2018-03-11 17:12:30 +00:00
Lawrence Jones
0dfdc11ed9 Automatically load Google auth
This change removes the hardcoded Google auth mechanism for the GCS
backend, instead using Google's provided client library to discover and
generate credential material.

Google recommend that client libraries use their common auth mechanism
in order to authorise requests against Google services. Doing so means
you automatically support various types of authentication, from the
standard GOOGLE_APPLICATION_CREDENTIALS environment variable to making
use of Google's metadata API if running within Google Container Engine.
2018-03-11 17:11:25 +00:00
Alexander Neumann
54c6837ec4 Merge pull request #1651 from qbit/obsd_inst
doc: Add install instructions for OpenBSD
2018-03-11 14:49:40 +01:00
Alexander Neumann
e085713b35 Return the first password/key which works
Closes #1663
2018-03-11 14:12:21 +01:00
Alexander Neumann
e77d8c64a7 Merge pull request #1661 from restic/fix-rest-content-length
rest: Really set Content-Length HTTP header
2018-03-10 20:34:30 +01:00
Alexander Neumann
a410fa16a1 Merge pull request #1667 from restic/improve-error-config
Return error message for config decryption failure
2018-03-10 20:34:25 +01:00
Alexander Neumann
b3e1089cf9 Return error message for config decryption failure
See #1663
2018-03-09 21:05:35 +01:00
Alexander Neumann
7f8e269891 Merge pull request #1662 from ebastos/version_password
Skip checking for password file issue #1632
2018-03-09 20:42:32 +01:00
Alexander Neumann
fcc9ce81ba rest: Really set Content-Length HTTP header 2018-03-09 20:21:34 +01:00
Eri Bastos
b9d643358a Skip checking for password file existence if command is 'version' - Issue #1632 2018-03-08 17:55:03 -04:00
Alexander Neumann
ab5ef600a2 Merge pull request #1660 from ncw/rest-fix
backend/rest: check HTTP error response for List
2018-03-08 21:55:58 +01:00
Nick Craig-Wood
04c4033695 backend/rest: check HTTP error response for List
Before this change restic would attempt to JSON decode the error
message resulting in confusing `Decode: invalid character 'B' looking
for beginning of value` messages.  Afterwards it will return `List
failed, server response: 400 Bad Request (400)`
2018-03-08 10:22:43 +00:00
Alexander Neumann
de37b68baa Move all unreleased changelog entries 2018-03-05 21:31:52 +01:00
Alexander Neumann
bdc206d440 Remove unneeded mkdir 2018-03-05 21:17:57 +01:00
Alexander Neumann
efe2e792b3 Correct changelog entries 2018-03-05 21:17:52 +01:00
Alexander Neumann
6f3c23eba7 Merge pull request #1653 from restic/fix-1652
lock: Ignore invalid lock file
2018-03-05 20:49:03 +01:00
Alexander Neumann
4b34bc3210 Reformat changelog entry 2018-03-05 20:40:40 +01:00
Alexander Neumann
6ed9100aa1 Add version number hint for the changelog entry 2018-03-05 20:34:07 +01:00
Alexander Neumann
c63b02d0f1 Move changelog entries to unreleased version 2018-03-05 20:32:10 +01:00
Alexander Neumann
d0205ec889 Add entry to changelog 2018-03-05 20:22:45 +01:00
Alexander Neumann
d8dcbc89d1 lock: Ignore invalid lock file
This commit fixes a bug introduced in
e9ea268847: When an invalid lock is
encountered (e.g. if the file is empty), the code used to ignore that,
but now returns the error.

Now, invalid files are ignored for the normal lock check, and removed
when `restic unlock --remove-all` is run.

Closes #1652
2018-03-05 20:19:57 +01:00
Alexander Neumann
be0a5b7f06 Merge pull request #1649 from jasperla/solaris
Minimal set of patches to get restic working on Solaris
2018-03-05 20:00:17 +01:00
Aaron Bieber
24ce08e122 doc: Add install instructions for OpenBSD 2018-03-05 06:31:57 -07:00
copart
864eaeab7c Added copr repositories for RHEL/Centos/Fedora 2018-03-04 15:50:30 -05:00
Jasper Lievisse Adriaanse
96311d1a2b Add support for illumos/Solaris
This does come without xattr/fuse support at this point.

NB: not hooking up the integration tests as restic won't compile without
    cgo with Go < 1.10.
2018-03-04 20:11:29 +00:00
Alexander Neumann
da77f4a2e2 Merge pull request #1647 from duzvik/aws-session-token
Change priority of AWS credential providers to accept AWS_SESSION_TOKEN
2018-03-04 20:54:56 +01:00
denis.uzvik
6bb1bcce03 Change priority of AWS credential providers to accept AWS_SESSION_TOKEN 2018-03-04 19:58:27 +02:00
Alexander Neumann
6edf28d1e1 Merge pull request #1639 from restic/improve-backend-save
backend: Improve/Cleanup Save()
2018-03-04 13:35:50 +01:00
Alexander Neumann
929afc63d5 Use int64 for the length in the RewindReader 2018-03-04 10:40:42 +01:00
Alexander Neumann
99f7fd74e3 backend: Improve Save()
As mentioned in issue [#1560](https://github.com/restic/restic/pull/1560#issuecomment-364689346)
this changes the signature for `backend.Save()`. It now takes a
parameter of interface type `RewindReader`, so that the backend
implementations or our `RetryBackend` middleware can reset the reader to
the beginning and then retry an upload operation.

The `RewindReader` interface also provides a `Length()` method, which is
used in the backend to get the size of the data to be saved. This
removes several ugly hacks we had to do to pull the size back out of the
`io.Reader` passed to `Save()` before. In the `s3` and `rest` backend
this is actively used.
2018-03-03 15:49:44 +01:00
Alexander Neumann
58306bfabb Merge pull request #1648 from duzvik/s3-bucketexists
Ignore s3 AccessDenied error, during creation of repository
2018-03-02 22:38:33 +01:00
denis.uzvik
f6890210aa Add entry to changelog 2018-03-02 11:06:06 +02:00
denis.uzvik
5873ab4031 Ignore s3 AccessDenied error, during creation of repository 2018-03-02 10:47:20 +02:00
Alexander Neumann
ab7a3a803d Update build.go from github.com/fd0/build-go 2018-02-28 21:19:28 +01:00
Alexander Neumann
1e868933c5 build.go: Allow specifying the temp dir to use 2018-02-27 21:56:42 +01:00
Alexander Neumann
21f67a0a13 Fix linebreak in GitHub changelog template 2018-02-26 21:41:37 +01:00
Alexander Neumann
272ccec7e1 Add VERSION for 0.8.3 2018-02-26 21:32:16 +01:00
Alexander Neumann
68bf1509bd Update manpages and auto-completion 2018-02-26 21:32:16 +01:00
Alexander Neumann
cfccd67600 Generate CHANGELOG.md for 0.8.3 2018-02-26 21:32:07 +01:00
Alexander Neumann
bc461d32e0 Add release date for 0.8.3 2018-02-26 21:31:50 +01:00
Alexander Neumann
ee4bfdf954 changelog: Fix spelling 2018-02-26 21:27:32 +01:00
Alexander Neumann
3037894f62 Add entry to changelog 2018-02-26 21:20:41 +01:00
Alexander Neumann
89075bdf6d Merge pull request #1643 from restic/fix-1641
Ignore files in the repo with invalid names
2018-02-26 21:18:05 +01:00
Alexander Neumann
c323f73bf9 Ignore files in the repo with invalid names
Closes #1641
2018-02-26 20:53:38 +01:00
Alexander Neumann
aef5e03731 Merge pull request #1638 from restic/fix-list-retry
backend/retry: return worker function error and abort
2018-02-25 21:20:08 +01:00
Alexander Neumann
fc1f74d32d Merge pull request #1640 from restic/fix-1637
mount: Ignore non-existing locks
2018-02-25 14:00:04 +01:00
Alexander Neumann
7d59df1ab8 mount: Ignore non-existing locks
Closes #1637
2018-02-25 13:11:03 +01:00
Alexander Neumann
2866f3f31c Add pull request to changelog entry 2018-02-24 14:53:46 +01:00
Alexander Neumann
dc1154c8ad Merge pull request #1556 from ifedorenko/check-subset
Add --read-data-subset flag to check command
2018-02-24 14:53:20 +01:00
Alexander Neumann
35a816e8ab Add entry to changelog 2018-02-24 13:34:42 +01:00
Alexander Neumann
93210614f4 backend/retry: return worker function error and abort
This is a bug fix: Before, when the worker function fn in List() of the
RetryBackend returned an error, the operation is retried with the next
file. This is not consistent with the documentation, the intention was
that when fn returns an error, this is passed on to the caller and the
List() operation is aborted. Only errors happening on the underlying
backend are retried.

The error leads to restic ignoring exclusive locks that are present in
the repo, so it may happen that a new backup is written which references
data that is going to be removed by a concurrently running `prune`
operation.

The bug was reported by a user here:
https://forum.restic.net/t/restic-backup-returns-0-exit-code-when-already-locked/484
2018-02-24 13:26:13 +01:00
Alexander Neumann
dfd37afee2 Merge pull request #1636 from kurin/pack-header
Refactor the eager-header reads for readability.
2018-02-23 17:36:45 +01:00
Toby Burress
08a5281bd4 Incorporate PR review comments. 2018-02-22 17:37:10 +00:00
Toby Burress
cdb48a8970 Add tests for the eager-header refactor. 2018-02-22 01:14:04 +00:00
Toby Burress
4fd5f0b8a9 Refactor the eager-header reads for readability.
This pulls the header reads into a function that works in terms of the
number of records requested.  This preserves the existing logic of
initially reading 15 records and then falling back if that fails.

In the event of a header with more than 15 records, it will read all
records, including the already-seen final 15 records.
2018-02-22 00:45:40 +00:00
Alexander Neumann
92ad6bf74f Add pull request to changelog 2018-02-21 19:52:16 +01:00
Alexander Neumann
2c7dd3edf4 Merge pull request #1635 from ifedorenko/1633-negative-load-offset
Fixed unexpected 'pack file cannot be listed' error
2018-02-21 19:51:38 +01:00
Igor Fedorenko
19e7803ac6 Fixed unexpected 'pack file cannot be listed' error
Fixes #1633

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-02-20 21:28:57 -05:00
Alexander Neumann
9f0605766c Add entry to changelog 2018-02-20 22:10:52 +01:00
Alexander Neumann
1a5d7a9965 Merge pull request #1634 from restic/update-blazer
Update github.com/kurin/blazer to 0.3.0
2018-02-20 22:01:30 +01:00
Alexander Neumann
296769355d Update github.com/kurin/blazer to 0.3.0
This commit will reduce the number of HTTP requests per file uploaded
from two to one.
2018-02-20 21:01:21 +01:00
Igor Fedorenko
07d080830e Add --read-data-subset flag to check command
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-02-18 23:31:27 -05:00
Alexander Neumann
c99eabfb37 Merge pull request #1625 from restic/update-blazer
Update B2 client library (github.com/kurin/blazer)
2018-02-18 19:18:52 +01:00
Alexander Neumann
842fe43590 Update github.com/kurin/blazer to 0.2.2 2018-02-18 14:53:23 +01:00
Alexander Neumann
be02008025 Merge pull request #1611 from qbit/master
On OpenBSD only root can set sticky bit
2018-02-18 12:57:54 +01:00
Alexander Neumann
29da86b473 Merge pull request #1623 from restic/backend-relax-restrictions
backend: Relax requirement for new files
2018-02-18 12:56:52 +01:00
Alexander Neumann
bad7215696 Add entry to CHANGELOG 2018-02-18 12:04:44 +01:00
Alexander Neumann
881ff5e554 Move changelog file, improve text 2018-02-18 11:51:45 +01:00
Alexander Neumann
86b7fd0335 Merge pull request #1624 from rawtaz/patch-1
Attempt to make issue template a bit clearer
2018-02-18 11:50:11 +01:00
rawtaz
70209d7d1d End both list items with a dot.
Changed my mind after checking other lists in the project's files, ending with a dot seems to be preferred here, and I like that better too.
2018-02-17 23:47:40 +01:00
rawtaz
f07552161c Remove potentially excessive dot
Some people like list items to not end with a comma or dot, some like it when they do. To keep things like and coherent I removed them in this case.
2018-02-17 23:40:30 +01:00
rawtaz
856f3a9135 Add forum URL back 2018-02-17 23:36:47 +01:00
Alexander Neumann
49e9bcadb7 Merge pull request #1560 from ifedorenko/1559-load-error-handling
Retry all repository file download errors
2018-02-17 23:25:28 +01:00
rawtaz
1b8823ef2e Attempt to make issue template a bit clearer 2018-02-17 23:19:58 +01:00
Alexander Neumann
b5062959c8 backend: Relax requirement for new files
Before, all backend implementations were required to return an error if
the file that is to be written already exists in the backend. For most
backends, that means making a request (e.g. via HTTP) and returning an
error when the file already exists.

This is not accurate, the file could have been created between the HTTP
request testing for it, and when writing starts. In addition, apart from
the `config` file in the repo, all other file names have pseudo-random
names with a very very low probability of a collision. And even if a
file name is written again, the way the restic repo is structured this
just means that the same content is placed there again. Which is not a
problem, just not very efficient.

So, this commit relaxes the requirement to return an error when the file
in the backend already exists, which allows reducing the number of API
requests and thereby the latency for remote backends.
2018-02-17 22:39:18 +01:00
Alexander Neumann
2fb4d44a4d Add VERSION for 0.8.2 2018-02-17 20:27:28 +01:00
Alexander Neumann
3c08f956a3 Update manpages and auto-completion 2018-02-17 20:27:28 +01:00
Alexander Neumann
1463a7aaaa Generate CHANGELOG.md for 0.8.2 2018-02-17 20:27:17 +01:00
Alexander Neumann
76b8e080c3 changelog: Add release date for 0.8.2 2018-02-17 20:26:43 +01:00
Alexander Neumann
a9f9243afc scripts/release: Fix usage for restic/builder
The restic/builder Docker container now has a slightly different usage.
2018-02-17 19:10:32 +01:00
Alexander Neumann
34bf9de339 Fix Go versions for Travis 2018-02-17 18:48:15 +01:00
Alexander Neumann
597f6be4e4 Disable CI tests on Go 1.8 on OS X 2018-02-17 18:39:16 +01:00
Alexander Neumann
aeeec0ff94 Update Go version for CI tests 2018-02-17 18:37:37 +01:00
Igor Fedorenko
ab040d8811 Introduced repository.DownloadAndHash helper
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-02-16 21:13:11 -05:00
Igor Fedorenko
d58ae43317 Reworked Backend.Load API to retry errors during ongoing download
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-02-16 21:12:14 -05:00
Alexander Neumann
b723094739 scripts/release: Check changelog subdir with date 2018-02-16 23:13:36 +01:00
Alexander Neumann
84dcc3c396 Update changelog/ for new version of calens
We've reworked calens[1] a bit, the changelog/releases file is not
needed any more. Insteady, the release date is stored in the dir name
which contains the changelog entries.

[1] https://github.com/restic/calens
2018-02-16 23:05:58 +01:00
Alexander Neumann
21747bc219 Remove punctuation from changelog entry title 2018-02-16 22:49:24 +01:00
Alexander Neumann
28b00891d2 scripts/release: Check Go version in container 2018-02-16 22:35:53 +01:00
Alexander Neumann
b4e00c324f scripts/release: Check release date first 2018-02-16 21:52:34 +01:00
Alexander Neumann
4eb9df63cf Merge pull request #1616 from rawtaz/fix-backup-doc
Restructure and complement exclude options doc
2018-02-14 22:11:14 +01:00
Alexander Neumann
fde674d214 Merge pull request #1615 from rawtaz/fix-forget-doc
Fix #1519 and formatting in `forget` doc
2018-02-14 22:11:10 +01:00
rawtaz
fbb08c14de Note that multiple --exclude-file may be used. 2018-02-14 22:05:52 +01:00
Leo R. Lundgren
8b95b48bae Fix grammar and formatting. 2018-02-14 02:59:47 +01:00
Leo R. Lundgren
71a4ccea71 Document refusal to process empty forget policy. 2018-02-14 02:59:41 +01:00
Leo R. Lundgren
5a9c7c51e6 Fix --keep-* options list formatting. 2018-02-14 02:59:33 +01:00
rawtaz
5abd133e26 Restructure and complement exclude options doc 2018-02-14 02:03:04 +01:00
Alexander Neumann
2debb5c0e0 Merge pull request #1613 from restic/strip-space-files-from
Strip spaces from items read via --files-from
2018-02-11 21:35:23 +01:00
Alexander Neumann
54c0794cf3 Merge pull request #1603 from jfparis/master
Add support for GOARM parameter when cross compiling
2018-02-11 21:03:29 +01:00
Alexander Neumann
cff3f3dc0d Add entry to CHANGELOG 2018-02-11 21:00:38 +01:00
Alexander Neumann
c5575c7ed9 Strip spaces from items read via --files-from
In #1590, it was mentioned that while lines read from exclude files via
`--exclude-file` have leading and trailing spaces stripped, this is not
the case for lines read via `--files-from`. This commit fixes that,
spaces are always stripped.
2018-02-11 20:56:11 +01:00
Aaron Bieber
99d88ad297 Disable the 'testSticky' test on OpenBSD. Only root can set sticky. 2018-02-11 07:46:31 -07:00
Alexander Neumann
7f0d964a05 Update build.go 2018-02-11 11:34:03 +01:00
Alexander Neumann
514f1b8917 Relax timeout backend test 2018-02-10 12:53:38 +01:00
Alexander Neumann
a7a99be460 Correct forum host name in issue template
Closes #1609
2018-02-10 11:13:22 +01:00
Alexander Neumann
e17f65b23a Update logo 2018-02-05 20:48:48 +01:00
Jean-François Paris
e86fb5eedd Add support for GOARM parameter when cross compiling
Go can cross compile to older ARM architectures (e.g. v5). In order to
do so one needs to use the GOARM parameter as documented in
https://github.com/golang/go/wiki/GoArm
2018-02-04 14:15:12 +00:00
Alexander Neumann
c8cb9a9509 Merge pull request #1579 from ifedorenko/RetryBackend-List
Implement RetryBackend.List()
2018-01-30 21:39:21 +01:00
Igor Fedorenko
aa333f4d49 Implement RetryBackend.List()
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-01-29 22:14:12 -05:00
Alexander Neumann
c40002246d Merge pull request #1598 from Phaeilo/s3-tutorial
Update AWS S3 tutorial, fixing #1593
2018-01-29 21:52:32 +01:00
Philip Huppert
77a6054d64 Update AWS S3 tutorial, fixing #1593 2018-01-29 20:47:27 +01:00
Alexander Neumann
94ec55ff2c Merge pull request #1595 from restic/remove-read-bandwidth
Remove archiver progress "data processed" bandwith
2018-01-28 10:15:53 +01:00
Alexander Neumann
2369da158f Merge pull request #1592 from ncw/helpful-tests
Make backend tests more helpful
2018-01-28 10:09:35 +01:00
Alexander Neumann
5379950c64 Add entry to changelog 2018-01-28 09:36:55 +01:00
Alexander Neumann
cbfa516575 Remove archiver progress "data processed" bandwith
This commit removes the bandwidth displayed during backup process. It is
misleading and seldomly correct, because it's neither the "read
bandwidth" (only for the very first backup) nor the "upload bandwidth".
Many users are confused about (and rightly so), c.f. #1581, #1033, #1591

We'll eventually replace this display with something more relevant when
 #1494 is done.
2018-01-28 09:30:26 +01:00
Nick Craig-Wood
fb62da1748 Make backend tests more helpful
* In TestList check that backend is empty first
  * Improve error message in TestBackend
2018-01-27 21:36:35 +00:00
Alexander Neumann
b1b1f6e04f Merge pull request #1594 from restic/gcs-use-http-transport
GS: Use generic http transport
2018-01-27 22:16:49 +01:00
Alexander Neumann
05f826a4fc Update Gopkg.lock 2018-01-27 21:48:41 +01:00
Alexander Neumann
649afb1198 Add entry to changelog 2018-01-27 20:16:51 +01:00
Alexander Neumann
5dc8d3588d GS: Use generic http transport
During the development of #1524 I discovered that the Google Cloud
Storage backend did not yet use the HTTP transport, so things such as
bandwidth limiting did not work. This commit does the necessary magic to
make the GS library use our HTTP transport.
2018-01-27 20:12:34 +01:00
Alexander Neumann
9d2aa0a71c Merge pull request #1524 from brycied00d/https_client_certificate
Support for TLS client certificate authentication
2018-01-27 20:12:09 +01:00
Alexander Neumann
c34db983d8 Read TLS client cert and key from the same file 2018-01-27 14:02:01 +01:00
Bryce Chidester
e805b968b1 Support for TLS client certificate authentication
This adds --tls-client-cert and --tls-client-key parameters and enables use
of that certificate/key pair when connecting to https servers.
2018-01-27 13:18:22 +01:00
Alexander Neumann
e706f1a8d1 Merge pull request #1589 from restic/fix-intermediate-index-upload
archiver: Fix intermediate index upload
2018-01-27 12:22:11 +01:00
Alexander Neumann
4219bfbcc9 Add entry to changelog 2018-01-26 22:05:53 +01:00
Alexander Neumann
f99c95c766 archiver: Fix intermediate index upload
A user discovered[1] that when the backup finishes during the upload of
an intermediate index, the upload is cancelled and the index never fully
saved, but the snapshot is saved and the backup finalizes without an
error. This lead to a situation where a snapshot references data that is
contained in the repo, but not referenced in any index, leading to
strange error messages.

This commit uses a dedicated context to signal the intermediate index
uploading routine to terminate after the last index has been uploaded.
This way, an upload running when the backup finishes is completed before
the routine terminates and the snapshot is saved.

[1] https://forum.restic.net/t/error-loading-tree-check-prune-and-forget-gives-error-b2-backend/406
2018-01-26 22:01:07 +01:00
Alexander Neumann
cccb2fc7e7 Merge pull request #1583 from restic/close-open-backend-files
Close backend files in case of errors
2018-01-26 21:57:28 +01:00
Alexander Neumann
ff3de66ddf Merge pull request #1582 from restic/optimize-debug-log
Optimize debug logs
2018-01-26 21:57:18 +01:00
Alexander Neumann
fb0d01ef1f Add pull request URL to changelog 2018-01-26 21:56:59 +01:00
Alexander Neumann
e11a183578 Merge pull request #1588 from restic/fix-sftp-without-tty
sftp: Allow running ssh without a tty
2018-01-26 21:56:41 +01:00
Alexander Neumann
ee6732d2c6 Add entry to changelog 2018-01-26 21:56:29 +01:00
Alexander Neumann
4b2f2b542d Merge pull request #1584 from restic/limit-index-file-size
Limit index file size
2018-01-26 21:53:12 +01:00
Alexander Neumann
6eb2d76435 index: Lower parallel load to 4 2018-01-26 21:10:38 +01:00
Alexander Neumann
f63c7048c7 index: Limit index files to 3000 pack files 2018-01-26 21:07:16 +01:00
Alexander Neumann
ca3c566a99 index: Check structure after saving a new index 2018-01-26 21:07:16 +01:00
Alexander Neumann
7719cf88d9 b2: Check timeout 2018-01-26 21:07:05 +01:00
Alexander Neumann
00e905ebe6 sftp: Allow running ssh without a tty 2018-01-26 19:21:14 +01:00
Alexander Neumann
164acceb55 Appveyor: Update Go version and download URL 2018-01-25 21:09:23 +01:00
Alexander Neumann
909d9273cc Close backend files in case of errors 2018-01-25 21:05:57 +01:00
Alexander Neumann
d62a664bb7 Merge pull request #1580 from ifedorenko/tests-errors-cause
Use errors.Cause in backend TestListCancel
2018-01-25 20:55:43 +01:00
Alexander Neumann
663c57ab4d debug: Remove manual Str() call Log() 2018-01-25 20:49:41 +01:00
Alexander Neumann
ed99f53786 debug: Auto-shorten IDs passed as parameters 2018-01-25 20:49:26 +01:00
Igor Fedorenko
abc4027083 Use errors.Cause in backend TestListCancel
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-01-25 08:53:50 -05:00
Alexander Neumann
d6212ee2d9 Fix entry to changelog 2018-01-24 20:54:51 +01:00
Alexander Neumann
9c55e8d69c Merge pull request #1549 from MJDSys/more_index_lookup_avoids
More optimizations to avoid calling Index.Lookup()
2018-01-24 20:53:30 +01:00
Alexander Neumann
1c9fca6d14 Combine changelog files for #1574 and #1575 2018-01-24 20:07:22 +01:00
Alexander Neumann
be205563be Merge pull request #1575 from ifedorenko/1567_optimize-repository-ListPack
Optimize Repository.ListPack()
2018-01-24 20:01:15 +01:00
Alexander Neumann
fc6c341b26 Merge pull request #1574 from ifedorenko/1567_optimize-pack-readHeader
Optimize pack readHeader() implementation
2018-01-24 19:54:14 +01:00
Alexander Neumann
f86c141b83 Merge pull request #1577 from EdwardBetts/patch-1
Correct a spelling mistake.
2018-01-24 19:44:49 +01:00
Alexander Neumann
a6d4888d48 Merge pull request #1573 from restic/update-dps
Update dependencies
2018-01-24 19:43:32 +01:00
Alexander Neumann
44550a88a0 Merge pull request #1571 from restic/rework-backend-list
Rework backend list
2018-01-24 19:43:07 +01:00
Edward Betts
b5c23695c9 Correct a spelling mistake. 2018-01-24 12:09:41 +00:00
Igor Fedorenko
0084e42cb6 Optimize Repository.ListPack()
Use pack file size returned by Backend.List() to avoid extra per-pack
Backend.Stat() requests

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-01-23 22:39:51 -05:00
Matthew Dawson
fe33c05a20 debug/log: Add benchmarks for calling the logging function
Add some benchmarks for calling Log, both with a static string
along with calling the ID.Str and ID.String functions.
2018-01-23 22:30:00 -05:00
Matthew Dawson
3789e55e20 repostiory/index: Remove logging from Lookup function.
The logging in these functions double the time they take to execute.
However, it is only really useful on failures, which are better
reported by the calling functions.

benchmark                                            old ns/op     new ns/op     delta
BenchmarkMasterIndexLookupSingleIndex-6              897           395           -55.96%
BenchmarkMasterIndexLookupMultipleIndex-6            2001          1090          -45.53%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       492           215           -56.30%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     1649          912           -44.69%

benchmark                                            old allocs     new allocs     delta
BenchmarkMasterIndexLookupSingleIndex-6              9              1              -88.89%
BenchmarkMasterIndexLookupMultipleIndex-6            19             1              -94.74%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       6              0              -100.00%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     16             0              -100.00%

benchmark                                            old bytes     new bytes     delta
BenchmarkMasterIndexLookupSingleIndex-6              160           96            -40.00%
BenchmarkMasterIndexLookupMultipleIndex-6            240           96            -60.00%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       48            0             -100.00%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     128           0             -100.00%
2018-01-23 22:28:38 -05:00
Matthew Dawson
4cec7e236a Add Changelog 2018-01-23 22:28:32 -05:00
Matthew Dawson
3a16148447 archiver/archiver: Use Index.Has() instead of Index.Lookup() in isKnownBlob
Index.Has() is a faster then Index.Lookup() for checking if a blob exists
in the index.  As the returned data is never used, this avoids a ton
of allocations.
2018-01-23 22:26:10 -05:00
Matthew Dawson
df2c03a6a4 repository/master_index: Optimize Index.Lookup()
When looking up a blob in the master index, with several
indexes present in the master index, a significant amount of time
is spent generating errors for each failed lookup.  However, these
errors are often used to check if a blob is present, but the contents
are not inspected making the overhead of the error not useful.

Instead, change Index.Lookup (and Index.LookupSize) to instead return
a boolean denoting if the blob was found instead of an error.  Also change
all the calls to these functions to handle the new function signature.

benchmark                                            old ns/op     new ns/op     delta
BenchmarkMasterIndexLookupSingleIndex-6              820           897           +9.39%
BenchmarkMasterIndexLookupMultipleIndex-6            12821         2001          -84.39%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       5378          492           -90.85%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     17026         1649          -90.31%

benchmark                                            old allocs     new allocs     delta
BenchmarkMasterIndexLookupSingleIndex-6              9              9              +0.00%
BenchmarkMasterIndexLookupMultipleIndex-6            59             19             -67.80%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       22             6              -72.73%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     72             16             -77.78%

benchmark                                            old bytes     new bytes     delta
BenchmarkMasterIndexLookupSingleIndex-6              160           160           +0.00%
BenchmarkMasterIndexLookupMultipleIndex-6            3200          240           -92.50%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       1232          48            -96.10%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     4272          128           -97.00%
2018-01-23 22:25:56 -05:00
Matthew Dawson
ebce4b2581 repository/index: Speed up benchmarks and tests
When setting up the index used for benchmarking, use math/rand instead of
crypto/rand since the generated ids don't need to be evenly distributed,
and not be secure against guessing.  As such, use a different random id
function (only available during tests) that uses math/rand instead.
2018-01-23 22:21:05 -05:00
Igor Fedorenko
953f3d55ee Optimize pack readHeader() implementation
Load pack header length and 15 header entries with single backend
request. This eliminates separate header Load() request for most pack
files and significantly improves index.New() performance.

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-01-23 22:02:25 -05:00
Alexander Neumann
7e6bfdae79 backend/rest: Implement REST API v2 2018-01-23 23:15:26 +01:00
Alexander Neumann
0f4cbea27d Document the REST v2 protocol 2018-01-23 23:15:26 +01:00
Alexander Neumann
59782e347c repository: Fix repack test
This reduces the chance of duplicate blobs, otherwise the tests fail
(make the contents of a blob depend on a pseudo-random number instead of
the size, sizes may be duplicate).
2018-01-23 23:14:05 +01:00
Alexander Neumann
e835abeceb backend/test: Reliably trigger timeout error 2018-01-23 23:14:05 +01:00
Alexander Neumann
8e812b7ac0 errors: Make Cause() unwrap *url.Error 2018-01-23 23:14:05 +01:00
Alexander Neumann
685ce719ad Add comments for List() 2018-01-23 23:14:05 +01:00
Alexander Neumann
2b39f9f4b2 Update dependencies
Among others, this updates minio-go, so that the new "eu-west-3" zone
for AWS is supported.
2018-01-23 19:40:42 +01:00
Alexander Neumann
b63de7c798 script/release: Tag versioned docker image 2018-01-21 21:41:14 +01:00
Alexander Neumann
b0c6e53241 Fix calls to repo/backend.List() everywhere 2018-01-21 21:15:09 +01:00
Alexander Neumann
e9ea268847 Change List() implementation for all backends 2018-01-21 21:15:09 +01:00
Alexander Neumann
dd91b13ff3 Vendor golang.org/x/sync 2018-01-21 18:35:37 +01:00
Alexander Neumann
c4e9d5d11e backend: Add tests for new List() function 2018-01-21 18:35:37 +01:00
Alexander Neumann
52230b8f07 backend: Rework List()
For a discussion see #1567
2018-01-21 18:35:37 +01:00
Alexander Neumann
a3d43a92b3 Merge pull request #1569 from restic/ext-rest-backend-test
REST: refactor backend tests
2018-01-21 18:02:09 +01:00
Alexander Neumann
2130897ce0 rest: Add test for external server 2018-01-20 10:25:47 +01:00
Alexander Neumann
67da240068 rest: Refactor backend tests 2018-01-20 10:25:37 +01:00
Alexander Neumann
1046eabf95 rest: Remove unneeded tempdir 2018-01-20 10:13:04 +01:00
Alexander Neumann
663e8284b2 init: Correct text (backend vs. repository)
Closes #1565
2018-01-20 09:51:49 +01:00
Alexander Neumann
5a43ca1ccb Add entry to CHANGELOG 2018-01-20 09:50:31 +01:00
Alexander Neumann
200415e0a1 Merge pull request #1564 from restic/sftp-password-prompt
sftp: Prompt for password, don't terminate on SIGINT
2018-01-20 09:48:17 +01:00
Alexander Neumann
ea50687e8b Merge pull request #1568 from martinseener/patch-1
Added hint where to place restic.exe on windows
2018-01-20 09:47:50 +01:00
Alexander Neumann
7335a2492f Add PR to changelog 2018-01-19 13:05:55 +01:00
Alexander Neumann
bcb6881ffb Merge pull request #1548 from ifedorenko/checker-backend-Test
checker: Optimize checker.Packs()
2018-01-19 13:05:35 +01:00
Alexander Neumann
37df829b3d CONTRIBUTING: clarify wording 2018-01-19 13:00:49 +01:00
Martin
39f1ba1897 Added hint where to place restic.exe on windows
I've added a hint for pre-compiled binaries on windows where to place them for easier use in the shell or in scripts without the hassle for absolute paths.
2018-01-19 12:24:09 +01:00
Igor Fedorenko
231076fa4a checker: Optimize checker.Packs()
Use result of single repository.List() to find both missing and
orphaned data packs. For 500GB repository this eliminates ~100K
repository.Test() calls and improves check time by >30M in my
environment (~45min before this change and ~7min after).

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-01-18 20:50:39 -05:00
Alexander Neumann
0bdb131521 Remove SuspendSignalHandler 2018-01-17 23:14:47 +01:00
Alexander Neumann
c31a5e7e5c Add argument to Skipf() 2018-01-17 23:14:37 +01:00
Alexander Neumann
05958caf6e sftp: Prompt for password, don't terminate on SIGINT
This is a follow-up on fb9729fdb9, which
runs the `ssh` in its own process group and selects that process group
as the foreground group. After the sftp connection is established,
restic switches back to the previous foreground process group.

This allows `ssh` to prompt for the password, but it won't receive
the interrupt signal (SIGINT, ^C) later on, because it is not in the
foreground process group any more, allowing a clean tear down.
2018-01-17 23:02:47 +01:00
Alexander Neumann
87e31799f2 Merge pull request #1483 from dstosberg/master
document how to create a full backup without running restic as root
2018-01-16 17:21:55 +01:00
Alexander Neumann
35528506a6 Merge pull request #1554 from restic/fix-988
fuse: Correct behavior for reading after EOF, add snapshot template string
2018-01-16 17:17:55 +01:00
Alexander Neumann
2544978aaf Add entry to changelog 2018-01-15 14:47:17 +01:00
Alexander Neumann
97565d68a2 fuse/mount: Add option for snapshot template 2018-01-15 14:47:17 +01:00
Alexander Neumann
f49f5c5903 fuse: Don't return an error for EOF 2018-01-13 10:14:10 +01:00
Alexander Neumann
d89f2e5226 fuse: Remove debug log message 2018-01-13 10:10:07 +01:00
Alexander Neumann
02f4f5dc66 Update build.go, correct version detection 2018-01-12 22:06:42 +01:00
Alexander Neumann
5723636b35 Add entry for changelog 2018-01-09 21:57:07 +01:00
Alexander Neumann
5632ca4f44 Merge pull request #1538 from MJDSys/make_lookup_internal
repository/index: Avoid creating an error when checking if an id is in the index.
2018-01-09 21:55:37 +01:00
Matthew Dawson
539599d1f1 repository/index: Optimize index.Has()
When backing up several million files (>14M tested here) with few changes,
a large amount of time is spent failing to find an id in an index and creating
an error to signify this.  Since this is checked using the Has method,
which doesn't use this error, this time creating the error is wasted.

Instead, directly check if the given id and type are present in the index.
This also avoids reporting all the packs containing this blob, further
reducing cpu usage.
2018-01-08 21:46:17 +01:00
Alexander Neumann
d77a326bb0 Add benchmark for Index.Has() 2018-01-08 21:10:30 +01:00
Dennis Stosberg
60a7a9d019 manual: document how to create a full backup without running restic as root 2018-01-08 19:59:15 +01:00
Alexander Neumann
99f0fce673 Merge pull request #1536 from restic/fix-restorer-permissions
restore: Fix directory permissions
2018-01-07 17:18:29 +01:00
Alexander Neumann
b708cfee3f Add entry to changelog 2018-01-07 15:16:16 +01:00
Alexander Neumann
b8620429e2 restorer: Set directory mode as last step
Closes #1512
2018-01-07 15:13:24 +01:00
Alexander Neumann
aab414b368 Add test case for #1512 2018-01-07 14:50:47 +01:00
Alexander Neumann
e120b17abd Update PR template 2018-01-07 11:53:30 +01:00
Alexander Neumann
5f43e17918 Merge pull request #1535 from ifedorenko/check-retry
fixed restic-check does not retry backend.Test failures
2018-01-07 09:43:49 +01:00
Alexander Neumann
489eef5a6f Merge pull request #1534 from restic/improve-error-restore
restore: Improve error message for more than one ID
2018-01-07 09:40:06 +01:00
Igor Fedorenko
8c550ca011 fixed restic-check does not retry backend.Test failures
added missing RetryBackend.Test implementation

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-01-06 23:22:35 -05:00
Alexander Neumann
032621289b errors: Add "Fatal" prefix for Fatalf() 2018-01-06 23:27:47 +01:00
Alexander Neumann
92316a9853 restore: Improve error message for more than one ID 2018-01-06 22:44:18 +01:00
Alexander Neumann
180741609e Merge pull request #1530 from juliangehring/patch-1
Fix code block formatting in FAQs
2018-01-06 18:46:14 +01:00
Julian Gehring
70250762f3 Fix code block formatting in FAQs
Fixes the formatting of a code block in the FAQs and makes it render properly.
2018-01-05 23:02:03 +00:00
Alexander Neumann
39ba17a2d6 Add entry to CHANGELOG 2018-01-05 19:29:02 +01:00
Alexander Neumann
cfe8c8c9cd Merge pull request #1529 from restic/fix-1528
Fix creating missing dirs in local/sftp repo
2018-01-05 19:28:17 +01:00
Alexander Neumann
b45fc89512 local/sftp: Create repo dirs on demand in Save() 2018-01-05 17:51:09 +01:00
Alexander Neumann
aabc0ccaa7 fs: Make HasPathPrefix work with relative paths 2018-01-05 17:38:52 +01:00
Alexander Neumann
2bc4d200d4 Merge pull request #1518 from restic/fix-1317-again
forget: Also run prune when only IDs are forgotten
2018-01-02 20:39:12 +01:00
Alexander Neumann
0247fe01c0 Update homepage URL in README 2018-01-01 21:50:44 +01:00
Alexander Neumann
c912b38bf0 forget: Also run prune when only IDs are forgotten 2018-01-01 21:27:40 +01:00
Alexander Neumann
559946c58a Add linebreak to VERSION generation 2017-12-30 09:55:56 +01:00
Alexander Neumann
a99637c613 Merge pull request #1511 from restic/improve-http-limiting
Improve limiting HTTP based backends
2017-12-29 20:57:02 +01:00
Alexander Neumann
36501dda73 Add changelog issue 2017-12-29 12:51:43 +01:00
Alexander Neumann
18ecd9df30 Improve limiting HTTP based backends 2017-12-29 12:43:49 +01:00
Alexander Neumann
c686dd0448 Merge pull request #1507 from restic/fix-fuse-list-timeout
fuse: Only reload list of snapshots once per minute
2017-12-28 14:06:59 +01:00
Alexander Neumann
6d91d468e7 Add entry to releases 2017-12-28 13:20:19 +01:00
Alexander Neumann
5f9b5b0219 Add entry to changelog 2017-12-28 13:20:03 +01:00
Alexander Neumann
3f7d85360a fuse: Only reload list of snapshots once per minute 2017-12-28 13:18:27 +01:00
Alexander Neumann
8d8456590c Update release script, add check for changelog/releases 2017-12-27 23:32:11 +01:00
Alexander Neumann
85f9f3e290 Fix changelog template for GitHub 2017-12-27 23:26:19 +01:00
Alexander Neumann
9e1180a29b Fix changelog IDs 2017-12-27 23:22:35 +01:00
Alexander Neumann
e17c1096a0 Add date for 0.8.1 2017-12-27 22:42:15 +01:00
Alexander Neumann
546d6f36b2 Add VERSION for 0.8.1 2017-12-27 22:31:03 +01:00
Alexander Neumann
6ecd14d780 Update manpages and auto-completion 2017-12-27 22:31:03 +01:00
Alexander Neumann
f6ed7dc013 Generate CHANGELOG.md for 0.8.1 2017-12-27 22:30:57 +01:00
Alexander Neumann
e290f2591e Merge pull request #1504 from restic/changelog-generator
Replace manual CHANGELOG with generated one
2017-12-27 22:00:15 +01:00
Alexander Neumann
75f90ca303 Run calens on Travis 2017-12-27 21:21:21 +01:00
Alexander Neumann
ca1430184f Replace CHANGELOG.md with generated version 2017-12-27 21:07:26 +01:00
Alexander Neumann
a297ab9d7c Add PrimaryURL to github template 2017-12-27 19:30:04 +01:00
Alexander Neumann
f078525d98 Add changelog entries 2017-12-27 19:22:06 +01:00
Alexander Neumann
e03cc81a9a Add changelog generation to release script 2017-12-27 19:21:55 +01:00
Alexander Neumann
af27f1dde5 Merge pull request #1503 from gliptak/patch-2
Correct golint warning on return
2017-12-27 12:52:31 +01:00
Gábor Lipták
32505c3916 Correct golint warning on return 2017-12-26 17:09:41 -05:00
Alexander Neumann
9a8d5a1bff Merge pull request #1499 from ametzler/improve-docs-exclude-396
doc: Improvements for --exclude
2017-12-24 21:10:28 +01:00
Alexander Neumann
740ee787c1 Merge pull request #1501 from gliptak/patch-1
Correct typos
2017-12-24 21:10:08 +01:00
Gábor Lipták
2eba0bfeec Correct typos 2017-12-24 12:06:52 -05:00
Andreas Metzler
d780ec4bce doc: Improvements for --exclude
Describe exclude pattern handling, adding multiple examples.

Closes #396
2017-12-24 15:39:18 +01:00
Alexander Neumann
6b564d21b3 Merge pull request #1493 from restic/retry-delete
backend: Retry deletes
2017-12-23 11:53:09 +01:00
Alexander Neumann
6c2b2a58ad backend: Retry deletes 2017-12-22 22:41:28 +01:00
Alexander Neumann
b80b68dcb3 doc: Correct setence 2017-12-22 17:59:52 +01:00
Alexander Neumann
29c92ca415 Merge pull request #1491 from restic/improve-b2-config
b2: Warn when account ID or key is empty
2017-12-20 18:06:58 +01:00
Alexander Neumann
bc04ce8e6b Merge pull request #1482 from restic/fix-1478
backend: Only return top-level files for most dirs
2017-12-20 18:06:47 +01:00
Alexander Neumann
6b6b75fa4a Merge pull request #1488 from da2x/patch-1
Doc: Specify `key passwd` usage
2017-12-19 21:19:38 +01:00
Alexander Neumann
84e493beba b2: Warn when account ID or key is empty
Closes #1490
2017-12-19 21:12:38 +01:00
Daniel Aleksandersen
323376efa2 Doc: Specify key passwd usage
Specify that `key passwd` is used to change a password.
2017-12-19 04:53:31 +01:00
Alexander Neumann
e353b00501 Merge pull request #1481 from restic/add-stdin-path
backup: Reject directories in filename for --stdin
2017-12-14 19:36:11 +01:00
Alexander Neumann
2510d770ab Add question about positive things to issue template 2017-12-14 19:22:23 +01:00
Alexander Neumann
7d8765a937 backend: Only return top-level files for most dirs
Fixes #1478
2017-12-14 19:14:16 +01:00
Alexander Neumann
81a04656c5 fuse: Cleanup node name 2017-12-13 20:21:18 +01:00
Alexander Neumann
2f26fb8834 Only print cache warning for terminals 2017-12-13 19:58:10 +01:00
Alexander Neumann
d3ebe1311f backup: Reject filenames with paths read from stdin 2017-12-13 19:55:53 +01:00
Alexander Neumann
42a8c19aae Merge pull request #1476 from harshavardhana/fix
Since upgrade to minio-go 4.0 remove workaround
2017-12-09 11:45:29 +01:00
Harshavardhana
27ccea6371 Since upgrade to minio-go 4.0 remove workaround
We added previously a code to fix the issue of chaining
credentials, we do not need this anymore since the
upstream minio-go already has this relevant change.
2017-12-09 02:01:42 -08:00
Alexander Neumann
4f46b4f393 Merge pull request #1475 from restic/update-minio-go
Update to minio-go v4
2017-12-09 10:12:05 +01:00
Alexander Neumann
221e741537 Fix broken link in README 2017-12-09 10:04:39 +01:00
Alexander Neumann
8b3b7bc5ef s3: Use context 2017-12-08 22:04:55 +01:00
Alexander Neumann
934ae1b559 Update to minio-go 4 2017-12-08 21:52:50 +01:00
Alexander Neumann
0e7e3cb714 Update minio-go 2017-12-08 20:45:59 +01:00
Alexander Neumann
95b6e4e9e9 Add entry to CHANGELOG 2017-12-08 20:29:50 +01:00
Alexander Neumann
3a5e040b7e Merge pull request #1439 from armhold/propagate-context
replace ad-hoc context.TODO() with gopts.ctx
2017-12-08 20:27:36 +01:00
Alexander Neumann
28c826868b Merge pull request #1471 from restic/fix-diff
Fix diff
2017-12-06 19:47:17 +01:00
George Armhold
1695c8ed55 use global context for check, debug, dump, find, forget, init, key,
list, mount, tag, unlock commands

gh-1434
2017-12-06 07:02:55 -05:00
George Armhold
366622f09a use global context for cat and ls
gh-1434
2017-12-06 05:48:39 -05:00
George Armhold
0dc31c03e1 remove check for context.Canceled
gh-1434
2017-12-06 05:38:29 -05:00
Alexander Neumann
0405e67f8b Change diff output slightly 2017-12-05 22:26:56 +01:00
Alexander Neumann
df350e1f6e Improve diff 2017-12-05 22:05:22 +01:00
Alexander Neumann
06cb3f7058 Merge pull request #1469 from zertrin/patch-1
Update usage help output in doc/manual_rest.rst
2017-12-04 21:36:16 +01:00
Alexander Neumann
56b884be17 Merge pull request #1468 from restic/update-deps
Update dependencies
2017-12-04 21:34:52 +01:00
Alexander Neumann
a25d280f3e Merge pull request #1462 from restic/add-diff
Add diff command
2017-12-04 21:34:44 +01:00
zertrin
2253a73837 Update usage help output in doc/manual_rest.rst
The usage help in the documentation was outdated relative to current version (0.8.0) with some commands outdated and some missing.
2017-12-04 22:57:00 +08:00
Alexander Neumann
946c8399e2 Update dependenciess
Exclude minio-go for now (pin to 3.x.y).
2017-12-03 21:22:33 +01:00
Alexander Neumann
9d0f13c4c0 Prominently mention SIGPIPE issue in CHANGELOG 2017-12-03 19:12:07 +01:00
Alexander Neumann
eb9e2bc79a manual: Document diff command 2017-12-03 18:33:35 +01:00
Alexander Neumann
0722c44ba1 Add entry to CHANGELOG 2017-12-03 18:33:35 +01:00
Alexander Neumann
2424012d75 Add 'diff' command 2017-12-03 18:33:35 +01:00
Alexander Neumann
82ded35706 node: Correctly compare times 2017-12-03 18:33:35 +01:00
Alexander Neumann
69fcb604c8 Merge pull request #1465 from restic/fuse-handle-others
fuse: Handle/format all node types correctly
2017-12-03 18:25:49 +01:00
Alexander Neumann
88607fc625 Correctly format all node types 2017-12-03 17:38:55 +01:00
Alexander Neumann
7092af6329 fuse: Handle sockets/fifos/devs correctly
Closes #1463
2017-12-03 17:25:00 +01:00
Alexander Neumann
23d7d91597 Merge pull request #1464 from restic/reenable-cache
Reenable cache
2017-12-03 16:15:13 +01:00
Alexander Neumann
ad82781743 Reenable cache
In 6341c7d72c, the cache was accidentally
disabled due to a bug, this commit reenables the cache.
2017-12-03 15:52:57 +01:00
George Armhold
20d78ab0d9 add context to repo.Flush() call in test code
gh-1434
2017-12-03 07:32:50 -05:00
George Armhold
be24237063 make retry code context-aware.
detect cancellation in backend, so that retry code does not keep trying
once user has hit ^c

gh-1434
2017-12-03 07:22:14 -05:00
George Armhold
d886cb5c27 replace ad-hoc context.TODO() with gopts.ctx, so that cancellation
can properly trickle down from cmd_*.

gh-1434
2017-12-03 07:22:14 -05:00
Alexander Neumann
63bb1933e5 Merge pull request #1461 from ljani/patch-1
Fix a typo
2017-12-03 09:51:08 +01:00
Jani
81e6a9d0d0 Fix a typo 2017-12-03 10:33:46 +02:00
Alexander Neumann
5d4110d2a7 Add entry to CHANGELOG 2017-12-01 21:52:07 +01:00
Alexander Neumann
0cedb3ac9f Merge pull request #1459 from restic/improve-s3-do-spaces
Improve s3 backend for DigitalOcean Spaces
2017-12-01 21:51:30 +01:00
Alexander Neumann
0b44c629f2 retry: Remove file after failed save 2017-11-30 22:05:14 +01:00
Alexander Neumann
2579fe6b7b Ignore SIGPIPE
Handling SIGPIPE made restic abort when a TCP connection was reset by a
server. This happened on DigitalOcean Spaces, which uses the s3 backend.
2017-11-30 21:23:43 +01:00
Alexander Neumann
812ce4bfc4 Add entry to CHANGELOG 2017-11-29 18:44:03 +01:00
Alexander Neumann
410efe0694 Merge pull request #1454 from restic/fix-cache-dir
Fix cache dir detection
2017-11-29 18:43:07 +01:00
Alexander Neumann
b2d944d5cb Merge pull request #1452 from restic/add-with-atime
backup: Do not save access time by default
2017-11-29 18:42:45 +01:00
Alexander Neumann
b846c3915c Add entry to CHANGELOG 2017-11-28 21:41:38 +01:00
Alexander Neumann
ffbc68aa2e Document access time handling 2017-11-28 21:36:20 +01:00
Alexander Neumann
eddb8549ef backup: By default, do not save the access time
This can be re-enabled with `--with-atime`.
2017-11-28 21:31:35 +01:00
Alexander Neumann
bb44855078 Add pull request template 2017-11-28 19:40:27 +01:00
Alexander Neumann
2567026ccb Merge pull request #1447 from fawick/fixMultipleRejectIfPresent
Fix multiple rejectIfPresent cancelling each other
2017-11-28 19:20:29 +01:00
Alexander Neumann
0cc8fc6f18 Merge pull request #1431 from n0npax/master
fix #1411
2017-11-27 21:54:06 +01:00
Alexander Neumann
cc81b916a6 Add entry to CHANGELOG 2017-11-27 21:40:13 +01:00
Fabian Wickborn
27fadd2c6e Document approach for multiple reject-if-present test 2017-11-27 21:38:15 +01:00
Alexander Neumann
dc38265b54 Merge pull request #1436 from restic/remove-old-cache
Remove old cache directories
2017-11-27 21:37:05 +01:00
Fabian Wickborn
1ea518d5ef cmd/restic: Use a dedicated cache for each rejectIfPresent 2017-11-27 17:33:53 +01:00
Fabian Wickborn
901cd5edef cmd/restic: Add test for rejectIfPresent bug
All RejectFuncs returned by rejectIfPresent share the same rejection
cache and hence might cancel each other out.
2017-11-27 17:26:19 +01:00
Alexander Neumann
e1fd47765b Fix scripts/release.go 2017-11-26 19:59:21 +01:00
Alexander Neumann
c02923fbfc Add VERSION for 0.8.0 2017-11-26 19:46:16 +01:00
Alexander Neumann
7c5ce83044 Update manpages and auto-completion 2017-11-26 19:46:16 +01:00
Alexander Neumann
37e2e9a844 Add version to CHANGELOG 2017-11-26 19:46:01 +01:00
Alexander Neumann
26e5db1849 Merge pull request #1446 from restic/fix-relative-restore
restore: Fix restore to relative path
2017-11-26 19:03:34 +01:00
Alexander Neumann
a2766ffe0c CI: return directly after an error occurred 2017-11-26 18:50:40 +01:00
Alexander Neumann
0f5e38609f restore: Fix restore to relative path 2017-11-26 18:36:48 +01:00
Alexander Neumann
f178cbf93d Merge pull request #1445 from restic/clean-node-name
restorer: Clean node names
2017-11-26 17:34:05 +01:00
Alexander Neumann
c8096ca8d2 Add entry to CHANGELOG 2017-11-26 15:31:09 +01:00
Alexander Neumann
27d29b9853 restorer: Make sure node names are clean 2017-11-26 15:28:45 +01:00
Alexander Neumann
8a171731ba restorer: Add tests for invalid node names 2017-11-26 15:28:37 +01:00
Alexander Neumann
abde9e2fba doc: Add --cacert to REST section 2017-11-26 10:09:54 +01:00
Alexander Neumann
6a4a328bbc Merge pull request #1443 from restic/fix-init-rest-cacert
rest: Use client for creating the repository
2017-11-25 22:12:18 +01:00
Alexander Neumann
8253fadc96 doc: Fix typo 2017-11-25 22:11:47 +01:00
Alexander Neumann
134abbd82b rest: Use client for creating the repository
Before, creating a new repo via REST would use the defaut HTTP client,
which is not a problem unless the server uses HTTPS and a TLS
certificate which isn't signed by a CA in the system's CA store. In this
case, all commands work except the 'init' command, which fails with a
message like "invalid certificate".
2017-11-25 20:56:40 +01:00
Alexander Neumann
fe557b022a Add entry to CHANGELOG 2017-11-25 12:12:08 +01:00
Alexander Neumann
cd8226130a Improve issue template 2017-11-25 11:43:44 +01:00
Alexander Neumann
1ebf0e8de8 Merge pull request #1437 from restic/fix-1292
s3: Document and remove default prefix
2017-11-25 11:34:26 +01:00
Alexander Neumann
37ea764000 cache: Add more documentation 2017-11-25 11:28:59 +01:00
Alexander Neumann
0fdb9a6129 Merge pull request #1426 from pmkane/master
fix the signal name for progress reports in the docs
2017-11-24 21:58:11 +01:00
Alexander Neumann
47b326b7b5 Merge pull request #1423 from harshavardhana/creds
Fix chaining of credentials for minio-go
2017-11-24 21:57:52 +01:00
Alexander Neumann
e2cf6eb434 Merge pull request #1428 from stephengroat/patch-1
use default brew formula
2017-11-24 21:38:04 +01:00
Alexander Neumann
f79698dcdd Merge pull request #1410 from armhold/deadlock2
unify behavior for max http connections across backends
2017-11-24 21:32:56 +01:00
Alexander Neumann
35a5307db3 Merge pull request #1415 from armhold/signals
also handle SIGPIPE in cleanup routines
2017-11-24 21:28:08 +01:00
Alexander Neumann
6341c7d72c cache: Add option to remove old cache dirs 2017-11-24 20:53:28 +01:00
Alexander Neumann
f4bab789b8 cache: Simplify cache dir creation 2017-11-24 20:53:26 +01:00
Alexander Neumann
fa893ee477 cache: Add detection code for old cache dirs 2017-11-24 20:53:26 +01:00
Alexander Neumann
014cec06f1 Add entry to CHANGELOG 2017-11-21 21:33:09 +01:00
Alexander Neumann
431ab5aa6a manual: Add hint about old default prefix 2017-11-21 21:33:09 +01:00
Alexander Neumann
262b0cd9d4 s3: Remove default prefix "/restic" 2017-11-21 21:33:09 +01:00
Alexander Neumann
e83ec17e95 s3: Correct comment 2017-11-20 22:21:39 +01:00
Alexander Neumann
ea593fca1b cache: Correctly return dir for Windows/darwin 2017-11-20 06:11:18 +01:00
Alexander Neumann
fe1f151ae1 cache: Return error during default dir detection 2017-11-20 06:10:42 +01:00
n0npax
b12bba4e2a fix #1411
replace panic during index save with fatal error
2017-11-19 11:40:47 +01:00
Stephen
e2005e02bb use default brew formula 2017-11-18 08:02:41 -08:00
Harshavardhana
41c8c946ba Fix chaining of credentials for minio-go
chaining failed because chaining provider
was only looking for subsequent credentials
provider after an error. Writer a new
chaining provider which proceeds to fetch
new credentials also under situations where
providers do not return but instead return
no keys at all.

Fixes https://github.com/restic/restic/issues/1422
2017-11-18 02:51:12 -08:00
Patrick Kane
fe08686558 fix the signal name for progress reports in the docs 2017-11-16 19:17:25 -08:00
George Armhold
0ed2401711 exit 1 if received signal is other than SIGINT
send cleanup msg to stderr, not stdout
gh-1413
2017-11-09 07:16:01 -05:00
Alexander Neumann
06bd606d85 Merge pull request #1414 from bket/1307
Fix test failing on OpenBSD #1307
2017-11-08 18:54:44 +01:00
George Armhold
c347431907 also handle SIGPIPE in cleanup routines
fixes gh-1413: restic fails to cleanup locks when bash pipeline fails
2017-11-05 07:14:27 -05:00
Björn Ketelaars
f63d7048f9 Fix test failing on OpenBSD #1307
Is seems that #1307 is similar to #1087, which describes a comparable
observation on Apple's new filesystem. #1389 Has been committed and
fixes the problem on Darwin.

Although I'm not sure if the root cause of the issue is the same the
solution is similar for OpenBSD, and leverages #1389.
2017-11-05 07:27:58 +01:00
Alexander Neumann
f39f7c76dd Merge pull request #1406 from michaeldorner/master
Minor fixes on README and LICENSE
2017-11-04 09:58:21 +01:00
George Armhold
0268d0e7d6 swift backend: limit http concurrency in Save(), Stat(), Test(), Remove(),
List().

move comment regarding problematic List() backend api (it's s3's ListObjects
that has a problem, NOT swift's ObjectsWalk).

As per discussion in PR #1399.
2017-11-02 18:29:32 -04:00
George Armhold
8515d093e0 swift backend: fix premature release of semaphore in Load() & document
concurrency issue in List().

refactor wrapReader from b2 -> semaphore so it can be used elsewhere.

As per discussion in PR #1399.
2017-11-02 12:38:17 -04:00
Michael Dorner
fe3f326d8d Update LICENSE
- improved formatting of license information (BSD 2-clase) so Github can automatically detect it.
2017-11-02 11:39:49 +01:00
Michael Dorner
8170db40c7 Update README.rst
- added Codecov badge
- added link to OSI for the license
2017-11-02 11:36:25 +01:00
George Armhold
99ac0da4bc s3 backend: limit http concurrency in Save(), Stat(), Test(), Remove()
NB: List() is NOT currently limited, as it would cause deadlock due to
be.client.ListObjects() implementation.

as per discussion in PR #1399
2017-11-01 09:40:54 -04:00
Alexander Neumann
7e2c93420f Merge pull request #1397 from restic/crypto-aead
crypto: Make crypto.Key implement cipher.AEAD
2017-11-01 13:21:10 +01:00
Alexander Neumann
6d46824fb0 Pass in a nil buffer to Open() 2017-11-01 10:30:40 +01:00
Alexander Neumann
bb435b39d9 crypto: Rework Seal/Open to use sliceForAppend 2017-11-01 10:30:40 +01:00
Alexander Neumann
2a67d7a6c2 crypto: Correct test function names 2017-11-01 10:30:40 +01:00
Alexander Neumann
ba43c8bab5 crypto: Fix nonce test, make it faster 2017-11-01 10:30:40 +01:00
Alexander Neumann
931e6ed2ac Use Seal/Open everywhere 2017-11-01 10:30:40 +01:00
Alexander Neumann
a5f0e9ab65 Remove custom Encrypt/Decrypt methods 2017-11-01 10:30:40 +01:00
Alexander Neumann
6fc133ad6a Run tests on Seal/Open 2017-11-01 10:30:40 +01:00
Alexander Neumann
e1b80859f2 Make crypto.Key implement cipher.AEAD 2017-11-01 10:30:39 +01:00
George Armhold
d069ee31b2 GS backend: limit http concurrency in Save(), Stat(), Test(), Remove(), List()
as per discussion in PR #1399
2017-10-31 08:01:43 -04:00
George Armhold
981752ade0 Azure backend: limit http concurrency in Stat(), Test(), Remove()
as per discussion in PR #1399
2017-10-31 07:32:30 -04:00
Alexander Neumann
d01d07fc0a Merge pull request #1400 from armhold/deadlock2
log unexpected errs from b2 ListCurrentObject()
2017-10-30 18:39:33 +01:00
Alexander Neumann
526aaca6f5 Merge pull request #1389 from wojas/fix-modtime-tests-on-apfs
Fix modtime tests on APFS: allow 1us difference
2017-10-29 14:20:40 +01:00
George Armhold
2f8147af59 log unexpected errs from b2 ListCurrentObject()
gh-1385
2017-10-29 08:53:39 -04:00
Konrad Wojas
f3016a9096 Darwin test fix: allow 1μs timestamp difference
On Darwin, allow a 1μs difference in restored timestamps, because
macOS <10.13 cannot restore with nanosecond precision and the current
version of Go (1.9.2) does not yet support the new syscall required
for this. (#1087 #1389)
2017-10-29 20:48:07 +08:00
Alexander Neumann
f854a41ba9 Merge pull request #1399 from armhold/deadlock2
prevent deadlock in List() for B2 when b2.connections=1
2017-10-29 09:26:46 +01:00
Alexander Neumann
ca3cadef5e Merge pull request #1398 from armhold/sftp-readdir
sftp ReadDir: add path to return error messages (gh-1323)
2017-10-29 09:26:42 +01:00
George Armhold
3304b0fcf0 prevent deadlock in List() for B2 when b2.connections=1
This is a fix for the following situation (gh-1188):

List() grabs a semaphore token upon entry, starts a goroutine, and
does not release the token until the routine exits (via a defer).

The goroutine iterates over the results from ListCurrentObjects(),
sending them one at a time to a channel, where they are ultimately
processed by be.Load().

Since be.Load() also needs a token, this will result in deadlock if
b2.connections=1.

This fix changes List() so that the token is only held during the call
to ListCurrentObjects().
2017-10-28 18:46:47 -04:00
George Armhold
d8938e259a sftp ReadDir: add path to return error messages (gh-1323)
fix missing "Close" string in debug log fmt
2017-10-28 14:16:27 -04:00
Alexander Neumann
53a554c89d Merge pull request #1395 from restic/small-crypto-improvements
Small improvements
2017-10-28 12:57:18 +02:00
Alexander Neumann
e71db01230 Improve crypto test 2017-10-28 12:09:33 +02:00
Alexander Neumann
178e946fc7 Rename KDFParams -> Params 2017-10-28 10:28:29 +02:00
Alexander Neumann
f3bff12939 Merge pull request #1394 from zcalusic/master
Use lowercase in messages from check/tag commands, too
2017-10-28 09:56:29 +02:00
Alexander Neumann
7a99418dc5 Merge pull request #1393 from armhold/lint-errcheck
detect errors from fs.Walk() in local backend List()
2017-10-28 09:56:11 +02:00
Alexander Neumann
c71ba466ea Merge pull request #1391 from armhold/b2-listmax
pass in defaultListMaxItems to b2Backend constructor
2017-10-28 09:54:57 +02:00
Alexander Neumann
8ce5d35543 Merge pull request #1390 from armhold/lint
small cleanup
2017-10-28 09:54:35 +02:00
Zlatko Čalušić
134f834c60 Use lowercase in messages from check/tag commands, too 2017-10-27 21:06:34 +02:00
George Armhold
8a37c07295 send errors from fs.Walk() to debug log
clarify non-err returns from Walk where err is already proved to be nil
2017-10-27 08:41:17 -04:00
George Armhold
bd0ada7842 go fmt 2017-10-26 16:37:11 -04:00
George Armhold
eea96f652d go fmt 2017-10-26 16:22:10 -04:00
George Armhold
38c3061df7 pass in defaultListMaxItems to b2Backend constructor
gh-1385
2017-10-26 14:22:16 -04:00
George Armhold
f5fa602482 detect and return error from file Close() in Node.createFileAt()
gh-1385
2017-10-26 13:53:31 -04:00
Alexander Neumann
e44ac55f63 Merge pull request #1387 from wojas/snapshots-filter-last
Add --last flag to snapshots command
2017-10-26 19:08:49 +02:00
Alexander Neumann
f1cfb73a8b Merge pull request #1384 from wojas/reduce-ticker-refresh
Reduce ticker refresh rate with RESTIC_PROGRESS_FPS env
2017-10-26 19:02:59 +02:00
Konrad Wojas
5b96885c6d Control progress rate with RESTIC_PROGRESS_FPS env
Add a RESTIC_PROGRESS_FPS environment variable to limit the interval
at which the progress indicator updates (allowed values: 1-60).

The default rate of 60 FPS can cause high terminal CPU load on some
systems, like iTerm2 on macOS with font anti-aliasing enabled.

Usage:

    RESTIC_PROGRESS_FPS=1 restic ...
    RESTIC_PROGRESS_FPS=60 restic ...
2017-10-26 14:46:56 +08:00
Konrad Wojas
c5da90a5b7 Add --last flag to snapshots command
Add --last flag to snapshots command to only show the last entry for any
(hostname, paths) combination.

This makes it easier to check when various paths were last backed up.
2017-10-26 14:02:29 +08:00
George Armhold
bcdebfb84e small cleanup:
- be explicit when discarding returned errors from .Close(), etc.
- remove named return values from funcs when naked return not used
- fix some "err" shadowing when redeclaration not needed
2017-10-25 12:03:55 -04:00
Alexander Neumann
359b273649 Merge pull request #1381 from tbm/broken-link
Remove Markdown syntax from documentation
2017-10-24 20:38:40 +02:00
Martin Michlmayr
2e2c8dc620 Remove Markdown syntax from documentation
Convert a link from Markdown to RST syntax.  This link must
have been missed during the recent conversion.
2017-10-24 14:40:33 +01:00
Alexander Neumann
8d37b723ca Upadte vendored dependencies 2017-10-22 10:07:36 +02:00
Alexander Neumann
315b7f282f Remove explicit version constraints 2017-10-22 10:07:23 +02:00
Alexander Neumann
a3f8e9dfa7 Add entry to CHANGELOG 2017-10-21 12:22:50 +02:00
Alexander Neumann
982810f7cc Merge pull request #1368 from TobyLL/master
Ignore comments (lines starting with #) in the --files-from file
2017-10-21 12:21:11 +02:00
Alexander Neumann
90b96d19cd Merge pull request #1365 from felix9/fix_1068
Fix failure to detect some legacy s3 repos
2017-10-21 12:19:58 +02:00
Alexander Neumann
6a52bb6f54 Merge pull request #1374 from restic/update-blazer
Update github.com/kurin/blazer
2017-10-21 11:26:49 +02:00
Alexander Neumann
cacaa4393f Merge pull request #1373 from restic/check-gopkg-lock
CI: Check the vendor dir and Gopkg.lock
2017-10-21 11:14:30 +02:00
Alexander Neumann
d63ab4e9a4 Merge pull request #1358 from prattmic/chunk_size
gs: add option to set chunk size
2017-10-21 11:13:48 +02:00
Alexander Neumann
ca6daec8dd CI: Check the vendor dir and Gopkg.lock 2017-10-21 10:50:38 +02:00
Alexander Neumann
c87f2420a6 Update github.com/kurin/blazer 2017-10-21 10:30:39 +02:00
Alexander Neumann
f5bbbc52f4 Update issue template 2017-10-21 10:17:58 +02:00
TobyLL
9e3dde8ec7 Fix whitespace 2017-10-19 15:52:06 +01:00
TobyLL
9dba182e51 Ignore comments (lines starting with #) in the --files-from file 2017-10-19 15:48:22 +01:00
Felix Lee
944fc857eb Fix failure to detect some legacy s3 repos
Sometimes s3 listobjects for a directory includes an entry for that
directory. The restic s3 backend doesn't expect that and returns
an error.

Symptom is:
  ReadDir: invalid key name restic/key/, removing prefix
     restic/key/ yielded empty string

I'm not sure when s3 does that; I'm unable to reproduce it myself.

But in any case, it seems correct to ignore that when it happens.

Fixes #1068
2017-10-18 13:45:31 -07:00
Alexander Neumann
7507a658ac Merge pull request #1362 from restic/fix-backend-retry
Fix Save() backend retry
2017-10-18 07:45:56 +02:00
Michael Pratt
9fa4f5eb6b gs: disable resumable uploads
By default, the GCS Go packages have an internal "chunk size" of 8MB,
used for blob uploads.

Media().Do() will buffer a full 8MB from the io.Reader (or less if EOF
is reached) then write that full 8MB to the network all at once.

This behavior does not play nicely with --limit-upload, which only
limits the Reader passed to Media. While the long-term average upload
rate will be correctly limited, the actual network bandwidth will be
very spikey.

e.g., if an 8MB/s connection is limited to 1MB/s, Media().Do() will
spend 8s reading from the rate-limited reader (performing no network
requests), then 1s writing to the network at 8MB/s.

This is bad for network connections hurt by full-speed uploads,
particularly when writing 8MB will take several seconds.

Disable resumable uploads entirely by setting the chunk size to zero.
This causes the io.Reader to be passed further down the request stack,
where there is less (but still some) buffering.

My connection is around 1.5MB/s up, with nominal ~15ms ping times to
8.8.8.8.

Without this change, --limit-upload 1024 results in several seconds of
~200ms ping times (uploading), followed by several seconds of ~15ms ping
times (reading from rate-limited reader). A bandwidth monitor reports
this as several seconds of ~1.5MB/s followed by several seconds of
0.0MB/s.

With this change, --limit-upload 1024 results in ~20ms ping times and
the bandwidth monitor reports a constant ~1MB/s.

I've elected to make this change unconditional of --limit-upload because
the resumable uploads shouldn't be providing much benefit anyways, as
restic already uploads mostly small blobs and already has a retry
mechanism.

--limit-download is not affected by this problem, as Get().Download()
returns the real http.Response.Body without any internal buffering.

Updates #1216
2017-10-17 21:12:04 -07:00
Alexander Neumann
ce4d71d626 backend: Add partial read failure to error backend 2017-10-17 22:11:38 +02:00
Alexander Neumann
8e2ef3f38b cache: Store first, then store in the backend
Store the file in the backend, then rewind the reader and store it
in the cache afterwards.
2017-10-17 22:01:20 +02:00
Alexander Neumann
8dc952775e backend: Correctly retry Save() calls
Make sure the given reader is an io.Seeker and rewind it properly each
time.
2017-10-17 21:46:38 +02:00
Alexander Neumann
99b6163e27 Add 'dump' to manual 2017-10-16 20:24:47 +02:00
Alexander Neumann
beaf55f1fc dump: Allow absolute paths 2017-10-16 20:22:01 +02:00
Alexander Neumann
980bb9059f Add entry to changelog 2017-10-16 20:15:19 +02:00
Alexander Neumann
0e7281eb71 Merge pull request #1346 from fawick/cmdCatFile
Add dump sub-command for dumping files to stdout
2017-10-16 20:13:44 +02:00
Alexander Neumann
0b6133d7b5 Merge pull request #1360 from mungomat/fuse_bugfix
fuse: bugfix: ignore empty tags
2017-10-16 20:07:05 +02:00
Tobias Klein
b57ca64275 fuse: bugfix: ignore empty tags 2017-10-16 16:22:08 +02:00
Alexander Neumann
faadbd734b Add entry to CHANGELOG 2017-10-14 20:29:47 +02:00
Alexander Neumann
88b0a93409 Merge pull request #1353 from restic/backend-retry
Retry failed backend requests
2017-10-14 20:28:57 +02:00
Alexander Neumann
4a995105a9 sftp: Fix Delete() 2017-10-14 16:08:15 +02:00
Alexander Neumann
7fe496f983 Ensure TestDelete runs last 2017-10-14 16:04:29 +02:00
Alexander Neumann
e56370eb5b Remove Deleter interface 2017-10-14 16:04:29 +02:00
Alexander Neumann
b8af7f63a0 backend test: Always remove files for TestList 2017-10-14 15:56:25 +02:00
Alexander Neumann
3eea555155 Add dependency cenkalti/backoff 2017-10-14 15:56:25 +02:00
Alexander Neumann
897c923cc9 Retry failed backend requests 2017-10-14 15:56:25 +02:00
Alexander Neumann
67193e3deb Improve error message when creating lock failed 2017-10-14 15:56:25 +02:00
Alexander Neumann
0e722efb09 backend: Add Delete() to restic.Backend interface 2017-10-14 15:56:25 +02:00
Alexander Neumann
3736f33ebf Merge pull request #1352 from tyll/manual_links
Fix links to manual
2017-10-14 14:54:16 +02:00
Fabian Wickborn
d1d9c3f9d7 Renamed cmd_catfile.go to cmd_dump.go 2017-10-14 13:55:21 +02:00
Fabian Wickborn
cd5cbe0910 Rename debug dump related variable and run function 2017-10-14 13:55:00 +02:00
Fabian Wickborn
814e992c0b Rename subcommand catfile to dump 2017-10-14 11:34:04 +02:00
Till Maas
660fe78735 Fix links to manual 2017-10-13 20:01:19 +02:00
Fabian Wickborn
87d084e18c Add subcommand dump 2017-10-12 20:24:34 +02:00
Alexander Neumann
9ce2a73fc5 Add issue and PR to the changelog entry 2017-10-12 20:20:06 +02:00
Alexander Neumann
f2314b26ba Move 'dump' to 'debug dump' 2017-10-12 20:18:45 +02:00
Alexander Neumann
74dcf41f25 Disable 'dump' for non-debug builds 2017-10-12 20:14:48 +02:00
Alexander Neumann
b6ba30186f Merge pull request #1336 from rmdashrf/rate-limit-backup
Add basic rate limiting to backup
2017-10-12 20:08:55 +02:00
rmdashrf
32637a0328 Basic rate limiting implementation.
Added `--limit-upload` and `--limit-download` flags to rate limit
backups and restores.
2017-10-11 20:01:20 -07:00
Alexander Neumann
0addd90e14 Merge pull request #1345 from harshavardhana/creds
Refactor credentials management to support multiple mechanisms.
2017-10-11 21:01:19 +02:00
Alexander Neumann
1b5ee5b10a Merge pull request #1343 from mungomat/fuse_speedup
fuse mount: speedup
2017-10-11 20:55:52 +02:00
Harshavardhana
042adeb5d0 Refactor credentials management to support multiple mechanisms.
This PR adds the ability of chaining the credentials provider,
such that restic as a tool attempts to honor credentials from
multiple different ways.

Currently supported mechanisms are

 - static (user-provided)
 - IAM profile (only valid inside configured ec2 instances)
 - Standard AWS envs (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
 - Standard Minio envs (MINIO_ACCESS_KEY, MINIO_SECRET_KEY)

Refer https://github.com/restic/restic/issues/1341
2017-10-09 12:51:39 -07:00
Tobias Klein
7e4ce0dacc fuse mount: speedup 2017-10-09 10:26:56 +02:00
Alexander Neumann
8ceb22fe8a Add entry to CHANGELOG 2017-10-08 09:48:22 +02:00
Alexander Neumann
c5553ec855 Merge pull request #1276 from fawick/supply_ca_cert
Add REST backend option to use CA root certificate
2017-10-08 09:47:23 +02:00
Alexander Neumann
bb3ed54291 Merge pull request #1337 from mungomat/fuse_updateRepo
fuse: mount and backup in parallel (#1330)
2017-10-08 09:45:28 +02:00
Alexander Neumann
513ba3b6f7 Merge pull request #1340 from jniggemann/fix-664
doc: Add info on docker container
2017-10-08 09:44:34 +02:00
Alexander Neumann
17d688afef Merge pull request #1339 from felix9/fix_1251
fixes #1251, race when writing indexes
2017-10-07 16:36:16 +02:00
Jan Niggemann
d81eee26b3 doc: Add info on docker container
fixes #664
2017-10-07 16:03:08 +02:00
Felix Lee
cc5ada63a4 fixes #1251, race when writing indexes 2017-10-07 05:11:42 -07:00
Alexander Neumann
88fb60e0b5 Improve issue template 2017-10-07 12:16:46 +02:00
Tobias Klein
02200acad0 fuse: mount and backup in parallel (#1330) 2017-10-07 11:48:48 +02:00
Alexander Neumann
1a2d190bdb Merge pull request #1334 from felix9/backup_tags
backup should use latest parent regardless of tags
2017-10-07 10:45:02 +02:00
Alexander Neumann
2db4ff168a Merge pull request #1327 from fawick/fix-1294
Create missing lock dir when saving lock
2017-10-07 10:40:29 +02:00
Alexander Neumann
a77c8cc5d2 Add entry to CHANGELOG 2017-10-07 10:23:56 +02:00
Alexander Neumann
b8866c1fe4 Merge pull request #1326 from fawick/rejectionCache
Cache evaluated directories in isExcludedByFile
2017-10-07 09:57:54 +02:00
Fabian Wickborn
f0f17db847 Cache evaluated directories in isExcludedByFile
Fixes #1271.
2017-10-06 22:11:22 +02:00
Alexander Neumann
a5c003acb0 Merge pull request #1333 from restic/fix-1328
Do not cache invalid/truncated files
2017-10-05 21:36:52 +02:00
Felix Lee
7b44fd0f9d fix #1143
Backup was choosing a parent snapshot that had the same tags, which
makes backup unnecessarily slow when there are newer snapshots with
different tags.

There's no reason parent has to have the same tags.

This change makes backup choose the newest snapshot instead.
2017-10-05 11:48:26 -07:00
Alexander Neumann
cebee0b8fa cache: Refuse to cache truncated files 2017-10-05 20:40:02 +02:00
Alexander Neumann
d886bc6c48 Ignore invalid index files, print warning 2017-10-05 20:39:53 +02:00
Alexander Neumann
d81adcfaa5 cache: Add file name to error message 2017-10-05 19:30:56 +02:00
Fabian Wickborn
6da9bfbbce Create missing lock dir when saving lock 2017-10-05 00:07:48 +02:00
Fabian Wickborn
69a6e622d0 Add REST backend option to use CA root certificate
Closes #1114.
2017-10-04 22:14:10 +02:00
Alexander Neumann
1dcfd64028 Merge pull request #1325 from antonlindstrom/lookup-group
Add group name in fillUser
2017-10-04 21:32:46 +02:00
Alexander Neumann
5d1c1f721e Merge pull request #1324 from antonlindstrom/password-feedback
Add password successful feedback
2017-10-04 21:31:24 +02:00
Alexander Neumann
fbc8bbf305 Merge pull request #1321 from damienstanton/master
fix broken installation link in README
2017-10-04 21:02:33 +02:00
Alexander Neumann
cdef55bb88 Merge pull request #1312 from hgfischer/issue-1119
nit: please remove dot imports #1119
2017-10-04 21:01:40 +02:00
Anton Lindstrom
26df48b2aa Add group name in fillUser
This does a lookup of the group name from the GID and adds it to the
Node.
2017-10-04 20:11:54 +02:00
Anton Lindstrom
a7baea0522 Output password successful on terminal stdout
This removes the conditions that checks if the password is supplied
through environment variable or file and outputs password is successful
on terminal and when --quiet is not supplied.
2017-10-04 14:55:04 +02:00
Anton Lindstrom
55e6003749 Add password successful feedback
This adds some feedback when entering the password on the command line.
When the password is entered and supplied through stdin (and stdout is a
terminal) then the a message saying `password is correct` if correct is
printed.
2017-10-04 13:45:05 +02:00
damienstanton
846acd5d4c use https 2017-10-03 14:28:17 -04:00
Herbert
43f8145858 Revert "Add .vscode to gitignore"
This reverts commit 1b5242b4f9.
2017-10-03 20:21:54 +02:00
damienstanton
79759928f6 fix broken installation link in README 2017-10-03 13:28:33 -04:00
Alexander Neumann
eb59d28154 Add entry to CHANGELOG 2017-10-03 18:54:07 +02:00
Alexander Neumann
79f63a2e74 Merge pull request #1320 from restic/add-snapshots-json-short-id
snapshots: Add short ID to JSON output
2017-10-03 18:53:17 +02:00
Alexander Neumann
6a62254048 Merge pull request #1299 from mungomat/fuse_addSnapshotIDs
fuse: Add a snapshot IDs directory (#1102)
2017-10-03 18:53:05 +02:00
Alexander Neumann
657a1d75af snapshots: Add short ID to JSON output
We're using the short ID in all output to users, so it should also be
included in the JSON output of the `snapshots` command.
2017-10-03 15:24:09 +02:00
Alexander Neumann
2694def56a Add entry to CHANGELOG 2017-10-03 14:52:10 +02:00
Alexander Neumann
7843341da3 Merge pull request #1319 from antonlindstrom/check-ok-output
Add explicit OK output to check command
2017-10-03 14:51:31 +02:00
Anton Lindstrom
d46314648e Add explicit OK output to check command
This adds additional output to the check command when no errors were
found. It means that when all checks have been completed, the following
output is displayed:

	No errors were found

The output is added to make sure that it is easier to understand that no
errors were found.

Full example output:

	Create exclusive lock for repository
	Load indexes
	Check all packs
	Check snapshots, trees and blobs
	No errors were found
2017-10-03 12:57:51 +02:00
Alexander Neumann
e45011af57 Merge pull request #1304 from jannickfahlbusch/showAmountOfSnapshots
Show the number of snapshots
2017-10-03 12:41:47 +02:00
Alexander Neumann
fb09884893 Merge pull request #1311 from felix9/dump-no-trees
"dump" command no longer has a "trees" option
2017-10-03 12:40:24 +02:00
Alexander Neumann
d1eecafa63 Merge pull request #1314 from jannickfahlbusch/fixTypo
Fix typo in comment
2017-10-03 12:39:24 +02:00
Alexander Neumann
b47d991f56 Merge pull request #1316 from restic/prune-wrong-data
prune: Warn about wrong plaintext blob ID
2017-10-03 12:39:04 +02:00
Alexander Neumann
553ea812a7 Add entry to CHANGELOG 2017-10-03 12:38:00 +02:00
Alexander Neumann
216e374310 Merge pull request #1315 from prattmic/cache_doc
Clarify cache location documentation
2017-10-03 12:36:53 +02:00
Alexander Neumann
034b0b8040 forget: Run prune for manually forgotten snapshots 2017-10-03 11:56:13 +02:00
Michael Pratt
5ab9e12b46 Clarify cache location documentation
PR #1287 changed the default cache location on darwin and windows.
Update the changelog and manual to reflect the new behavior.

Since the cache hasn't been included in an official release yet, I've
just changed the main cache changelog entry.

Fixes #1309
2017-10-03 11:44:09 +02:00
Alexander Neumann
abe6e0d22d Merge pull request #1300 from jniggemann/refactor-manual
doc: Refactors the documentation
2017-10-03 11:41:00 +02:00
Jan Niggemann
f5b550191c doc: Refactors the documentation
This commit refactors the documentation according to my proposal in #1273
and the discussion I had with fd0 on IRC.

The bits from the manual that I could not immediately put into the new
structure are contained in manual_rest.rst Anything else is still there,
nothing has been deleted.

I changed the heading markup to follow the convention used in Python’s
Style Guide for documentation, this convention is explained in a comment
at the top of every file.

I also added a paragraph on installing restic on Debian.
2017-10-03 11:21:53 +02:00
Jannick Fahlbusch
3dcacb3730 Fix typo in comment 2017-10-02 18:19:22 +02:00
Herbert
033589a66b Use rtest on these as well to keep codebase consistent 2017-10-02 17:48:08 +02:00
Jannick Fahlbusch
e46a647c45 Just print the number of snapshots 2017-10-02 17:05:59 +02:00
Alexander Neumann
f26492fc2d prune: Warn about wrong plaintext blob ID 2017-10-02 16:27:08 +02:00
Herbert
3473c3f7b6 Remove all dot-imports 2017-10-02 15:06:39 +02:00
Herbert
1b5242b4f9 Add .vscode to gitignore 2017-10-02 15:06:23 +02:00
Felix Lee
6d897def1b also fix man page 2017-10-01 18:33:41 -07:00
Jannick Fahlbusch
2f10e25738 Show the amount of snapshots 2017-10-01 17:14:54 +02:00
Felix Lee
555bd257bd dump command no longer has 'trees' option 2017-10-01 08:04:52 -07:00
Alexander Neumann
3afd974dea Add entry to CHANGELOG 2017-10-01 10:35:15 +02:00
Alexander Neumann
f4120c9d45 Merge pull request #1301 from restic/update-blazer
Update vendored dependencies
2017-10-01 10:33:54 +02:00
Alexander Neumann
61cb1cc6f8 Update vendored dependencies
This includes github.com/kurin/blazer 0.2.0, which resolves #1291
2017-10-01 10:13:39 +02:00
Tobias Klein
49bc1d0b3b fuse: Add a snapshot IDs directory (#1102) 2017-09-30 16:22:14 +02:00
Alexander Neumann
ba23d24dd1 Merge pull request #1298 from restic/fix-1288
sftp: Handle/Document tilde character
2017-09-30 12:28:09 +02:00
Alexander Neumann
556a63de19 sftp: Return error when path starts with a tilde (~) 2017-09-30 10:34:23 +02:00
Alexander Neumann
fae3c4d437 faq: Add entry about Synology NAS' sftp path 2017-09-30 10:30:21 +02:00
Alexander Neumann
89c2ed2a1c manual: Document that sftp does not expand tilde 2017-09-30 10:24:28 +02:00
Alexander Neumann
23f1cb06d6 Fix FAQ 2017-09-30 10:21:56 +02:00
Alexander Neumann
ac92e2dd2d Add to CHANGELOG entry 2017-09-29 21:45:37 +02:00
Alexander Neumann
bf58425351 Merge pull request #1287 from prattmic/cachedir
cache: OS-specific cache directories
2017-09-29 21:42:34 +02:00
Alexander Neumann
a3dc0ab398 Merge pull request #1295 from bachp/patch-2
Fix wrong quotes for command in installation.
2017-09-29 21:41:13 +02:00
Pascal Bach
224ebdb8b9 Fix wrong quotes for command in installation. 2017-09-29 10:08:24 +02:00
Alexander Neumann
cf80d295f3 Merge pull request #1285 from bachp/patch-1
Add instruction on how to install restic using Nix
2017-09-28 22:03:19 +02:00
Michael Pratt
2133869127 cache: OS-specific cache directories
Windows, and to a lesser extent OS X, don't conform to XDG and have
their own preferred locations for caches.

On Windows, use %LOCALAPPDATA%/restic (i.e., ~/AppData/Local/restic). I
can't find authoritative documentation from Microsoft recommending
specifically which of %APPDATA%, %LOCALAPPDATA%, and %TEMP% should be
used for caches, but %LOCALAPPDATA% is where browsers store their
caches, so it seems like a good fit.

On OS X, use ~/Library/Caches/restic, which is recommended by the Apple
documentation. They do suggest using the application "bundle identifier"
as the base folder name, but restic doesn't have one, so I just used
"restic".
2017-09-27 21:16:22 -07:00
Pascal Bach
97330ac621 Add instruction on how to install restic using Nix 2017-09-27 22:41:20 +02:00
Alexander Neumann
1ee1559506 Add release script 2017-09-26 22:14:58 +02:00
Alexander Neumann
eccc336319 Add entry to CHANGELOG 2017-09-26 14:18:37 +02:00
Alexander Neumann
7fe657ec71 Merge pull request #1282 from restic/rework-autogeneration
Rework generation of manpages and completion files
2017-09-26 14:16:41 +02:00
Alexander Neumann
77c07bfd19 Remove integration test for manpage 2017-09-26 13:16:55 +02:00
Alexander Neumann
4de938d97a Update manpages and auto-completion 2017-09-26 13:16:08 +02:00
Alexander Neumann
dad1c87afe Rework generation of manpages and completion files
This commit removes the `manpages` and `autocomplet` commands and
replaces them with the more generic `generate` command. Also, zsh
completion file support was added.
2017-09-26 13:12:12 +02:00
Alexander Neumann
801dbb6d03 Merge pull request #1281 from prattmic/gcs_perms
gs: allow backend creation without storage.buckets.get
2017-09-26 09:38:33 +02:00
Michael Pratt
fa0be82da8 gs: allow backend creation without storage.buckets.get
If the service account used with restic does not have the
storage.buckets.get permission (in the "Storage Admin" role), Create
cannot use Get to determine if the bucket is accessible.

Rather than always trying to create the bucket on Get error, gracefully
fall back to assuming the bucket is accessible. If it is, restic init
will complete successfully. If it is not, it will fail on a later call.

Here is what init looks like now in different cases.

Service account without "Storage Admin":

Bucket exists and is accessible (this is the case that didn't work
before):

$ ./restic init -r gs:this-bucket-does-exist:/
enter password for new backend:
enter password again:
created restic backend c02e2edb67 at gs:this-bucket-does-exist:/

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

Bucket exists but is not accessible:

$ ./restic init -r gs:this-bucket-does-exist:/
enter password for new backend:
enter password again:
create key in backend at gs:this-bucket-does-exist:/ failed:
service.Objects.Insert: googleapi: Error 403:
my-service-account@myproject.iam.gserviceaccount.com does not have
storage.objects.create access to object this-bucket-exists/keys/0fa714e695c8ecd58cb467cdeb04d36f3b710f883496a90f23cae0315daf0b93., forbidden

Bucket does not exist:

$ ./restic init -r gs:this-bucket-does-not-exist:/
create backend at gs:this-bucket-does-not-exist:/ failed:
service.Buckets.Insert: googleapi: Error 403:
my-service-account@myproject.iam.gserviceaccount.com does not have storage.buckets.create access to bucket this-bucket-does-not-exist., forbidden

Service account with "Storage Admin":

Bucket exists and is accessible: Same

Bucket exists but is not accessible: Same. Previously this would fail
when Create tried to create the bucket. Now it fails when trying to
create the keys.

Bucket does not exist:

$ ./restic init -r gs:this-bucket-does-not-exist:/
enter password for new backend:
enter password again:
created restic backend c3c48b481d at gs:this-bucket-does-not-exist:/

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
2017-09-25 22:25:51 -07:00
Alexander Neumann
7e8bc8d362 Merge pull request #1279 from restic/fix-eof-error
cache: Synchronize downloading
2017-09-25 16:39:20 +02:00
Alexander Neumann
0bb2a8e0d0 cache: Synchronize downloading
This commit adds code to synchronize downloading files to the cache.
Before, requests that came in for files currently downloading would fail
because the file was not completed in the cache. Now, the code waits
until the download is completed.

Closes #1278
2017-09-25 15:58:20 +02:00
Alexander Neumann
2e72b57f2f Correct debug message 2017-09-25 14:35:37 +02:00
Alexander Neumann
bff1039e3a Add entry to CHANGELOG 2017-09-25 13:17:44 +02:00
Alexander Neumann
5a999cb77f Merge pull request #1040 from restic/add-cache
Add local cache
2017-09-25 13:13:07 +02:00
Alexander Neumann
3a2539e0ac doc: Update manpages 2017-09-24 23:13:04 +02:00
Alexander Neumann
e262f35d0a cache: Auto-remove invalid files 2017-09-24 23:11:47 +02:00
Alexander Neumann
176bfa6529 backend: Improve ReadAt 2017-09-24 23:11:23 +02:00
Alexander Neumann
240c4cf2fd cache: In case of an error, fall back backend 2017-09-24 23:11:23 +02:00
Alexander Neumann
db5ec5d876 repo: Automatically cache tree-only pack files 2017-09-24 23:11:23 +02:00
Alexander Neumann
e1dfaf5d87 cache: Allow proactive caching of tree packs
This commit adds a function to the cache which can decide to proactively
load the complete pack file and store it in the cache. This is helpful
for pack files containing only tree blobs, as it is likely that the same
file is accessed again in the future.
2017-09-24 23:11:23 +02:00
Alexander Neumann
5436154f0d cache: Add PerformReadahead 2017-09-24 23:11:23 +02:00
Alexander Neumann
809e218d20 cache: Improve debug logs 2017-09-24 23:11:23 +02:00
Alexander Neumann
1eaad6cebb index: Add TreePacks() 2017-09-24 21:54:53 +02:00
Alexander Neumann
56fccecd06 prune: Repack mixed pack files 2017-09-24 21:54:53 +02:00
Alexander Neumann
3890a947ca Clear data files in cache 2017-09-24 21:54:53 +02:00
Alexander Neumann
e299272378 repo: Try cached pack files first 2017-09-24 21:54:53 +02:00
Alexander Neumann
70248bd05a repo: Clear indexes 2017-09-24 21:54:53 +02:00
Alexander Neumann
7a5fde8f5a repository: Save pack files for trees in cache 2017-09-24 21:54:53 +02:00
Alexander Neumann
62ba9f1950 check: Disable cache by default 2017-09-24 21:54:53 +02:00
Alexander Neumann
610b676444 Automatically exclude current restic cache 2017-09-24 21:54:53 +02:00
Alexander Neumann
58699e3c90 Write CACHEDIR.TAG to cache base directory 2017-09-24 21:54:53 +02:00
Alexander Neumann
9be24a1c9f Add cache
This commits adds rudimentary support for a cache directory, enabled by
default. The cache directory is created if it does not exist. The cache
is used if there's anything in it, newly created snapshot and index
files are written to the cache automatically.
2017-09-24 21:54:53 +02:00
Alexander Neumann
5ace41471e Merge pull request #1277 from prattmic/gcs_cleanup
Document GCS permission requirements
2017-09-24 20:49:02 +02:00
Michael Pratt
3b2106ed30 gs: document required permissions
In the manual, state which standard roles the service account must
have to work correctly, as well as the specific permissions required,
for creating even more specific custom roles.
2017-09-24 11:25:57 -07:00
Michael Pratt
5f4f997126 gs: minor comment cleanups
* Remove a reference to S3.
* Config can only be used for GCS, not other "gcs compatibile servers".
* Make comments complete sentences.
2017-09-24 10:10:56 -07:00
Alexander Neumann
49d397a419 Merge pull request #1275 from fawick/sort_snapshots
Always sort snapshots lists ascending by timestamp
2017-09-24 16:32:40 +02:00
Fabian Wickborn
ea1ab96749 Always sort snapshots lists ascending by timestamp
Fixes #1219.
2017-09-24 13:01:13 +02:00
Alexander Neumann
24c62e719a Add entry to CHANGELOG 2017-09-23 22:15:10 +02:00
Alexander Neumann
9c6b7f688e Merge pull request #1270 from restic/sftp-allow-password-prompt
sftp: Allow password entry
2017-09-23 22:13:04 +02:00
Alexander Neumann
d41dce5ecb Merge pull request #1272 from jniggemann/doc-faq-add-prio
doc: FAQ: Add info on IO and CPU prioritization
2017-09-23 20:08:17 +02:00
Jan Niggemann
52a3eafede doc: FAQ: Add info on IO and CPU prioritization 2017-09-23 19:32:07 +02:00
Alexander Neumann
55dfc85159 manual: Add hint for RESTIC_PASSWORD_FILE 2017-09-23 19:16:07 +02:00
Alexander Neumann
a7a478a19e doc: Correct FAQ 2017-09-23 19:15:21 +02:00
Alexander Neumann
2080afd9de Merge pull request #1259 from jniggemann/doc-add-restic-check-advice
adds advice to run restic check regularly
2017-09-23 14:07:55 +02:00
Alexander Neumann
9aa136b982 Merge pull request #1260 from jniggemann/doc-add-info-on-special-items
Doc add info on special items
2017-09-23 14:07:20 +02:00
Alexander Neumann
3a191f37cb Add entry to CHANGELOG 2017-09-23 14:05:55 +02:00
Alexander Neumann
429106340f Merge pull request #1267 from harshavardhana/possible-fix-memory
Implement Size() and Len() to know the optimal size.
2017-09-23 14:04:15 +02:00
Alexander Neumann
530c73b457 Merge pull request #1269 from mrzv/forget-compact
Add --compact option to forget
2017-09-23 14:02:34 +02:00
Alexander Neumann
fb9729fdb9 sftp: Allow password entry
This was a bit tricky: We start the ssh binary, but we want it to ignore
SIGINT. In contrast, restic itself should process SIGINT and clean up
properly. Before, we used `setsid()` to give the ssh process its own
process group, but that means it cannot prompt the user for a password
because the tty is gone.

So, now we're passing in two functions that ignore SIGINT just before
the ssh process is started and re-install it after start.
2017-09-23 11:43:33 +02:00
Alexander Neumann
45a09c76ff Allow suspending SIGINT handler 2017-09-23 11:12:56 +02:00
Dmitriy Morozov
efd65a1b65 Update manpage for forget 2017-09-22 16:35:58 -07:00
Dmitriy Morozov
ae60188eb9 Add --compact option to forget 2017-09-22 16:32:59 -07:00
Jan Niggemann
3b904525d9 manual: Add info on special items (device files) 2017-09-22 22:58:26 +02:00
Jan Niggemann
1e31f5202f manual: Add info on special items
Add info about handling of symlinks and bind-mounts.

Closes: #1014
2017-09-22 22:51:31 +02:00
Jan Niggemann
f12d41138a Add advice to run check regularly 2017-09-22 22:27:10 +02:00
Harshavardhana
98369f6a5d Implement Size() and Len() to know the optimal size. 2017-09-22 12:09:17 -07:00
Alexander Neumann
8f9bf1995b Merge pull request #1265 from restic/improve-packers
Improve packers, prepare for cache
2017-09-22 16:16:10 +02:00
Alexander Neumann
e7de3b5f9d Merge pull request #1266 from JaCoB1123/document_hostname_option
Add note about rescan to hostname flag (fixes #1221)
2017-09-22 15:40:24 +02:00
Alexander Neumann
3541d06d07 repo: Split packers for tree and data
The code now bundles tree blobs and data blobs into different pack
files, so we'll end up with pack files that either only contain data or
trees. This is in preparation to adding a cache (#1040), because
tree-only pack files can easily be cached later on.
2017-09-22 15:36:47 +02:00
Alexander Neumann
db0e3cd772 repo: Remove packer limits
This commit simplifies finding a packer: The first open packer is taken,
and the upper limit for the pack file is removed.
2017-09-22 15:36:47 +02:00
Alexander Neumann
d3fee08f9a Merge pull request #1263 from restic/cleanups
Small cleanups
2017-09-22 15:36:13 +02:00
Jan Bader
727fb6eabe Add note about rescan to hostname flag (fixes #1221) 2017-09-22 14:29:04 +02:00
Alexander Neumann
d610c60991 repo: Remove unused sync.Pool 2017-09-22 12:37:10 +02:00
Alexander Neumann
3f6e11d26e Allow sorting nodes in trees 2017-09-22 12:37:05 +02:00
Alexander Neumann
a4577769ae Merge pull request #1262 from mafgh/bytes
fix duplicate bytes in prune output
2017-09-22 11:59:10 +02:00
Stefan Völkel
7f927d4774 fix duplicate bytes in prune output 2017-09-22 10:07:24 +02:00
Alexander Neumann
e091673f8f Merge pull request #1258 from restic/sftp-remove-stat
local/sftp: Remove unneeded stat() call
2017-09-21 22:37:12 +02:00
Alexander Neumann
9842eff887 local/sftp: Remove unneeded stat() call 2017-09-21 21:47:03 +02:00
Alexander Neumann
c40b3d3983 Add entry to CHANGELOG 2017-09-21 20:34:17 +02:00
Alexander Neumann
ac5eefdee4 Merge pull request #1249 from mungomat/fuse_symlink
fuse: added symlink 'latest' to snapshots-dir
2017-09-21 20:33:10 +02:00
Alexander Neumann
bf508643a5 Merge pull request #1257 from restic/update-deps
Update vendored dependencies
2017-09-21 20:19:25 +02:00
Alexander Neumann
02fc16e97d Update vendored dependencies 2017-09-21 17:48:45 +02:00
Tobias Klein
1a83a739dc fuse: added symlink 'latest' to snapshots-dir 2017-09-21 16:41:20 +02:00
Alexander Neumann
81473f4538 Add VERSION file for 0.7.3 2017-09-20 20:50:07 +02:00
Alexander Neumann
e1a847e4d1 Add new version to CHANGELOG 2017-09-20 20:49:55 +02:00
Alexander Neumann
0f426c3795 Merge pull request #1254 from jniggemann/doc_metadata
documents metadata handling, fixes #647
2017-09-20 20:24:57 +02:00
Alexander Neumann
6df3d169b8 Add entry to CHANGELOG 2017-09-20 11:05:35 +02:00
Alexander Neumann
5479daa6d4 Merge pull request #1247 from restic/fix-1246
Fix backend List()
2017-09-20 11:04:05 +02:00
Jan Niggemann
397fec0152 documents metadata handling, fixes #647 2017-09-19 15:49:54 +02:00
Alexander Neumann
d7e644272f prune: Add plausibility check 2017-09-19 10:50:07 +02:00
Alexander Neumann
e91749bbb0 Merge pull request #1245 from anarcat/faq
add explanation of restic automation
2017-09-19 10:46:55 +02:00
Antoine Beaupré
bcd1e45ba7 fix typo, add note about file permissions 2017-09-18 08:55:18 -04:00
Alexander Neumann
4c6b626db6 backend: Improve TestList 2017-09-18 13:18:42 +02:00
Alexander Neumann
835ba16c27 b2: Add pagination for List() 2017-09-18 12:13:35 +02:00
Alexander Neumann
3b6a580b32 backend: Make pagination for List configurable 2017-09-18 12:01:54 +02:00
Alexander Neumann
01c486d486 Merge pull request #1250 from dvrkps/patch-1
travis: update go versions
2017-09-17 20:59:35 +02:00
Alexander Neumann
6342a08a16 Merge pull request #1248 from mungomat/fuse_typo
fuse: typo
2017-09-17 20:57:42 +02:00
Davor Kapsa
94c8ee11f8 travis: update go versions 2017-09-17 19:02:22 +02:00
Tobias Klein
9b38980ed9 fuse: typo 2017-09-17 17:39:28 +02:00
Alexander Neumann
649c536250 backend: Improve test for pagination in list 2017-09-17 11:36:45 +02:00
Alexander Neumann
dd49e2b12d Azure: Fix List(), use pagination marker 2017-09-17 11:32:05 +02:00
Alexander Neumann
f61dab1774 backend: Add test for List() 2017-09-17 11:09:16 +02:00
Alexander Neumann
40edf00182 gs: implement pagination 2017-09-17 11:08:51 +02:00
Alexander Neumann
c35518a865 Azure/GS: Remove ReadDir() 2017-09-17 11:05:30 +02:00
Antoine Beaupré
7a0b4428e3 add explanation of restic automation
every time i look at restic, i block on this and figured it may be useful for others
2017-09-16 10:17:36 -04:00
Alexander Neumann
c784a15aaa Merge pull request #1244 from restic/fix-swift-backend-tests
Ignore "not exist" errors for swift backend tests
2017-09-16 14:54:32 +02:00
Alexander Neumann
ce180de9b8 Merge pull request #1243 from restic/improve-error-reporting
Improve error reporting
2017-09-16 14:54:30 +02:00
Alexander Neumann
fca9a523e9 Merge pull request #1241 from restic/fix-timestamp-check
Use .Equal() instead of == for time.Time
2017-09-16 14:54:26 +02:00
Alexander Neumann
8a3889be11 Merge pull request #1240 from restic/config-autocomplete-dir
Correct bash completion file path
2017-09-16 14:54:20 +02:00
Alexander Neumann
2a1633621b Ignore "not exist" errors for swift backend tests 2017-09-16 13:59:55 +02:00
Alexander Neumann
e2deeceb1b Update manpage 2017-09-16 11:29:37 +02:00
Alexander Neumann
d4e994de7b Improve error reporting
This will print the error (including a stack trace) if available before
exiting.
2017-09-16 10:55:13 +02:00
Alexander Neumann
a60e751217 Use .Equal() instead of == for time.Time
Closes #1238
2017-09-15 20:57:35 +02:00
Alexander Neumann
81c5d8a968 Correct bash completion file path 2017-09-15 20:45:16 +02:00
Alexander Neumann
5b1e4df177 Add version to CHANGELOG 2017-09-13 17:14:43 +02:00
Alexander Neumann
4d80744cbb Add VERSION file for 0.7.2 2017-09-13 17:14:26 +02:00
Alexander Neumann
e243d4b7ee Merge pull request #1236 from restic/update-deps
Update dependencies
2017-09-13 14:51:07 +02:00
Alexander Neumann
dce35fcb00 Merge pull request #1232 from prattmic/patch-1
doc: remove broken link
2017-09-13 14:18:46 +02:00
Alexander Neumann
e45a21b0b6 Merge pull request #1231 from restic/fix-local-datadir-create
local: Fix creating data dirs
2017-09-13 14:15:30 +02:00
Alexander Neumann
fda563d606 Update dependencies 2017-09-13 14:09:48 +02:00
Alexander Neumann
f3b49987f8 Add entry to CHANGELOG 2017-09-13 14:04:55 +02:00
Alexander Neumann
c8c01a5cae Merge pull request #1223 from mrzv/snapshots-compact
Add --compact option to snapshots
2017-09-13 14:02:31 +02:00
Michael Pratt
f7ece90129 doc: remove broken link
The link to SFTP is broken, remove it.

I don't even bother to put a new link, since SFTP is literally the next section.
2017-09-12 21:02:34 -07:00
Alexander Neumann
0f25ef9498 Merge pull request #1230 from shayneholmes/update-short-help
Update style in short help commands
2017-09-11 22:25:31 +02:00
Alexander Neumann
5bf2228596 local: Fix creating data dirs 2017-09-11 21:48:25 +02:00
Alexander Neumann
227b01395f local: Add test for open non-existing dir 2017-09-11 21:34:26 +02:00
Shayne Holmes
9f52fe1a10 Update manpages
This is a programmatic change; just ran `restic man-page`
2017-09-11 12:05:51 -07:00
Shayne Holmes
affc6c3390 Correct manpage command 2017-09-11 11:21:44 -07:00
Alexander Neumann
951a34dcbf Add entry to CHANGELOG 2017-09-11 19:02:36 +02:00
Alexander Neumann
36eaa22ed0 Merge pull request #1205 from mungomat/backup_time
Backup time
2017-09-11 19:01:41 +02:00
Alexander Neumann
62df316356 Merge pull request #1194 from mungomat/bugfix_bucker_always
fix: bucker
2017-09-11 19:01:23 +02:00
Shayne Holmes
00797fdd85 Synchronize manual with help output
Two commands in the output weren't mentioned in the manual.
2017-09-11 09:33:19 -07:00
Shayne Holmes
9eb39cef05 Capitalize short help commands
Unify existing Cobra help command, and git-help's style.
2017-09-11 09:32:44 -07:00
Shayne Holmes
ee6150f67c Change short help messages to imperative voice
Unify the output of `restic help`.
2017-09-11 09:26:13 -07:00
Alexander Neumann
9fa909ccd6 Update golden files 2017-09-11 17:52:22 +02:00
Alexander Neumann
b1af544b1d Merge pull request #1224 from restic/improve-exclude-caches
Allow multiple exclude-if-present
2017-09-11 17:31:24 +02:00
Dmitriy Morozov
7d5b17ac72 Update man page for snapshots 2017-09-10 15:28:06 -07:00
Dmitriy Morozov
7a221f2473 Run changes through gofmt 2017-09-10 15:09:28 -07:00
Dmitriy Morozov
bdbe956c5c Add --compact option to snapshots
With --compact, snapshots doesn't list directories and puts all tags on a single
line. This way each snapshot takes up exactly one line.
2017-09-10 13:06:43 -07:00
Alexander Neumann
8e5b1e6f2f Add --group-by to manual 2017-09-10 21:00:51 +02:00
Alexander Neumann
257a454515 Add entry to CHANGELOG 2017-09-10 20:55:02 +02:00
Alexander Neumann
b6aeea425b Merge pull request #1196 from mungomat/forget_groupByTagsOnly
forget: group-by-tags-only
2017-09-10 20:52:15 +02:00
Alexander Neumann
c8e05d1f2a Add entry to CHANGELOG 2017-09-10 20:29:08 +02:00
Alexander Neumann
a8aa4eb06c Rename parameter filename -> path 2017-09-10 20:28:21 +02:00
Alexander Neumann
c1a02cc081 Merge pull request #1228 from restic/fix-1204-3
Always use long name for keys
2017-09-10 19:39:23 +02:00
Alexander Neumann
e66adc42da Always use long name for keys
Otherwise the code panics if a file with a short name is tried.
2017-09-10 15:35:10 +02:00
Alexander Neumann
89938bc21c Update manual pages 2017-09-10 15:33:20 +02:00
Alexander Neumann
0b2947dedb Add test for rejectByPattern 2017-09-10 15:31:58 +02:00
Alexander Neumann
47ddd34266 Improve test 2017-09-10 15:20:41 +02:00
Alexander Neumann
2fdca5d310 Improve debug message 2017-09-10 15:16:05 +02:00
Alexander Neumann
e5d4e33509 Improve error message if no targets specified 2017-09-10 15:14:11 +02:00
Alexander Neumann
e117f613af Move device test into new RejectFunc 2017-09-10 15:13:40 +02:00
Alexander Neumann
0dfdf02885 Rework pattern excludes 2017-09-10 14:34:28 +02:00
Alexander Neumann
4a0129fc2b Rename excludeByFile -> rejectIfPresent 2017-09-10 14:25:58 +02:00
Alexander Neumann
a9c705009c Move reject functions to new file 2017-09-10 14:25:25 +02:00
Alexander Neumann
d937ad8cf6 Rename FilenameCheck to RejectFunc
We already have the opposite: pipe.SelectFunc(item string, fi
os.FileInfo) bool, so RejectFunc is a good name.
2017-09-10 14:21:51 +02:00
Alexander Neumann
1a08a8219f Merge pull request #1227 from restic/fix-1204-2
Handle invalid key file
2017-09-10 14:12:51 +02:00
Tobias Klein
9924c311c9 added test cases 2017-09-10 12:23:28 +02:00
Alexander Neumann
e846e14965 Ignore files with invalid name in the repo 2017-09-10 11:00:07 +02:00
Alexander Neumann
36e70228f2 Handle invalid key file 2017-09-10 10:55:01 +02:00
Tobias Klein
a677f1139a removed unnacessary line 2017-09-10 10:41:07 +02:00
Alexander Neumann
6f8eba9c28 Merge pull request #1222 from damiencourousse/manual-typo-fix
manual: typo fix
2017-09-09 21:40:00 +02:00
Alexander Neumann
c22c582546 Allow multiple exclude-if-present 2017-09-09 21:24:29 +02:00
Alexander Neumann
ea75509d6e Print warning for non-existing items 2017-09-09 21:12:41 +02:00
Tobias Klein
ed30bd7b76 gofmt 2017-09-09 18:19:19 +02:00
Damien Couroussé
7090c5ceeb manual: typo fix 2017-09-09 16:58:17 +02:00
Tobias Klein
bee09c1a0f test 2017-09-09 16:33:51 +02:00
Tobias Klein
8f9ef4402b error in case of unknown grouping option 2017-09-09 15:55:37 +02:00
Tobias Klein
f26c0cb70f testcase updated 2017-09-09 15:33:12 +02:00
Tobias Klein
81d7ecba2b manual updated 2017-09-09 13:26:35 +02:00
Tobias Klein
087c3fe1dc tests updated 2017-09-09 13:26:35 +02:00
Tobias Klein
43ff971dfd new sub-option for backup: time
New option to specify the timestamp for a backup
2017-09-09 13:26:35 +02:00
Alexander Neumann
5c75a98053 Merge pull request #1220 from restic/fix-1204
Fix panic when file name is too short
2017-09-09 11:49:23 +02:00
Alexander Neumann
7ce47402fb Merge pull request #1170 from fawick/exclude_caches
Add option to exclude contents of cache directories
2017-09-09 10:56:12 +02:00
Alexander Neumann
1e48141648 Fix panic when file name is too short
Closes #1204
2017-09-09 10:50:32 +02:00
Fabian Wickborn
dbda892542 Add option to exclude directories with a tagfile
The option is named --exclude-if-present and accepts a parameter
filename[:content]. Directories are excluded and their contents is not
backed up if they contain a file with the specified name and,
optionally, that starts with the specified content. The tagfile itself
is never excluded.

There is also a shortcut --exclude-caches that works in the same way as
the likewise-named option of tar(1): Directories are recognized as cache
if they contain a file named "CACHEDIR.TAG.

Closes #317.
2017-09-09 09:57:42 +02:00
Alexander Neumann
b46774be21 Merge pull request #1214 from ricardoseriani/fix-manual.rst-key-remove
Fix manual.rst to use key remove instead of key rm
2017-09-07 21:16:46 +02:00
Tobias Klein
1073bfba37 flexible grouping option for the forget-command 2017-09-06 20:14:18 +02:00
Alexander Neumann
5dfb4d1195 Merge pull request #1209 from restic/handle-colliding-names
Resolve name collisions
2017-09-05 22:33:32 +02:00
Alexander Neumann
0a2219c5f7 Travis: Reduce workload by removing Go tip 2017-09-05 22:08:23 +02:00
Alexander Neumann
ff3149831e Merge pull request #1210 from prattmic/glob_test
filter: document recursive wildcards
2017-09-05 21:50:27 +02:00
Alexander Neumann
c935d0558c Add entry to CHANGELOG 2017-09-05 21:48:13 +02:00
Alexander Neumann
83eb075e3a Resolve name collisions
At the moment when two items to be saved have the same directory name,
restic only saves the first one to the repo. Let's say we have a
structure like this:

    dir1
    └── subdir
        └── file
    dir2
    └── subdir
        └── file

When restic is run on `dir1/subdir` and `dir2/subdir`, it will only save
the first `subdir`:

    $ restic backup dir1/subdir dir2/subdir
    [...]

    $ restic ls -l latest
    drwxr-xr-x  1000   100      0 2017-08-27 20:56:39 /subdir
    -rw-r--r--  1000   100     17 2017-08-27 20:56:39 /subdir/file

That's obviously a bad thing, caused by an early decision to strip the
full path to the files/dirs to save and only leave the last directory.

This commit partly resolves this by handling colliding names and
resolving the conflicts. Restic will now append a counter to the file
(`-123`) until the conflict is resolved. So in the example above, we'll
end up with the following structure:

    $ restic ls -l latest
    drwxr-xr-x  1000   100      0 2017-08-27 20:56:39 /subdir
    -rw-r--r--  1000   100     17 2017-08-27 20:56:39 /subdir/file
    drwxr-xr-x  1000   100      0 2017-08-27 20:56:46 /subdir-1
    -rw-r--r--  1000   100     17 2017-08-27 20:56:46 /subdir-1/file

This partly addresses #549 and closes #1179.

At first I thought that the obvious correction would be to archive the
full path. But it turns out that collisions may still occur: Suppose you
have a file named `foo` in the current directory, and the parent directory
also contains a file `foo`. Archiving these with restic also causes a
collision, since restic strips the `../` from the first file:

    $ restic backup ../foo foo

This also happens with `tar`, which does not handle the collision and
will happily archive two files called `foo`.

So, the best way forward is to handle name collisions and archive the
whole path. The latter will be tackled in a separate PR.
2017-09-05 21:47:02 +02:00
Ricardo Seriani
204c2bf09c Fix manual to use key remove instead of key rm 2017-09-05 16:22:42 -03:00
Alexander Neumann
2444522243 Add test for colliding names 2017-09-05 21:10:02 +02:00
Michael Pratt
92eb1cbffd filter: document recursive wildcards
Match/ChildMatch accept a ** pattern which is not noted in the doc
string, nor do any of the docs or tests specify whether the match is
greedy (i.e., can 'foo/**/bar' match paths with additional intermediate
bar directories?).

Add a note to the doc string and add test cases for greedy matches.
2017-09-04 14:38:48 -07:00
Alexander Neumann
8c40ae5a03 Add entry to CHANGELOG 2017-09-04 21:58:33 +02:00
Alexander Neumann
fa2ee78a5c Merge pull request #1044 from lloeki/982-improve-restore
Improve restore
2017-09-04 21:51:12 +02:00
Tobias Klein
e4a5cdc5bc forget: group-by-tags-only 2017-09-03 17:11:25 +02:00
Tobias Klein
2d73a273af saving a variable 2017-09-03 17:09:55 +02:00
Tobias Klein
761af08889 fix: bucker
bucker "always" does not return a unique id in case of exact same timestamps
2017-09-03 17:09:55 +02:00
Alexander Neumann
0ee1650f82 Merge pull request #1191 from prattmic/profile
debug: properly handle interrupted profiles
2017-09-03 09:49:46 +02:00
Alexander Neumann
0e647417f3 Add entry to CHANGELOG 2017-09-03 09:49:37 +02:00
Alexander Neumann
d1bf5a4882 Merge pull request #1203 from myfreeweb/master
Handle SIGINFO on all supported platforms
2017-09-03 09:44:24 +02:00
Alexander Neumann
b8414b240c Add entry to CHANGELOG 2017-09-03 09:44:15 +02:00
Greg V
3fbdd12b04 Handle SIGINFO on all supported platforms
Not just darwin
2017-09-02 22:06:31 +03:00
Alexander Neumann
a3f6bf3e5a Merge pull request #1202 from restic/fix-manpages
Fix manpages, update Go version
2017-09-02 10:11:43 +02:00
Alexander Neumann
3a5805db50 Update Go versions for CI 2017-09-02 09:29:02 +02:00
Alexander Neumann
de8c64e767 Use deterministic date for man pages 2017-09-02 09:27:11 +02:00
Alexander Neumann
73d6b15095 Merge pull request #1201 from gjmf/patch-1
Fixed word-o. ("Package", not "packet".)
2017-09-01 21:21:18 +02:00
Alexander Neumann
5d396b9302 Merge pull request #1200 from molivier/patch-1
Update manual.rst
2017-09-01 21:21:16 +02:00
Graham Freeman
61d2519111 Fixed word-o. ("Package", not "packet".)
Fixed a word-o. homebrew is a package manager, not a packet manager. :)
2017-08-31 13:34:53 -07:00
Matthieu OLIVIER
e61c94a846 Update manual.rst
`key remove` becomes `key rm`.
2017-08-31 18:26:04 +02:00
Alexander Neumann
7ed0f61f3f Merge pull request #1189 from FiloSottile/patch-2
doc/design: fix keys.data MAC format description
2017-08-29 20:44:31 +02:00
Alexander Neumann
85055d1c68 Merge pull request #1187 from FiloSottile/patch-1
internal/crypto: small simplifications
2017-08-29 20:43:15 +02:00
Michael Pratt
e4c469c149 debug: properly handle interrupted profiles
By default (i.e., without profile.NoShutdownHook), profile.Start listens
for SIGINT and will stop the profile and call os.Exit(0).

restic already listens for SIGINT and runs its own cleanup handlers
before calling os.Exit(0).

As is, these handlers are racing when an interrupt occurs, and in my
experience, restic tends to win the race, resulting in an unusable
profile.

Eliminate the race and properly stop profiles on interrupt by disabling
package profile's signal handler and instead stop the profile in a
restic cleanup handler.
2017-08-28 22:03:26 -07:00
Filippo Valsorda
9940e8d9f1 internal/crypto: small simplifications
* append operates on len, not cap (not a bug since len is set to cap above, but let's avoid the confusion)
* no need to extend ciphertext again to cap after we made it big enough
* make consistent use of ciphertext[:ivSize] vs iv[:]
* make all input problems errors and impossible/catastrophic cases panics
2017-08-29 00:30:06 +02:00
Filippo Valsorda
3dccca1f27 doc/design: fix keys.data MAC format description
"not including the last 32 byte" was wrong, should have been 16 bytes. But the whole description is redundant anyway.
2017-08-29 00:22:11 +02:00
Alexander Neumann
22e96a37f8 Merge pull request #1184 from prattmic/docs
Doc cleanup and mention restore include/exclude
2017-08-28 21:14:48 +02:00
Alexander Neumann
48b1ab5aaf Merge pull request #1182 from restic/fix-1167
local: do not create dirs below data/ for non-existing dir
2017-08-28 21:13:24 +02:00
Alexander Neumann
0230fa188f Add entry to CHANGELOG 2017-08-28 21:13:14 +02:00
Alexander Neumann
4118ce876e Merge pull request #1185 from prattmic/gcs_panic
gs: fix nil dereference
2017-08-28 21:11:30 +02:00
Michael Pratt
9537bc561d gs: fix nil dereference
info can be nil if err != nil, resulting in a nil dereference while
logging:

$ # GCS config
$ ./restic init
debug enabled
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x935947]

goroutine 1 [running]:
github.com/restic/restic/internal/backend/gs.(*Backend).Save(0xc420012690, 0xe84e80, 0xc420010448, 0xb57149, 0x3, 0xc4203fc140, 0x40, 0xe7be40, 0xc4201d8f90, 0xa0, ...)
	src/github.com/restic/restic/internal/backend/gs/gs.go:226 +0x6d7
github.com/restic/restic/internal/repository.AddKey(0xe84e80, 0xc420010448, 0xc4202f0360, 0xc42000a1b0, 0x4, 0x0, 0xa55b60, 0xc4203043e0, 0xa55420)
	src/github.com/restic/restic/internal/repository/key.go:235 +0x4a1
github.com/restic/restic/internal/repository.createMasterKey(0xc4202f0360, 0xc42000a1b0, 0x4, 0xa55420, 0xc420304370, 0x6a6070)
	src/github.com/restic/restic/internal/repository/key.go:62 +0x60
github.com/restic/restic/internal/repository.(*Repository).init(0xc4202f0360, 0xe84e80, 0xc420010448, 0xc42000a1b0, 0x4, 0x1, 0xc42030a440, 0x40, 0x32a4573d3d9eb5, 0x0, ...)
	src/github.com/restic/restic/internal/repository/repository.go:403 +0x5d
github.com/restic/restic/internal/repository.(*Repository).Init(0xc4202f0360, 0xe84e80, 0xc420010448, 0xc42000a1b0, 0x4, 0xe84e40, 0xc42004ad80)
	src/github.com/restic/restic/internal/repository/repository.go:397 +0x12c
main.runInit(0xc420018072, 0x16, 0x0, 0x0, 0x0, 0xe84e40, 0xc42004ad80, 0xc42000a1b0, 0x4, 0xe7dac0, ...)
	src/github.com/restic/restic/cmd/restic/cmd_init.go:47 +0x2a4
main.glob..func9(0xeb5000, 0xedad70, 0x0, 0x0, 0x0, 0x0)
	src/github.com/restic/restic/cmd/restic/cmd_init.go:20 +0x8e
github.com/restic/restic/vendor/github.com/spf13/cobra.(*Command).execute(0xeb5000, 0xedad70, 0x0, 0x0, 0xeb5000, 0xedad70)
	src/github.com/restic/restic/vendor/github.com/spf13/cobra/command.go:649 +0x457
github.com/restic/restic/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xeb3e00, 0xc420011650, 0xa55b60, 0xc420011660)
	src/github.com/restic/restic/vendor/github.com/spf13/cobra/command.go:728 +0x339
github.com/restic/restic/vendor/github.com/spf13/cobra.(*Command).Execute(0xeb3e00, 0x25, 0xc4201a7eb8)
	src/github.com/restic/restic/vendor/github.com/spf13/cobra/command.go:687 +0x2b
main.main()
	src/github.com/restic/restic/cmd/restic/main.go:72 +0x268

(The error was likely because I had just enabled the GCS API. Subsequent
runs were fine.)
2017-08-27 21:36:04 -07:00
Michael Pratt
ae43c47ca8 doc: add mention of restore --exclude/--include
There is a lot more detail that could be added here, but it is worth
getting things off the ground with at least a mention that it is
possible to restore individual files.

Updates #396
2017-08-27 18:36:00 -07:00
Michael Pratt
2fa4060991 doc: fix inconsistencies
* Replace references to ~/shared/work/web which should be ~/web.

* Replace references to ~/tmp which should be /tmp.

* Restore /home/art to /tmp/restore-art instead of /tmp/restore-work,
  which is clearly a copy/paste from the command above.
2017-08-27 18:35:01 -07:00
Alexander Neumann
f9a934759f sftp: Improve error handling for non-existing dir 2017-08-27 20:53:04 +02:00
Alexander Neumann
3686b1ffe5 local: Create directories below data/ if it exists 2017-08-27 20:52:58 +02:00
Alexander Neumann
ea017a49c3 local: Add test for #1167
It was discovered that restic creates directories when a non-existing
directory is specified as a local repository.
2017-08-27 20:38:46 +02:00
Alexander Neumann
3559f9c776 Merge pull request #1174 from pwaring/patch-1
Update minimum Go version
2017-08-25 21:26:13 +02:00
Paul Waring
637f57ca71 Update minimum Go version
Minimum version is now 1.8 according to build.go (from latest master, cloned a few minutes ago):

```
paul@voga:~/third-dev/restic$ go run build.go 
Go version go1.7.4 detected, restic requires at least Go 1.8
exit status 1
```
2017-08-24 19:51:11 +01:00
Alexander Neumann
4e60156b45 Add entry to CHANGELOG 2017-08-18 19:50:32 +02:00
Alexander Neumann
af9946b098 Merge pull request #1164 from ricardoseriani/fix-key-remove-command
Change key rm command to key remove
2017-08-18 19:49:50 +02:00
Ricardo Seriani
b7d4b0f821 Update man pages 2017-08-17 16:44:28 -03:00
Ricardo Seriani
62ed776a8c Change key rm command to key remove
Change key rm command to key remove, to follow manual and other commands
2017-08-17 11:03:26 -03:00
Loic Nageleisen
f880ff21aa Fixing restore with excluded
An exclude filter is basically a 'wildcard but foo', so even if a
childMayMatch, other children of a dir may not, therefore childMayMatch
does not matter, but we should not go down unless the dir is selected
for restore.
2017-08-16 15:25:02 +02:00
Loic Nageleisen
4a36993c19 Smarter filter when children won't match
This improves restore performance by several orders of magniture by not
going through the whole tree recursively when we can anticipate that no
match will ever occur.
2017-08-16 15:25:02 +02:00
Alexander Neumann
d87b2f189d Merge pull request #1157 from ceh/defer-file-close-after-err-check
internal: check error before deferring file Close()
2017-08-13 19:50:05 +02:00
Alexander Neumann
f9a097a8c0 Merge pull request #1158 from ceh/fix-contributing-typo
Fix contribution typo
2017-08-13 19:49:03 +02:00
Alexander Neumann
d43358b6dd Correct URL to forum 2017-08-13 19:47:54 +02:00
Alexander Neumann
8058f196e1 Merge pull request #1156 from dimejo/patch-1
Readme: Correct link formatting
2017-08-13 19:47:31 +02:00
Emil Hessman
e13e6f34d2 Fix contribution typo 2017-08-13 19:35:53 +02:00
Emil Hessman
c2ff7150aa internal: check error before deferring file Close()
If there is an error, file will be `nil`. We should check the returned error before deferring file `Close()`.
2017-08-13 19:28:13 +02:00
dimejo
a899621930 Readme: Correct link formatting 2017-08-13 15:07:40 +02:00
Alexander Neumann
a0966e1d1d Update README 2017-08-11 21:38:18 +02:00
Alexander Neumann
e2464382ed Update issue template 2017-08-11 21:36:10 +02:00
Alexander Neumann
095bc79dc3 Correct README #2 2017-08-09 21:58:20 +02:00
Alexander Neumann
1fd3c2488e Correct README 2017-08-09 21:57:55 +02:00
Alexander Neumann
2ee8485886 Update README 2017-08-09 21:56:41 +02:00
Alexander Neumann
b67c178672 Merge pull request #1149 from restic/azure-support
Add Azure blob storage as backend
2017-08-09 21:30:35 +02:00
Alexander Neumann
7ac4f0a525 Merge pull request #1134 from restic/gcs-support
Add backend for Google Cloud Storage (GCS)
2017-08-09 21:28:36 +02:00
Alexander Neumann
c4613c51d1 Add note about Go 1.8 2017-08-09 20:33:30 +02:00
Alexander Neumann
77bf17076b Add entry to CHANGELOG 2017-08-09 20:23:29 +02:00
Alexander Neumann
8dd6beba15 gs: Add section to the manual 2017-08-09 20:22:07 +02:00
Alexander Neumann
a345386967 Add a section to the CHANGELOG 2017-08-09 20:15:08 +02:00
Alexander Neumann
bdd43bd430 Add a section to the manual 2017-08-09 20:13:34 +02:00
Alexander Neumann
1716501598 CI: Make sure the GCS backend tests run on Travis 2017-08-06 21:47:56 +02:00
Alexander Neumann
d9a5b9178e gs: Rework path initialization 2017-08-06 21:47:56 +02:00
Alexander Neumann
8ca6a9a240 Vendor dependencies for GCS 2017-08-06 21:47:56 +02:00
Dipta Das
ba75a3884c Add Google Cloud Storage as backend
Environment variables:
GOOGLE_PROJECT_ID=gcp-project-id
GOOGLE_APPLICATION_CREDENTIALS=path-to-json-file

Environment variables for test:
RESTIC_TEST_GS_PROJECT_ID=gcp-project-id
RESTIC_TEST_GS_APPLICATION_CREDENTIALS=path-to-json-file
RESTIC_TEST_GS_REPOSITORY=gs:us-central1/test-bucket

Init repository:
$ restic -r gs🪣/[prefix] init
2017-08-06 21:47:55 +02:00
Alexander Neumann
d91d89eef6 azure: Create container if it does not exist 2017-08-06 21:47:04 +02:00
Alexander Neumann
a726c91116 azure: Rework path initialization 2017-08-06 21:47:04 +02:00
Alexander Neumann
d00fe95f10 Upgrade min Go version to 1.8 2017-08-06 21:47:04 +02:00
Alexander Neumann
072b7a014e azure: User internal errors package 2017-08-06 21:47:04 +02:00
Alexander Neumann
618ce115d7 Azure: Use default HTTP transport 2017-08-06 21:47:04 +02:00
Alexander Neumann
d973aa82fe Vendor dependencies for azure backend 2017-08-06 21:47:04 +02:00
Dipta Das
3a85b6b7c6 Add Azure Blob Storage as backend
Environment variables:
AZURE_ACCOUNT_NAME=storage-account-name
AZURE_ACCOUNT_KEY=storage-account-key

Environment variables for test:
RESTIC_TEST_AZURE_ACCOUNT_NAME=storage-account-name
RESTIC_TEST_AZURE_ACCOUNT_KEY=storage-account-key
RESTIC_TEST_AZURE_REPOSITORY=azure:restic-test-container

Init repository:
$ restic -r azure:container-name:/prefix/dir init
2017-08-06 21:47:04 +02:00
Alexander Neumann
2c22ff175c Merge pull request #1150 from restic/removed-date-from-manpage
Fix manpage generation
2017-08-06 21:46:44 +02:00
Alexander Neumann
6bc43a4198 manpage: Remove auto gen tag from man page 2017-08-06 21:31:01 +02:00
Alexander Neumann
e348b3deeb manpage: Do not panic when no command is given 2017-08-06 21:01:49 +02:00
Alexander Neumann
6724b9a583 Merge pull request #1148 from restic/update-simple-scrypt
Lock simple-scrypt library to master branch
2017-08-05 20:04:33 +02:00
Alexander Neumann
41c35b2218 Lock simple-scrypt library to master branch
The master branch includes a fix for i386, otherwise the calibration
panics. See https://github.com/restic/restic/issues/676 for details.
2017-08-05 19:24:56 +02:00
Alexander Neumann
4477d76f03 Merge pull request #1147 from restic/add-manpage
Add new command 'manpage'
2017-08-05 12:39:42 +02:00
Alexander Neumann
14f5f6235a Add entry to CHANGELOG 2017-08-05 12:05:53 +02:00
Alexander Neumann
739350fd8e backup: Do not print hostname in help text
This is necessary so that the manpage generation is deterministic and we
can test if the man pages are up to date when the CI tests run.
2017-08-05 12:05:53 +02:00
Alexander Neumann
14ed97102b Add instructions for developers 2017-08-05 12:05:53 +02:00
Alexander Neumann
db389058fa Add generated manual pages 2017-08-05 12:05:53 +02:00
Alexander Neumann
b557d04007 CI: Fix test for forbidden packages 2017-08-05 12:05:53 +02:00
Alexander Neumann
52c5da997b Add CI test for manpages 2017-08-05 12:05:53 +02:00
Alexander Neumann
57d198f99a Vendor dependencies for manpage generation 2017-08-05 11:08:49 +02:00
Alexander Neumann
a3ab17b470 Add 'manpage' command to generate manual pages 2017-08-05 10:57:01 +02:00
Alexander Neumann
9bf3141893 Add entry to CHANGELOG 2017-08-01 22:02:08 +02:00
Alexander Neumann
d35eb6a0c3 Merge pull request #1144 from wjkohnen/close-exclude-files
Close exclude files and check errors
2017-08-01 22:01:43 +02:00
Johannes Kohnen
37aad2e3aa Close exclude files and check errors 2017-08-01 17:34:27 +00:00
Alexander Neumann
efc5d0699a Merge pull request #1139 from donat-b/pwfile-doesnt-exist
Error message in case PasswordFile is missing
2017-07-27 15:41:50 +02:00
donat
893bc9f777 Error message in case PasswordFile is missing 2017-07-27 14:23:08 +03:00
Alexander Neumann
61b8729ef9 Merge pull request #1138 from stapelberg/patch-1
fix typo: explicitely → explicitly
2017-07-27 09:01:47 +02:00
Michael Stapelberg
b89d3cc4d0 fix typo: explicitely → explicitly 2017-07-27 08:24:53 +02:00
Alexander Neumann
e8cc11ea34 Add entry to CHANGELOG 2017-07-26 22:29:39 +02:00
Alexander Neumann
2e804511ca Merge pull request #1133 from middelink/fix-1132
Force restic to ask the password when adding a key.
2017-07-26 22:11:37 +02:00
Alexander Neumann
b6790c491b Merge pull request #1131 from restic/test-cleanups
Rework withTestEnvironment
2017-07-26 22:06:06 +02:00
Pauline Middelink
c95e2b009e Simpify cmd_backup and cmd_init now we have the password in gopts. 2017-07-24 23:32:55 +02:00
Pauline Middelink
d5615a67c8 Refactor password resolving.
Instead of determining the password lazily during ReadPassword(), do so now in
cobra.PersistentPreRunE() so we can store the result in the globalOptions and
reuse/override when applicable without having to worry about the environment
or flag options interfering.
2017-07-24 23:05:37 +02:00
Pauline Middelink
d9b9bbd4a8 Force restic to ask the password when adding a key.
As `restic key add` uses the same `ReadPasswordTwice()` as the
rest of restic, it is sensitive to the environment variable
RESTIC_PASSWORD or --password-file= override.

When asking for the new key, temporary remove these 2 overrides, forcing
the password to be asked.
2017-07-24 22:00:44 +02:00
Alexander Neumann
d780b1eede Rework withTestEnvironment
Switch from a function passed as a parameter to a cleanup function,
which is also executed when the test function panics, so no temporary
directories are left behind.
2017-07-24 21:32:34 +02:00
Alexander Neumann
608adf15a3 Merge pull request #1130 from middelink/fix-fuse-test
Fuse testing leaves test mountpoint around.
2017-07-24 21:26:05 +02:00
Pauline Middelink
1717391f6c Fuse testing leaves test mountpoint around. Move it under the testing tree which is removed after each test. 2017-07-24 20:33:39 +02:00
Alexander Neumann
2e6e9ff6f8 Merge pull request #1129 from restic/move-restic-pkg
Move restic package to internal/restic
2017-07-24 19:19:28 +02:00
Alexander Neumann
23c903074c Move restic package to internal/restic 2017-07-24 17:43:32 +02:00
Alexander Neumann
94030a12cf Add entry to CHANGELOG 2017-07-23 16:54:56 +02:00
Alexander Neumann
f63d7de9da Merge pull request #1126 from restic/switch-to-default-go-git-layout
Use idiomatic default Go git repo layout
2017-07-23 16:54:53 +02:00
Alexander Neumann
13ee6792df Add remark about GOPATH on forked repos 2017-07-23 16:36:13 +02:00
Alexander Neumann
6302444f34 Remove linebreak from Errorf() format string 2017-07-23 15:51:44 +02:00
Alexander Neumann
61c5e4b54a Fix glyphcheck 2017-07-23 15:51:39 +02:00
Alexander Neumann
d6118871be Update other files 2017-07-23 14:40:05 +02:00
Alexander Neumann
94b27e8933 Fix paths for tests 2017-07-23 14:39:57 +02:00
Alexander Neumann
05500dc5f8 Update documentation 2017-07-23 14:32:31 +02:00
Alexander Neumann
c5a72971fe Remove Vagrantfile 2017-07-23 14:30:33 +02:00
Alexander Neumann
5bc6486e3b Update docs 2017-07-23 14:25:39 +02:00
Alexander Neumann
59e18bce0a Fix build.go 2017-07-23 14:25:39 +02:00
Alexander Neumann
898c5b6df5 Fix integration tests 2017-07-23 14:25:39 +02:00
Alexander Neumann
9cd422791a Update build.go 2017-07-23 14:25:38 +02:00
Alexander Neumann
91edebf1fe Vendor dependencies with dep 2017-07-23 14:25:38 +02:00
Alexander Neumann
df8a5792f1 Remove Dockerfile 2017-07-23 14:25:38 +02:00
Alexander Neumann
cda7b417cd Remove envrc 2017-07-23 14:25:38 +02:00
Alexander Neumann
d2ac35af26 Remove vendor 2017-07-23 14:25:37 +02:00
Alexander Neumann
6caeff2408 Run goimports 2017-07-23 14:21:03 +02:00
Alexander Neumann
83d1a46526 Moves files 2017-07-23 14:19:13 +02:00
Alexander Neumann
d1bd160b0a Merge pull request #1061 from bclermont/docker-image
add docker image
2017-07-22 11:52:22 +02:00
Alexander Neumann
bc88a8bb03 Add entry to CHANGELOG 2017-07-22 11:52:16 +02:00
Alexander Neumann
04cfb984ae Add VERSION file for 0.7.1 2017-07-22 11:04:32 +02:00
Alexander Neumann
02a245941a Adapt CHANGELOG for 0.7.1 2017-07-22 11:03:44 +02:00
Alexander Neumann
7fb1352aa1 Merge pull request #1124 from restic/use-minio-300
Set minio-go to v3.0.0
2017-07-22 11:01:57 +02:00
Alexander Neumann
4c555bad2e Set minio-go to v3.0.0 2017-07-22 10:19:52 +02:00
Alexander Neumann
75c789bab4 Merge pull request #1122 from restic/swift-remove-range-test
swift: Remove check for byte range
2017-07-21 23:05:05 +02:00
Alexander Neumann
626d020e62 swift: Remove check for byte range
Closes #1084
Closes #1094
2017-07-21 20:45:25 +02:00
Alexander Neumann
3830117735 Merge pull request #1121 from restic/fix-swift-test
swift: Increase backend test delay for removed file
2017-07-21 20:37:10 +02:00
Alexander Neumann
042cee8e36 Merge pull request #1117 from donat-b/password-file-env
Set default value for password-file flag from env
2017-07-21 19:47:49 +02:00
Alexander Neumann
03cc5b47e9 appveyor: Update Go version to 1.8.3 2017-07-21 19:42:34 +02:00
Alexander Neumann
46fa45942e swift: Increase backend test delay for removed file 2017-07-21 19:42:34 +02:00
Alexander Neumann
0cb4104aa7 Fix Go report card URLs (thanks @tyll) 2017-07-20 22:31:41 +02:00
donat
f2bbc5fbc4 Set default value for password-file flag from env
Allows defining password file path as RESTIC_PASSWORD_FILE=/foo
2017-07-20 10:47:02 +03:00
Alexander Neumann
16340ce811 Merge pull request #1090 from middelink/fix-1081
Update HasTags() and HasPaths() to follow #1081 feature request
2017-07-19 17:11:18 +02:00
Alexander Neumann
2cf8153f4a Add entry to CHANGELOG 2017-07-19 17:09:02 +02:00
Alexander Neumann
2f00287e45 Merge pull request #1112 from restic/fix-chmod-not-supported
Ignore error for Chmod() on FS that don't support it
2017-07-19 17:05:18 +02:00
Alexander Neumann
0de17f64e9 Add entry to CHANGELOG 2017-07-19 17:02:53 +02:00
Alexander Neumann
c30838878f Merge pull request #1115 from restic/fix-prune-index
prune: Fix newly created index
2017-07-19 17:01:11 +02:00
Alexander Neumann
bd31281f1e prune: Fix newly created index
This fixes a bug introduced in c79fb6fcdd
where the index file after a prune contains pack files that do not exist
any more. Restic will detect this during backup and abort with an error
message until the user runs `prune` or `rebuild-index` again.
2017-07-18 23:10:30 +02:00
Alexander Neumann
7fc54ed98e Improve test case for prune 2017-07-18 23:07:29 +02:00
Alexander Neumann
0abdcedcab Ignore error for Chmod() on FS that don't support it
See #1079
2017-07-18 21:47:30 +02:00
Alexander Neumann
6c05353086 Add entry to CHANGELOG 2017-07-17 22:02:06 +02:00
Alexander Neumann
e7575bf380 Merge pull request #1108 from restic/update-deps
Update vendored deps
2017-07-17 22:01:52 +02:00
Alexander Neumann
89ace85903 s3: Use streaming API and remove workarounds 2017-07-17 20:43:53 +02:00
Alexander Neumann
68a91d66b7 s3: Use new API for CopyObject 2017-07-17 20:43:45 +02:00
Alexander Neumann
724b5bf4fe Update minio-go 2017-07-17 20:19:04 +02:00
Alexander Neumann
d6da9211bc Update vendored deps (except minio-go) 2017-07-17 20:00:44 +02:00
Alexander Neumann
f45abac27f Merge pull request #1107 from bclermont/fix-s3-panic
Fix S3 panic on Invalid configuration
2017-07-17 18:50:46 +02:00
Bruno Clermont
00b9a1d87d evaluate open error 2017-07-17 11:33:19 +03:00
Alexander Neumann
20b835b5a4 Improve help text 2017-07-16 21:41:13 +02:00
Alexander Neumann
7bb1a474df Add entry to CHANGELOG 2017-07-16 21:41:13 +02:00
Alexander Neumann
750ee35dbf Add more examples to the manual 2017-07-16 21:40:53 +02:00
Alexander Neumann
fda5e1f543 Adress code review comments 2017-07-16 21:40:53 +02:00
Alexander Neumann
78d090aea5 Implement TagList and TagLists as pflag.Value 2017-07-16 21:40:53 +02:00
Alexander Neumann
7362569cf5 Use TagLists for all commands 2017-07-16 21:40:53 +02:00
Alexander Neumann
f5b1c7e5f1 Add TagList 2017-07-16 21:40:53 +02:00
Pauline Middelink
c554cdac4c Update HasTags() and HasPaths() to follow #1081 idea
Replace all but 3 occurences of StringSliceVar to StringArrayVar. This will
prevent the flag parser to interpretate the given values as CSV string.

Both --tag, --keep-tag and --path can be given multiple times, the command will
match snapshots matching ANY of the tags/paths. Only when a value is given which
contains a comma separated list of tags/paths, ALL elements need to match.
2017-07-16 21:40:53 +02:00
Alexander Neumann
41b624ea1b Merge pull request #1105 from restic/improve-sftp-open
sftp: Improve check for data subdirs
2017-07-16 15:41:13 +02:00
Alexander Neumann
7cdcaadcf5 Add entry to CHANGELOG 2017-07-16 15:11:26 +02:00
Alexander Neumann
4ad33d3c3b sftp: Improve check for data subdirs 2017-07-16 15:10:06 +02:00
Alexander Neumann
2778ac21de Merge pull request #1103 from tobya/docsupdate
Update links to design.md to design.rst
2017-07-16 10:39:53 +02:00
Toby Allen
cb3cd57926 Update links to design.md to design.rst 2017-07-15 19:35:45 +01:00
Alexander Neumann
ba6815d413 Merge pull request #1100 from fawick/master
Allow absolute target path in build.go
2017-07-15 10:11:45 +02:00
Fabian Wickborn
52004cdde8 Allow absolute target path in build.go
Fixes #1099.
2017-07-14 11:54:46 +02:00
Alexander Neumann
1d2045cb61 Test error for os.PathError
See https://github.com/restic/restic/issues/1079#issuecomment-315177469
for details.
2017-07-13 21:29:29 +02:00
Alexander Neumann
357e2e404a Merge pull request #1080 from restic/fix-1079
local: Ignore ENOTSUP error for chmod
2017-07-13 20:12:47 +02:00
Alexander Neumann
38e5640cda Add CHANGELOG entry 2017-07-09 21:43:05 +02:00
Alexander Neumann
c4c731bd9a Merge pull request #1082 from Habbie/siginfo
support SIGINFO on Darwin
2017-07-09 21:41:29 +02:00
Alexander Neumann
04d27acd60 Add entry to CHANGELOG 2017-07-05 20:54:37 +02:00
Alexander Neumann
80f0303b21 Merge pull request #1086 from kamsz/iam
Add support for IAM instance profile
2017-07-05 20:53:31 +02:00
Kamil Szczygieł
d651d9b427 more verbose debug 2017-07-05 19:21:57 +02:00
Kamil Szczygieł
3b2648bd5e iam instance profile 2017-07-05 16:19:25 +02:00
Peter van Dijk
73cc11f000 support SIGINFO on Darwin 2017-07-03 20:39:42 +02:00
Alexander Neumann
637de0149c Add entry to CHANGELOG 2017-07-03 19:49:18 +02:00
Alexander Neumann
855575e5a7 Merge pull request #1077 from restic/create-subdirs
local/sftp: Auto-create subdirs of data/ on init/open
2017-07-03 19:47:58 +02:00
Alexander Neumann
ed2999a163 Merge pull request #1075 from restic/migrate-s3-continue
s3: Improve migration to new layout
2017-07-03 19:47:22 +02:00
Alexander Neumann
a18c16e19e local: Ignore ENOTSUP error for chmod
Closes: #1079
2017-07-03 19:45:56 +02:00
Alexander Neumann
9032ab2eec local/sftp: Create dirs on open() 2017-07-02 19:35:45 +02:00
Alexander Neumann
03f66b8d74 Create subdirs below data/ 2017-07-02 19:35:45 +02:00
Alexander Neumann
8c30ae7c65 Add entry to CHANGELOG 2017-07-02 11:21:05 +02:00
Alexander Neumann
453c9c9199 s3 migrate layout: Retry on errors 2017-07-02 11:15:20 +02:00
Alexander Neumann
993e370f92 s3 migrate layout: Ignore already renamed files 2017-07-02 10:47:50 +02:00
Alexander Neumann
2bcd3a3acc s3 migrate layout: Rename key files last 2017-07-02 10:47:20 +02:00
Alexander Neumann
c54c632ca1 s3 migrate layout: Force old layout for rename 2017-07-02 10:47:03 +02:00
Alexander Neumann
28a4a35625 Allow migrate to run althoug check failed 2017-07-02 10:29:41 +02:00
Alexander Neumann
e7577d7bb4 Add stub to CHANGELOG 2017-07-01 15:11:36 +02:00
Alexander Neumann
27ea0623d7 Add VERSION file for 0.7.0 2017-07-01 14:12:07 +02:00
Alexander Neumann
390e2bbddc Merge pull request #1070 from restic/warn-unsupported-repo-type
Return an error for invalid backend schemes
2017-06-30 22:15:17 +02:00
Alexander Neumann
b50fc08f39 Add entry to CHANGELOG 2017-06-30 22:15:00 +02:00
Alexander Neumann
b2ce7e8d84 Return an error for invalid backend schemes
Closes #1021
2017-06-30 21:28:39 +02:00
Alexander Neumann
2b1c6d3cf8 Merge pull request #1066 from restic/update-minio-go
Update minio-go
2017-06-30 20:40:43 +02:00
Alexander Neumann
c658305a1b Correct path for rest-server 2017-06-27 21:19:48 +02:00
Alexander Neumann
63235d8f94 Update minio-go 2017-06-26 22:06:57 +02:00
Bruno Clermont
d702227af0 install fuse and ca-certificates 2017-06-23 10:38:19 +02:00
Bruno Clermont
b7251dbea5 add docker image 2017-06-23 10:28:18 +02:00
Alexander Neumann
144b7f3386 doc: Correct path in manual 2017-06-22 19:54:55 +02:00
Alexander Neumann
9583dc820f Merge pull request #1051 from restic/refactor-crypto
crypto: Make Encrypt/Decrypt a method of *Key
2017-06-21 19:26:11 +02:00
Alexander Neumann
a03076f2d8 Merge pull request #1056 from restic/fix-1053
prune: Delete invalid/incomplete pack files
2017-06-21 19:25:55 +02:00
Alexander Neumann
d76fa22b4b prune: Delete invalid/incomplete pack files
Closes #1053
2017-06-20 22:53:49 +02:00
Alexander Neumann
f960831f10 crypto: Make Encrypt/Decrypt a method of *Key 2017-06-20 22:14:51 +02:00
Alexander Neumann
b0fb95dfc9 backend tests: Use delayedRemove() 2017-06-19 20:02:49 +02:00
Alexander Neumann
bca9566849 Merge pull request #1050 from restic/extend-fuse-mount
fuse: Add more directories
2017-06-19 19:52:45 +02:00
Alexander Neumann
8760de42fe Merge pull request #1046 from restic/s3-split-open
s3: Split Create() from Open()
2017-06-19 19:52:40 +02:00
Alexander Neumann
2c02efd1fe fuse: Reduce code duplication, add MetaDir 2017-06-18 21:32:07 +02:00
Alexander Neumann
4b4a63ed44 fuse: Add tags dir 2017-06-18 21:32:07 +02:00
Alexander Neumann
64f434eca4 fuse: Add hosts dir 2017-06-18 21:32:07 +02:00
Alexander Neumann
f4e85a53e7 fuse: Add '.' and '..' entries to all directories 2017-06-18 21:32:07 +02:00
Alexander Neumann
f8176a74ec fuse: Rename DirSnapshots -> SnapshotsDir 2017-06-18 21:32:07 +02:00
Alexander Neumann
e60a96a71a swift: Increase delete timeout to 20s 2017-06-18 21:31:48 +02:00
Alexander Neumann
216e2607ca Add entry to CHANGELOG 2017-06-18 21:18:11 +02:00
Alexander Neumann
53f8026018 Merge pull request #1048 from restic/cleanup-fuse-mount
Cleanup/fix fuse mount
2017-06-18 18:41:02 +02:00
Alexander Neumann
de92ce7a88 Merge pull request #1049 from restic/fix-backend-tests-delayed-remove
backend tests: Add configurable delay for delayed remove
2017-06-18 18:31:38 +02:00
Alexander Neumann
eb8041b943 backend tests: Add configurable delay for delayed remove 2017-06-18 17:36:57 +02:00
Alexander Neumann
9c6e9bcf33 fuse: Add build tags for unsupported OS 2017-06-18 17:02:07 +02:00
Alexander Neumann
154816ffd0 fuse: Fix file test 2017-06-18 16:29:00 +02:00
Alexander Neumann
c86e425df6 fuse: Fix file inode 2017-06-18 16:28:55 +02:00
Alexander Neumann
3883c7a190 fuse: Fix blob length cache 2017-06-18 16:28:39 +02:00
Alexander Neumann
a66760d86d fuse: Fix inode handling 2017-06-18 15:11:32 +02:00
Alexander Neumann
52752659c1 fuse: Rewrite fuse implementation 2017-06-18 14:59:44 +02:00
Alexander Neumann
f676c0c41b index: Add Each() to MasterIndex 2017-06-18 14:52:14 +02:00
Alexander Neumann
f31e993f09 fuse: Reenable integration tests 2017-06-18 14:23:35 +02:00
Alexander Neumann
56f610e548 fuse: Remove struct SnapshotWithId 2017-06-18 14:11:33 +02:00
Alexander Neumann
052a6a0acc Move snapshot filter function to restic package 2017-06-18 13:18:12 +02:00
Alexander Neumann
77037e33c9 Move snapshot finding functions to new file 2017-06-18 13:06:52 +02:00
Alexander Neumann
5a34799554 Move Snapshots struct and policy to other files 2017-06-18 13:05:47 +02:00
Alexander Neumann
47282abfa4 fuse: Use Mutex instead of RWMutex 2017-06-17 23:00:38 +02:00
Alexander Neumann
c9cc724b31 s3: Split Create() from Open() 2017-06-17 22:15:58 +02:00
Alexander Neumann
0d3674245b Merge pull request #1043 from restic/fix-gcs
s3: Fix GCS
2017-06-17 10:35:10 +02:00
Alexander Neumann
82b21cdf4a Merge pull request #1027 from restic/s3-set-retry
s3: Allow setting the number of retries for minio-go
2017-06-17 10:34:36 +02:00
Alexander Neumann
c4592f577a Merge pull request #1036 from restic/prune-remove-invalid-files
prune: Remove invalid files
2017-06-16 22:52:44 +02:00
Alexander Neumann
3cd851e578 Update github.com/minio/minio-go 2017-06-16 22:29:40 +02:00
Alexander Neumann
e074833a7d Merge pull request #1045 from restic/prune-fix-progress
prune: Fix progress information
2017-06-16 20:21:55 +02:00
Alexander Neumann
c5f1a83cb4 prune: Fix progress information 2017-06-16 19:03:26 +02:00
Alexander Neumann
1baaa778ee Add entry to CHANGELOG 2017-06-16 12:27:44 +02:00
Alexander Neumann
6a948d5afd s3: Fix backend for Google Cloud Storage 2017-06-16 11:25:06 +02:00
Alexander Neumann
ea66ae0811 s3: Fix IsNotExist() 2017-06-16 10:54:46 +02:00
Alexander Neumann
bf8a155fb1 Update github.com/minio/minio-go 2017-06-16 10:53:38 +02:00
Alexander Neumann
4ae59bef96 prune: Remove invalid files
Closes #1029
2017-06-15 20:56:22 +02:00
Alexander Neumann
eadf5dcb2d Merge pull request #1038 from restic/s3-prevent-close
Improve GCS support
2017-06-15 20:54:52 +02:00
Alexander Neumann
91a24e8229 Merge pull request #1035 from restic/fix-1032
prune: Remove files as the last step
2017-06-15 20:22:42 +02:00
Alexander Neumann
e3c979a7a4 Merge pull request #1034 from restic/fix-1030
prune: Fix status string for narrow terminals
2017-06-15 20:22:33 +02:00
Alexander Neumann
05365706c0 backend/tests: Correct error message and delayed remove 2017-06-15 20:05:35 +02:00
Alexander Neumann
bbca31b661 test/s3: Retry connection to Minio server 2017-06-15 19:51:55 +02:00
Alexander Neumann
eb7fc12e01 backend tests: Delay listing for swift backend 2017-06-15 19:41:07 +02:00
Alexander Neumann
98ae7b1210 s3: Save config in backend 2017-06-15 16:41:09 +02:00
Alexander Neumann
51877cecf7 s3: Prevent closing of the reader for GCS 2017-06-15 16:39:42 +02:00
Alexander Neumann
9053b2000b s3: Delete ignores error if the object doesn't exist 2017-06-15 16:27:19 +02:00
Alexander Neumann
dd6ce5f9d8 Remove backend.Closer, use ioutil.NopCloser() instead 2017-06-15 15:58:23 +02:00
Alexander Neumann
9a8301fc74 prune: Fix status string for narrow terminals
Closes #1030
2017-06-15 15:41:07 +02:00
Alexander Neumann
aabe2a0a30 Merge pull request #1002 from restic/test-codecov
Remove codecov config file
2017-06-15 15:09:50 +02:00
Alexander Neumann
c79fb6fcdd prune: Delete repacked files as the very last step 2017-06-15 14:46:50 +02:00
Alexander Neumann
af9ba3be91 backend: Add IsNotExist 2017-06-15 13:40:27 +02:00
Alexander Neumann
6f24d038f8 prune: Only remove data after index has been uploaded
Closes #1032
2017-06-15 13:12:46 +02:00
Alexander Neumann
cf65893c4b s3: Allow setting the number of retries for minio-go
https://github.com/restic/restic/issues/1013#issuecomment-307883970
2017-06-12 21:09:37 +02:00
Alexander Neumann
bd7d5a429f Merge pull request #1025 from restic/fix-1013
s3: Switch back to high-level API for upload
2017-06-12 19:58:12 +02:00
Alexander Neumann
7b54f6e642 Add entry to CHANGELOG 2017-06-12 19:56:50 +02:00
Alexander Neumann
422c0dfb5e s3: Exit test loop for minio server on success 2017-06-11 20:49:56 +02:00
Alexander Neumann
73b296918b s3: Reorder debug messages
This way the semaphore token acquisition can be observed in the debug
log.
2017-06-11 20:49:53 +02:00
Alexander Neumann
907c201693 debug: Add version number to debug log 2017-06-11 20:48:46 +02:00
Alexander Neumann
58de8bf392 swift/rest: Reduce number of connections 2017-06-11 20:48:46 +02:00
Alexander Neumann
a89a7a783a s3: Correct comment on the connections option 2017-06-11 20:48:46 +02:00
Alexander Neumann
c422010597 s3: Fix test 2017-06-11 20:48:46 +02:00
Alexander Neumann
08e1d9ffad s3: Switch back to high-level API, limit connections 2017-06-11 20:48:42 +02:00
Alexander Neumann
a4e8dc3371 s3: Improve error message in debug log 2017-06-11 11:22:25 +02:00
Alexander Neumann
19da56a6ea debug: Add log before panic() 2017-06-11 11:22:25 +02:00
Alexander Neumann
d3c06c39f9 debug: Fix EOF detection in HTTP transport 2017-06-11 11:22:25 +02:00
Alexander Neumann
6301620428 s3: Add more debug logs 2017-06-11 11:22:25 +02:00
Alexander Neumann
a6f157f346 Merge pull request #1024 from restic/remove-unused
Remove unused code/variables
2017-06-11 11:18:02 +02:00
Alexander Neumann
8d4417ec92 Remove unused code/variables 2017-06-11 09:29:53 +02:00
Alexander Neumann
0b55be2581 prune: Fix debug log 2017-06-10 22:16:42 +02:00
Alexander Neumann
88a59fd0ca options: Handle uint 2017-06-10 21:07:10 +02:00
Alexander Neumann
539674614b Merge pull request #1019 from restic/fix-1017
ls: Print names with percent correctly
2017-06-10 12:43:46 +02:00
Alexander Neumann
9d1b9157d4 ls: Print names with percent correctly
Closes #1017
2017-06-10 12:17:21 +02:00
Alexander Neumann
5f449045d2 Merge pull request #1003 from fwilhe/contributing-md-link
Fix relative link to CONTRIBUTING.md
2017-06-09 20:56:21 +02:00
Alexander Neumann
3e4d236751 Merge pull request #1010 from restic/update-minio-go
Update github.com/minio/minio-go
2017-06-09 20:55:49 +02:00
Alexander Neumann
4fe6593fbe Merge pull request #1011 from restic/fix-1009
pack: Handle small files
2017-06-09 20:53:52 +02:00
Florian Wilhelm
635633379a Fix link to CONTRIBUTING.md 2017-06-09 00:36:31 +02:00
Alexander Neumann
48fecd791d pack: Handle more invalid header cases 2017-06-08 21:04:07 +02:00
Alexander Neumann
a325a20fb4 s3: Increase wait time for minio server 2017-06-08 20:50:56 +02:00
Alexander Neumann
1f0916b01b Merge pull request #1004 from restic/add-migrate-s3
Add 'migrate' command, change s3 layout
2017-06-08 20:48:27 +02:00
Alexander Neumann
eb767ab15f pack: Handle small files 2017-06-08 20:40:12 +02:00
Alexander Neumann
92c0aa3854 Merge pull request #998 from restic/fix-820
fuse: Add cache for blob sizes
2017-06-08 20:21:26 +02:00
Alexander Neumann
a61016cb55 Update github.com/minio/minio-go 2017-06-08 19:40:06 +02:00
Alexander Neumann
eb7ddd6e11 Add entry to CHANGELOG 2017-06-08 19:21:52 +02:00
Alexander Neumann
ff3d2e42f4 migrate: Be a bit more verbose 2017-06-08 19:19:45 +02:00
Alexander Neumann
1aab123b6c Merge pull request #1008 from chaquotay/patch-1
Fixing tiny typo
2017-06-08 19:04:14 +02:00
Stephan Müller
d11f8d294f Fixing tiny typo 2017-06-08 13:27:22 +02:00
Alexander Neumann
04ded881f6 s3: Change the default layout to "default" 2017-06-07 23:08:20 +02:00
Alexander Neumann
4f9bf5312b Add migrate
This commits adds a 'migrate' command and a migration to move s3
repositories from the 's3legacy' to the 'default' layout.
2017-06-07 23:08:02 +02:00
Alexander Neumann
7cf8f59987 layout: Add String() and Name() 2017-06-07 21:59:41 +02:00
Alexander Neumann
b8b5c8e8c9 s3: Rename struct to Backend 2017-06-07 21:59:01 +02:00
Alexander Neumann
a46baf7685 s3: Remove cache 2017-06-07 20:51:45 +02:00
Alexander Neumann
f2a51aa37c Add entry to CHANGELOG 2017-06-07 20:51:08 +02:00
Alexander Neumann
233eaf8ee9 fuse: Improve semantics of the blob size cache
Wrap it in a struct and add a Lookup() function to make clear that it
is only queried, not changed, so we don't have any race conditions.
2017-06-07 20:04:58 +02:00
Alexander Neumann
067be2c551 fuse: Add cache for blob sizes
Closes: #820
2017-06-07 20:04:15 +02:00
Alexander Neumann
550e1feaec Merge pull request #999 from restic/backend-use-semaphore
backends: Use new semaphore
2017-06-07 19:48:32 +02:00
Alexander Neumann
f90ce23f30 Merge pull request #994 from restic/add-context
Add context.Context to the backend
2017-06-07 19:11:56 +02:00
Alexander Neumann
29f8f8fe68 Update github.com/kurin/blazer
Reduces cost-intensive list_files API calls.
2017-06-07 19:10:05 +02:00
Alexander Neumann
48c1e7b00d Fix location tests 2017-06-06 21:12:38 +02:00
Alexander Neumann
2175ccedd2 Remove codecov config file 2017-06-06 21:02:19 +02:00
Alexander Neumann
d4e74f20aa Add context to dump command 2017-06-06 00:37:25 +02:00
Alexander Neumann
aa5bc39311 swift: Use semaphore 2017-06-06 00:33:25 +02:00
Alexander Neumann
46049b4236 rest: Use semaphore 2017-06-06 00:26:29 +02:00
Alexander Neumann
683ebef6c6 s3: Use semaphore 2017-06-06 00:17:39 +02:00
Alexander Neumann
5010e95c23 Add error handling to semaphore 2017-06-06 00:17:21 +02:00
Alexander Neumann
46b7a270a6 Add context parameters to tests 2017-06-05 23:56:59 +02:00
Alexander Neumann
cf497c2728 Add context to restic packages 2017-06-04 14:35:14 +02:00
Alexander Neumann
16fcd07110 Add a Context to the backend 2017-06-04 14:02:44 +02:00
Alexander Neumann
a9a2798910 Merge pull request #993 from restic/improve-search-performance
Improve find
2017-06-04 12:44:29 +02:00
Alexander Neumann
9cd664caa3 Add entry to CHANGELOG 2017-06-04 11:50:38 +02:00
Alexander Neumann
a90e0c6595 find: Check trees only once 2017-06-04 11:42:40 +02:00
Alexander Neumann
7b5efaf7b0 find: Move functions to struct 2017-06-04 11:38:46 +02:00
Alexander Neumann
3b7ca4ac35 find: Improve debug log 2017-06-04 11:22:56 +02:00
Alexander Neumann
40a61b82ce Merge pull request #978 from restic/add-backblaze-backend
Add Backblaze B2 backend
2017-06-03 14:54:04 +02:00
Alexander Neumann
028f43299a Merge pull request #975 from restic/add-swift-backend
Add swift backend
2017-06-03 14:52:47 +02:00
Alexander Neumann
3a4727f0f5 Add entry to CHANGELOG.md 2017-06-03 14:28:29 +02:00
Alexander Neumann
fec89f95fb Improve swift backend 2017-06-03 14:28:18 +02:00
Bartłomiej Święcki
5681d41f76 Implement OpenStack swift backend
This commit implements support for OpenStack swift
storage server, tested on OVH public cloud storage.

Special thanks to jayme-github <tuxnet@gmail.com>
who helped with the implementation.
2017-06-03 14:26:29 +02:00
Alexander Neumann
efd61d97ef Vendor github.com/ncw/swift 2017-06-03 14:25:57 +02:00
Alexander Neumann
3ed56f2192 Add entry to CHANGELOG.md 2017-06-03 14:24:59 +02:00
Alexander Neumann
122462b9b1 Add Backblaze B2 backend
This is based on prior work by Joe Turgeon <arithmetric@gmail.com>
@arithmetric.
2017-06-03 14:24:59 +02:00
Alexander Neumann
2217b9277e Vendor github.com/kurin/blazer 2017-06-03 14:24:59 +02:00
Alexander Neumann
b5e0e3631b Addd nev version section 2017-06-03 14:10:28 +02:00
Alexander Neumann
be68e43871 Fix link 2017-06-02 22:08:04 +02:00
Alexander Neumann
f6034c0882 Merge pull request #990 from tmcarr/fix_readme_links
Fix the links in the readme to render in RST
2017-06-02 21:59:41 +02:00
Travis Carr
f693781bf0 Fix the links in the readme to render right. 2017-06-02 12:32:13 -07:00
Alexander Neumann
3ae9be987f Add VERSION file for 0.6.1 2017-05-31 23:52:13 +02:00
Alexander Neumann
ec0975c388 Add VERSION file for 2017-05-31 23:51:02 +02:00
Alexander Neumann
c2ce484e93 Add version to CHANGELOG 2017-05-31 23:50:54 +02:00
Alexander Neumann
e5c7c314a7 Add section about reproducible build to README
In addition, the build script isn't needed any more.
2017-05-31 23:48:56 +02:00
Alexander Neumann
6d36dcd46e Merge pull request #987 from Thor77/minor-doc-fix
[docs] Fix paragraph not indented correctly in #Autocomplete
2017-05-31 23:23:27 +02:00
Thor77
96c9ecd20e Fix paragraph not indented correctly 2017-05-31 21:40:47 +02:00
Alexander Neumann
997be9a036 Remove PR 2017-05-31 21:34:18 +02:00
Alexander Neumann
31fd8e98b9 Add Entry to CHANGELOG 2017-05-31 21:33:45 +02:00
Alexander Neumann
aa0f874c8d s3: Simplify IsNotExist() 2017-05-31 21:23:01 +02:00
Alexander Neumann
5c59484d2b s3: Return only basename in List() 2017-05-31 21:22:55 +02:00
Alexander Neumann
fba6211c99 Merge pull request #986 from restic/fix-regression-985
Allow many idle connections per host
2017-05-31 20:49:50 +02:00
Alexander Neumann
a8386e7d71 Add entry to CHANGELOG 2017-05-31 19:53:54 +02:00
Alexander Neumann
04b262d8f1 Allow many idle connections per host
Closes #985
2017-05-31 19:39:19 +02:00
Alexander Neumann
4dbbc24a44 Update Go version 2017-05-30 23:05:13 +02:00
Alexander Neumann
725d50554a Merge pull request #981 from restic/reproducible-builds
build.go: Strip temporary path, allow reproducible builds
2017-05-29 23:49:49 +02:00
Alexander Neumann
ed91cafce2 Add entry to CHANGELOG 2017-05-29 23:46:48 +02:00
Alexander Neumann
de48a5ac9c build.go: Strip temporary path, allow reproducible builds 2017-05-29 23:27:25 +02:00
Alexander Neumann
1d167f4680 Merge tag 'v0.6.0'
v0.6.0
2017-05-29 21:35:27 +02:00
Alexander Neumann
efad7ee197 Add VERSION file for 0.6.0 2017-05-29 21:31:41 +02:00
Alexander Neumann
820c88ea73 Add VERSION file for 0.6.0 2017-05-29 21:12:52 +02:00
Alexander Neumann
e7f031c9b3 Merge pull request #976 from restic/backend-fixes
Misc fixes for the backend/test code
2017-05-28 13:30:56 +02:00
Alexander Neumann
f3f6924b61 backend/test: Loose requirement about early error 2017-05-28 13:06:27 +02:00
Alexander Neumann
c5244abad9 rest: Improve error messages 2017-05-28 12:33:47 +02:00
Alexander Neumann
1f5954e2c1 layout: Test DefaultLayout for empty path prefix 2017-05-28 12:33:47 +02:00
Alexander Neumann
e046a2a6da sftp: Use path instead of filepath 2017-05-28 12:33:47 +02:00
Alexander Neumann
8395b53400 backend/test: Reduce verbosity in logs 2017-05-28 12:33:47 +02:00
Alexander Neumann
24ec14738d backend/test: Skip offset == length test 2017-05-28 12:33:47 +02:00
Alexander Neumann
79477fdfe4 backend/test: Randomize test suite 2017-05-28 12:33:47 +02:00
Alexander Neumann
7ec0543af3 testing: Add id to error message in panic 2017-05-28 10:17:04 +02:00
Alexander Neumann
e73e3cb4ba Merge pull request #974 from restic/remove-noninteractive-progress
Remove regular status printing for non terminals
2017-05-25 18:56:55 +02:00
Alexander Neumann
317d9c4559 Add entry to the changelog 2017-05-25 17:06:06 +02:00
Alexander Neumann
5247de552a Remove regular status printing for non terminals 2017-05-25 17:03:48 +02:00
Alexander Neumann
37b107b90b build script: Check for dirty work directory 2017-05-25 15:50:37 +02:00
Alexander Neumann
17ff41af47 Update version number in CHANGELOG.md 2017-05-25 15:42:44 +02:00
Alexander Neumann
772f647a73 Add VERSION file for 0.6.0-rc.1 2017-05-25 15:27:45 +02:00
Alexander Neumann
5c4facd9dc Update CHANGELOG.md 2017-05-25 15:26:22 +02:00
Alexander Neumann
40ad1df6ee Merge pull request #971 from restic/add-switch-enable-cgo
build.go: Add --enable-cgo
2017-05-25 11:28:40 +02:00
Alexander Neumann
c49da4c6f7 build.go: Add --enable-cgo 2017-05-25 11:03:29 +02:00
Alexander Neumann
d1cc87ba28 Merge pull request #966 from restic/unify-repo-layout
WIP: Unify repository layout
2017-05-22 21:15:38 +02:00
Alexander Neumann
89be1d496a Improve issue template wording 2017-05-21 10:09:49 +02:00
Alexander Neumann
69e114c5c4 Merge pull request #968 from mcastilho/patch-2
Fixing CPUProfile
2017-05-17 21:13:42 +02:00
Alexander Neumann
5d9a4ce90c Merge pull request #967 from middelink/gofmt
Fix a number of warnings from go report
2017-05-17 21:12:59 +02:00
Marcio Castilho
84f1037b8f Update global_debug.go
The variable was wrong and the CPU profile was never being created.
2017-05-17 11:07:33 -04:00
Pauline Middelink
120af801cf Fix golint warnings
(except the exported fields/functions without comments)
2017-05-17 01:39:39 +02:00
Pauline Middelink
f3d09ce7c8 Fix vet warnings 2017-05-17 01:34:33 +02:00
Pauline Middelink
2b9323529f Fix gofmt -s warnings 2017-05-17 01:28:39 +02:00
Pauline Middelink
971ecee171 Fix ineffassign mistakes 2017-05-17 01:25:52 +02:00
Alexander Neumann
4640d1a28e Add codecov.yml 2017-05-16 20:52:42 +02:00
Alexander Neumann
8b461a7456 Make TestFlags less verbose 2017-05-16 20:49:18 +02:00
Alexander Neumann
2fa1238b8a Fix filenames for layout tests 2017-05-16 20:45:17 +02:00
Alexander Neumann
55ae5dab2b design: Add s3.layout option 2017-05-15 23:37:16 +02:00
Alexander Neumann
fa41183a53 s3: Add s3.layout option and layout auto detection 2017-05-15 23:37:02 +02:00
Alexander Neumann
959aa0f595 Simplify layout detection 2017-05-15 23:36:23 +02:00
Alexander Neumann
069752cb42 Make layout default Dirname() consistent
Always return a trailing slash now.
2017-05-15 23:35:52 +02:00
Alexander Neumann
0c537837d9 Rename testdata files 2017-05-15 22:39:50 +02:00
Alexander Neumann
f19852a738 Rename CloudLayout -> RESTLayout
The REST backend uses a special layout without subdirs below data/. This
layout is just used there and nowhere else, and our REST server
implementation uses the default layout for the on disk storage. So we
remove the REST layout from the auto detection code.
2017-05-15 22:05:43 +02:00
Alexander Neumann
61cade6222 Rename layout s3 -> s3legacy 2017-05-15 21:59:28 +02:00
Alexander Neumann
a4e3a0dd97 design: Remove cloud layout 2017-05-15 21:51:19 +02:00
Alexander Neumann
22a6cd3a26 Merge pull request #964 from restic/update-minio-go
Update github.com/minio/minio-go
2017-05-15 21:15:28 +02:00
Alexander Neumann
7f10828dfa Update github.com/minio/minio-go 2017-05-15 20:47:50 +02:00
Alexander Neumann
04264c6338 Remove doc/PKGBUILD 2017-05-15 19:48:57 +02:00
Alexander Neumann
13393c76dc Merge pull request #957 from middelink/fix-953
Change backup policy to be inclusive
2017-05-15 17:19:04 +02:00
Alexander Neumann
13c12ca83d Fix PKGBUILD
Closes #963
2017-05-15 17:17:23 +02:00
Pauline Middelink
7fffd408af Do not use singleton counter.
Revert change for running the numbered tests as subtests.
2017-05-15 08:53:59 +02:00
Alexander Neumann
089b04c8a9 Merge pull request #962 from restic/improve-s3-partial-download
Improve s3 backend
2017-05-14 22:00:16 +02:00
Alexander Neumann
b192dacc13 Add text to the issue template 2017-05-14 21:37:45 +02:00
Alexander Neumann
250a45ab15 s3: Move interfaces to function 2017-05-14 20:36:26 +02:00
Alexander Neumann
3b44b87137 s3: Remove file after usage in test 2017-05-14 20:34:22 +02:00
Alexander Neumann
26c16b9fd3 s3: Correctly return token in Load 2017-05-14 20:19:12 +02:00
Alexander Neumann
0bd40bae6e s3: Prevent net/http client from closing the reader 2017-05-14 20:19:12 +02:00
Alexander Neumann
246ccf09b9 s3: add getRemainingSize 2017-05-14 20:19:12 +02:00
Alexander Neumann
9452f416bf s3: Use low level API for saving files
benchmark                        old ns/op       new ns/op       delta
    BenchmarkBackendMinio/Save-4     184482294       40663344        -77.96%
    BenchmarkBackendS3/Save-4        35030825568     54475455819     +55.51%

    benchmark                        old MB/s     new MB/s     speedup
    BenchmarkBackendMinio/Save-4     90.95        412.64       4.54x
    BenchmarkBackendS3/Save-4        0.48         0.31         0.65x

    benchmark                        old allocs     new allocs     delta
    BenchmarkBackendMinio/Save-4     631            560            -11.25%
    BenchmarkBackendS3/Save-4        646            584            -9.60%

    benchmark                        old bytes     new bytes     delta
    BenchmarkBackendMinio/Save-4     66818060      50735         -99.92%
    BenchmarkBackendS3/Save-4        66834000      73024         -99.89%
2017-05-14 20:19:12 +02:00
Alexander Neumann
1e0e6ee573 s3: Use low-level API with a Range header for Load
benchmark                                         old ns/op      new ns/op      delta
    BenchmarkBackendMinio/LoadFile-4                  9213315        11001787       +19.41%
    BenchmarkBackendMinio/LoadPartialFile-4           4176619        3479707        -16.69%
    BenchmarkBackendMinio/LoadPartialFileOffset-4     4391521        3139214        -28.52%
    BenchmarkBackendS3/LoadFile-4                     2886070905     2505907501     -13.17%
    BenchmarkBackendS3/LoadPartialFile-4              762702722      735694398      -3.54%
    BenchmarkBackendS3/LoadPartialFileOffset-4        789724328      1108989142     +40.43%

    benchmark                                         old MB/s     new MB/s     speedup
    BenchmarkBackendMinio/LoadFile-4                  1821.21      1525.15      0.84x
    BenchmarkBackendMinio/LoadPartialFile-4           1004.49      1205.67      1.20x
    BenchmarkBackendMinio/LoadPartialFileOffset-4     955.34       1336.45      1.40x
    BenchmarkBackendS3/LoadFile-4                     5.81         6.70         1.15x
    BenchmarkBackendS3/LoadPartialFile-4              5.50         5.70         1.04x
    BenchmarkBackendS3/LoadPartialFileOffset-4        5.31         3.78         0.71x

    benchmark                                         old allocs     new allocs     delta
    BenchmarkBackendMinio/LoadFile-4                  406            204            -49.75%
    BenchmarkBackendMinio/LoadPartialFile-4           225            206            -8.44%
    BenchmarkBackendMinio/LoadPartialFileOffset-4     227            207            -8.81%
    BenchmarkBackendS3/LoadFile-4                     600            388            -35.33%
    BenchmarkBackendS3/LoadPartialFile-4              416            302            -27.40%
    BenchmarkBackendS3/LoadPartialFileOffset-4        417            303            -27.34%

    benchmark                                         old bytes     new bytes     delta
    BenchmarkBackendMinio/LoadFile-4                  29475         13904         -52.83%
    BenchmarkBackendMinio/LoadPartialFile-4           4218838       13958         -99.67%
    BenchmarkBackendMinio/LoadPartialFileOffset-4     4219175       14332         -99.66%
    BenchmarkBackendS3/LoadFile-4                     114152        97424         -14.65%
    BenchmarkBackendS3/LoadPartialFile-4              4265416       56212         -98.68%
    BenchmarkBackendS3/LoadPartialFileOffset-4        4266520       56308         -98.68%
2017-05-14 20:19:12 +02:00
Alexander Neumann
be0e53c07b tests: Add test for backend Save() from file 2017-05-14 19:55:32 +02:00
Alexander Neumann
9dd58196e3 Merge pull request #961 from restic/add-backend-benchmarks
Add backend benchmarks
2017-05-14 19:49:33 +02:00
Alexander Neumann
66b4999765 tests: Remove code generation, use reflection
This simplifies the code.
2017-05-14 12:59:07 +02:00
Alexander Neumann
403e201e1a tests: Improve robustness of config tests 2017-05-14 12:50:20 +02:00
Alexander Neumann
77a55fbe5c tests: Add documentation 2017-05-14 11:48:30 +02:00
Alexander Neumann
90c1608d88 sftp: Add Benchmarks 2017-05-14 11:48:30 +02:00
Alexander Neumann
7a51640262 rest: Add benchmarks 2017-05-14 11:48:30 +02:00
Alexander Neumann
13946e7db7 mem: Add benchmarks 2017-05-14 11:48:30 +02:00
Alexander Neumann
e009c002ba local: Add benchmark 2017-05-14 11:48:30 +02:00
Alexander Neumann
d24e0cc6cc s3: Add benchmarks 2017-05-14 11:48:30 +02:00
Alexander Neumann
5b8131e2d3 tests: Add benchmark for Save 2017-05-14 11:48:30 +02:00
Alexander Neumann
8fc25cc567 tests: Add benchmarks for partial file (+offset) 2017-05-14 11:48:30 +02:00
Alexander Neumann
77ebb95d3d tests: Add BenchmarkLoadFile 2017-05-14 11:48:30 +02:00
Alexander Neumann
f142b1c22f tests: Add benchmarks to test suite 2017-05-14 11:48:30 +02:00
Alexander Neumann
1c9159d6a0 Merge pull request #960 from restic/travis-run-tests-against-s3
Enable CI tests against AWS S3
2017-05-14 10:26:16 +02:00
Alexander Neumann
82e15dc6dc Allow skipping the test against s3
If RESTIC_TEST_S3_REPOSITORY is not available it means we're probably
running on Travis for a third-party PR, so the access credentials for S3
are not exposed.
2017-05-14 00:33:19 +02:00
Alexander Neumann
ee68f9298b tests: Ensure that backend tests cannot be skipped on Travis 2017-05-14 00:33:15 +02:00
Alexander Neumann
5c6ec78789 s3: Remove log line 2017-05-13 19:58:25 +02:00
Alexander Neumann
4ac0d3ad40 backend tests load: Use reader with Size() method 2017-05-13 19:56:11 +02:00
Harshavardhana
c7209ef231 api: Failed call retry with region only when http.StatusBadRequest.
Fixes https://github.com/minio/minio-go/issues/677
2017-05-13 19:53:24 +02:00
Alexander Neumann
0249c16b04 Revert "minio: Apply fix"
This reverts commit f43d34899d.
2017-05-13 19:41:55 +02:00
Alexander Neumann
f43d34899d minio: Apply fix
https://github.com/minio/minio-go/issues/677
2017-05-13 13:18:42 +02:00
Alexander Neumann
19daefd04e Merge pull request #938 from restic/rework-backend-tests
WIP: rework backend integration tests
2017-05-12 22:09:57 +02:00
Alexander Neumann
ec5e984ed9 Increase waiting time for network services 2017-05-12 21:45:18 +02:00
Alexander Neumann
88de3cfecc rest: Wait until REST server is started 2017-05-12 21:33:34 +02:00
Alexander Neumann
a654f41ddb Add docstring for package test 2017-05-12 21:19:32 +02:00
Alexander Neumann
fbf2462325 Rename backend test functions 2017-05-12 21:06:49 +02:00
Alexander Neumann
85e4831198 Remove stuff from run_integration_tests.go 2017-05-12 21:04:04 +02:00
Alexander Neumann
091dc29f72 doc: Fix s3 tutorial 2017-05-12 21:04:04 +02:00
Alexander Neumann
59d1986660 s3: Use random prefix for tests 2017-05-12 21:04:04 +02:00
Alexander Neumann
db4fa48f66 rest: Add context 2017-05-12 21:04:04 +02:00
Alexander Neumann
f2b83ece10 s3: Ignore errors for terminated minio server 2017-05-12 21:04:04 +02:00
Alexander Neumann
860b52273e s3: Add tests for AWS 2017-05-12 21:04:04 +02:00
Alexander Neumann
bafceb56fb s3: Convert to test suite, run Minio automatically 2017-05-12 21:04:04 +02:00
Alexander Neumann
9271b3662a sftp: ignore error 2017-05-12 21:04:04 +02:00
Alexander Neumann
554013ca9f rest: Convert to test suite 2017-05-12 21:04:04 +02:00
Alexander Neumann
3250fdc2ca mem: Convert to test suite 2017-05-12 21:04:04 +02:00
Alexander Neumann
3678d34b16 sftp: Convert to test suite 2017-05-12 21:04:04 +02:00
Alexander Neumann
16276853a1 local: Convert to test suite 2017-05-12 21:04:04 +02:00
Alexander Neumann
f10c24e404 Improve test helpers 2017-05-12 21:04:04 +02:00
Alexander Neumann
6f5fd72738 Implement test suite for backend tests 2017-05-12 21:03:55 +02:00
Pauline Middelink
103a491ac0 Make houndci-bot happy. 2017-05-11 22:38:12 +02:00
Pauline Middelink
929f90344e Change backup policy to be inclusive, meaning all given policies
are evaluated for each snapshot, thereby making sure that each
keep-* is able to retain its most recent snapshot. Thereby insuring
that weeklies keep Sundays around and monthlies keep the last day of
the month around.

Added testcase to make sure when multiple --keep-tags are given,
ALL of them need to match.
2017-05-11 22:34:50 +02:00
Alexander Neumann
a963052d64 Add custom HTTP transport 2017-05-11 21:51:30 +02:00
Alexander Neumann
898613e14f debug: Add RoundTripper 2017-05-11 21:51:30 +02:00
Alexander Neumann
b84e63d503 backend.LoadAll: Check errors from io.Copy() 2017-05-11 21:51:30 +02:00
Alexander Neumann
63870d2830 Check error for Close() 2017-05-11 21:51:30 +02:00
Alexander Neumann
54e46f5984 Add MinimalData option for backend tests 2017-05-11 21:51:30 +02:00
Alexander Neumann
0096eca7fe Install log buffer for global logs
This hides logged message from the net/http library and only shows them
in case an error occurs.
2017-05-11 21:51:30 +02:00
Pauline Middelink
ce3acbd30a Remove unused SnapshotFilter and FilterSnapshots(). 2017-05-11 21:07:58 +02:00
Alexander Neumann
c5ae5524ff Remove codecov.yml 2017-05-10 20:52:06 +02:00
Alexander Neumann
fbf5a8123b Merge pull request #952 from restic/remove-temp-files
Unlink tempfiles before using them
2017-05-10 20:25:07 +02:00
Alexander Neumann
edbd6ad584 Add fs.TempFile and fs.RemoveIfExists 2017-05-10 19:48:22 +02:00
Alexander Neumann
faf11c4a46 Fix link in README.rst
closes #950
2017-05-08 21:31:11 +02:00
Alexander Neumann
68a9e5f963 Merge pull request #947 from restic/correct-codecov
codecov.io: Add path fixes
2017-05-07 10:59:20 +02:00
Alexander Neumann
fa7f0ef44f codecov.io: Add path fixes 2017-05-07 10:41:32 +02:00
Alexander Neumann
bcfe7afbfd Merge pull request #945 from middelink/fix-942
Add bash autocompletion generation
2017-05-07 10:30:10 +02:00
Alexander Neumann
46e6e8984d Merge pull request #941 from restic/improve-mkdir
local/sftp: Auto create directories if needed
2017-05-07 10:26:39 +02:00
Alexander Neumann
0f057bd440 Merge pull request #943 from middelink/fix-names
Small changes to cmd_forget and snapshot_filter
2017-05-07 10:15:21 +02:00
Alexander Neumann
ba91a76f5f Merge pull request #946 from restic/update-minio-go
Update github.com/minio/minio-go
2017-05-07 09:54:55 +02:00
Alexander Neumann
c09e005cc8 Update github.com/minio/minio-go
This corrects a severe performance regression when the current minio
server is used as a backend, see
https://github.com/minio/minio/issues/4275
2017-05-07 09:36:49 +02:00
Pauline Middelink
2c42629c51 Add bash autocompletion generation
Fix #942
2017-05-06 12:39:23 +02:00
Pauline Middelink
f93e5a39e5 Small textual changes to apply, to reflect actual meaning.
Added tests for the ExpirePolicy operations.
2017-05-04 16:39:41 +02:00
Pauline Middelink
02eafe94b3 Use global context in cmd_forget, like all other cmds 2017-05-04 16:35:35 +02:00
Alexander Neumann
7d914768bb local/sftp: Auto create directories if needed
As reported in #940
2017-05-03 21:19:22 +02:00
Alexander Neumann
db6541b24c Merge pull request #936 from Thor77/manual-fix-rst-links
Fix links broken during conversion to rst
2017-04-30 19:08:20 +02:00
Thor77
228566c0a1 Fix links broken during conversion to rst 2017-04-30 18:25:45 +02:00
Alexander Neumann
1e4bcf3367 design.rst: Fix tree IDs in example
Closes #913
2017-04-29 20:15:30 +02:00
Alexander Neumann
51e7d26053 Merge pull request #935 from restic/add-glyphcheck
CI: Add glyphcheck, rework REST server install
2017-04-27 16:16:07 +02:00
Alexander Neumann
e42627d2cb CI: Add glyphcheck, rework REST server install 2017-04-27 16:01:09 +02:00
Alexander Neumann
548d4eed95 Correct backend test for len = 0 2017-04-26 20:47:15 +02:00
Alexander Neumann
8a5034d289 Merge pull request #930 from restic/update-cobra
forget: don't sort flags
2017-04-21 20:01:47 +02:00
Alexander Neumann
5a68641f27 Add saythanks.io 2017-04-21 19:32:37 +02:00
Alexander Neumann
ce28584dda Don't sort flags for forget command 2017-04-21 19:25:21 +02:00
Alexander Neumann
ab870dd691 Update cobra und spf13 2017-04-21 19:19:14 +02:00
Alexander Neumann
aed0126102 Merge pull request #927 from restic/open-check-config
Test if a config file on open
2017-04-19 19:29:27 +02:00
Alexander Neumann
d0db1bf9b3 Test if a config file on open 2017-04-19 18:56:01 +02:00
Alexander Neumann
90ba606633 Add console hilighting to README.rst 2017-04-19 18:32:32 +02:00
Alexander Neumann
5b80cb8b6b Merge pull request #922 from restic/update-deps
Update vendored dependencies
2017-04-19 18:30:41 +02:00
Alexander Neumann
e6ca604d24 build.go: Also copy header files 2017-04-19 17:31:07 +02:00
Alexander Neumann
9940b7f853 Update golang.org/x/* 2017-04-18 21:40:17 +02:00
Alexander Neumann
faca9276e9 Update spf13/pflag 2017-04-18 21:40:17 +02:00
Alexander Neumann
5a7e463ef6 Update spf13/cobra 2017-04-18 21:40:17 +02:00
Alexander Neumann
27c5a2825a Update restic/chunker to v0.1.0 2017-04-18 21:40:17 +02:00
Alexander Neumann
e9d939ff2d Update code for pkg/xattr v0.2.0 2017-04-18 21:39:55 +02:00
Alexander Neumann
88e8eaa575 Update pkg/xattr to v0.2.0 2017-04-18 21:32:55 +02:00
Alexander Neumann
3bbcf89105 Update pkg/errors to v0.8.0 2017-04-18 21:30:02 +02:00
Alexander Neumann
1794bdc663 Update minio/minio-go to v2.0.4 2017-04-18 21:29:56 +02:00
Alexander Neumann
bcf97ebf1f Merge pull request #920 from jpmens/patch-2
Small typo in "Frankfurt"
2017-04-18 11:55:04 +02:00
JP Mens
a1496547fd Small typo in Frankfurt 2017-04-18 11:42:40 +02:00
Alexander Neumann
0befa06cd0 Merge pull request #919 from Phaeilo/s3-tutorial
Added AWS S3 tutorial.
2017-04-17 22:44:12 +02:00
Alexander Neumann
407b8c179b doc: Use "code-block console" for session blocks 2017-04-17 22:43:04 +02:00
Alexander Neumann
aafee25d58 Convert AWS S3 tutorial to rst 2017-04-17 22:41:23 +02:00
Philip Huppert
adb602dbfe Added AWS S3 tutorial. 2017-04-17 22:14:39 +02:00
Alexander Neumann
085b6bede8 doc: Add hline 2017-04-17 22:13:43 +02:00
Alexander Neumann
096aab592b doc: Move doc/references/* to doc/ 2017-04-17 22:07:06 +02:00
Alexander Neumann
2f117982e9 Add symlink to make old link to Design.md work 2017-04-17 21:17:15 +02:00
Alexander Neumann
b05603eb6c Merge pull request #913 from Thor77/switch-to-sphinx
Switch documentation to Sphinx
2017-04-17 21:00:22 +02:00
Thor77
9344b0b3a8 Documentation: Refactor and switch to Sphinx
* convert documentation to reStructuredText
* switch from mkdocs to sphinx
* rewrite README.md (now .rst) to include less information and be more
attractive
2017-04-17 20:53:38 +02:00
Alexander Neumann
89ea1171db Merge pull request #918 from restic/fix-917
s3: Correct layout prefix, reduce connections
2017-04-17 20:22:39 +02:00
Alexander Neumann
28968caf33 s3: Correct prefix for layout 2017-04-17 20:06:55 +02:00
Alexander Neumann
16fd1c2352 s3: Correct layout handle url/path 2017-04-17 20:06:55 +02:00
Alexander Neumann
b942f61272 s3: Reduce connection limit, prevents timeouts 2017-04-17 20:06:55 +02:00
Alexander Neumann
e3d1badfcf Add more questions to github issue template 2017-04-17 18:30:20 +02:00
Alexander Neumann
76f6a9e597 Merge pull request #916 from restic/fix-915
Ignore empty lines in excludes file, allow multiple files
2017-04-17 10:20:12 +02:00
Alexander Neumann
69dd1d2544 backup: Allow specifying multiple exclude files 2017-04-16 20:52:41 +02:00
Alexander Neumann
dd65ac56ef filter: Ignore empty patterns 2017-04-16 20:49:20 +02:00
Alexander Neumann
c796d84fca Ignore empty lines in excludes file
Closes #915
2017-04-16 20:46:52 +02:00
Alexander Neumann
525db875b0 Merge pull request #898 from restic/prepare-cloud-backends
Prepare more cloud backends, add backend layouts
2017-04-15 11:30:25 +02:00
Alexander Neumann
e73038cbcb Merge pull request #912 from restic/update-go
Update Go
2017-04-15 11:19:20 +02:00
Alexander Neumann
00a8edb4a0 Merge pull request #911 from restic/fix-735
Implement MkdirAll() for Windows
2017-04-15 11:19:17 +02:00
Alexander Neumann
ee7e0d05f5 Remove old Go versions from run_integration_tests.go 2017-04-15 11:06:50 +02:00
Alexander Neumann
6357dc8a91 Update Go versions 2017-04-15 10:58:43 +02:00
Alexander Neumann
5bd95b3ce1 Implement MkdirAll() for Windows
Closes #735
2017-04-15 10:56:17 +02:00
Alexander Neumann
be06983c80 options: Fix sorting (and test) 2017-04-14 00:45:54 +02:00
Alexander Neumann
55bdc1fa16 Add documentation for new options 2017-04-14 00:14:11 +02:00
Alexander Neumann
a634c22ae0 Add hidden 'options' command to list all opts 2017-04-14 00:00:03 +02:00
Alexander Neumann
859ee23d2e options: Register local and sftp backends 2017-04-14 00:00:03 +02:00
Alexander Neumann
b7671dafc8 options: Allow registering 2017-04-14 00:00:02 +02:00
Alexander Neumann
541484d142 s3: Use Layout 2017-04-14 00:00:02 +02:00
Alexander Neumann
f531ca3b48 layout: Fix corner cases 2017-04-14 00:00:02 +02:00
Alexander Neumann
0da7264e75 rest: Convert to Layout 2017-04-14 00:00:02 +02:00
Alexander Neumann
7f3bcdb4cc layout: prepare use for REST backend 2017-04-14 00:00:02 +02:00
Alexander Neumann
5eaa51eeff Remove unused assignments 2017-04-14 00:00:02 +02:00
Alexander Neumann
7b64b890d7 Simplify code 2017-04-14 00:00:02 +02:00
Alexander Neumann
ccc201ea5f remove unused code 2017-04-14 00:00:02 +02:00
Alexander Neumann
e6578857cf sftp/local: Fix listing files 2017-04-14 00:00:02 +02:00
Alexander Neumann
320c22f1f5 backend/layout: Add Basedir() 2017-04-14 00:00:02 +02:00
Alexander Neumann
e2af5890f3 backend: Add test for listing files with layouts 2017-04-14 00:00:02 +02:00
Alexander Neumann
36b1c0898c sftp: Add OS X sftp-server path 2017-04-13 23:59:59 +02:00
Alexander Neumann
94441dcbee Merge pull request #908 from restic/handle-nested-mountpoints
Check allowed devices per path
2017-04-13 22:00:10 +02:00
Alexander Neumann
c723cdf808 Check allowed devices per path
Closes #645
Closes #702
2017-04-13 21:03:05 +02:00
Alexander Neumann
c2ee0d9c84 sftp: Skip tests if server binary is not available 2017-04-10 22:51:00 +02:00
Alexander Neumann
e8780f1ec6 sftp: Add layout tests 2017-04-10 22:42:33 +02:00
Alexander Neumann
783fd73ea1 local: Rename local_layout_test 2017-04-10 22:42:33 +02:00
Alexander Neumann
74eb293733 sftp: Remove legacy filename/dirname methods 2017-04-10 22:42:33 +02:00
Alexander Neumann
42ea4d257b sftp: first step of conversion to Layout 2017-04-10 22:42:33 +02:00
Alexander Neumann
a849edf19a local: remove double Close() 2017-04-10 22:42:33 +02:00
Alexander Neumann
0cbd59856c layout: Add IsNotExist 2017-04-10 22:42:33 +02:00
Alexander Neumann
698ba57597 backend/tests: Print error stacktrace if available 2017-04-10 22:42:33 +02:00
Alexander Neumann
27ce6a85e9 sftp: Rework Open/Create 2017-04-10 22:42:33 +02:00
Alexander Neumann
ae290ab374 sftp: Rename Dir -> Path 2017-04-10 22:42:33 +02:00
Alexander Neumann
ab602c9d14 sftp: Add Layout 2017-04-10 22:40:24 +02:00
Alexander Neumann
2e53af1b75 sftp: Rename Open/Create 2017-04-10 22:39:13 +02:00
Alexander Neumann
a725e065d9 Merge pull request #903 from restic/fix-hardlinks
Add new field DeviceID and tests
2017-04-07 21:05:20 +02:00
Alexander Neumann
db7e23b423 Skip /dev/null on darwin 2017-04-07 20:37:20 +02:00
Alexander Neumann
dc8493535d Merge pull request #905 from restic/fix-json-output
Remove short option conflict
2017-04-07 20:30:18 +02:00
Alexander Neumann
10a395ca33 Make tests runnable on os x/darwin 2017-04-06 20:36:09 +02:00
Alexander Neumann
522c7ade91 Add test for double defined flags 2017-04-06 19:44:53 +02:00
Alexander Neumann
a3d6099892 Remove short option conflict 2017-04-06 19:14:38 +02:00
Alexander Neumann
280028290e Disable tests on darwin 2017-04-05 21:35:29 +02:00
Alexander Neumann
c195139d31 Only run tests on unix 2017-04-05 20:51:26 +02:00
Alexander Neumann
6f1b03415c Fix hardlinks 2017-04-05 20:45:24 +02:00
Alexander Neumann
ffcb015581 Add new field DeviceID and tests 2017-04-05 20:42:15 +02:00
Alexander Neumann
67a2ac292b Merge pull request #902 from jgfrm/issue893
storage of device id of files
2017-04-05 17:44:03 +02:00
jgfrm
6674b2a70c Moved assignment of device id outside the switch 2017-04-05 11:33:08 +02:00
jgfrm
c855d6bb9a storage of device id of files 2017-04-04 21:28:25 +02:00
Alexander Neumann
1086528ab7 sftp: Fix errors import 2017-04-03 21:42:41 +02:00
Alexander Neumann
c26dd6b76f sftp: Integrate command 2017-04-03 21:05:42 +02:00
Alexander Neumann
d3b6f75848 sftp: Add SplitShellArgs 2017-04-03 08:57:33 +02:00
Alexander Neumann
d1efdcd78e Add integration test for layouts 2017-04-02 20:35:17 +02:00
Alexander Neumann
95ab5adda1 local: Expose layout as extended option 2017-04-02 20:29:00 +02:00
Alexander Neumann
c5eb36fe9d layout: improve error message for ParseLayout 2017-04-02 20:28:42 +02:00
Alexander Neumann
e3e3a8a695 local: Add layout tests 2017-04-02 20:01:40 +02:00
Alexander Neumann
24ebf95f33 local: Automatically detect layout 2017-04-02 20:01:40 +02:00
Alexander Neumann
54465c92cc layout: Allow passing in a default layout 2017-04-02 20:01:36 +02:00
Alexander Neumann
f7c4b3a922 Fix layout detection 2017-04-02 19:18:03 +02:00
Alexander Neumann
50dfa64a54 Add layout name parser 2017-04-02 17:57:28 +02:00
Alexander Neumann
c6b8ffbb61 Add layout auto detection 2017-04-02 17:25:22 +02:00
Alexander Neumann
3e81dcdfc2 Add cloud and s3 layout 2017-03-26 22:20:10 +02:00
Alexander Neumann
782b740c95 local: Remove unused code 2017-03-26 22:14:37 +02:00
Alexander Neumann
3fd6fa6f86 local: Use Layout for filename generation 2017-03-26 21:53:26 +02:00
Alexander Neumann
6a201f7962 backend: Add Layout 2017-03-26 21:52:49 +02:00
Alexander Neumann
80a864c52c test: Add TempDir() helper 2017-03-26 20:40:45 +02:00
Alexander Neumann
c8eea49909 debug: Allow creating insecure repositories
Uses low-security KDF parameters for scrypt(). Do not use in production!
2017-03-26 10:55:08 +02:00
Alexander Neumann
95c354fe81 doc: s3 backend deviations, cloud repo layout 2017-03-26 10:55:08 +02:00
Alexander Neumann
6935f82389 Merge pull request #897 from restic/add-extended-options
Add extended options
2017-03-26 10:53:36 +02:00
Alexander Neumann
719bb18316 Parse extended options 2017-03-25 17:52:16 +01:00
Alexander Neumann
a8a7701f60 options: Add namespace to Apply() 2017-03-25 17:52:16 +01:00
Alexander Neumann
2924ebc124 options: Add Apply() 2017-03-25 17:52:16 +01:00
Alexander Neumann
f587a5f4f0 options: fail when key is configured twice 2017-03-25 17:52:16 +01:00
Alexander Neumann
946b4f4b86 Add extended options via -o/--option 2017-03-25 17:52:12 +01:00
Alexander Neumann
d0a5e86da1 Add free-form options parsing 2017-03-25 17:45:11 +01:00
Alexander Neumann
2e0b19f63f location: Move to backend/location 2017-03-25 17:45:11 +01:00
Alexander Neumann
9861f3d435 location: Reformat tests, use sub tests 2017-03-25 17:45:10 +01:00
Alexander Neumann
482a6e9840 local: Add Config struct 2017-03-25 17:45:10 +01:00
Alexander Neumann
6a8700d86c Merge pull request #896 from restic/fix-895
doc: Add section about how to change the sftp port
2017-03-25 09:31:29 +01:00
Alexander Neumann
1733b24f9d doc: Add section about how to change the sftp port
Closes #895
Related to #785
2017-03-25 09:07:50 +01:00
Alexander Neumann
c85055bb03 Merge pull request #887 from restic/rest-integration-tests
WIP: rest: Make backend honor the REST protocol
2017-03-18 15:21:05 +01:00
Alexander Neumann
280f05b174 Run rest-server 2017-03-18 11:11:31 +01:00
Alexander Neumann
990b0f1c15 Travis: Run rest-server for integration tests 2017-03-16 21:59:21 +01:00
Alexander Neumann
6f76a6db66 rest: Make backend honor the REST protocol 2017-03-16 21:50:26 +01:00
Alexander Neumann
0c2834edb7 Merge pull request #883 from trbs/s3_perf_cache_stat
Cache size of last ReadAt on S3 for performance
2017-03-15 20:59:29 +01:00
trbs
f1ba45723c introduce RWLock for caching obj.Stat() 2017-03-14 23:05:51 +01:00
trbs
b523eef294 Cache size of last ReadAt on S3 for performance
Each obj.Stat() call adds another request to the S3 endpoint
for some commands a lot of ReadAt calls are made for the same
object in S3. This patch essentially cuts the number of calls
to S3 in this case in half. Speeding up the progress and lowering
costs to S3.
2017-03-14 14:02:50 +01:00
Alexander Neumann
bf30b2831b Merge pull request #878 from middelink/fix-825
Fix restoring files ownership as non-root user
2017-03-12 18:20:28 +01:00
Pauline Middelink
642cd3bebf Fix restoring files as non-root user
As we cannot reliably detect in advance if we can set ownership, permissions,
timestamps or ext attributes, execute ALL the requested changes before
returning the first error we found.

Report total errors at end of restore and stop printing entire stacktraces
where just the error message is sufficient.

Fixes #655
2017-03-12 16:51:58 +01:00
Alexander Neumann
057dd6c590 Merge tag 'v0.5.0'
v0.5.0
2017-03-11 21:14:39 +01:00
Alexander Neumann
0bd7db9efe Add VERSION file for 0.5.0 2017-03-11 12:36:35 +01:00
Alexander Neumann
f678c97346 Add VERSION file for 0.5.0 2017-03-11 12:34:27 +01:00
Alexander Neumann
887e81188f Merge pull request #877 from middelink/fix-875
Fix crash at restoring pre-existing hardlinked files
2017-03-11 10:13:43 +01:00
Alexander Neumann
8d0140aabe Merge pull request #876 from middelink/fix-836
Restore does not truncate files it restores
2017-03-11 10:12:53 +01:00
Alexander Neumann
2a8f275a28 Manual: Document TMPDIR env variable
Closes #873
2017-03-11 10:02:33 +01:00
Alexander Neumann
55c3150946 Manual: Fix section about JSON 2017-03-11 09:59:36 +01:00
Pauline Middelink
b56e16acd0 Fix crash at restoring pre-existing hardlinked files
Remove target file, ignore non existing file errors.

Small memory saving: Only keep inodes around for files with a link count > 1.
(We will/can never be asked to restore a hardlinked file with one of the
files having a link count of 1.)

Closes #836
2017-03-11 00:06:58 +01:00
Pauline Middelink
b75186533d Restore does not truncate files it restores
Missing O_TRUNC when doing O_CREAT.

Closes #875
2017-03-10 22:46:13 +01:00
Alexander Neumann
8958efba60 Merge pull request #871 from middelink/fix-869
Refactor output of `find` to allow for json and normal output.
2017-03-10 20:03:26 +01:00
Pauline Middelink
a9707a5728 Refactor output of find to allow for json and normal output.
Rather complicated solution becaused I wanted to retain the streaming
character of the output, which means for json I have to manually add
headers and footers per snapshot scanned + a list around the whole
set.

As the json ouput is now partly handcrafted, add proper testing to catch
unintentional changes to the output, making it non-json compliant.

Closes #869
2017-03-10 16:23:58 +01:00
Alexander Neumann
7635feb591 Add cross-compilation to README and Manual 2017-03-09 21:24:38 +01:00
Alexander Neumann
087c2917aa Check version in build.go
Print a sensible error for Go < 1.7 (the "context" package is missing
there)
2017-03-09 21:24:30 +01:00
Alexander Neumann
deddedd88f Merge pull request #872 from middelink/fix-787
Suppress (non error) output of `forget` during -q
2017-03-09 21:23:36 +01:00
Pauline Middelink
b38294f236 Suppress (non error) output of forget during -q
Closes #787
2017-03-09 20:20:17 +01:00
Alexander Neumann
fdb9872cef Add cross-compilation to README and Manual 2017-03-09 11:37:44 +01:00
Alexander Neumann
f8dd5d5088 Check version in build.go
Print a sensible error for Go < 1.7 (the "context" package is missing
there)
2017-03-09 11:29:24 +01:00
Alexander Neumann
9e0207e534 Add CHANGELOG.md and instructions for contributors 2017-03-09 10:40:24 +01:00
Alexander Neumann
9eae789cd2 Add VERSION file for 0.5.0-rc.1 2017-03-09 10:28:33 +01:00
Alexander Neumann
d6104935d7 Add VERSION file for 0.5.0rc1 2017-03-09 10:20:11 +01:00
Alexander Neumann
406af5916c Also include the source in SHA256SUMS 2017-03-09 10:19:45 +01:00
Alexander Neumann
93e4e4f4fb Merge pull request #866 from middelink/widespread-tags
Add --tag filtering to every command, where applicable
2017-03-09 09:55:26 +01:00
Pauline Middelink
8a05de537f Refactor prune and rebuild_index
Factor out and reuse `rebuildIndex()` in cmd_rebuild_index and cmd_prune.

Use contexts.
2017-03-08 20:30:52 +01:00
Pauline Middelink
8a92687d9a Refactor find and ls commands
Implement filtering by using `FindFilteredSnapshots()` to iterate over the snapshots

Refactor cmd_ls' `PrintNode()` into format.go, reuse its pretty printing in both `find`
and `ls` commands.

Use contexts.
2017-03-08 20:29:31 +01:00
Pauline Middelink
3432e7edcd Refactor tag to use FindFilteredSnapshots() 2017-03-08 20:28:44 +01:00
Pauline Middelink
3c6c17abcd Refactor forget and snapshots command
Implement filtering by using `FindFilteredSnapshots()` to iterate over the snapshots

Refactor cmd_snapshots' `PrintSnapshots()` so its pretty printing can be used from
both `forget` and `snapshots`.

Use contexts.
2017-03-08 20:24:58 +01:00
Pauline Middelink
11d237c252 New helper function FindFilteredSnapshots to iterate over snapshots
This helper function takes a set of filters and/or a list of snapshots
from the commandline. It returns a channel of *Snapshot.

When snapshot ids are given, they are checked for validity and their
corresponding Snapshots returned. The snapshot id "latest" is handled
special to return either the last snapshot (no filters) or the last
snapshot matching the filters.

When no arguments are given, the filters are applied over all available
snapshots and these are returned.
2017-03-08 20:19:12 +01:00
Pauline Middelink
0f7b6ec5ac Adapt key command to context world. 2017-03-08 20:17:30 +01:00
Pauline Middelink
b4526c4e6e Enable the use of context in restic
Set up a cancelble context in global options, hook it into the ctrl-C handler
for proper cancel propegation.

Bump up minimal requirement for Go to version 1.7 in documentation
and test-build files.
2017-03-08 20:12:16 +01:00
Pauline Middelink
3eaaa0f286 Correct some typo's in comments. 2017-03-08 20:09:24 +01:00
Pauline Middelink
b1c8071163 Add filtering to mount command 2017-03-08 19:59:19 +01:00
Pauline Middelink
3468108d4c Implement --tag processing to backup and restore command
Add `tags` argument to `FindLatestSnapshot()`
2017-03-08 19:55:58 +01:00
Alexander Neumann
33c8dd4ee5 Merge pull request #867 from middelink/optimize
Optimize List() pipeline
2017-03-08 11:50:47 +01:00
Alexander Neumann
5e2c4caa32 Merge pull request #861 from middelink/fix-841
Refactor cmd_forget
2017-03-08 09:56:01 +01:00
Pauline Middelink
7c989ca487 Optimize List() pipeline 2017-03-07 20:58:36 +01:00
Pauline Middelink
e9a2982ecd Add --path for snapshot filtering by path.
Add `--group-by-tags` for grouping on host,tags,dirs instead of host,dirs.

Borrow the snapshot printing from cmd_snapshot.

Closes #841
2017-03-07 19:28:42 +01:00
Alexander Neumann
f5a55a81f7 Merge pull request #865 from restic/handle-empty-snapshots
Refuse to create empty snapshots
2017-03-07 11:21:48 +01:00
Alexander Neumann
340f2c80a0 Merge pull request #864 from restic/find-case-insensitive
find: Add option to ignore case
2017-03-07 11:20:33 +01:00
Alexander Neumann
bb144436c7 Add test for empty snapshot 2017-03-07 11:17:15 +01:00
Alexander Neumann
b50d3ba805 Refuse to create empty snapshots
Closes #862
2017-03-07 11:12:34 +01:00
Alexander Neumann
00e7158381 Merge pull request #860 from middelink/factor-out
Create a helper function to get the terminal width
2017-03-07 10:59:39 +01:00
Alexander Neumann
221bef48c0 find: Add option to ignore case
Closes #859
2017-03-07 10:58:09 +01:00
Pauline Middelink
afcc1ba706 Create a helper function to get the terminal width
Rationale: contain terminal access to one file.
2017-03-06 11:23:00 +01:00
Alexander Neumann
bf88a62a16 Merge pull request #858 from middelink/snapshot-layout
Fix layout issue in cmd_snapshot "ascii art"
2017-03-06 09:10:35 +01:00
Alexander Neumann
cc140744d6 Merge pull request #857 from middelink/fix-856
Fix SamePaths() and make it into a receiver function
2017-03-06 09:10:33 +01:00
Pauline Middelink
354e8ffb82 Fix layout issue in cmd_snapshot "ascii art"
The layouter does not account for multi tags when determining the
need for ascii art.

36fd8178  2017-03-03 21:35:04  abuseio.polyware.nl    NL          /
                                                      A       └──

vs

36fd8178  2017-03-03 21:35:04  abuseio.polyware.nl    NL      ┌── /
                                                      A       └──
2017-03-06 02:49:15 +01:00
Pauline Middelink
e1c828be3e Fix SamePaths() and make it into a receiver function
Add `HasPath(paths []string) bool` to Snapshot for testing if the
snapshot has at least the paths given to the function.

Reimplemented SamePaths(paths []string) so it does what the name implies,
compare if all given paths are in the snapshot.
2017-03-06 02:21:58 +01:00
Alexander Neumann
d50dc9f649 Merge pull request #855 from middelink/fix-851
Add `tag` command to restic cli to manipulate tags on existing snapshots.
2017-03-05 20:20:20 +01:00
Alexander Neumann
07695b3622 Documentation fixes 2017-03-05 20:12:25 +01:00
Pauline Middelink
be15a9261a Add design and user documentation for the restic tag command 2017-03-05 19:55:23 +01:00
Pauline Middelink
1fa2313aef Snapshot: Add Original ID
The Original ID is used when the snapshot is modified (e.g. by `tag`
command). Adjust integration testing to assert correctness.
2017-03-05 19:55:23 +01:00
Pauline Middelink
26e266a951 Fix type of ID field in cmd_snapshots type Snapshot 2017-03-05 19:55:22 +01:00
Pauline Middelink
208edaa3d1 Snapshot: Add AddTags() and RemoveTags()
Both prevent duplicate tags.
2017-03-05 19:55:22 +01:00
Pauline Middelink
f6a258b4a8 Add tag: Manipulate tags on existing snapshots
Add integration testing.
2017-03-05 19:55:22 +01:00
Alexander Neumann
db08581352 Add hint for other backend URI formats 2017-03-05 16:58:24 +01:00
Alexander Neumann
0afeb68e6c Merge pull request #854 from middelink/dynamic-host,tag-columns-in-cmd_snapshots
Dynamic host,tag columns widths in snapshots command
2017-03-05 16:11:38 +01:00
Alexander Neumann
a809c9ac5f Merge pull request #853 from middelink/cmd_backup-tag-integration-test
restic backup --tag integration test
2017-03-05 16:08:02 +01:00
Pauline Middelink
45e9f35654 Make columns for host and tags size width dynamicly on their content. 2017-03-05 05:32:01 +01:00
Pauline Middelink
edd5c8b44d Add integration test to make sure cmd_backup adds tags when required. 2017-03-05 05:24:40 +01:00
Pauline Middelink
7238a3ee89 Changed cmd_snapshots to be testable (no more using os.Stdout) 2017-03-05 05:24:11 +01:00
Alexander Neumann
bbcab800c9 Merge pull request #850 from middelink/fix-848
Add progressbar to repack and blob remove phases of prune cmd.
2017-03-04 18:47:03 +01:00
Pauline Middelink
5564c78e53 English typo: rewriten > rewritten. 2017-03-04 17:43:58 +01:00
Pauline Middelink
792b81725e Add progressbar to repack and blob remove phases of prune cmd. 2017-03-04 17:38:34 +01:00
Alexander Neumann
b101efe26e Merge pull request #845 from restic/update-pkg-xattr
Update github.com/pkg/xattr
2017-03-04 16:49:07 +01:00
Alexander Neumann
becc34a159 Merge pull request #847 from middelink/fix-846
Display the proper amount of bytes we will be pruning from the repo.
2017-03-04 16:47:20 +01:00
Pauline Middelink
1273c6f3d4 Display the proper amount of bytes we will be pruning from the repo. 2017-03-04 15:17:44 +01:00
Alexander Neumann
82458d4de0 Update github.com/pkg/xattr
Closes #843
2017-03-04 14:23:15 +01:00
Alexander Neumann
7066cc17bb Merge pull request #844 from welpo/patch-1
Fix Minio Server URL
2017-03-04 13:51:05 +01:00
welpo
12ed2f65e3 Fix Minio Server URL
The previous link gave a 404
2017-03-04 13:19:50 +01:00
Alexander Neumann
90bc187355 Merge pull request #840 from middelink/master
Display absolute paths when displaying the output of ls and find.
2017-03-03 12:09:20 +01:00
Pauline Middelink
5ecaaea90b Really use absolute pathnames, not all systems use /. 2017-03-03 11:14:39 +01:00
Alexander Neumann
f9fc8674eb Merge pull request #837 from restic/index
Misc improvements
2017-03-03 09:44:06 +01:00
Pauline Middelink
039e81b04b Fix unit test, we need to check for absolute paths now. 2017-03-02 23:30:56 +01:00
Pauline Middelink
efb4315a1e Display absolute paths when displaying the output of ls and find. 2017-03-02 22:41:11 +01:00
Alexander Neumann
f53d33ba34 Make ArchiveReader a struct 2017-03-02 15:45:35 +01:00
Alexander Neumann
9b776dc7ab Use new Index implementation for rebuild-index 2017-03-02 15:23:59 +01:00
Alexander Neumann
1d64a1dcbb Merge pull request #835 from restic/fix-834
Allow filtering absolute paths
2017-03-02 15:22:17 +01:00
Alexander Neumann
7c92994f10 Clarify variable name 2017-03-02 14:52:18 +01:00
Alexander Neumann
bf97cc7efa Allow filtering absolute paths
Before, the restorer called the filter function with a relative path,
this prevented anchoring absolute patterns (which just never matched).
Now call the restore function with an absolute virtual path, starting at
the filepath separator.

Closes #834
2017-03-02 14:50:54 +01:00
Alexander Neumann
4f5e9e939b Merge pull request #829 from restic/fix-822
Ignore empty lines in --files-from
2017-02-27 21:15:35 +01:00
Alexander Neumann
92ad35848a Fix Travis Go versions 2017-02-27 20:40:10 +01:00
Alexander Neumann
bb69b20aff Travis: Configure Go versions 2017-02-27 20:39:00 +01:00
Alexander Neumann
80e93621e1 Travis: Configure Go versions 2017-02-27 20:37:47 +01:00
Alexander Neumann
31ff506309 Ignore empty lines in --files-from
Closes #822
2017-02-27 19:42:00 +01:00
Alexander Neumann
4a51ddf741 Add section to the manual about same directory names
Related to #652
2017-02-21 20:46:26 +01:00
Alexander Neumann
3d1dc636d0 Merge pull request #817 from restic/add-forget-prune
Add `--prune` switch to `forget`
2017-02-21 11:40:30 +01:00
Alexander Neumann
8609ba28d0 Add News section 2017-02-21 11:12:07 +01:00
Alexander Neumann
685f5ebbd1 Add --prune switch to forget 2017-02-21 10:58:30 +01:00
Alexander Neumann
4e2f8145f5 Update Appveyor 2017-02-18 19:09:21 +01:00
Alexander Neumann
11b63d3417 Update Travis 2017-02-18 19:08:11 +01:00
Alexander Neumann
2c81bc35dc Merge pull request #814 from restic/fix-archiver
Correct archiver behavior in case of errors
2017-02-18 19:00:33 +01:00
Alexander Neumann
b8ce1b4e69 Correct archiver behavior in case of errors 2017-02-18 17:46:06 +01:00
Alexander Neumann
23c2717ab2 Merge pull request #809 from restic/fix-xattr
WIP: fix panic
2017-02-18 15:00:49 +01:00
Alexander Neumann
132afbe83b Correct error check for ENOTSUP, add errors.Wrap() 2017-02-18 14:36:37 +01:00
Alexander Neumann
ef52d15edd Continue if extended attribute cannot be read 2017-02-18 14:35:11 +01:00
Alexander Neumann
6df2f9e5ba Add support for extended attributes on FreeBSD 2017-02-18 14:35:11 +01:00
Alexander Neumann
eb9be4e884 Use github.com/pkg/xattr for extended attributes 2017-02-18 14:35:06 +01:00
Alexander Neumann
0674f32d79 Merge pull request #766 from jgfrm/nissue25
Add support for extended attributes (e.g. ACL)

Closes #766
2017-02-16 11:45:24 +01:00
Jaap Gordijn
49cae0904f Add support for extended attributes (e.g. ACL) 2017-02-16 11:44:51 +01:00
Alexander Neumann
40685a0e61 Merge pull request #803 from ibib/adds-json-support
Adds JSON support for the snapshots command
2017-02-16 11:26:14 +01:00
Alexander Neumann
4772a4986b Merge pull request #808 from restic/fix-807
restore: Make sure buffer is large enough
2017-02-15 19:58:03 +01:00
Alexander Neumann
c973a1f875 Merge pull request #806 from oysols/consistent-cli
Fix inconsistencies in CLI
2017-02-15 15:26:27 +01:00
Alexander Neumann
50d066befb restore: Make sure buffer is large enough 2017-02-15 15:19:28 +01:00
Øystein Olsen
c35e48291d Fix correct number of arguments for key command 2017-02-13 16:16:34 +01:00
Øystein Olsen
511278b66a Fix wrong description of rebuild-index command in help text 2017-02-13 16:07:29 +01:00
Øystein Olsen
514a11346d Add long description of list command in help text 2017-02-13 16:06:27 +01:00
Øystein Olsen
2eb75bb941 Consistently refer to 'the' instead of 'a' repository in help text 2017-02-13 16:05:25 +01:00
Øystein Olsen
9922ce97bf Use lowercase consistently in help text 2017-02-13 16:02:47 +01:00
ibib
3ed4127297 Adds JSON support for the snapshots command 2017-02-12 21:43:39 +01:00
Alexander Neumann
c83e608cce Merge pull request #800 from restic/jgfrm-hardlinks-fuse
fuse: correct nlink count for directories
2017-02-11 22:09:33 +01:00
Jaap Gordijn
3e2ae15882 Manual: Add section about hard links for fuse 2017-02-11 21:54:54 +01:00
Jaap Gordijn
3047702ded Correct hardlinks for fuse directories 2017-02-11 21:54:23 +01:00
Alexander Neumann
05cae4911d Merge pull request #798 from restic/fix-797
Checker: Propagate errors properly
2017-02-11 14:55:50 +01:00
Alexander Neumann
8c34eaad15 Improve error message 2017-02-11 14:28:15 +01:00
Alexander Neumann
0492eabff1 Improve error messages 2017-02-11 14:24:11 +01:00
Alexander Neumann
7797e084f9 checker: Pass on error loading an index 2017-02-11 14:22:14 +01:00
Alexander Neumann
b40aa66985 errors: Add method Wrapf 2017-02-11 14:22:04 +01:00
Alexander Neumann
76c06c5f2a Add check for modified index 2017-02-11 14:13:58 +01:00
Alexander Neumann
83538c745a Merge pull request #792 from restic/fix-791
s3: Increase MaxIdleConnsPerHost
2017-02-11 11:20:20 +01:00
Alexander Neumann
f266741f40 s3/rest: raise connection limit to 40 2017-02-11 10:40:51 +01:00
Alexander Neumann
4795a5c5d1 Merge pull request #795 from restic/fix-767
fuse: Add options --allow-other and --allow-root
2017-02-11 10:38:33 +01:00
Alexander Neumann
c14cb62cd7 fuse: Add options --allow-other and --allow-root
Closes #767
2017-02-10 21:58:10 +01:00
Alexander Neumann
b3ec01521e Merge pull request #794 from restic/fix-789
Use non-formatting functions of errors for strings
2017-02-10 21:42:59 +01:00
Alexander Neumann
6483df5ee4 Merge pull request #793 from restic/add-host-parameter
backup: Add `--hostname` parameter
2017-02-10 21:42:56 +01:00
Alexander Neumann
6275d69a36 fuse: Show link count 2017-02-10 21:16:48 +01:00
Alexander Neumann
6300c8df56 Merge pull request #763 from jgfrm/issue25
Support hard links
2017-02-10 20:58:39 +01:00
Jaap Gordijn
366bf4eb0c Support hard links
Closes #152
2017-02-10 20:58:19 +01:00
Alexander Neumann
21b358c742 backend tests: Always close reader 2017-02-10 20:49:46 +01:00
Alexander Neumann
7a0303f7ae s3: Make sure to return connection token 2017-02-10 20:49:37 +01:00
Alexander Neumann
64165ea4c8 s3: Hold connection semaphore until Close() 2017-02-10 20:17:52 +01:00
Alexander Neumann
c8fc789393 Use non-formatting functions of errors for strings
Commands used:

    $ gofmt -w -r 'errors.Fatalf(x) -> errors.Fatal(x)' src
    $ gofmt -w -r 'errors.Errorf(x) -> errors.New(x)' src

Closes #789
2017-02-10 19:39:49 +01:00
Alexander Neumann
f145e1de0f backup: Add --hostname parameter 2017-02-10 19:37:33 +01:00
Alexander Neumann
36dee7d892 s3: Increase MaxIdleConnsPerHost 2017-02-10 19:25:42 +01:00
Alexander Neumann
98ae30b513 Update Dockerfile 2017-02-10 17:25:48 +01:00
Alexander Neumann
073edd914d Merge pull request #783 from opennota/master
Fix some typos
2017-02-09 10:43:27 +01:00
Alexander Neumann
316b520ffb Merge pull request #782 from restic/fix-backup-stdin
Make sure backups read from stdin have a file name
2017-02-09 10:42:49 +01:00
opennota
25e459659a Fix some typos 2017-02-09 06:43:10 +07:00
Alexander Neumann
fdebb022e4 Make sure backups read from stdin have a file name 2017-02-08 22:37:02 +01:00
Alexander Neumann
ed1739acbd Merge pull request #779 from restic/benchmark-checker
checker: Reduce memory usage
2017-02-07 11:03:49 +01:00
Alexander Neumann
1f81919d4a checker: Reduce memory usage
benchmark              old bytes     new bytes     delta
    BenchmarkChecker-4     25551348      4288037       -83.22%
2017-02-06 21:19:27 +01:00
Alexander Neumann
436b5dc20c Add Blob.String() 2017-02-06 19:50:27 +01:00
Alexander Neumann
0c867b21ff Add benchmark for checker 2017-02-06 19:34:40 +01:00
Alexander Neumann
4cacb622eb Merge pull request #773 from opennota/master
Allow --files-from to take a dash for stdin (fixes #769)
2017-02-06 14:36:52 +01:00
opennota
24acb09a2a Allow --files-from to take a dash for stdin (fixes #769) 2017-02-06 18:43:44 +07:00
Alexander Neumann
ec45cdba84 Merge pull request #778 from restic/fix-777
prune: Close backend reader after download
2017-02-05 20:01:44 +01:00
Alexander Neumann
4ca134a41c prune: Close backend reader after download
Closes #777
2017-02-05 15:40:30 +01:00
Alexander Neumann
b85eae2aea Merge pull request #775 from restic/improve-backup-stdin
Add more tests for reading backups from stdin
2017-02-05 15:29:29 +01:00
Alexander Neumann
aee58a8c17 Add more tests for reading backups from stdin 2017-02-04 16:38:33 +01:00
Alexander Neumann
22f3e21266 Start error message with a lower case character 2017-02-03 17:06:06 +01:00
Alexander Neumann
d0de1ed2e4 Merge pull request #771 from restic/fix-770
Return an error if password is not set for stdin
2017-02-03 16:47:00 +01:00
Alexander Neumann
807fcf07d9 Return an error if password is not set for stdin
Closes #770
2017-02-03 15:53:07 +01:00
Alexander Neumann
f5faff9020 Merge pull request #768 from brikou/download_link
Add note about pre-compiled binaries
2017-02-03 11:11:05 +01:00
Brikou Carré
769a52df16 Bring back "Build restic" paragraph 2017-02-03 10:41:54 +01:00
Brikou Carré
dfb94290ae Add link to release page in doc 2017-02-03 09:09:49 +01:00
Alexander Neumann
f366a636e6 Fix build_release_binaries.sh 2017-02-02 10:23:18 +01:00
Alexander Neumann
a7c2f28f06 Add VERSION file for 0.4.0 2017-02-02 10:21:29 +01:00
Alexander Neumann
79e198451c Don't set GOMAXPROCS
This was a temporary fix for Go earlier than 1.5 to run code on all
avaialble cores. We don't need that any more since we require at least
Go 1.6.
2017-02-02 10:18:07 +01:00
Alexander Neumann
844c959912 Merge pull request #750 from restic/document-rest-create-repo
Document creating a new repo via REST
2017-02-02 10:03:07 +01:00
Alexander Neumann
19ec8f4a77 Add paragraph about code contributions 2017-02-01 12:00:38 +01:00
Alexander Neumann
f5aeda359c Merge pull request #764 from olgeni/typos
Fix typo.
2017-02-01 10:53:46 +01:00
olgeni
5a5687a506 Fix typo. 2017-01-31 10:09:59 +01:00
Alexander Neumann
7e9bfa51e9 REST: Make create repository idempotent 2017-01-30 19:33:17 +01:00
Alexander Neumann
c7d60279f7 Merge pull request #762 from restic/fix-759
Add 'index' to list of options for 'cat' command
2017-01-30 18:26:32 +01:00
Alexander Neumann
7d49c65dd0 Merge pull request #761 from restic/fix-758
Remove inconsistencies regarding the `cat` command
2017-01-30 18:25:49 +01:00
Alexander Neumann
32dfbad414 Merge pull request #760 from restic/fix-756
Allow listing blobs
2017-01-30 18:25:47 +01:00
Alexander Neumann
abd44ca7c5 Add 'index' to list of options for 'cat' command 2017-01-30 10:53:17 +01:00
Alexander Neumann
44f4ff9d37 Remove 'tree' from help text for 'cat' command 2017-01-30 10:50:52 +01:00
Alexander Neumann
658bee17e9 Clean up documentation, remove inconsistencies 2017-01-30 10:48:59 +01:00
Alexander Neumann
524ce01423 Remove 'cat tree' command 2017-01-30 10:48:50 +01:00
Alexander Neumann
41f59ffc78 Fix 'cat' command for tree blobs 2017-01-30 10:48:21 +01:00
Alexander Neumann
d7f52fd7e5 Allow listing blobs
Closes #756
2017-01-30 10:28:17 +01:00
Alexander Neumann
1f9db97d33 Merge pull request #728 from cit/improve-snapshot-output
Add box-drawing character to the snapshots command
2017-01-29 11:00:13 +01:00
Alexander Neumann
f806d8818d Add DELETE 2017-01-29 10:55:47 +01:00
Alexander Neumann
1a538509d0 Document creating a new repo via REST
Closes #736
2017-01-28 10:31:44 +01:00
Alexander Neumann
b2d00b2a86 Merge pull request #749 from restic/normalise-backend-api
Normalise the backend API
2017-01-27 13:30:30 +01:00
Alexander Neumann
31f6093513 Fix tests 2017-01-27 12:47:34 +01:00
Alexander Neumann
b9bddeff39 Normalise the backend API
This makes the following changes, before:

    type backend interface {
        // Test a boolean value whether a File with the name and type exists.
        Test(t FileType, name string) (bool, error)

        // Remove removes a File with type t and name.
        Remove(t FileType, name string) error
    }

After:

    type backend interface {
        // Test a boolean value whether a File with the name and type exists.
        Test(h Handle) (bool, error)

        // Remove removes a File with type t and name.
        Remove(h Handle) error
    }
2017-01-26 22:02:22 +01:00
Florian Adamsky
54c2f622a4 Add box-drawing character to the snapshots command
Remove underscore of variable name

Format code with gofmt

Change snapshot output according to the discussion
2017-01-26 21:54:27 +01:00
Alexander Neumann
d55b56edd3 Merge pull request #748 from restic/fix-747
rest backend: Do not close the reader
2017-01-26 20:20:55 +01:00
Alexander Neumann
8d6fdb7a3e Fix packer manager test on Windows 2017-01-25 17:51:34 +01:00
Alexander Neumann
dceaae33ed PackerManager: Remove unused Finalize() method 2017-01-25 17:12:06 +01:00
Alexander Neumann
e8995b85b8 rest backend: Do not close the reader
Closes #747
2017-01-25 17:12:06 +01:00
Alexander Neumann
925a3cfad7 backend: Check that backends do not close the reader 2017-01-25 17:12:03 +01:00
Alexander Neumann
c99a44b122 Fix Random() function 2017-01-25 13:26:16 +01:00
Alexander Neumann
439d3107f9 Merge pull request #746 from restic/improve-fuse-memory
fuse: Improve memory usage
2017-01-24 14:13:01 +01:00
Alexander Neumann
afc593676a fuse: Improve memory usage
Discard blobs that aren't in use any more. This greatly reduces memory
usage and will probably only trigger on sequential read (e.g. for
restore via fuse).

Closes #480
2017-01-24 12:38:44 +01:00
Alexander Neumann
17d7af6ccc Merge pull request #745 from restic/fix-742
Fix restore/fuse with larger files
2017-01-24 12:12:35 +01:00
Alexander Neumann
0b982d3316 Add restore test with larger files 2017-01-24 11:51:21 +01:00
Alexander Neumann
26da14f315 fuse: improve tests 2017-01-24 11:42:50 +01:00
Alexander Neumann
2de7e03698 repository.LoadBlob: Read correct number of bytes 2017-01-24 11:42:50 +01:00
Alexander Neumann
3d2fbed55f Add test for LoadBlob 2017-01-24 11:42:50 +01:00
Alexander Neumann
0e445ec0f5 checker: Use TestRepository 2017-01-24 11:42:50 +01:00
Alexander Neumann
31055d88a5 Add debug messages 2017-01-24 11:42:42 +01:00
Alexander Neumann
0d125725bc Merge pull request #741 from restic/rework-backend-api
Rework backend API
2017-01-23 20:18:31 +01:00
Alexander Neumann
8b09b5b3cd Merge pull request #740 from restic/add-debug-profiles
Add debug memory/cpu profile options
2017-01-23 20:18:28 +01:00
Alexander Neumann
2d8a699515 Merge pull request #739 from restic/fix-archiver-test
Fix Archiver test: Clean up temp files
2017-01-23 20:18:25 +01:00
Alexander Neumann
0d95507909 Fix test for PackerManager 2017-01-23 19:00:15 +01:00
Alexander Neumann
8e722d8fee Fix saving pack: close temp file before removing 2017-01-23 18:45:15 +01:00
Alexander Neumann
03292d10cc backend: Rename Get() -> Load() 2017-01-23 18:11:10 +01:00
Alexander Neumann
cfc9e8b2fa backends: Remove Load() 2017-01-23 17:54:12 +01:00
Alexander Neumann
f382696ccf repository: Use ReadAt() instead of Load() 2017-01-23 17:54:12 +01:00
Alexander Neumann
e8fcc7e74c repack: Use Get() instead of Load()
In addition, use a tempfile instead of a buffer.
2017-01-23 17:54:12 +01:00
Alexander Neumann
2bd9c9247c checker: Remove Load() from test error backend 2017-01-23 17:54:12 +01:00
Alexander Neumann
fc235317fe backend: Use Get instead of Load for ReaderAt 2017-01-23 17:54:12 +01:00
Alexander Neumann
82d9163955 backend: Ensure Reader is closed on error 2017-01-23 17:54:12 +01:00
Alexander Neumann
4a354befe5 Fix checker test 2017-01-23 17:54:12 +01:00
Alexander Neumann
212936eb52 Make backend.LoadAll() similar to ioutil.ReadAll() 2017-01-23 17:54:12 +01:00
Alexander Neumann
05afedd950 Add backend.Get() 2017-01-23 17:54:11 +01:00
Alexander Neumann
a36c01372d Use streaming functions for saving data in repo 2017-01-23 17:54:11 +01:00
Alexander Neumann
9b48da5b4e Change backend Save() function signature 2017-01-23 17:54:11 +01:00
Alexander Neumann
c93f79f0f3 Add hashing package 2017-01-23 17:54:11 +01:00
Alexander Neumann
89a5152f7d Fix Archiver test: Clean up temp files 2017-01-23 17:53:59 +01:00
Alexander Neumann
47bd9cdf2f Add options for creating a memory or CPU profile 2017-01-23 17:52:26 +01:00
Alexander Neumann
84255f4f4f Vendor github.com/pkg/profile 2017-01-23 17:41:07 +01:00
Alexander Neumann
668a36a652 Add option for debug pprof service 2017-01-23 17:27:42 +01:00
Alexander Neumann
30ff7413be Merge pull request #737 from restic/fix-734
Index: Store pack ID
2017-01-22 22:41:16 +01:00
Alexander Neumann
af1cc0717b Add integration test for forget and prune 2017-01-22 22:23:30 +01:00
Alexander Neumann
5e3365d233 Index: Store pack ID 2017-01-22 22:10:36 +01:00
Alexander Neumann
4f780a01f9 Index: Test pack ID 2017-01-22 22:09:56 +01:00
Alexander Neumann
dc6a832cc3 Correct BenchmarkIndexSave 2017-01-22 09:59:19 +01:00
Alexander Neumann
164ba823e5 Merge pull request #731 from restic/improve-memory-usage
Improve memory usage
2017-01-20 15:56:31 +01:00
Alexander Neumann
8dd7fe82ff Add TestIndexSave 2017-01-20 14:46:14 +01:00
Alexander Neumann
6c1032548b Correct FAQ 2017-01-19 11:47:33 +01:00
Alexander Neumann
f0eeb16b33 Merge pull request #729 from restic/add-faq
Add FAQ document
2017-01-19 11:42:38 +01:00
Alexander Neumann
c88b0d20e8 Correct typo 2017-01-18 21:57:59 +01:00
Alexander Neumann
fbecae7362 Add FAQ document 2017-01-18 21:48:39 +01:00
Alexander Neumann
38ba5fbceb Merge pull request #727 from MirkoDziadzka/fix-typo
fix typo: outputcomprehensive
2017-01-18 11:46:56 +01:00
Mirko Dziadzka
d47758a540 fix typo 2017-01-18 10:46:04 +01:00
Alexander Neumann
dac18e3bf8 Improve BenchmarkIndexSave 2017-01-17 13:00:59 +01:00
Alexander Neumann
c4f44c7bcb Reduce memory consuption of TestCreateSnapshot 2017-01-17 12:56:20 +01:00
Alexander Neumann
73ad3d418d Index: Remove unneeded allocation 2017-01-17 12:46:41 +01:00
Alexander Neumann
36276c41b2 Add Benchmark for IndexSave 2017-01-17 10:40:58 +01:00
Alexander Neumann
d40f566e41 Index: Use slices instead of maps, reduce data 2017-01-17 10:40:58 +01:00
Alexander Neumann
cd9b526203 Preallocate pack entries list 2017-01-17 10:40:58 +01:00
Alexander Neumann
caabc4ec44 Reduce memory usage while decoding index 2017-01-17 10:40:58 +01:00
Alexander Neumann
73e7a2bea8 Add BenchmarkLoadIndex 2017-01-17 10:40:58 +01:00
Alexander Neumann
e463587bad Add BenchmarkDecodeIndex 2017-01-17 10:40:57 +01:00
Alexander Neumann
e571b6a656 Use the same buffer for decryption 2017-01-17 10:40:57 +01:00
Alexander Neumann
710499cf46 Add benchmark for LoadAndDecrypt 2017-01-17 10:40:57 +01:00
Alexander Neumann
32a5c2c1f6 Add a few functions to calculate Blob buffer len 2017-01-17 10:40:57 +01:00
Alexander Neumann
91dcb958e0 Fix tests 2017-01-17 10:40:57 +01:00
Alexander Neumann
9a5b9253c4 LoadBlob: use buffer as scratch space
benchmark               old bytes     new bytes     delta
    BenchmarkLoadBlob-4     1010128       2256          -99.78%
2017-01-17 10:40:57 +01:00
Alexander Neumann
215af5c60a Add LoadBlob benchmark 2017-01-17 10:40:57 +01:00
Alexander Neumann
8734c2466c Fix call to debug.Log() 2017-01-17 10:40:57 +01:00
Alexander Neumann
0556687584 Reduce memory usage for prune 2017-01-17 10:40:57 +01:00
Alexander Neumann
094e80f4a4 Merge pull request #723 from ulziibuyan/ls-latest-cmd
Added latest keyword in ls command.
2017-01-15 12:40:58 +01:00
Uzi
c13a0953c8 User interface inconsistency fixed 2017-01-14 11:19:47 +08:00
Alexander Neumann
9f1f174c99 Add script to build release binaries 2017-01-12 19:52:03 +01:00
Alexander Neumann
7e0a4c66e7 build.go: Make binaries completely static (disables cgo) 2017-01-12 19:51:26 +01:00
Alexander Neumann
62b1056860 build.go: Allow setting the output file name 2017-01-12 19:51:08 +01:00
Alexander Neumann
a3181dbead CI: cross-compile for OpenBSD 2017-01-12 19:50:41 +01:00
Uzi
02c02283cf Added latest keyword in ls command. 2017-01-12 19:24:08 +08:00
Alexander Neumann
4d93da9f68 Add VERSION file for 0.3.3 2017-01-08 10:46:43 +01:00
Alexander Neumann
4a6086a14b Merge pull request #718 from mholt/flag-priority
CLI options now override env vars
2017-01-02 20:31:20 +01:00
Matthew Holt
0a34a2d5d8 Consider the environment 2017-01-02 12:21:30 -07:00
Matthew Holt
a394b675b0 CLI options now override env vars 2017-01-02 11:14:22 -07:00
Alexander Neumann
04846b10bc Merge pull request #717 from restic/fix-367
Only add entries to indexes inside PackerManager
2017-01-02 17:18:59 +01:00
Alexander Neumann
f9501e97a2 Only add entries to indexes inside PackerManager
This was a nasty bug. Users reported that restic aborts with panic:

    panic: store new item in finalized index

The code calling panic() is in the Store() method of an index and guards
the failure case that an index is to be modified while it has already
been saved in the repo.

What happens here (at least that's what I suspect): PackerManager calls
Current() on a MasterIndex, which yields one index A. Concurrently,
another goroutine calls Repository.SaveFullIndex(), which in turn calls
MasterIndex.FullIndexes(), which (among others) yields the index A. Then
all indexes are marked as final. Then the other goroutine is executed
which adds an entry to the index A, which is now marked as final. Then
the panic occurs.

The commit solves this by removing MasterIndex.Current() and adding a
Store() method that stores the entry in one non-finalized index. This
method uses the same RWMutex as the other methods (e.g. FullIndexes()),
thereby ensuring that the full indexes can only be processed before or
after Store() is called.

Closes #367
2017-01-02 14:14:51 +01:00
Alexander Neumann
3ef788765a Merge pull request #715 from zcalusic/master
Document REST backend
2017-01-02 11:13:35 +01:00
Alexander Neumann
8e16931949 Merge pull request #716 from zcalusic/rest-server-new-location
Rest server moved to https://github.com/restic/rest-server
2017-01-02 11:12:37 +01:00
Zlatko Čalušić
2267aca296 Rest server moved to https://github.com/restic/rest-server 2017-01-01 16:22:46 +01:00
Zlatko Čalušić
c70bc7ed0b Document REST backend
Closes #644
2016-12-31 13:14:44 +01:00
Alexander Neumann
8e3b81c5ec Merge pull request #713 from restic/update-travis
Update .travis.yml
2016-12-30 17:21:27 +01:00
Alexander Neumann
30975f7116 Update appveyor configuration 2016-12-30 17:07:42 +01:00
Alexander Neumann
0ef463d56a Update .travis.yml 2016-12-30 15:21:49 +01:00
Alexander Neumann
5132f5bfe6 Merge pull request #709 from restic/fix-708
Make sure cleanup is executed before exiting
2016-12-28 18:28:07 +01:00
Alexander Neumann
80457018d7 Make sure cleanup is executed before exiting
Closes #708
2016-12-28 10:53:31 +01:00
Alexander Neumann
b0997d05fb Merge pull request #704 from restic/remove-timestamp
Remove timestamp from `version` command
2016-12-19 22:22:43 +01:00
Alexander Neumann
3add2f0acb Merge pull request #703 from sjoerdsimons/master
Avoid duplicate backup paths
2016-12-19 22:21:18 +01:00
Alexander Neumann
166d1811a1 Remove timestamp from version command
This enables reproducible builds, for details see
https://reproducible-builds.org/docs/timestamps/
2016-12-19 21:14:12 +01:00
Sjoerd Simons
e1fc455079 Avoid duplicate backup paths
Target directories from the from-files argument get added to the command
line args, after which all command line args were appended to the same
variable again causing duplicates. Split the used variables to avoid
this.

Signed-off-by: Sjoerd Simons <sjoerd@luon.net>
2016-12-18 23:23:57 +01:00
Alexander Neumann
98237bf942 Add VERSION file for 0.3.2 2016-12-18 18:53:03 +01:00
Alexander Neumann
75f21f23ff Merge pull request #700 from restic/debug-panic
Make sure SaveFile always returns a node
2016-12-14 21:29:04 +01:00
Alexander Neumann
9885aeac3b Make sure SaveFile always returns a node 2016-12-14 18:56:11 +01:00
Alexander Neumann
85c87b9ab9 Add VERSION file for 0.3.1 2016-12-13 21:36:22 +01:00
Alexander Neumann
51cd78e16c Merge pull request #691 from restic/fix-604
Correctly save modified files
2016-12-10 17:31:20 +01:00
Alexander Neumann
e6a40af06d Treat changed files as a warning, not an error 2016-12-10 17:14:13 +01:00
Alexander Neumann
3fcbb4ac25 Use new Node if file has changed
Closes #604
2016-12-10 16:54:20 +01:00
Alexander Neumann
7d71bad4eb Test if modified files are correctly saved 2016-12-10 16:36:58 +01:00
Alexander Neumann
dbdfed6343 Merge pull request #690 from zcalusic/master
Even if file changes size during backup, still save it
2016-12-10 12:36:56 +01:00
Alexander Neumann
5e48c1fadc Merge pull request #688 from restic/fix-686
Save snapshot after saving all pack files
2016-12-10 12:33:58 +01:00
Zlatko Čalušić
deb6dd7f72 Even if file changes size during backup, still save it
Previously such files (typically log files) wouldn't be backed up at
all!

The proper behaviour is to backup what we can, and warn the operator
that file is possibly not complete. But it is a warning, not an error.

Closes #689
2016-12-10 12:24:45 +01:00
Alexander Neumann
c265673c8e Save snapshot after saving all pack files
Closes #686
2016-12-10 11:49:09 +01:00
Alexander Neumann
0fceeb20f1 Merge pull request #685 from jannic/patch-1
Update debug message
2016-12-06 08:16:33 +01:00
Jan Niehusmann
c5897e0d62 Update debug message
Since client.BucketExists was changed to return a separate 'found' value, instead of reporting an error when the bucket doesn't exist, the error code path does no longer imply a call to client.MakeBucket. So the second part of the debug message, "...trying to create the bucket" doesn't apply any more.
Also, changed the name of the return value from 'ok' to 'found', matching the API documentation at https://docs.minio.io/docs/golang-client-api-reference#BucketExists.
2016-12-05 23:12:30 +01:00
Alexander Neumann
8d13f22c50 Merge pull request #683 from jannic/pr1
Omit "archived as %v" messages in quiet mode.
2016-12-03 11:15:24 +01:00
Alexander Neumann
1815536534 Update build.go 2016-12-03 11:14:30 +01:00
Jan Niehusmann
9267c25aa0 Omit "archived as %v" messages in quiet mode. 2016-12-03 10:28:49 +01:00
Alexander Neumann
281cbbdf2e Merge pull request #682 from jpmens/patch-1
Small typo in dry-run of remove snapshot
2016-12-03 10:11:58 +01:00
JP Mens
5996d671a0 Small typo in dry-run of remove snapshot 2016-12-02 17:33:05 +01:00
Alexander Neumann
ef9b974bcd Merge pull request #681 from zcalusic/master
Stop trying to detect Go version
2016-12-02 11:15:29 +01:00
Zlatko Čalušić
7e66b73ce0 Stop trying to detect Go version
It fails on pre-release versions, anyway.  It's enough to mention the oldest
supported version in README.md.  Anything older than two latest Go releases
is bad idea, anyway, 'cause it's unsupported by Go development team.

Closes #680
2016-12-01 20:06:23 +01:00
Alexander Neumann
505a2097ad Manual: Add note about s3 bucket locations 2016-11-27 20:18:57 +01:00
Alexander Neumann
07380878fb Merge pull request #678 from restic/fix-676
Update github.com/elithrar/simple-scrypt
2016-11-19 19:22:44 +01:00
Alexander Neumann
3b29ae3c99 Update github.com/elithrar/simple-scrypt
Closes #676
2016-11-19 17:13:13 +01:00
Alexander Neumann
e5617b5fd1 Merge pull request #675 from restic/parent-check-hostname
Use the hostname filter to find a parent snasphot
2016-11-19 12:42:40 +01:00
Alexander Neumann
11f23ae663 Merge pull request #673 from Novex/restore-directory-metadata-for-existing-directories
Don't consider a pre-existing directory in the restore path to be a failure
2016-11-19 12:42:31 +01:00
Alexander Neumann
2828003d60 Test that existing files and dirs are restored 2016-11-15 21:41:41 +01:00
Alexander Neumann
16cef3b4c6 Use the hostname filter to find a parent snasphot
Closes #674
2016-11-15 21:04:51 +01:00
Alexander Neumann
699f39e3cf FindLatestSnapshot: Rename parameter to clarify meaning 2016-11-15 21:03:54 +01:00
Seb Patane
33b6a7381b Don't consider a pre-existing directory in the restore path to be a failure
* When a directory already exists, CreateDirAt returns an error stating so
  * This means that the restoreMetadata step is skipped, so for directories which already exist no file permissions, owners, groups, etc will be restored on them
* Not returning the error if it's a "directory exists" error means the metadata will get restored
  * It also removes the superfluous "error for ...: mkdir ...: file exists" messages
* This makes the behaviour of directories consistent with that of files (which always have their content & metadata restored, regardless of whether they existed or not)
2016-11-14 17:53:09 +10:00
Alexander Neumann
190673b24a Merge pull request #657 from AlexanderThaller/read_backup_files_from_file
Read files to backup from a file
2016-11-12 21:47:11 +01:00
Alexander Thaller
b7b03dbd4a Added new flag to backup subcommand that reads the files to backup from a file 2016-11-12 15:45:32 +01:00
Alexander Neumann
56009dd16e Merge pull request #670 from restic/remove-fadvise
Remove fadvise
2016-11-10 23:42:21 +01:00
Alexander Neumann
b56bde3f61 Remove fadvise
This commit removes the use of FADV_DONTNEED, which also purges active
cached pages for other processes.
2016-11-10 22:21:22 +01:00
Alexander Neumann
b1ed74eb43 Merge pull request #669 from zcalusic/master
Fix REST backend HTTP keepalive
2016-11-10 21:05:14 +01:00
Zlatko Čalušić
d8f0e7cbd1 Fix REST backend HTTP keepalive
This is subtle.  A combination od fast client disk (read: SSD) with lots
of files and fast network connection to restic-server would suddenly
start getting lots of "dial tcp: connect: cannot assign requested
address" errors during backup stage.  Further inspection revealed that
client machine was plagued with TCP sockets in TIME_WAIT state.  When
ephemeral port range was finally exhausted, no more sockets could be
opened, so restic would freak out.

To understand the magnitude of this problem, with ~18k ports and default
timeout of 60 seconds, it means more than 300 HTTP connections per
seconds were created and teared down.  Yeah, restic-server is that
fast. :)

As it turns out, this behavior was product of 2 subtle issues:

1) The body of HTTP response wasn't read completely with io.ReadFull()
   at the end of the Load() function.  This deactivated HTTP keepalive,
   so already open connections were not reused, but closed instead, and
   new ones opened for every new request.  io.Copy(ioutil.Discard,
   resp.Body) before resp.Body.Close() remedies this.

2) Even with the above fix, somehow having MaxIdleConnsPerHost at its
   default value of 2 wasn't enough to stop reconnecting.  It is hard to
   understand why this would be so detrimental, it could even be some
   subtle Go runtime bug.  Anyhow, setting this value to match the
   connection limit, as set by connLimit global variable, finally nails
   this ugly bug.

I fixed several other places where the response body wasn't read in
full (or at all).  For example, json.NewDecoder() is also known not to
read the whole body of response.

Unfortunately, this is not over yet. :( The check command is firing up
to 40 simultaneous connections to the restic-server.  Then, once again,
MaxIdleConnsPerHost is too low to support keepalive, and sockets in the
TIME_WAIT state pile up.  But, as this kind of concurrency absolutely
kill the poor disk on the server side, this is a completely different
bug then.
2016-11-10 09:32:07 +01:00
Alexander Neumann
5e721afb5d doc/mkdocs: Improve code hilighting
Additionally, refresh the restic sample output.
2016-11-08 20:23:39 +01:00
Alexander Neumann
149c01a86a Merge pull request #659 from restic/device-freebsd
fs.DeviceID(): Return errors whehn fi is nil
2016-11-05 13:35:16 +01:00
Alexander Neumann
51322a1055 selectFunc: handle nil 2016-11-05 12:38:33 +01:00
Alexander Neumann
c5bc802ff0 fs.DeviceID(): Return errors when fi is nil 2016-11-05 12:38:17 +01:00
Alexander Neumann
6b88d3b5d0 Merge pull request #651 from justinclift/issue649v1
Remove redundant check of error var e
2016-10-26 16:06:14 +02:00
Justin Clift
ecc1f92787 Remove redundant check of error var e
As per #649
2016-10-25 18:10:53 +01:00
Alexander Neumann
d4f76fbe26 Merge pull request #650 from restic/forget-remove-index-load
forget: do not load index
2016-10-24 14:33:11 +02:00
Alexander Neumann
1dd72693f9 forget: Remove unneeded index loading 2016-10-24 14:01:23 +02:00
Alexander Neumann
fe1013e779 cmds/ls: Format timestamp 2016-10-19 22:11:37 +02:00
Alexander Neumann
84ca5172f0 Remove Debian UID from GPG key printout 2016-10-17 13:10:16 +02:00
Alexander Neumann
7c49255c2a Add hints how to use the go tool and direnv 2016-10-17 13:09:56 +02:00
Alexander Neumann
a5a9c42185 Merge pull request #646 from stakewinner00/master
don't print status info when running in the background
2016-10-15 20:09:42 +02:00
David
5f8a6cea6f don't print status info if running in the background
clean

fix OS issues & format code

fix issues
2016-10-15 18:12:19 +00:00
Alexander Neumann
50212805aa Merge pull request #643 from restic/update-poly1305
Update golang.org/x/crypto/poly1305
2016-10-14 15:51:57 +02:00
Alexander Neumann
cd7feb0148 Update golang.org/x/crypto/poly1305 2016-10-14 12:44:06 +02:00
Alexander Neumann
974f2f78a9 Merge pull request #641 from restic/fix-640
Improve error message for 'forget'
2016-10-13 20:41:29 +02:00
Alexander Neumann
250b36eeb1 Improve error message for 'forget'
$ bin/restic forget /d 7 /w 4 /m 12
    argument "/d" is not a snapshot ID, ignoring
    argument "7" is not a snapshot ID, ignoring
    argument "/w" is not a snapshot ID, ignoring
    argument "4" is not a snapshot ID, ignoring
    argument "/m" is not a snapshot ID, ignoring
    cound not find a snapshot for ID "12", ignoring
2016-10-10 20:55:02 +02:00
Alexander Neumann
6f72164bbe Merge pull request #638 from hmsdao/patch-fixpath
Added long paths fix for samba network shares
2016-10-05 17:07:53 +02:00
Daniel Örn
ba8d960c8f using backtics instead of doublequotes 2016-10-05 08:26:32 +02:00
Daniel Örn
84421a7c68 structured file with gofmt 2016-10-05 07:30:46 +02:00
Daniel Örn
5c7325f44a Added long paths fix for samba network shares 2016-10-05 07:09:56 +02:00
Alexander Neumann
c45b498a8b Merge pull request #637 from ckemper67/s3-join
Use path.Join to create the s3 object name within the bucket.
2016-10-03 22:38:28 +02:00
Christian Kemper
a4261dcc9c Use path.Join to create the s3 object name within the bucket.
path.Join already automatically skips empty path segments when
joining, so this simplifies the s3Path code.
2016-10-02 16:56:07 -07:00
Alexander Neumann
d1ecdf7441 Add VERSION file for 0.3.0 2016-10-02 18:24:14 +02:00
Alexander Neumann
088ca033f8 Reword README 2016-10-02 16:08:05 +02:00
Alexander Neumann
5b7dd32c20 Manual: Reword section about fuse support 2016-10-02 16:03:02 +02:00
Alexander Neumann
eb94395f3d Merge pull request #635 from restic/fix-633
Fix short-hand option clash
2016-09-29 21:39:05 +02:00
Alexander Neumann
22f5fc5739 Improve help text for slice options 2016-09-29 20:39:55 +02:00
Alexander Neumann
e994cacbfe Fix short-hand option clash 2016-09-29 20:37:45 +02:00
Alexander Neumann
3114d41cb7 Merge pull request #632 from restic/rework-debug
Rework debug message printing
2016-09-28 21:11:02 +02:00
Alexander Neumann
968b2ece43 Add section to the manual about debug message filters 2016-09-28 20:22:22 +02:00
Alexander Neumann
feed54caef Remove timing, simplify function matching 2016-09-28 20:10:40 +02:00
Alexander Neumann
4eddcb344e Update calls to debug.Log() 2016-09-28 19:56:03 +02:00
Alexander Neumann
2ae06a7a01 Rework debug log function 2016-09-28 19:56:03 +02:00
Alexander Neumann
25945718a1 Fix recursive call to debug.Log 2016-09-28 19:56:03 +02:00
Alexander Neumann
254188f38f Merge pull request #631 from restic/switch-to-cobra
Switch to cobra/pflag for CLI
2016-09-28 19:54:59 +02:00
Alexander Neumann
3601c39177 Add comments 2016-09-27 20:22:01 +02:00
Alexander Neumann
02f7bb0d4c Add mousetrap library for Windows 2016-09-27 20:13:22 +02:00
Alexander Neumann
565d72ef36 Use cobra for all commands 2016-09-27 19:53:03 +02:00
Alexander Neumann
3806623c23 Vendor cobra and pflag 2016-09-27 19:52:48 +02:00
Alexander Neumann
0fa12839a5 Remove go-flags 2016-09-27 19:52:48 +02:00
Alexander Neumann
a257a613d7 Fix debug log 2016-09-27 19:52:48 +02:00
Alexander Neumann
0a752b9fab test helpers: Always print stack trace 2016-09-27 19:50:26 +02:00
Alexander Neumann
eeec0d63c2 Merge pull request #630 from restic/remove-unused
Remove unused bits and pieces
2016-09-24 12:03:26 +02:00
Alexander Neumann
04d6b5da2f Remove more unused bits 2016-09-21 20:45:18 +02:00
Alexander Neumann
1dfd3b8aa3 Remove unused bits and pieces
Reported by https://github.com/dominikh/go-unused
2016-09-21 20:22:32 +02:00
Alexander Neumann
0873821b98 Add section about --one-file-system to manual 2016-09-18 20:18:52 +02:00
Alexander Neumann
0a9cbd47c7 Merge pull request #626 from rfjakob/master
Add "-x", "--one-file-system" option
2016-09-18 20:03:58 +02:00
Alexander Neumann
b61027b48d Merge pull request #627 from restic/fix-fuse-test
fuse: fix tests for snapshots with same timestamps
2016-09-18 19:55:12 +02:00
Jakob Unterwurzacher
53701891a1 Add "-x", "--one-file-system" option
Equivalent to rsync's "-x" option.

Notes to the naming:

"--exclude-other-filesystems"
is used by Duplicity,

"--one-file-system"
is used rsync and tar.

This latter should be more familiar to the user.
2016-09-18 18:52:30 +02:00
Alexander Neumann
68b462d057 fuse: Add test for same timestamps 2016-09-18 18:30:25 +02:00
Alexander Neumann
649f789190 fuse: Fix test for timestamps with same second 2016-09-18 18:13:39 +02:00
Alexander Neumann
7b3e319398 Merge pull request #625 from restic/fix-624
fuse: correctly handle snapshots
2016-09-18 15:35:50 +02:00
Alexander Neumann
5494c1858e fuse: correctly handle snapshots
The fuse code kept adding snapshots to the top-level dir "snapshots". In
addition, snapshots with the same timestamp (same second) were not added
correctly, they will now be suffixed by an incrementing counter, e.g.:

    dr-xr-xr-x 1 fd0 users 0 Sep 18 15:01 2016-09-18T15:01:44+02:00
    dr-xr-xr-x 1 fd0 users 0 Sep 18 15:01 2016-09-18T15:01:48+02:00
    dr-xr-xr-x 1 fd0 users 0 Sep 18 15:01 2016-09-18T15:01:48+02:00-1

Closes #624
2016-09-18 15:04:39 +02:00
Alexander Neumann
c5763e59d5 Merge pull request #623 from restic/fix-622
Improve error messages for open repo
2016-09-18 14:04:30 +02:00
Alexander Neumann
b090c73bd4 Remove wrapper functions in errors package
This way, our own errors package does not appear in the stack traces.
2016-09-18 13:28:59 +02:00
Alexander Neumann
2b9a408ccc Return a fatal for location.Parse 2016-09-18 13:28:41 +02:00
Alexander Neumann
83c35bd6b5 Do not print stack trace when open repo failed
Closes #622
2016-09-18 13:24:46 +02:00
Alexander Neumann
98b012a04e Merge pull request #620 from restic/watch-529
Add verbose error when marshalling a node fails
2016-09-17 11:05:00 +02:00
Alexander Neumann
a9af896ddd Add verbose error when marshalling a node fails
This code is introduced to watch for issue #529, in which two users
describe that restic failed encoding a time in a node to JSON with the
error message:

    panic: json: error calling MarshalJSON for type *restic.Node: json: error calling MarshalJSON for type time.Time: Time.MarshalJSON: year outside of range [0,9999]

The error message now is:

    panic: Marshal: json: error calling MarshalJSON for type *restic.Node: node /home/fd0/shared/work/restic/restic/.git/hooks/applypatch-msg.sample has invalid ModTime year -1: -0001-01-02 03:04:05.000000006 +0053 LMT
2016-09-17 10:43:04 +02:00
Alexander Neumann
309dca8179 Merge pull request #619 from restic/update-deps
Update all vendored dependencies
2016-09-15 22:50:44 +02:00
Alexander Neumann
8144cd24d6 Add golang.org/x/crypto/ed25519 2016-09-15 22:36:49 +02:00
Alexander Neumann
0ce8191be5 Add golang.org/x/crypto/curve25519 2016-09-15 22:36:29 +02:00
Alexander Neumann
595f2582fa Update golang.org/x/sys/unix 2016-09-15 22:35:45 +02:00
Alexander Neumann
da83bd8265 Upadte golang.org/x/net/context 2016-09-15 22:34:06 +02:00
Alexander Neumann
799cc37c22 Update golang.org/x/crypto/ssh 2016-09-15 22:33:32 +02:00
Alexander Neumann
35ba817128 Update golang.org/x/crypto/scrypt 2016-09-15 22:32:38 +02:00
Alexander Neumann
29a61950dd Update golang.org/x/crypto/poly1305 2016-09-15 22:32:17 +02:00
Alexander Neumann
acd39eaab5 Update golang.org/x/crypto/pbkdf2 2016-09-15 22:31:49 +02:00
Alexander Neumann
3d55b54f3d Update github.com/pkg/sftp 2016-09-15 22:31:18 +02:00
Alexander Neumann
daae3500dd Update branch for github.com/kr/fs 2016-09-15 22:30:27 +02:00
Alexander Neumann
64fe9ec048 Update github.com/jessevdk/go-flags 2016-09-15 22:29:49 +02:00
Alexander Neumann
cb80a70aca Update bazil.org/fuse 2016-09-15 22:26:23 +02:00
Alexander Neumann
24398d2b9d Merge pull request #618 from restic/rework-ci-fuse-tests
Cleanup CI tests for fuse
2016-09-15 21:53:44 +02:00
Alexander Neumann
d4a2d70089 Retry umount for integration tests 2016-09-15 21:37:50 +02:00
Alexander Neumann
9add72e9d6 Exclude unneeded test run without fuse tests 2016-09-15 21:37:50 +02:00
Alexander Neumann
e7fc908ff1 Run fuse tests on Linux 2016-09-15 21:25:59 +02:00
Alexander Neumann
4ffca0f4b4 Improve integration tests for fuse 2016-09-15 21:17:20 +02:00
Alexander Neumann
a0f3e94655 fuse: handle duplicate timestamps for snapshots
This closes #606, which fails because several snapshots are created with
exactly the same timestamp, and the code checks that for each snapshot
there is a dir in the fuse mount. This fails for colliding timestamps,
so we now add a suffix "-1", "-2" etc for each duplicate timestamp.
2016-09-15 21:15:49 +02:00
Alexander Neumann
6485a6cdc0 Simplify mount logic 2016-09-15 19:59:07 +02:00
Alexander Neumann
931f5cdd33 Merge pull request #616 from restic/add-snapshot-tags
Add tags to snapshots
2016-09-14 20:58:12 +02:00
Alexander Neumann
3975d76f23 Correct filenames for expire policy tests 2016-09-13 21:20:41 +02:00
Alexander Neumann
bf6602bc1b Update golden file 2016-09-13 21:19:57 +02:00
Alexander Neumann
a85ffc66ae Add documentation for tags 2016-09-13 21:09:55 +02:00
Alexander Neumann
828267aaa3 Fix status for stdin archiver 2016-09-13 21:01:29 +02:00
Alexander Neumann
a77c615909 Fix 'forget' command with tags 2016-09-13 20:56:18 +02:00
Alexander Neumann
cfdf4c92f7 Add --keep-tag to forget command 2016-09-13 20:37:11 +02:00
Alexander Neumann
0f9fb37c78 Add tags to forget command 2016-09-13 20:20:55 +02:00
Alexander Neumann
673bce936e Add tags to 'backup' and 'snapshots' commands 2016-09-13 20:20:52 +02:00
Alexander Neumann
1f83635267 Add tags to snapshots and filter 2016-09-13 20:12:55 +02:00
Alexander Neumann
2d7e1b5804 Merge pull request #615 from kerel-fs/fix/manual
doc/Manual: Update usage help listing
2016-09-12 21:20:18 +02:00
Fabian P. Schmidt
085cf36199 doc/Manual: Update usage help listing 2016-09-12 20:42:38 +02:00
Alexander Neumann
ceb4a3ecc0 Merge pull request #613 from restic/read-password-from-file
Read password from file
2016-09-12 20:37:08 +02:00
Alexander Neumann
cf7795ce64 Merge pull request #614 from restic/improve-prune-stats
Improve statistics for `prune`
2016-09-12 20:37:00 +02:00
Alexander Neumann
223dc78acb Improve statistics for prune
Sample:

    counting files in repo
    building new index for repo
    [0:00] 100.00%  22 / 22 packs
    repository contains 22 packs (1377 blobs) with 90.610 MiB bytes
    processed 1377 blobs: 0 duplicate blobs, 0B duplicate
    load all snapshots
    find data that is still in use for 1 snapshots
    [0:00] 100.00%  1 / 1 snapshots
    found 409 of 1377 data blobs still in use, removing 968 blobs
    will delete 10 packs and rewrite 10 packs, this frees 64.232 MiB
    creating new index
    [0:00] 100.00%  7 / 7 packs
    saved new index as df467c6e
    done

Closes #581
2016-09-12 14:26:47 +02:00
Alexander Neumann
f63cd12569 Document new option 2016-09-12 14:10:36 +02:00
Alexander Neumann
65afeba19a Add option to read the password from a file 2016-09-12 14:09:22 +02:00
Alexander Neumann
791f73e0db Merge pull request #608 from rosetree/patch-1
Fix a small typo in `stdin` example
2016-09-05 19:27:26 +02:00
Micha Rosenbaum
8ded453ab0 Fix a small typo in stdin example
s/--stdin-filenam/--stdin-filename/
2016-09-05 19:01:53 +02:00
Alexander Neumann
e443454c4b Add OS and Arch to 'version' output 2016-09-04 15:46:50 +02:00
Alexander Neumann
1dd9a58e5a Merge pull request #600 from restic/restructure
WIP: restructure code
2016-09-04 15:36:26 +02:00
Alexander Neumann
b628bcee27 Remove redundant ParseID 2016-09-04 14:38:18 +02:00
Alexander Neumann
dfc0cbf3a8 Use one test password 2016-09-04 14:30:14 +02:00
Alexander Neumann
512a92895f Rename WithTestEnvironment -> Env 2016-09-04 14:29:04 +02:00
Alexander Neumann
6ab425f130 Remove SetupRepo 2016-09-04 13:24:51 +02:00
Alexander Neumann
f5b9ee53a3 Fix mock.Repository 2016-09-04 13:18:25 +02:00
Alexander Neumann
ea073f58cf Correct comment 2016-09-04 13:08:09 +02:00
Alexander Neumann
bef5c4acb8 Add mock.Repository, Rework SetupRepo 2016-09-04 12:52:43 +02:00
Alexander Neumann
b5b3c0eaf8 Add repository.SaveTree 2016-09-03 21:10:25 +02:00
Alexander Neumann
1fb80bf0e2 Fix fuse mount 2016-09-03 21:10:25 +02:00
Alexander Neumann
436332d5f2 LoadDataBlob -> LoadBlob 2016-09-03 21:10:25 +02:00
Alexander Neumann
fe8c12c798 Replace repolitoy.SaveAndEncrypt to SaveBlob() 2016-09-03 21:10:25 +02:00
Alexander Neumann
1cc59010f5 Remove LoadJSONPack, un-export loadBlob 2016-09-03 21:10:25 +02:00
Alexander Neumann
878c1cd936 Add more comments 2016-09-03 21:10:25 +02:00
Alexander Neumann
5170c4898a Address hound comments 2016-09-03 21:10:25 +02:00
Alexander Neumann
2054e3c026 Fix tests 2016-09-03 21:10:25 +02:00
Alexander Neumann
ffbe05af9b Rework crypto, use restic.Repository everywhere 2016-09-03 21:10:25 +02:00
Alexander Neumann
84f95a09d7 Introduce LoadTreeBlob and LoadDataBlob 2016-09-03 21:10:25 +02:00
Alexander Neumann
573410afab Fix archiver test 2016-09-03 21:10:25 +02:00
Alexander Neumann
619939ccd9 Reorder methods in interface Repository 2016-09-03 21:10:25 +02:00
Alexander Neumann
714a5d1dc4 Move tree walker to restic/walk 2016-09-03 21:10:25 +02:00
Alexander Neumann
bc42dbdf87 Create package restic/errors 2016-09-03 21:10:24 +02:00
Alexander Neumann
765b5437bd Fix command 'dump' 2016-09-03 21:10:24 +02:00
Alexander Neumann
5d7b38cabf Remove sentinel errors 2016-09-03 21:10:24 +02:00
Alexander Neumann
debf1fce54 Remove IDSize, TestRandomID -> NewRandomID 2016-09-03 21:10:24 +02:00
Alexander Neumann
0045f2fb61 Remove functions 2016-09-03 21:10:24 +02:00
Alexander Neumann
5764b55aee Rename Node.FileType -> Type 2016-09-03 21:10:24 +02:00
Alexander Neumann
5e3a41dbd2 Rename struct member FileType -> Type 2016-09-03 21:10:24 +02:00
Alexander Neumann
88d0f24ce7 Reduce lock timeout to zero 2016-09-03 21:10:24 +02:00
Alexander Neumann
eb6e3ba8b3 Fix imported package 2016-09-03 21:10:24 +02:00
Alexander Neumann
528c301891 Last fixes for integration tests 2016-09-03 21:10:24 +02:00
Alexander Neumann
f7ae0cb78f Fix cmds/restic for new structure 2016-09-03 21:10:24 +02:00
Alexander Neumann
3695ba5882 Tests pass for restic/ 2016-09-03 21:10:24 +02:00
Alexander Neumann
4c95d2cfdc wip 2016-09-03 21:10:24 +02:00
Alexander Neumann
cc6a8b6e15 wip 2016-09-03 21:10:24 +02:00
Alexander Neumann
51d8e6aa28 wip 2016-09-03 21:10:24 +02:00
Alexander Neumann
f0600c1d5f wip 2016-09-03 21:10:24 +02:00
Alexander Neumann
90da66261a Copy ID from backend to restic 2016-09-03 21:10:24 +02:00
Alexander Neumann
82c2dafb23 Copy interfaces and basic types to package restic/ 2016-09-03 21:10:24 +02:00
Alexander Neumann
bfdd26c541 Remove (unused) cache implementation 2016-09-03 21:10:24 +02:00
Alexander Neumann
e699f6d1bd Update doc comment 2016-09-03 21:10:24 +02:00
Alexander Neumann
fae65ebc61 Merge pull request #602 from restic/update-chunker
Update chunker
2016-09-03 21:10:18 +02:00
Alexander Neumann
f744c3534d Update chunker 2016-09-03 20:56:21 +02:00
Alexander Neumann
9ce40761c8 Remove coveralls.io 2016-09-03 11:06:09 +02:00
Alexander Neumann
48924009fe Add codecov.io 2016-09-03 10:44:37 +02:00
Alexander Neumann
d497fb6966 Merge pull request #599 from restic/remove-lowlevel-syscall
Replace lowlevel syscall to restore symlink times
2016-08-31 19:28:23 +02:00
Alexander Neumann
5bc7f150f8 Merge pull request #598 from restic/update-minio-go
Update minio-go
2016-08-31 19:28:17 +02:00
Alexander Neumann
a6eda344a4 Update minio-go 2016-08-31 18:08:43 +02:00
Alexander Neumann
1aa52e5e1e Replace lowlevel syscall to restore symlink times 2016-08-30 21:45:16 +02:00
Alexander Neumann
769f06cea2 Merge pull request #580 from restic/remove-juju-errors
Change errors library
2016-08-30 21:23:53 +02:00
Alexander Neumann
8d90588020 Add better error message for 'cat' 2016-08-30 21:19:04 +02:00
Alexander Neumann
9cf63c99cf Wrap errors #3 2016-08-29 22:16:58 +02:00
Alexander Neumann
4a0f77650b Wrap errors #2 2016-08-29 21:54:50 +02:00
Alexander Neumann
b53679a24d Wrap errors 2016-08-29 21:38:34 +02:00
Alexander Neumann
b06845c545 Always use errors.Cause() for testing error values 2016-08-29 19:52:03 +02:00
Alexander Neumann
c55b6ee544 Add restic.Fatal/f
This is a new error which implements the restic.Fataler interface.
Errors of this type are written to stderr, the restic exits. For all
other errors, restic prints the stack trace (if available).
2016-08-29 19:52:00 +02:00
Alexander Neumann
045f545085 repository: Handle errors correctly 2016-08-29 19:23:50 +02:00
Alexander Neumann
038b63f7f7 CI: Check for packages importing "errors" from stdlib 2016-08-29 19:23:50 +02:00
Alexander Neumann
d3f4c816c7 Print error stack if available 2016-08-29 19:23:50 +02:00
Alexander Neumann
72aa6be38d Replace fmt.Errorf() by errors.Errorf() 2016-08-29 19:23:50 +02:00
Alexander Neumann
444a268ce0 Replace stdlib errors with github.com/pkg/errors 2016-08-29 19:23:50 +02:00
Alexander Neumann
17a38faa43 Drop dependency github.com/juju/errors 2016-08-29 19:23:50 +02:00
Alexander Neumann
24385ff56e Merge pull request #597 from restic/fix-panic-596
Fix panic for debug.Log() with empty string
2016-08-29 17:13:37 +02:00
Alexander Neumann
f51bc8e9b9 Fix panic for debug.Log() with empty string 2016-08-28 22:43:05 +02:00
Alexander Neumann
6f5bf45212 Merge pull request #595 from restic/fix-cat
Fix the cat command
2016-08-28 22:28:25 +02:00
Alexander Neumann
3af8f53097 Allow 'cat' for tree blobs 2016-08-28 21:23:46 +02:00
Alexander Neumann
6c6b0e2395 cat: Add warning when pack was modified 2016-08-28 21:21:04 +02:00
Alexander Neumann
26351522c5 Merge pull request #594 from restic/fix-checker
Remove check for filemode 0
2016-08-28 21:09:02 +02:00
Alexander Neumann
dec2e4788e Remove flaky test 2016-08-28 21:06:27 +02:00
Alexander Neumann
f9cd736b33 Fix flaky test 2016-08-28 21:04:35 +02:00
Alexander Neumann
553dd00741 Merge pull request #592 from restic/fix-587
Fix panic when parsing sftp URIs
2016-08-28 20:14:17 +02:00
Alexander Neumann
88634dac3a Remove check for filemode 0 2016-08-28 20:04:09 +02:00
Alexander Neumann
83924d0864 Improve error message when sftp fails
Also add a prefix for all errors written to stderr by the client
2016-08-28 19:56:46 +02:00
Alexander Neumann
22bde5b277 sftp: Add debug log messages 2016-08-28 19:47:12 +02:00
Alexander Neumann
cdbdf74811 Remove debug output for tests 2016-08-28 19:30:56 +02:00
Alexander Neumann
db16702263 Report errors to stderr for tests 2016-08-28 19:30:56 +02:00
Alexander Neumann
5dd137d53e Improve error handling with the ssh subprocess 2016-08-28 19:30:56 +02:00
Alexander Neumann
8de06bd453 Vendor github.com/pkg/errors 2016-08-28 19:30:56 +02:00
Alexander Neumann
a7e64afc0d Update sftp library 2016-08-28 19:30:56 +02:00
Alexander Neumann
ed09887d9e Fix panic when parsing sftp URIs
Closes #587
2016-08-28 19:30:56 +02:00
Alexander Neumann
d097d40237 Merge pull request #593 from restic/correct-backend-errors
local/sftp: Fix broken error handling
2016-08-28 19:30:50 +02:00
Alexander Neumann
196bbbd25b local/sftp: Fix broken error handling
This yields the error messages for a full backup location:

    panic: write /home/fd0/mnt/temp/tmp/temp-987810174: no space left on device

Closes #540

Also connected to #574
2016-08-28 18:54:58 +02:00
Alexander Neumann
93e62c6f18 Merge pull request #591 from viric/packs-not-files
On prune report, packs instead of files + fix counter
2016-08-27 21:57:09 +02:00
Lluís Batlle i Rossell
3acf03986a On prune report, packs instead of files + fix counter 2016-08-27 20:04:35 +02:00
Alexander Neumann
12a904eb4b Fix reading password from stdin
This fixes a bug introduced in #585, it must by checked for stdin and
stdout separately whether it is a terminal.
2016-08-27 18:31:46 +02:00
Alexander Neumann
7f06ec98b8 Merge pull request #585 from trbs/progress_without_terminal
show progress every second when run non interactively
2016-08-27 10:10:18 +02:00
Alexander Neumann
d62264c837 Merge pull request #584 from restic/fix-panic
Add more safety checks for Unpacker
2016-08-27 10:09:57 +02:00
Alexander Neumann
b2a67d458c Remove unneeded packs without repacking 2016-08-25 22:35:22 +02:00
Alexander Neumann
de88fb2022 Simplify pack.List 2016-08-25 22:25:55 +02:00
trbs
71263b5090 show progress every second when run non interactively 2016-08-25 22:13:47 +02:00
Alexander Neumann
3fd1e4a992 Add backend.ReaderAt 2016-08-25 21:49:00 +02:00
Alexander Neumann
9f752b8306 Rework function for listing packs 2016-08-25 21:08:16 +02:00
Alexander Neumann
e07ae7631c Add more safety checks for Unpacker 2016-08-23 22:21:29 +02:00
Alexander Neumann
9fd941f6fc Merge pull request #583 from stuertz/windowsoutput
Fix progress output on Windows
2016-08-23 21:18:09 +02:00
Jan Stürtz
91c458bf74 Fixed gofmt 2016-08-22 22:07:10 +02:00
Jan Stürtz
374b1144de Dont't guess the max width, get it from the terminal 2016-08-22 17:27:58 +02:00
Jan Stürtz
f05b0871e9 fixed maxlen computation (off by one) on small terminals 2016-08-22 17:27:03 +02:00
Jan Stürtz
4cb8fe3210 Fixed style hints from hound
- no else, when if has a return
- Improve Comment on Function
2016-08-21 23:10:28 +02:00
Jan Stürtz
08eb5b42eb Fix progress output on Windows
The windows cmd shell is not aware of ANSI escape sequences and
does print them uninterpreted to the console. This is ugly.
Added a function to generate platform specific string for the escape sequence. On Windows this will be 79 white spaces with
a trailing \r.
2016-08-21 22:38:22 +02:00
Alexander Neumann
1c703e4161 Merge pull request #579 from restic/debug-544
Properly close connections to s3 backend on Stat()
2016-08-21 17:10:07 +02:00
Alexander Neumann
ebd3723a06 Properly close the minio object on Stat()
Closes #544
2016-08-21 16:15:41 +02:00
Alexander Neumann
06b23edb39 Fix code for newer minio-go 2016-08-21 16:14:58 +02:00
Alexander Neumann
e893be3dec Update minio-go 2016-08-21 16:14:22 +02:00
Alexander Neumann
ca14942c80 Merge pull request #578 from restic/fix-build-on-arm
Fix build on linux/arm
2016-08-21 15:09:46 +02:00
Alexander Neumann
11d01fcd32 Merge pull request #577 from restic/dynamic-scrypt
Dynamically calibrate scrypt parameters
2016-08-21 15:00:24 +02:00
Alexander Neumann
5061607e77 x/sys/unix: Manually add FADV_* constants for Linux/arm 2016-08-21 14:59:15 +02:00
Alexander Neumann
69d8fe5b4f Add check for cross-compilation 2016-08-21 14:21:19 +02:00
Alexander Neumann
916efa4e1a Merge pull request #576 from restic/fix-documentation-forget
Improve documentation, add explanation and weekly
2016-08-21 13:51:38 +02:00
Alexander Neumann
a3492d69dd Use low-security scrypt KDF parameters for testing 2016-08-21 13:42:04 +02:00
Alexander Neumann
8e24c51233 Fix commets for constants 2016-08-21 13:13:05 +02:00
Alexander Neumann
d8107f77aa Limit the number of key files checked on SearchKey 2016-08-21 13:10:16 +02:00
Alexander Neumann
79e950b710 Remove dead code 2016-08-21 13:10:15 +02:00
Alexander Neumann
f0d7f3f1bd Calibrate scrypt for the current hardware
Closes #17
2016-08-21 13:10:08 +02:00
Alexander Neumann
9afec53c55 Remove crypto reader/writer (unused) 2016-08-21 13:10:08 +02:00
Alexander Neumann
11098d6eb0 Move KDF() to kdf.go 2016-08-21 13:10:08 +02:00
Alexander Neumann
7e6fc15ece Vendor github.com/elithrar/simple-scrypt 2016-08-21 13:10:08 +02:00
Alexander Neumann
78c0995853 Improve documentation, add explanation and weekly 2016-08-21 11:53:05 +02:00
Alexander Neumann
84c14e623d Merge pull request #575 from restic/remove-constants
Remove POSIX constants, reduce code duplication
2016-08-21 11:02:06 +02:00
Alexander Neumann
d965d703d1 Reduce duplicate code in wrappers for os 2016-08-21 10:42:07 +02:00
Alexander Neumann
b20921d836 Use constants from /x/sys/unix 2016-08-21 10:36:20 +02:00
Alexander Neumann
a78493f549 Update golang.org/x/sys/unix 2016-08-21 10:35:12 +02:00
Alexander Neumann
2be0aa9dbc Merge pull request #518 from restic/implement-prune
Implement prune
2016-08-21 09:22:22 +02:00
Alexander Neumann
aa29c68189 Fix progress for new index 2016-08-20 20:44:57 +02:00
Alexander Neumann
d3da30e8fb Use UTC for snapshot time based tests 2016-08-20 18:49:02 +02:00
Alexander Neumann
3337b5d3c4 Add prune/forget to the manual 2016-08-20 18:38:16 +02:00
Alexander Neumann
458448357c Add help texts which cross-line prune/forget 2016-08-20 18:33:24 +02:00
Alexander Neumann
27d0909302 forget: Remove message when no policy is specified 2016-08-20 18:15:36 +02:00
Alexander Neumann
5f0ebb71b2 forget: Allow filtering for a hostname 2016-08-20 17:59:47 +02:00
Alexander Neumann
00f647dc92 forget: Join paths by ":" 2016-08-20 17:59:10 +02:00
Alexander Neumann
8e7202bd6a Rename function in debug 'dump' command 2016-08-20 17:54:27 +02:00
Alexander Neumann
5cf7c827b8 forget: Do nothing if no policy is configured 2016-08-20 17:53:03 +02:00
Alexander Neumann
71f7f4f543 Add ExpirePolicy.Empty() 2016-08-20 17:51:48 +02:00
Alexander Neumann
bf47dba1c4 Add 'forget' command 2016-08-20 17:43:25 +02:00
Alexander Neumann
cbd457e557 Add Hourly expire functions 2016-08-20 15:55:23 +02:00
Alexander Neumann
6cf4b81558 Add functions to filter snapshots 2016-08-20 15:22:40 +02:00
Alexander Neumann
bb84d351f1 Revert "ID: move Str() to non-pointer receiver"
This reverts commit f102406cd7.
2016-08-19 20:45:19 +02:00
Alexander Neumann
a107e3cc84 Correct comment 2016-08-19 20:36:24 +02:00
Alexander Neumann
e934966b54 Merge pull request #573 from restic/fix-osxfuse-travis
Fix osxfuse on Travis/darwin
2016-08-19 19:23:24 +02:00
Alexander Neumann
bd9f23f1d2 Fix osxfuse on Travis/darwin 2016-08-19 19:04:02 +02:00
Alexander Neumann
2a2fb74ba8 Merge pull request #569 from restic/fix-568
Use the platform-independent function for joining
2016-08-19 17:53:09 +02:00
Alexander Neumann
bd819a5e81 Fix panic 2016-08-16 21:59:43 +02:00
Alexander Neumann
162629571d Add BenchmarkFindUsedBlobs 2016-08-16 21:30:14 +02:00
Alexander Neumann
2c04ad3c29 TestCreateSnapshot: free buffer 2016-08-16 21:30:14 +02:00
Alexander Neumann
238d3807e9 prune: Format duplicate bytes properly 2016-08-16 21:30:14 +02:00
Alexander Neumann
7f9d227725 Use progress in prune command 2016-08-16 21:30:14 +02:00
Alexander Neumann
8de6e5a627 Add progress option to index 2016-08-16 21:30:14 +02:00
Alexander Neumann
8d735cf6a9 Explicitely specify supersedes for new index 2016-08-16 21:30:14 +02:00
Alexander Neumann
29bb845f0e Rebuild index at the end of prune 2016-08-16 21:30:14 +02:00
Alexander Neumann
1bb2d59e38 Add Save() method to Index 2016-08-16 21:30:14 +02:00
Alexander Neumann
3ceb2ad3cf Progress: Call OnUpdate before OnDone 2016-08-16 21:30:14 +02:00
Alexander Neumann
009c803c8a prune: Use new Index 2016-08-16 21:30:14 +02:00
Alexander Neumann
c0ef1ec6fd Add RemovePack for index 2016-08-16 21:30:14 +02:00
Alexander Neumann
69c2e8ce7e Add first version of the prune command 2016-08-16 21:30:14 +02:00
Alexander Neumann
f102406cd7 ID: move Str() to non-pointer receiver 2016-08-16 21:30:14 +02:00
Alexander Neumann
302619a11a Move interfaces to package restic/types 2016-08-16 21:30:14 +02:00
Alexander Neumann
80bcae44e2 Decouple ListAllPacks from repository 2016-08-16 21:30:14 +02:00
Alexander Neumann
1f263a7683 Decouple index/ and repository/ 2016-08-16 21:30:14 +02:00
Alexander Neumann
3b57075109 Add global interface Repository 2016-08-16 21:30:14 +02:00
Alexander Neumann
3fa7304e94 Add interfaces to ListAllPacks 2016-08-16 21:30:14 +02:00
Alexander Neumann
47950b82a0 Add test for loading index from documentation 2016-08-16 21:30:14 +02:00
Alexander Neumann
9ecf7070af Implement Lookup() and Save() for new Index 2016-08-16 21:30:14 +02:00
Alexander Neumann
2310773798 Compute negative offsets ourselves in the s3 backend 2016-08-16 21:30:14 +02:00
Alexander Neumann
a60e3b5030 Make backend tests less verbose 2016-08-16 21:30:14 +02:00
Alexander Neumann
b350b443d0 Stop backend tests early on failure 2016-08-16 21:30:14 +02:00
Alexander Neumann
2c517e4a33 Add Index structures for Blobs 2016-08-16 21:30:14 +02:00
Alexander Neumann
4bdd59b4ad Index: Add DuplicateBlobs() 2016-08-16 21:30:14 +02:00
Alexander Neumann
f5daf33322 Add pack size to ListAllPacks 2016-08-16 21:30:14 +02:00
Alexander Neumann
1058a91b39 Add option to create duplicate blobs in TestCreateSnapshot 2016-08-16 21:30:14 +02:00
Alexander Neumann
240b8f273a Add more index tests 2016-08-16 21:30:14 +02:00
Alexander Neumann
6808523d34 Add String() for Blob 2016-08-16 21:30:14 +02:00
Alexander Neumann
bad6184ab5 Add new Index data structure 2016-08-16 21:30:14 +02:00
Alexander Neumann
6b384287f3 Return error when it occurs 2016-08-16 21:30:14 +02:00
Alexander Neumann
ef33cf12ca Fix Unpacker for packs < 2048 byte 2016-08-16 21:30:14 +02:00
Alexander Neumann
a5cbbb8b5a Fix BufferLoader for negative offset 2016-08-16 21:30:14 +02:00
Alexander Neumann
71924fb7c0 Add tests for Load() with negative offset 2016-08-16 21:30:14 +02:00
Alexander Neumann
b0565015cc Remove ReadSeeker 2016-08-16 21:30:14 +02:00
Alexander Neumann
fa283c6ecd Remove unused GetReader() 2016-08-16 21:30:14 +02:00
Alexander Neumann
94d157d97a Introduce interface pack.Loader 2016-08-16 21:30:14 +02:00
Alexander Neumann
f72f3dbc6a Buffer last 2048 bytes of a file for unpack 2016-08-16 21:28:55 +02:00
Alexander Neumann
3c3a180417 Move RandomID() to backend package 2016-08-16 21:28:55 +02:00
Alexander Neumann
fd6c854a21 Add TestResetRepository and BenchmarkCreateSnapshot 2016-08-16 21:28:55 +02:00
Alexander Neumann
e9cddc0be5 Fix TestFindUsedBlobs 2016-08-16 21:28:55 +02:00
Alexander Neumann
d7e5f11b78 Export FindUsedBlobs 2016-08-16 21:28:55 +02:00
Alexander Neumann
2b1b6d8c2a Export ListAllPacks 2016-08-16 21:28:55 +02:00
Alexander Neumann
acc2fa5816 Fix TestRepack
* Decrease number of blobs for use in test
 * Fail the test when there's a duplicate blob
2016-08-16 21:28:54 +02:00
Alexander Neumann
6285f31604 Use pack.BlobSet instead of backend.IDSet 2016-08-16 21:28:54 +02:00
Alexander Neumann
3cca831b2e Fix invalid type in newly created packs 2016-08-16 21:28:54 +02:00
Alexander Neumann
cff6fea32a Fix 'cat' command 2016-08-16 21:28:54 +02:00
Alexander Neumann
17e1872544 Switch order of parameters to repo.LoadBlob() 2016-08-16 21:28:54 +02:00
Alexander Neumann
246302375d Index: Add multiple packs per blob, pack.Type
Change the index so that a blob can be contained in multiple packs.

Require passing the blob type to all lookup functions.
2016-08-16 21:28:54 +02:00
Alexander Neumann
231da4ff80 Remove old repacking code 2016-08-16 21:28:54 +02:00
Alexander Neumann
1b4b469440 Add pack.Handle and pack.Handles 2016-08-16 21:28:54 +02:00
Alexander Neumann
35e3762e37 Remove dead code 2016-08-16 21:28:54 +02:00
Alexander Neumann
7e732dbd2d Allow multiple entries in the index 2016-08-16 21:28:54 +02:00
Alexander Neumann
8b4d4ec25f Fix TestCreateSnapshot, do not store duplicate data 2016-08-16 21:28:54 +02:00
Alexander Neumann
035d0aeb31 Do not create duplicate content for tests 2016-08-16 21:28:54 +02:00
Alexander Neumann
f1bc181c5b Add more checks for tests 2016-08-16 21:28:54 +02:00
Alexander Neumann
50b724ca23 Fix stylistic issues with FindUsedBlobs 2016-08-16 21:28:54 +02:00
Alexander Neumann
6227821b4e Move functions to correct file 2016-08-16 21:28:54 +02:00
Alexander Neumann
810056c2bc Correct packages for tests 2016-08-16 21:28:54 +02:00
Alexander Neumann
34b3e3a095 Split index/repack functions to different files 2016-08-16 21:28:54 +02:00
Alexander Neumann
bdd085e9f1 Prevent loops when finding used blobs 2016-08-16 21:28:54 +02:00
Alexander Neumann
ffc3503e6f Add first version of FindUsedBlobs 2016-08-16 21:28:54 +02:00
Alexander Neumann
51b16ad57d Add handy functions to backend.IDSet 2016-08-16 21:28:54 +02:00
Alexander Neumann
723592d923 Move FindUsedBlobs to package restic 2016-08-16 21:28:54 +02:00
Alexander Neumann
22aa17091b Add test for FindUsedBlobs 2016-08-16 21:28:54 +02:00
Alexander Neumann
4720a7d807 Allow specifying chunker polynomial for tests 2016-08-16 21:28:54 +02:00
Alexander Neumann
d5323223f4 Change repository Init() function to allow better testing 2016-08-16 21:28:54 +02:00
Alexander Neumann
fe79177b40 Make TestCreateSnapshot return the snapshot itself 2016-08-16 21:28:54 +02:00
Alexander Neumann
5c32ae15c2 Move test checking repo code to checker package 2016-08-16 21:28:54 +02:00
Alexander Neumann
6c2334f505 Make TestCreateSnapshot less verbose 2016-08-16 21:28:54 +02:00
Alexander Neumann
b55ac2afd6 Make test files in test repo less random 2016-08-16 21:28:54 +02:00
Alexander Neumann
d9012b4a64 Add trees recursively to test snapshot 2016-08-16 21:28:54 +02:00
Alexander Neumann
952f124238 Use RandReader instead of rand directly
This is a fix to be backwards-compatible with Go < 1.6.
2016-08-16 21:28:54 +02:00
Alexander Neumann
14db71d3fa Move RandReader to repository package 2016-08-16 21:28:54 +02:00
Alexander Neumann
f59ffcaeae Correct comment 2016-08-16 21:28:54 +02:00
Alexander Neumann
d609e4a986 Extended plaintext buffer if necessary 2016-08-16 21:28:54 +02:00
Alexander Neumann
0e6c72ad1d Implement Repack() 2016-08-16 21:28:54 +02:00
Alexander Neumann
d5f42201c5 Fix test for Repack 2016-08-16 21:28:54 +02:00
Alexander Neumann
122a0944a6 Do not repack blobs that shouldn't be kept 2016-08-16 21:28:54 +02:00
Alexander Neumann
fa26ecc8f9 Make rebuild-index use the code in package repository 2016-08-16 21:28:54 +02:00
Alexander Neumann
00139648a0 Implement Repack() 2016-08-16 21:28:54 +02:00
Alexander Neumann
6ba38e9a38 Add tests for Repack() 2016-08-16 21:28:54 +02:00
Alexander Neumann
812cb0ba77 Update Go version in manual 2016-08-16 21:24:48 +02:00
Alexander Neumann
b5c397435c Merge pull request #571 from restic/raise-go-version
Require Go 1.6 or greater
2016-08-16 21:20:59 +02:00
Alexander Neumann
043424824c Only test cross-compilation on Go 1.7 2016-08-16 21:02:30 +02:00
Alexander Neumann
c88c48a29f Do not build toolchain with gox for Go >= 1.5 2016-08-16 20:51:46 +02:00
Alexander Neumann
2fa93b291a Update default Go version in Dockerfile 2016-08-16 20:51:31 +02:00
Alexander Neumann
1ad4d1aafd Require Go 1.6 or greater 2016-08-16 20:32:58 +02:00
Jan Stürtz
b108966b12 Fix 567 (#570)
* Patch for  https://github.com/restic/restic/issues/567
Backup also files on windows with longer pathnames than 255 chars (e.g. from node).

as fd0 says "So, as far as I can see, we need to have custom methods for all functions that accept a path, so that on Windows we can substitute the normal (possibly relative) path used within restic by an (absolute) UNC path, and only then call the underlying functions like os.Stat(), os.Lstat(), os.Open() and so on.

I've already thought about adding a generic abstraction for the file system (so we can mock this easier in tests), and this looks like a good opportunity to build it."

* fixed building tests

* Restructured patches
Add Wrapper for filepath.Walk

* using \\?\ requires absolute pathes to be used.
Now all tests run

* used gofmt on the code

* Restructured Code. No patches dir, integrate the file functions into restic/fs/

There is still an issue, because restic.fs.Open has a different api the os.Open, which returns the result of OpenFile, but takes only a string

* Changed the last os.Open() calls to fs.Open() after extending the File interface

* fixed name-clash of restic.fs and fuse.fs detected by travis

* fixed fmt with gofmt

* c&p failure: removed fixpath() call.

* missing include

* fixed includes in linux variant

* Fix for Linux. Fd() is required on File interface

* done gofmt
2016-08-15 21:59:13 +02:00
Alexander Neumann
1fe8deeb6e Use new URI syntax in documentation 2016-08-11 20:37:49 +02:00
Alexander Neumann
fa4570bde8 Always use forward slashes in file names 2016-08-11 19:41:47 +02:00
Alexander Neumann
f6c2787d80 Use the platform-independent function for joining 2016-08-11 19:37:22 +02:00
Alexander Neumann
4b8b625b90 Merge pull request #562 from damekr/implement-gomaxprocs-env
Issue-535: restic respect GOMAXPROCS env variable depending on go version
2016-08-11 19:09:54 +02:00
damekr
be00d91967 Respect GOMAXPROCS variable
Closes #535
2016-08-08 21:37:20 +02:00
Alexander Neumann
e4a9905d6f Merge pull request #563 from restic/fix-build-script
Invert go version test for ldflags
2016-08-04 19:37:30 +02:00
Alexander Neumann
68ec29e7ec Invert go version test for ldflags 2016-08-03 22:04:03 +02:00
Alexander Neumann
d860ce0570 Merge pull request #559 from vrischmann/master
Fix the debug environment variable name in the manual
2016-08-02 22:12:18 +02:00
Alexander Neumann
fc9b27c533 Revert "Fix TestCreateSnapshot, do not generate duplicate data"
This reverts commit 628fb0fb72.
2016-08-02 22:11:55 +02:00
Vincent Rischmann
d4a9b546c1 Fix the debug environment variable name in the manual 2016-08-01 22:23:42 +02:00
Alexander Neumann
628fb0fb72 Fix TestCreateSnapshot, do not generate duplicate data 2016-08-01 22:01:34 +02:00
Alexander Neumann
2de233fe8b Merge pull request #558 from vrischmann/master
Detect a devel version correctly in LDFlags()
2016-08-01 21:49:20 +02:00
Vincent Rischmann
d2834b61fb Detect a devel version correctly in LDFlags() 2016-08-01 20:47:33 +02:00
Alexander Neumann
c7f5ac22eb add VERSION file for 0.2.0 2016-07-30 11:24:52 +02:00
Alexander Neumann
959df5cc14 Merge pull request #554 from restic/debug-fuse-panic
Fix fuse panic with empty files
2016-07-29 21:51:58 +02:00
Alexander Neumann
e575494353 Correct goreportcard badge URLs 2016-07-29 21:35:06 +02:00
Alexander Neumann
c0fb2c306d Merge pull request #553 from restic/update-minio-go
Update minio-go
2016-07-29 21:23:39 +02:00
Alexander Neumann
8418fed18e Handle empty files correctly 2016-07-29 21:18:32 +02:00
Alexander Neumann
3de989b7bb Fix panic with empty files 2016-07-29 21:05:36 +02:00
Alexander Neumann
5afda94a3c Handle reads with large offsets 2016-07-29 20:55:09 +02:00
Alexander Neumann
56dd4c0595 Update minio-go 2016-07-29 20:28:44 +02:00
Alexander Neumann
a9729eeb1b Merge pull request #552 from benagricola/fix-connection-leak
Explicitly Close() obj after ReadFull()
2016-07-29 20:28:22 +02:00
Ben Agricola
edb1843f24 Explicitly Close() obj after ReadFull()
Signed-off-by: Ben Agricola <bagricola@squiz.co.uk>
2016-07-29 14:18:02 +01:00
Alexander Neumann
e1960cadb2 Merge pull request #548 from mappu/patch-1
idset.go: micro-optimise away redundant scan
2016-07-28 20:05:23 +02:00
mappu
32985f7904 idset.go: micro-optimise away redundant scan 2016-07-26 09:36:31 +12:00
Alexander Neumann
e8e45fe2e3 Merge pull request #545 from restic/fix-528
Don't report valid types as invalid
2016-07-20 21:17:43 +02:00
Alexander Neumann
6b7ddf1b03 Don't report valid types as invalid
Closes #528
2016-07-20 20:46:57 +02:00
Alexander Neumann
b8c7622a8a Merge pull request #543 from mappu/master
Updates for build.go
2016-07-17 21:32:13 +02:00
mappu
983e509388 build.go: gofmt 2016-07-16 10:14:41 +12:00
mappu
c9400d5c61 build.go: Support cross-compilation via new --goos and --goarch flags 2016-07-16 09:42:41 +12:00
mappu
f967e90a96 build.go: Strip harder (add -w flag) 2016-07-16 09:42:22 +12:00
mappu
38d4522ea5 build.go: Add go1.7 to list of linkers requiring -Xfoo=bar syntax 2016-07-16 09:42:02 +12:00
Alexander Neumann
c9ab75a44c Fix build tag in run_integration_tests.go 2016-06-29 09:36:40 +02:00
Alexander Neumann
8b47ca5f98 Merge pull request #537 from xet7/master
Fix typo in Manual.md
2016-06-26 13:58:12 +02:00
Lauri Ojansivu
4e98c951e0 Fix typo in Manual.md 2016-06-26 14:35:48 +03:00
Alexander Neumann
814424fa6e Merge pull request #531 from restic/update-minio-go
Update minio-go
2016-06-08 22:00:16 +02:00
Alexander Neumann
902f619a06 Fix call to minio.New()
The last parameter changed semantics from `insecure` to `secure`.
2016-06-08 21:33:18 +02:00
Alexander Neumann
d66a98c2db Update minio-go
This fixes #520.
2016-06-08 21:11:48 +02:00
Alexander Neumann
e1b5593e07 Merge pull request #525 from koolhead17/miniorestic
Added Minio.io configuration steps to run as backend for restic.
2016-05-31 22:32:33 +02:00
koolhead17
789b8c8b49 Added minor modification to the doc as suggested by the author. 2016-05-25 02:26:00 +05:30
Alexander Neumann
795e3d5b6c Merge pull request #503 from gerdus/restore-latest
Add option to restore latest snapshot with optional path and source filters
2016-05-11 20:48:56 +02:00
Gerdus van Zyl
73e9cac5c4 gofmt + small doc fix 2016-05-10 22:20:03 +02:00
Gerdus van Zyl
8010a0d90c fix 2016-05-10 21:57:30 +02:00
Gerdus van Zyl
3cb68ddb0d Add option to restore latest snapshot with optional path and source filters
eg restic -r r1 restore latest --target restore2 --path "D:\dev\restic\bin\s1"
path and source filters also added to snapshot cmd
eg restic -r r1 snapshots --source nucore --path="D:\dev\restic\bin\s1"

Add option to restore latest snapshot with optional path and source filters

eg restic -r r1 restore latest --target restore2 --path "D:\dev\restic\bin\s1"
path and source filters also added to snapshot cmd
eg restic -r r1 snapshots --source nucore --path="D:\dev\restic\bin\s1"
2016-05-10 21:41:26 +02:00
Gerdus van Zyl
49f82f54b0 rebase, change source to host and add description to manual 2016-05-10 21:40:32 +02:00
Alexander Neumann
6bc7a71e55 Merge pull request #516 from restic/fix-flaky-test
Fix flaky worker cancel test
2016-05-09 22:12:39 +02:00
Alexander Neumann
2c1e590e47 Merge pull request #509 from restic/read-from-stdin
Allow reading data from stdin
2016-05-09 22:12:22 +02:00
Alexander Neumann
84f7d28abf Merge pull request #515 from viric/fix_traverse_order
Traverse paths in the same order as parent snapshot
2016-05-09 22:10:36 +02:00
Alexander Neumann
cb75737770 Merge pull request #514 from viric/fix_parent_search
Better backup parent snapshot search. Part of #513
2016-05-09 22:10:32 +02:00
Alexander Neumann
fb45ea139d Add barrier 2016-05-09 21:29:13 +02:00
Alexander Neumann
bce0bbeda2 Add Benchmark for ArchiveReader 2016-05-09 21:16:59 +02:00
Alexander Neumann
c6d934a685 Fix flaky worker cancel test 2016-05-09 20:41:55 +02:00
Alexander Neumann
143fde66bc Add documentation 2016-05-09 20:11:32 +02:00
Alexander Neumann
4146c09a04 Add test for ArchiveReader() 2016-05-09 20:11:32 +02:00
Alexander Neumann
43f9c2d36e backup: Save file size when reading from stdin 2016-05-09 20:11:32 +02:00
Alexander Neumann
5e0813ca04 fuse: Use correct file size in case it's zero 2016-05-09 20:11:32 +02:00
Alexander Neumann
6ee9baa9c5 fuse: Add debug logs 2016-05-09 20:11:32 +02:00
Alexander Neumann
7c76ff3aaf Allow reading backups from stdin 2016-05-09 20:11:32 +02:00
Alexander Neumann
deae1e7e29 Merge pull request #511 from restic/cleanups
Cleanups and test functions
2016-05-09 20:10:39 +02:00
Lluís Batlle i Rossell
4818a8e356 Fix gofmt 2016-05-09 16:31:59 +02:00
Lluís Batlle i Rossell
83aa63365a Not exporting baseNameSlice. Noone else wants it. 2016-05-09 14:46:14 +02:00
Lluís Batlle i Rossell
aed73be93d Improve comment according to hound guidelines 2016-05-09 14:44:03 +02:00
Lluís Batlle i Rossell
4ea62ecbcc Traverse paths in the same order as parent snapshot
This is the 2nd partial fix to #513.

The archivepipe requires the snapshot paths and the backup paths to be
traversed in the same order, and they were sorted differently: the backup paths
by full path, and the snapshot by basename path.
2016-05-09 14:32:17 +02:00
Lluís Batlle i Rossell
60c8c90d35 Better backup parent snapshot search. Part of #513
I look for the newest snapshot that contains all supplied paths to backup.
2016-05-09 12:42:12 +02:00
Alexander Neumann
c523b38abb Remove darwin/arm 2016-05-08 23:42:13 +02:00
Alexander Neumann
f928b30caa CI: Use gox with -osarch
This allows not to cross-compile to darwin/arm, which fails at the
moment.
2016-05-08 23:25:30 +02:00
Alexander Neumann
20afed4058 Checker: handle symlinks 2016-05-08 23:16:17 +02:00
Alexander Neumann
a2224e380b Address style issues identified by Hound 2016-05-08 22:38:38 +02:00
Alexander Neumann
31030baca3 Add comment 2016-05-08 13:51:33 +02:00
Alexander Neumann
173940cbdf Add repository.ListPack 2016-05-08 13:51:21 +02:00
Alexander Neumann
6fc3590838 Remove repository.SaveFrom() 2016-05-08 13:13:29 +02:00
Alexander Neumann
43f7a1fcd9 Correct log statement 2016-05-08 13:09:36 +02:00
Alexander Neumann
7faf272996 Progress: Use reference to sync.Once 2016-05-08 13:04:58 +02:00
Alexander Neumann
514a43f74b Add more tests 2016-05-08 12:25:01 +02:00
Alexander Neumann
6655511ab8 checker: test file mode 2016-05-08 12:25:01 +02:00
Alexander Neumann
168cfc2f6d Add testing helper functions 2016-05-08 12:25:01 +02:00
Alexander Neumann
6cfa0d502d Add LoadAllSnapshots() 2016-05-08 12:25:01 +02:00
Alexander Neumann
a996dbb9d6 check: Add more checks for nodes 2016-05-08 12:25:01 +02:00
Alexander Neumann
7572586ded Fix image URL for readthedocs.org 2016-05-08 12:24:09 +02:00
Alexander Neumann
a0ab9f2fdf Merge pull request #507 from restic/debug-minio-on-darwin
Update minio-go
2016-05-08 12:20:25 +02:00
Alexander Neumann
1b50d55d0c Update minio-go 2016-05-08 11:24:24 +02:00
Alexander Neumann
7dc7f0d295 Vagrantfile: Fix network for darwin 2016-05-07 23:38:41 +02:00
Alexander Neumann
3f8da47a0c Fix restic s3 backend for new minio-go version 2016-05-07 23:38:41 +02:00
Alexander Neumann
72fdd0bc09 Update minio-go 2016-05-07 23:38:41 +02:00
Alexander Neumann
be04a3b683 Travis: Update Go version, set ulimit 2016-05-07 23:38:30 +02:00
Alexander Neumann
60f1fbe35b Update domain for readthedocs 2016-05-05 13:25:06 +02:00
Alexander Neumann
4531456be5 Merge pull request #497 from Thor77/excludeFileExpandEnv
Expand environment-variables in exclude-files
2016-04-18 21:40:30 +02:00
Alexander Neumann
59ec393be1 Merge pull request #502 from restic/update-poly1305
Update crypto library
2016-04-18 21:40:16 +02:00
Alexander Neumann
306c0fea16 Update crypto library
This allows building restic with Go 1.3 on ARM.

Closes #501
2016-04-18 21:23:00 +02:00
Alexander Neumann
039019689a Merge pull request #500 from restic/fix-499
Fix exclude filters with trailing slash
2016-04-18 21:02:27 +02:00
Alexander Neumann
1eb896ae6f Merge pull request #498 from restic/fix-travis-drawin
Fix CI tests
2016-04-18 20:58:45 +02:00
Alexander Neumann
22338903bf Fix golint 2016-04-18 20:33:45 +02:00
Alexander Neumann
baece5eeb3 Add error checking for CI tests 2016-04-18 20:24:12 +02:00
Alexander Neumann
b2846ea49d Add error handling 2016-04-18 20:24:12 +02:00
Alexander Neumann
aa43b69651 Better error reporting for CI tests 2016-04-18 20:24:12 +02:00
Alexander Neumann
6fe25548bd Add another filter test 2016-04-17 22:04:42 +02:00
Alexander Neumann
9002eaa259 Fix exclude filters with trailing slash 2016-04-17 21:54:12 +02:00
Alexander Neumann
2e3c541237 Rework Go program to run CI tests 2016-04-17 17:55:13 +02:00
Alexander Neumann
ead6d11ecf Backend tests: remove debug 2016-04-17 17:39:14 +02:00
Alexander Neumann
41e3e12f4b Travis: correct exclude for darwin 2016-04-17 13:08:57 +02:00
Alexander Neumann
d4b202243a Update Go versions 2016-04-16 22:50:36 +02:00
Alexander Neumann
87250c4489 Call minio server with env variables 2016-04-16 22:49:23 +02:00
Alexander Neumann
4a576af855 Fix CI tests on darwin 2016-04-16 22:27:34 +02:00
Thor77
4fb6669196 add documentation for environment-var expanding in exclude-files 2016-04-16 22:07:36 +02:00
Thor77
9644399074 add environment-var expanding for exclude-files 2016-04-16 22:04:29 +02:00
Alexander Neumann
5b33a7a903 Merge pull request #496 from jzacsh/jzacsh-nit-on-doc-goget
doc: point to correct `gb` arg for `go get`
2016-04-14 08:58:12 +02:00
Jonathan Zacsh
1fe0e30d71 doc: point to correct db arg for go get
per side-chat in https://github.com/restic/restic/issues/494#issuecomment-209591073
2016-04-13 15:52:08 -04:00
Alexander Neumann
75dd9e0fee Merge pull request #495 from restic/fix-494
Umount fuse in tests
2016-04-13 20:48:50 +02:00
Alexander Neumann
23d7464306 Umount fuse in tests
This corrects the order when the fuse mount is terminated by closing the
done channel: Before, restic would close the fuse connection and only
afterwards try to remove the mount, that does not work.

Closes #494
2016-04-13 20:18:54 +02:00
Alexander Neumann
32a5778602 Merge pull request #490 from Thor77/backupExcludeFile
add backup --exclude-file
2016-04-06 00:09:46 +02:00
Thor77
b4493b4640 add documentation for --exclude[-file] and patterns 2016-04-01 15:56:52 +02:00
Thor77
1c1eacfc94 add backup --exclude-file 2016-04-01 13:53:22 +02:00
Alexander Neumann
aac2405e95 Merge pull request #489 from xmaka/xmaka-patch-1
add debian stable help
2016-03-31 22:01:14 +02:00
xmaka
90765a7dac add debian stable help
add a little clue to debian stable users, to install 'go' from the repositories
2016-03-31 21:02:05 +02:00
Alexander Neumann
008337aad4 Merge pull request #487 from restic/use-fadvise
Purge read and written data from OS cache
2016-03-31 19:27:24 +02:00
Alexander Neumann
380e9b8119 Fix Makefile 2016-03-31 19:20:57 +02:00
Alexander Neumann
ddfadae6f6 Fix compilation for Go 1.3 2016-03-28 16:09:28 +02:00
Alexander Neumann
c8f46ce81d fs: Require Go1.4 for Linux 2016-03-28 15:48:18 +02:00
Alexander Neumann
c30f4a9134 fs: remove unneeded code 2016-03-28 15:33:10 +02:00
Alexander Neumann
b7713d2d34 local backend: Drop file content from cache after write 2016-03-28 15:31:25 +02:00
Alexander Neumann
5b5bb070b9 fs: Split out ClearCache from File 2016-03-28 15:31:25 +02:00
Alexander Neumann
feb664620a Use fadvise() to not cache the content of files read 2016-03-28 15:26:46 +02:00
Alexander Neumann
d2df2ad92d Add dep: golang.org/x/sys/unix 2016-03-28 15:25:45 +02:00
Alexander Neumann
9e81b158bf Add hints on how to get started 2016-03-12 11:28:10 +01:00
Alexander Neumann
49eb55c457 Merge pull request #484 from mholt/patch-1
Change ErrNoKeyFound message
2016-03-11 21:26:19 +01:00
Matt Holt
e6ba9e5849 Change ErrNoKeyFound message
For #438. I was just going to change it to "wrong password" but then I saw that it might actually be the case that no key could be found, so I changed it to what I did. Let me know if you'd like something different!
2016-03-11 08:21:01 -07:00
Alexander Neumann
a747cf994e Merge pull request #482 from restic/update-chunker
Update chunker
2016-03-10 11:02:19 +01:00
Alexander Neumann
afd0eb7f67 Merge pull request #481 from restic/fix-memory-usage
Use tempfiles for not-yet-uploaded pack files
2016-03-06 16:28:44 +01:00
Alexander Neumann
e4a6dd8c8c Use newRandReader instead of rand.New()
This needs to be done since for Go < 1.6 rand.Rand does not implement
io.Reader.
2016-03-06 14:21:02 +01:00
Alexander Neumann
18c3024171 Unexport NewPackerManager 2016-03-06 14:20:48 +01:00
Alexander Neumann
1e1368eea3 Add randReader for tests
This can be removed once we require Go 1.6.
2016-03-06 13:59:06 +01:00
Alexander Neumann
cda7616c82 Remove tempdir for packerManager 2016-03-06 13:14:06 +01:00
Alexander Neumann
015cea0c50 PackerManager: Remove debug comment 2016-03-06 12:35:21 +01:00
Alexander Neumann
c0b5f5a8af Fix all code which uses repository.New() 2016-03-06 12:34:23 +01:00
Alexander Neumann
f956f60f9f PackerManager: use tempfiles instead of memory buffers 2016-03-06 12:26:25 +01:00
Alexander Neumann
f893ec57cb Add test and benchmark for PackerManager 2016-03-05 15:58:39 +01:00
Alexander Neumann
9e24238cdd Update chunker 2016-03-05 13:46:20 +01:00
Alexander Neumann
4dac6d45fd Merge pull request #459 from restic/debug-434
Debug issue #434
2016-02-27 14:02:59 +01:00
Alexander Neumann
8d1a5731f3 Remove integration test for 'optimize' 2016-02-27 13:38:05 +01:00
Alexander Neumann
04b3ce00e2 Remove command 'optimize'
It was discovered that the current code may delete still-referenced
blobs, so we'll remove the command for now.

This closes #434
2016-02-27 13:12:22 +01:00
Alexander Neumann
9386bfbafa checker: Do not use reference in checker errors 2016-02-27 13:10:35 +01:00
Alexander Neumann
a613e23e34 checker: Use backend.IDSet instead of custom struct 2016-02-27 13:10:35 +01:00
Alexander Neumann
5ce1375ddd Rename non-exported function 2016-02-27 13:10:35 +01:00
Alexander Neumann
4cefd456bb Refactor rebuild-index code
This code reads all pack headers from all packs and rebuilds the index
from scratch. Afterwards, all indexes are removed. This is needed
because in #434 the command `optimize` produced a broken index that
did not contain a blob any more. Running `rebuild-index` should fix
this.
2016-02-27 13:10:35 +01:00
Alexander Neumann
bc911f4609 cmd_dump: Only load pack header 2016-02-27 13:06:21 +01:00
Alexander Neumann
090920039f pack: Add test with backend.NewReadSeeker
This uses the new backend ReadSeeker with the unpacker.
2016-02-27 13:06:21 +01:00
Alexander Neumann
21a99397ff worker: fix tests
The test failed on Windows, probably because the machine used for CI was
too slow. The new test doesn't depend on timing any more.
2016-02-27 13:06:21 +01:00
Alexander Neumann
482fc9f51d Add backend.readSeeker
This struct implements an io.ReadSeeker on top of a backend. This is the
easiest way to make the packer compatible to the new backend without
loading a complete pack into a bytes.Buffer.
2016-02-27 13:06:21 +01:00
Alexander Neumann
b114ab7108 cmd_dump: Allow dumping all blobs in all packs
I had the suspicion that one of my repositories contained redundant
data blobs that is not recorder in the index. In order to check this I
needed to dump information about the pack files without consulting the
index. The code here iterates over all packs and dumps the headers, and
dumps information about them in JSON (so we can use other tools to parse
that information again).
2016-02-27 13:06:21 +01:00
Alexander Neumann
4cb4a3ac7f Add separate goroutine that closes the output chan
This allows iterating over the output channel without having to start
another Goroutine outside of the worker pool. This also removes the need
for calling Wait().
2016-02-27 13:06:21 +01:00
Alexander Neumann
ee422110c8 Make worker pools input/output chans symmetric
Input and output channel are now both of type `chan Job`, this makes it
possible to chain multiple worker pools together.
2016-02-27 13:06:21 +01:00
Alexander Neumann
e5ee4eba53 Add worker pool
A worker pool is needed whenever something should be done concurrently.
This small library makes it easy to create a worker pool by specifying
channels, concurrency and a function that should be executed for each
job and returns a result and an error.
2016-02-27 13:06:21 +01:00
Alexander Neumann
1e0b7dbdd2 Merge pull request #477 from restic/rest-backend
rest backend: Remove indirection on http.Client
2016-02-25 22:24:00 +01:00
Alexander Neumann
5e152b7753 Merge pull request #476 from restic/fix-475
Ignore invalid index files
2016-02-25 17:56:49 +01:00
Alexander Neumann
17f5b524a6 local: Replace matching code with proper Readdir() 2016-02-24 22:43:04 +01:00
Alexander Neumann
4ae16d7661 repository: Use backend.ID to load index
This commit uses ParallelWorkFuncParseID() to load all indexes and
ignores file names with invalid format.

This fixes #475.
2016-02-24 22:41:32 +01:00
Alexander Neumann
77d85cee52 Merge pull request #472 from restic/update-chunker
Update chunker
2016-02-24 21:25:15 +01:00
Alexander Neumann
d84dec47bf Merge pull request #473 from restic/fix-coveralls
Fix coveralls.io service
2016-02-23 23:51:24 +01:00
Alexander Neumann
6c9170da51 Merge pull request #471 from fawick/move-cmd-folders
Refactor src/restic/cmd to src/cmds
2016-02-23 23:24:09 +01:00
Alexander Neumann
2ce49ea0ee Update code to use the new Chunker interface 2016-02-23 23:14:35 +01:00
Alexander Neumann
3db569c45a Update chunker 2016-02-23 23:14:35 +01:00
Alexander Neumann
98985019f9 Set GOPATH for goveralls 2016-02-23 23:03:38 +01:00
Alexander Neumann
699cb5ed8f Let goveralls fail if it needs to
Maybe this fixes coverage reporting...
2016-02-23 22:16:01 +01:00
Fabian Wickborn
6005bd9833 Fix relative paths in integrations tests 2016-02-23 09:21:50 +01:00
Fabian Wickborn
ee494ab939 Use relocated command folders in build.go and run_integration_tests.go
The resulting command structure is almost compatible to that of that gb
reference project (example-gsftp), as the subfolder for commands is
'cmds' instead of 'cmd'.
2016-02-23 09:18:09 +01:00
Fabian Wickborn
442780f214 Move commands to src/cmds 2016-02-23 07:21:28 +01:00
Alexander Neumann
9c47a8abfc Merge pull request #467 from fawick/master
Merging restic-server
2016-02-22 20:48:18 +01:00
Fabian Wickborn
dd5680dab6 restic-server: Create tmp folder 2016-02-22 20:14:11 +01:00
Fabian Wickborn
1cdbc8e1aa restic-server: Fix folder permissions 2016-02-22 20:14:11 +01:00
Fabian Wickborn
e4168fdde5 restic-server: Reduce memory footprint for saving blobs
Before, the restic-server read the whole blob (up to 8MB) into memory
prior to writing it to disk. Concurrent writes consumed a lot
of memory. This change writes the blob to a tmp file directly and
renames it afterwards in case there where no errors.
2016-02-22 20:14:11 +01:00
Fabian Wickborn
4749e610af restic-server: Fix content length for HEAD requests 2016-02-22 20:14:11 +01:00
Fabian Wickborn
51d86370a5 Fixes for the PR
- Removed external dependencies for test
- Prevent building restic-server w/ Go 1.3

Go versions 1.0, 1.1., and 1.2 are going to fail as well, but they
are "excluded" by README.md already.
2016-02-22 20:13:55 +01:00
Fabian Wickborn
d86c093480 Merged the restic-server by @bchapuis
Commit ID in fd0/restic-server at time of merge is
07fae00e7ddd8751b150e2ebf0bff8b2871c77ce
2016-02-22 20:12:50 +01:00
Alexander Neumann
bb7b9ef3fc Merge pull request #466 from ckemper67/sftp-path-clean
Cleaned up the sftp parsing logic.
2016-02-22 19:04:56 +01:00
Alexander Neumann
5dd65a5c19 Merge pull request #464 from restic/rest-backend
Add REST backend
2016-02-22 18:52:37 +01:00
Christian Kemper
6eb97ca6cc Cleaned up the sftp parsing logic.
Simplified and cleaned up the sftp parsing logic. Added support to
path.Clean the directory. Added additional tests.
2016-02-21 10:50:36 -08:00
Alexander Neumann
9a822285eb Merge pull request #468 from ckemper67/s3-logging
Added missing handle to the s3.Stat log message output
2016-02-21 19:13:10 +01:00
Christian Kemper
c2716755f1 Added missing handle to the s3.Stat log message output 2016-02-21 09:59:27 -08:00
Alexander Neumann
7087efaa79 rest backend: Remove indirection on http.Client 2016-02-21 17:06:35 +01:00
Alexander Neumann
921c2f6069 rest backend: Improve documentation 2016-02-21 17:03:27 +01:00
Alexander Neumann
8ad98e8040 rest backend: Fixes 2016-02-21 16:35:25 +01:00
Alexander Neumann
f7a10a9b9c backend tests: Test accessing config
This commit adds real testing for accessing the config file with
different names.
2016-02-21 16:02:13 +01:00
Alexander Neumann
bd621197f8 Add rest backend to ui parser 2016-02-21 15:33:13 +01:00
Alexander Neumann
ec34da2d66 Add rest backend to location 2016-02-21 15:33:13 +01:00
Alexander Neumann
c2348ba768 Add REST backend
This is a port of the original work by @bchapuis in
https://github.com/restic/restic/pull/253
2016-02-21 15:33:13 +01:00
Alexander Neumann
75d69639e6 .gitignore: Add /vendor/pkg 2016-02-21 15:33:13 +01:00
Alexander Neumann
9485fd0c4d Merge pull request #462 from restic/update-documentation
doc: Reduce text in README, add to Manual
2016-02-21 14:01:57 +01:00
Alexander Neumann
9b93b3a72c doc: Reduce text in README, add to Manual 2016-02-21 13:37:55 +01:00
Alexander Neumann
eaa2f899d5 Merge pull request #455 from restic/mkdocs
Add mkdocs for readthedocs.org
2016-02-21 13:14:14 +01:00
Alexander Neumann
1edf9c1ee4 doc: Add paragraph about mkdocs 2016-02-21 13:04:45 +01:00
Alexander Neumann
45e9561b48 doc: Add paragraph about documentation version 2016-02-21 12:58:42 +01:00
Alexander Neumann
7de8bf6c27 Manual: Correct headings, add section about debug 2016-02-21 12:56:25 +01:00
Alexander Neumann
1c2992e2e5 Update manual 2016-02-21 12:52:31 +01:00
Alexander Neumann
dc994699d9 Remove obsolete structure image 2016-02-21 12:29:13 +01:00
Alexander Neumann
a13f9f14d0 Add something to the front page 2016-02-21 12:28:46 +01:00
Alexander Neumann
e71e2c74f8 README: Add readthedocs badge 2016-02-21 12:28:46 +01:00
Alexander Neumann
7c4bd662cb Manual: Fix shell blocks and ToC 2016-02-21 12:28:46 +01:00
Alexander Neumann
6559fa7382 Add mkdocs for readthedocs.org 2016-02-21 12:28:46 +01:00
Alexander Neumann
b9eea24728 Merge pull request #460 from restic/add-github-issue-template
Add issue template
2016-02-21 12:23:24 +01:00
Alexander Neumann
8f33afead4 Add issue template 2016-02-21 00:35:58 +01:00
Alexander Neumann
625c987d23 Move sftp test 2016-02-20 20:53:40 +01:00
Alexander Neumann
933479047f Merge pull request #458 from restic/use-gb-vendor
Properly vendor dependencies with gb-vendor
2016-02-20 19:19:34 +01:00
Alexander Neumann
eef73d466d Properly vendor dependencies with gb-vendor 2016-02-20 18:33:06 +01:00
Alexander Neumann
134d129986 Merge pull request #445 from restic/switch-to-gb
Switch to gb
2016-02-20 18:23:40 +01:00
Alexander Neumann
4dffd3de66 Update .travis.yml 2016-02-20 17:56:11 +01:00
Alexander Neumann
cc8a929d43 Vagrantfile: Update to new structure (and Go version)
Also add /.vagrant to .gitignore
2016-02-20 17:44:48 +01:00
Alexander Neumann
1ab8220022 Update CONTRIBUTING.md 2016-02-20 17:31:39 +01:00
Alexander Neumann
6e31f4bb19 Dockerfile: Update for gb 2016-02-20 17:31:39 +01:00
Alexander Neumann
9bda164ed3 Update Appveyor configuration 2016-02-20 17:31:22 +01:00
Alexander Neumann
c51889c157 Ignore vendor/ directory for gofmt tests 2016-02-20 17:31:22 +01:00
Alexander Neumann
b3c2febf79 Add output of build.go to gitignore 2016-02-20 17:31:22 +01:00
Alexander Neumann
96e66bb3e9 Update build.go and run_integration_tests.go 2016-02-20 17:31:22 +01:00
Alexander Neumann
841326d713 Move build.go and run_integration_tests.go to root 2016-02-20 17:31:21 +01:00
Alexander Neumann
7b6629802b Move "doc" to root dir 2016-02-20 17:31:21 +01:00
Alexander Neumann
c0bd660a9e Rename package
* github.com/restic/restic -> restic
2016-02-20 17:31:21 +01:00
Alexander Neumann
0a8ef79dad Move top-level files 2016-02-20 17:31:21 +01:00
Alexander Neumann
b63399d606 Move things around for gb
This moves all restic source files to src/, and all vendored
dependencies to vendor/src.
2016-02-20 17:31:20 +01:00
Alexander Neumann
273d028a82 Merge pull request #451 from fawick/master
Support custom SSH ports in URL of SFTP-repo
2016-02-16 21:13:42 +01:00
Fabian Wickborn
2d94e71117 Support custom SSH ports in URL of SFTP-repo 2016-02-15 19:17:41 +01:00
Alexander Neumann
d918d0f0c3 Merge pull request #443 from ckemper67/fix-387
Introduced a configurable object path prefix for s3 repositories to address #387
2016-02-14 22:43:40 +01:00
Alexander Neumann
517eff7e48 Merge pull request #447 from ckemper67/fix-442
Fix #442: set blocks and blocksize in fuse
2016-02-14 22:25:37 +01:00
Christian Kemper
1f81865847 set Blocks and BlockSize in Attr using a default blocksize of 512 to address #442. 2016-02-14 13:14:08 -08:00
Christian Kemper
f3f1404849 always use "restic" as the default prefix for parsing.
improved test error message to make it easier to find the problematic pattern
2016-02-14 11:26:46 -08:00
Christian Kemper
24b7514fe0 cleaner sharing of s3: and s3:// configuration 2016-02-14 09:45:58 -08:00
Christian Kemper
91dc14c9fc fix Hound warning 2016-02-14 09:27:00 -08:00
Christian Kemper
32c2cafa89 Simplify creation of the Config by moving it to a separate function. Simplify the parsing logic
by sharing the handling of s3: and s3://
2016-02-14 09:18:22 -08:00
Christian Kemper
74608531c7 strip off the trailing slash for the object prefix 2016-02-14 07:16:50 -08:00
Christian Kemper
48f85fbb09 replaced if-else chain with switch 2016-02-14 07:01:14 -08:00
Christian Kemper
535dfaf097 address first round of review comments 2016-02-14 06:40:15 -08:00
Christian Kemper
8f5ff379b7 Introduced a configurable object path prefix for s3 repositories.
Prepends the object path prefix to all s3 paths and allows to have multiple independent
restic backup repositories in a single s3 bucket.

Removed the hardcoded "restic" prefix from s3 paths.

Use "restic" as the default object path prefix for s3 if no other prefix gets specified.
This will retain backward compatibility with existing s3 repository configurations.

Simplified the parse flow to have a single point where we parse the bucket name and the prefix within the bucket.

Added tests for s3 object path prefix and the new default prefix to config_test and location_test.
2016-02-14 06:05:38 -08:00
Alexander Neumann
b7260cafac CONTRIBUTING: Add paragraph about PRs and tests 2016-02-14 11:49:10 +01:00
Alexander Neumann
3d27751b69 Merge pull request #441 from restic/add-note-gccgo
README: Add note about gccgo
2016-02-13 20:21:58 +01:00
Alexander Neumann
4053fe0a9b Merge pull request #440 from restic/fix-431
sftp: Use os.IsNotExist() for Test()
2016-02-13 19:45:14 +01:00
Alexander Neumann
e75856a7d2 README: Add note about gccgo
Closes #432
2016-02-13 19:20:25 +01:00
Alexander Neumann
50380f8c14 Merge pull request #439 from restic/fix-402
Add cleanup handler to restore terminal state
2016-02-13 19:14:59 +01:00
Alexander Neumann
4032af0b78 sftp: Use os.IsNotExist() for Test()
The sftp library introduced a change so that the error returned for
(among others) Lstat() can be used with os.IsNotExist() to test whether
the target file does not exist.
2016-02-13 19:11:41 +01:00
Alexander Neumann
2bb55f017d Update pkg/sftp library 2016-02-13 19:11:35 +01:00
Alexander Neumann
1287b307ac Add cleanup handler to restore terminal state
Closes #402
2016-02-13 18:29:26 +01:00
Alexander Neumann
24618305cc Merge pull request #429 from restic/fix-428
backup: Hide status output for narrow terminals
2016-02-10 18:02:18 +01:00
Alexander Neumann
4ec5050cbb backup: Hide status output for narrow terminals
closes #428
2016-02-10 17:28:48 +01:00
Alexander Neumann
eccbcb73a1 Merge pull request #425 from restic/fix-loadall
backend.LoadAll: return nil on expected error
2016-02-08 00:37:04 +01:00
Alexander Neumann
e781e1cf1d Merge pull request #424 from restic/fix-backup
Fix backup of root directory
2016-02-08 00:06:19 +01:00
Alexander Neumann
e9a21c1dc6 backend.LoadAll: return nil on expected error
The current code returns io.ErrUnexpectedEOF, but it is the normal,
expected behaviour of the function LoadAll() to load until the item is
completely loaded. Therefore, the io.ErrUnexpectedEOF is not returned to
the caller.
2016-02-07 23:48:54 +01:00
Alexander Neumann
a37ed45534 Add test for LoadAll with too large buffer
LoadAll() should not pass on io.ErrUnexpectedEOF, since the occurrence
of this error is normal.
2016-02-07 23:48:03 +01:00
Alexander Neumann
26484d0c7b pipe: Ignore excluded directories
When top-level directories are ignored, they were still added to the
top-level Dir{} object. This commit ignores them.
2016-02-07 23:23:16 +01:00
Alexander Neumann
68db75b4e3 pipe/archiver: Add more debug messages 2016-02-07 23:22:06 +01:00
Alexander Neumann
9048eb676b pipe: join replaced paths with original path
When saving `/`, it was replaced with the contents, but without the
proper path. So `/` was replaced by [`boot`, `bin`, `home`, ...], but
without prefixing the entry name with the proper path.
2016-02-07 22:18:37 +01:00
Alexander Neumann
6a5b022939 archiver: Add error reporting for directories
When an error occurred while walking a directory, this error wasn't
reported to the user before.
2016-02-07 22:18:00 +01:00
Alexander Neumann
57a24b2cdf Merge pull request #421 from restic/fix-fuse-with-special-names
fuse: Replace special node names with their content
2016-02-07 20:43:29 +01:00
Alexander Neumann
da47389483 Merge pull request #420 from restic/archiver-unique-paths
archiver: deduplicate list of paths to save
2016-02-07 20:31:18 +01:00
Alexander Neumann
4c329110c5 Merge pull request #419 from restic/fix-backup-root
Fix backup of "/"
2016-02-07 20:31:08 +01:00
Alexander Neumann
46fbae0d71 Merge pull request #418 from benmur/show-pack-name-on-rebuild-index-error
Handle pack loading errors in rebuild-index
2016-02-07 19:59:01 +01:00
Alexander Neumann
1835e988cf fuse: Replace special node names with their content
This is the companion fix for #419 and allows mounting repositories with
special directory names directly below the snapshot.

Closes #403
2016-02-07 19:51:29 +01:00
Alexander Neumann
537347d9b5 archiver: deduplicate list of paths to save 2016-02-07 19:35:35 +01:00
Alexander Neumann
811dbfa52d Make TestWalkerPath absolute before walking 2016-02-07 19:34:02 +01:00
Rached Ben Mustapha
ba35ca522a Handle pack loading errors in rebuild-index
Errors returned from backend.LoadAll() were not handled, leading to
these fatal errors from the unpacker trying to read the size from the end of
an empty buffer:

`seeking to read header length failed: bytes.Reader.Seek: negative position`

This change takes care of returning on error, as well as showing which pack
failed to load and validating pack integrity.
2016-02-07 18:30:47 +00:00
Alexander Neumann
c6a1f2e2f3 pipe: Handle special paths gracefully
This fixes handling special paths "." and "/". When such a path name is
found, it is replaced by the contents of the path before walking.
2016-02-07 19:30:00 +01:00
Alexander Neumann
604c27f001 Test pipe walker for invalid paths
It was discovered that when restic is instructed to save `/`, the tree
structures in the repository contain an invalid node.

When saving the dir `/home/user`, the following structure is created:

    snapshot
         -> tree
             nodes: ["user"]
             [...]

When the root directory `/` is saved, the structure is as follows:

    snapshot
         -> tree
             nodes: ["/"]
             [...]

This behavior is caused by the walker in pipe.go sending a node with the
name "." to the archiver, so this commit adds a test for invalid node
names.
2016-02-07 19:29:44 +01:00
Alexander Neumann
0535490618 Merge pull request #413 from restic/reduce-travis-matrix
Reduce jobs run on Travis
2016-02-06 14:10:54 +01:00
Alexander Neumann
9c0fc4930b Merge pull request #412 from restic/activate-hound
Enable HoundCI checking for Go
2016-02-06 13:37:13 +01:00
Alexander Neumann
a45f2cb205 Reduce jobs run on Travis
Skip building and testing restic with Go 1.3, 1.4 and 1.6 on osx.
2016-02-06 13:04:49 +01:00
Alexander Neumann
3fd5b5975a Merge pull request #411 from restic/fix-appveyor-sourceforge
Always use NetCologne SourceForge mirror
2016-02-06 13:01:55 +01:00
Alexander Neumann
9175f0b6af Enable HoundCI checking for Go 2016-02-05 21:15:46 +01:00
Alexander Neumann
1160d03279 Always use NetCologne SourceForge mirror
The one automatically selected by the SourceForge CDN fails currently:

    appveyor DownloadFile http://downloads.sourceforge.net/project/gnuwin32/tar/1.13-1/tar-1.13-1-bin.zip -FileName tar.zip
    Error downloading file: Unable to connect to the remote server Command exited with code 2
2016-02-05 21:13:14 +01:00
Alexander Neumann
789e0df49e Merge pull request #408 from benmur/validate-pack-checksums
checker: Validate pack checksums before unpacking
2016-02-05 19:45:21 +01:00
Rached Ben Mustapha
83bbf21f1a checker: Validate pack checksums before unpacking
This avoids reading a possibly invalid size at the end of a corrupted pack
2016-02-04 22:55:39 +01:00
Alexander Neumann
7a8054d678 Add link to record of talk at C4 Cologne 2016-02-04 19:56:41 +01:00
Alexander Neumann
c0bbb7254d Merge pull request #406 from restic/fix-405
Test and fix for #405
2016-02-02 20:53:52 +01:00
Alexander Neumann
4f1f03cdb9 Move testing for known blobs to Archiver
This removes the list of in-flight blobs from the master index and
instead keeps a list of "known" blobs in the Archiver. "known" here
means: either already processed, or included in an index. This property
is tested atomically, when the blob is not in the list of "known" blobs,
it is added to the list and the caller is responsible to make this
happen (i.e. save the blob).
2016-02-01 23:50:56 +01:00
Alexander Neumann
382c766983 Move test for #405: Test Archiver instead of Repo 2016-02-01 23:50:41 +01:00
Alexander Neumann
f5f6e9cf37 Add test to reproduce #405 2016-02-01 23:35:01 +01:00
Alexander Neumann
cf88b33383 Merge pull request #398 from restic/update-minio-go
Update minio-library
2016-01-29 13:14:37 +01:00
Alexander Neumann
57615edd3a Merge pull request #400 from restic/update-go-appveyor
Update Go version for CI tests
2016-01-29 13:13:42 +01:00
Alexander Neumann
49cb88d158 Add note about lightning talk at FOSDEM 2016-01-28 22:40:52 +01:00
Alexander Neumann
1464d84cf5 Update minio-go
This should work with Go 1.3/1.4 again
2016-01-28 22:38:29 +01:00
Alexander Neumann
74ce027924 Merge pull request #397 from restic/remove-readcloser
Remove backend.ReadCloser, this is not used any more
2016-01-28 22:31:30 +01:00
Alexander Neumann
39f698886a Update Travis Go version 2016-01-28 22:28:17 +01:00
Alexander Neumann
9f8f2bc874 Update Go version for appveyor 2016-01-28 00:00:28 +01:00
Alexander Neumann
f8daadc5ef Update minio-library
This addresses #388
2016-01-27 23:23:47 +01:00
Alexander Neumann
f2371db2a9 repository: remove decryptReadCloser 2016-01-27 22:47:09 +01:00
Alexander Neumann
7d1775e000 Remove backend.ReadCloser 2016-01-27 22:35:18 +01:00
Alexander Neumann
ce4a7f16ca Merge pull request #395 from restic/rework-backend-interface
WIP: Rework backend interface
2016-01-27 22:11:20 +01:00
Alexander Neumann
322eca86bc mem backend: remove unused code 2016-01-27 21:33:48 +01:00
Alexander Neumann
3d06e6083a CI: download minio for the correct os and arch 2016-01-26 23:52:39 +01:00
Alexander Neumann
b64006221c CI: Download minio server, do not compile latest master 2016-01-26 23:50:27 +01:00
Alexander Neumann
1fde872016 CI: only build minio on Go 1.5.1 and above 2016-01-26 22:36:06 +01:00
Alexander Neumann
2701eabe39 Remove ContinuousReader 2016-01-26 22:35:51 +01:00
Alexander Neumann
c388101217 s3: Unexport structure 2016-01-26 22:19:44 +01:00
Alexander Neumann
1528d1ca83 sftp: Reduce duplicate code, add error check 2016-01-26 22:16:24 +01:00
Alexander Neumann
0bbad683c5 local: split out tempfile write function 2016-01-26 22:12:53 +01:00
Alexander Neumann
9ec435d863 local: remove duplicate code 2016-01-26 22:09:29 +01:00
Alexander Neumann
9b1c4b2dd6 local: Remove mutex and hash of open files 2016-01-26 22:08:20 +01:00
Alexander Neumann
7196971159 Remove unneeded HashingReader implementation 2016-01-26 22:00:11 +01:00
Alexander Neumann
eb1669a061 Add a lot of comments 2016-01-26 21:56:13 +01:00
Alexander Neumann
c34aa72538 Remove duplicate function str2id 2016-01-26 21:52:02 +01:00
Alexander Neumann
da883d6196 Cleanups, move Hash() to id.go 2016-01-26 21:49:33 +01:00
Alexander Neumann
b482df04ec Add more documentation 2016-01-26 21:49:22 +01:00
Alexander Neumann
5fcb5ae549 Reduce number of tests for Load() 2016-01-24 21:40:54 +01:00
Alexander Neumann
a0d484113a backends: Do not sort strings
Closes #305
2016-01-24 21:32:45 +01:00
Alexander Neumann
d9c87559b5 s3/local backend: Fix error for overwriting files 2016-01-24 21:13:24 +01:00
Alexander Neumann
1547d3b656 Remove Create() everywhere 2016-01-24 20:23:50 +01:00
Alexander Neumann
ea29ad6f96 Remove last ocurrence of Create() 2016-01-24 19:30:14 +01:00
Alexander Neumann
1a95e48389 Remove unneeded special readers 2016-01-24 18:58:15 +01:00
Alexander Neumann
ac2fe4e04f Remove BlobWriter 2016-01-24 18:53:39 +01:00
Alexander Neumann
cfdd3a853d Remove usage of CreateEncryptedBlob() 2016-01-24 18:52:11 +01:00
Alexander Neumann
01e40e62bf repo: Use Save() instead of Create() 2016-01-24 18:50:41 +01:00
Alexander Neumann
35f9eae6c3 local backend: do not call Sync() on directory
This fails at least on Windows.
2016-01-24 18:01:00 +01:00
Alexander Neumann
fe565e17c3 Key: Use Save() instead of Create() 2016-01-24 17:52:44 +01:00
Alexander Neumann
4735a7f9b5 Improve random reader for tests 2016-01-24 17:47:45 +01:00
Alexander Neumann
54f8860612 backends: Add Save() 2016-01-24 16:59:38 +01:00
Alexander Neumann
ed172c06e0 backends: Add Save() function 2016-01-24 01:15:35 +01:00
Alexander Neumann
adbe9e2e1c backend: Remove GetReader 2016-01-24 01:00:27 +01:00
Alexander Neumann
2c3a6a6fa9 cmd_rebuild_index: Remove calls to GetReader() 2016-01-24 00:42:04 +01:00
Alexander Neumann
61551b0591 cmd_cat: Remove calls to GetReader() 2016-01-24 00:42:04 +01:00
Alexander Neumann
280d580ae2 checker: Use Load() instead of GetReader() 2016-01-24 00:42:04 +01:00
Alexander Neumann
782a1bf7b0 repository: remove GetDecryptReader() 2016-01-24 00:12:17 +01:00
Alexander Neumann
3191778d33 repository: Use Load() instead of GetReader() 2016-01-24 00:12:09 +01:00
Alexander Neumann
9bfa633187 repository/key: Use Load() instead of GetReader() 2016-01-23 23:48:19 +01:00
Alexander Neumann
9209dcfa26 Add LoadAll() 2016-01-23 23:41:55 +01:00
Alexander Neumann
919b40c6cf Add Stat() method to backend interface 2016-01-23 23:27:58 +01:00
Alexander Neumann
10b03eee27 Add comment 2016-01-23 23:27:40 +01:00
Alexander Neumann
0b50f9e02c Move MemoryBackend to backend/mem 2016-01-23 19:50:11 +01:00
Alexander Neumann
f05a32509e Add "Test" prefix to backend test functions 2016-01-23 19:12:02 +01:00
Alexander Neumann
e4f2e4a203 Remove old s3 tests 2016-01-23 19:11:47 +01:00
Alexander Neumann
6ba56befad Abort fuse integration test on error
Before, the fuse integration test was run and the tests were never
finished, because the testing code did not detect any errors when the
fusermount binary returned an error. This commit fixes it.
2016-01-23 19:10:43 +01:00
Alexander Neumann
15c8b85a4b Add tests for s3 backend 2016-01-23 18:46:04 +01:00
Alexander Neumann
c6db567e3f Add sftp tests 2016-01-23 18:30:02 +01:00
Alexander Neumann
4952f86682 Add test for to prevent double create 2016-01-23 18:07:15 +01:00
Alexander Neumann
16b7cc7655 Remove redundant local tests 2016-01-23 17:45:33 +01:00
Alexander Neumann
99fab793c0 Remove timestamp from generated tests 2016-01-23 17:43:49 +01:00
Alexander Neumann
9423767827 Update test generate script, add tests to membackend 2016-01-23 17:42:26 +01:00
Alexander Neumann
e966df3fed Add Load() to MemBackend 2016-01-23 17:19:55 +01:00
Alexander Neumann
3aafa21887 Fix MockBackend.Load() 2016-01-23 17:19:47 +01:00
Alexander Neumann
9a490f9e01 Implement package-local tests 2016-01-23 17:08:03 +01:00
Alexander Neumann
0a24261afb Add Load() for all existing backends 2016-01-23 14:12:12 +01:00
Alexander Neumann
8b7bf8691d backend: Remove Get()
This is the first commit that removes the (redundant) Get() method of
the backend interface. Get(x, y) is equivalent to GetReader(x, y, 0, 0).
2016-01-23 13:13:05 +01:00
Alexander Neumann
d3a6e2a991 Drop requirement from List()
Closes #305
2016-01-23 12:47:16 +01:00
Alexander Neumann
171cd0dfe1 Add backend.Handle, add comments 2016-01-23 12:46:20 +01:00
Alexander Neumann
4d7e802c44 Merge pull request #392 from restic/fix-build
Allow saving duplicate blobs in the repacker
2016-01-17 22:02:09 +01:00
Alexander Neumann
109a120b39 Fix RandomReader 2016-01-17 21:27:51 +01:00
Alexander Neumann
f53008d916 Allow saving duplicate blobs in the repacker
This adds code to the master index to allow saving duplicate blobs
within the repacker. In this mode, only the list of currently in flight
blobs is consulted, and not the index. This correct because while
repacking, a unique list of blobs is saved again to the index.
2016-01-17 21:14:55 +01:00
Alexander Neumann
34c1056efc Merge pull request #358 from episource/iss358_pack_not_referened_add_test
Closes #365
Closes #358
2016-01-17 20:07:56 +01:00
Alexander Neumann
00e7a76ecc Merge branch 'iss358_pack_not_referenced_fix' of https://github.com/episource/restic into episource-358 2016-01-17 20:07:31 +01:00
Alexander Neumann
e689d499e7 Improve RandomReader 2016-01-17 19:46:48 +01:00
Alexander Neumann
5df9bdec9a Merge pull request #366 from restic/howeyc-s3-minio
rebase: Switch s3 library to allow for s3 compatible backends
2016-01-17 19:21:13 +01:00
Alexander Neumann
c722851f92 Update Dockerfile 2016-01-17 18:50:50 +01:00
Alexander Neumann
877f3f61a0 Add flag to disable cross-compilation 2016-01-17 18:49:43 +01:00
Alexander Neumann
1dd4c52a8b Add comments, configure flag library 2016-01-17 18:48:05 +01:00
Alexander Neumann
c6e1696f07 Fix debug message 2016-01-17 18:48:05 +01:00
Alexander Neumann
1483e15e4e Update s3 library (again) 2016-01-17 18:48:05 +01:00
Alexander Neumann
6a56d5b87b Repo: Add more debug 2016-01-17 18:48:05 +01:00
Alexander Neumann
289aee9448 Adapt s3 backend to new library 2016-01-17 18:48:05 +01:00
Alexander Neumann
0e9236475b Update s3 library (again) 2016-01-17 18:48:05 +01:00
Alexander Neumann
181480b68b Update s3 library 2016-01-17 18:48:05 +01:00
Alexander Neumann
61e66e936f Fix imports 2016-01-17 18:48:05 +01:00
Alexander Neumann
314182e7e0 Add debug, do not create bucket if it already exists 2016-01-17 18:48:05 +01:00
Alexander Neumann
69e6e9e5c7 Update s3 library (again) 2016-01-17 18:48:05 +01:00
Alexander Neumann
fc347ba60f Add new test with multiple writes for backends 2016-01-17 18:48:05 +01:00
Alexander Neumann
26eb859663 Dockerfile: Add sftp server binary 2016-01-17 18:48:05 +01:00
Alexander Neumann
338ad42273 location: fix tests 2016-01-17 18:48:05 +01:00
Alexander Neumann
5722ccfcda Fix s3 backend, add more tests 2016-01-17 18:48:05 +01:00
Alexander Neumann
0237b0d972 Update s3 library again 2016-01-17 18:48:05 +01:00
Alexander Neumann
a850041cf0 ContReader: Remove debug output 2016-01-17 18:48:05 +01:00
Alexander Neumann
5071f28d55 ReadCloser: Call close if reader implements it 2016-01-17 18:48:05 +01:00
Alexander Neumann
e0361b1f9f Add ContinuousReader 2016-01-17 18:48:05 +01:00
Alexander Neumann
f319354174 Update s3 library again 2016-01-17 18:48:05 +01:00
Alexander Neumann
a73c4bd5a7 update s3 library for bugfix 2016-01-17 18:48:05 +01:00
Alexander Neumann
d79c85af62 Fix s3 tests 2016-01-17 18:48:05 +01:00
Alexander Neumann
407819e5a9 s3: properly integrate minio-go lib 2016-01-17 18:48:05 +01:00
Alexander Neumann
2c15597e24 walker: print errors 2016-01-17 18:48:05 +01:00
Alexander Neumann
a17b6bbb64 Update minio-go library 2016-01-17 18:48:05 +01:00
Alexander Neumann
1922a4272c s3: fix usage
Ignore error response for existing bucket, add more debug code.
2016-01-17 18:48:05 +01:00
Alexander Neumann
2b10791df2 location: Fix test 2016-01-17 18:48:05 +01:00
Alexander Neumann
1ad5c3813c correct CI s3 test server url 2016-01-17 18:48:05 +01:00
Alexander Neumann
7d5f8214cf use new backend open with config 2016-01-17 18:48:05 +01:00
Alexander Neumann
2b0b44c5ce s3: implement open with config 2016-01-17 18:48:05 +01:00
Alexander Neumann
f7c9091970 sftp: implement open with config 2016-01-17 18:48:05 +01:00
Alexander Neumann
7b1e8fdd06 local: correct comment 2016-01-17 18:48:05 +01:00
Alexander Neumann
d257dedf42 rename LocationParse -> Parse 2016-01-17 18:48:05 +01:00
Alexander Neumann
3d2a714b5a Update minio-go library 2016-01-17 18:48:05 +01:00
Alexander Neumann
de933a1d48 Rename URI -> Config/Location 2016-01-17 18:48:05 +01:00
Alexander Neumann
566a15285a Add repository location parsing code 2016-01-17 18:48:05 +01:00
Alexander Neumann
43cf95e3c6 Correctly stop the minio server after the tests 2016-01-17 18:48:05 +01:00
Alexander Neumann
0b12ceabe9 Dockerfile: Install go in home dir
This allows cross-compilation with gox with Go < 1.5
2016-01-17 18:48:05 +01:00
Alexander Neumann
e96f28c536 Output stderr when minio server failed 2016-01-17 18:48:05 +01:00
Alexander Neumann
d5e36bd2f0 Only run minio server for Go >= 1.5.1 2016-01-17 18:48:05 +01:00
Alexander Neumann
34e8f63f77 Increase debug output for minio server 2016-01-17 18:47:24 +01:00
Alexander Neumann
3e422c8776 Add debug output, listen on localhost 2016-01-17 18:47:24 +01:00
Alexander Neumann
edfb31f4fe s3: Run integration test with minio server 2016-01-17 18:47:24 +01:00
Alexander Neumann
8562a1bb2f Dockerfile: Also install minio 2016-01-17 18:46:08 +01:00
Alexander Neumann
fa7192fdfb CI: save cross-compiled binaries in /tmp 2016-01-17 18:46:08 +01:00
Alexander Neumann
c22c0f2706 Add Dockerfile that resembles the Travis environment 2016-01-17 18:46:08 +01:00
Alexander Neumann
5736742c3e s3: Open() creates bucket if it does not exist 2016-01-17 18:46:08 +01:00
Alexander Neumann
248f991ad4 s3: don't remove the bucket on Delete() 2016-01-17 18:46:08 +01:00
Alexander Neumann
55f10eb1c1 Fix s3 test with local minio server instance 2016-01-17 18:46:08 +01:00
Alexander Neumann
d0ca118387 Fix usage of the done chan 2016-01-17 18:46:08 +01:00
Chris Howey
69a9adc4c3 Use local instance of minio server.
Need to figure out how to have tests automatically start and kill
server.
2016-01-17 18:46:08 +01:00
Chris Howey
e2445f4c97 GetPartialObject does not work. 2016-01-17 18:46:08 +01:00
Chris Howey
ed2a4ba1d5 Fix s3 backend test 2016-01-17 18:46:08 +01:00
Chris Howey
6d1552af51 Switch s3 library to allow for s3 compatible backends. Fixes #315 2016-01-17 18:46:08 +01:00
Alexander Neumann
c969de7fad Merge pull request #390 from restic/fix-travis
Fix travis
2016-01-16 14:39:51 +01:00
Alexander Neumann
b8c300e61e Remove run_tests.go from Makefile 2016-01-16 14:37:23 +01:00
Alexander Neumann
2499bbb09d Also specify new -X syntax for go1.6 2016-01-16 14:08:13 +01:00
Alexander Neumann
7c70d5c1bd Build toolchain for gox only on older Versions of Go 2016-01-16 13:40:16 +01:00
Alexander Neumann
f90381910b Remove Go tip, add 1.6beta2 2016-01-16 13:39:12 +01:00
Alexander Neumann
172c31ff45 Use gotestcover instead of homebrew run_tests.go 2016-01-16 13:32:23 +01:00
Alexander Neumann
bbfd1dd0c0 Fix ignore tip build failure 2016-01-16 13:23:45 +01:00
Alexander Neumann
8d71e5d698 Travis CI: Update Go version, add tip 2016-01-16 13:00:28 +01:00
Alexander Neumann
0f69169262 OpenChaos lecture 2016-01-13 20:16:47 +01:00
Alexander Neumann
72bcebbfb1 Remove (broken) sourcegraph and waffle badges 2016-01-07 21:09:32 +01:00
Philipp Serr
0fde09a866 Lock MasterIndex and InFlight store together
fixes: #358
2015-12-28 18:40:43 +01:00
Philipp Serr
e7bf936d2b Increase number of chunks and test repetitions 2015-12-28 18:33:28 +01:00
Philipp Serr
3d7f72311a Provoke unreferenced packs using fewer goroutines
TestParallelSaveWithDuplication has been reworked to provoke
unreferenced packs using fewer goroutines than before and create
only one bytes.Reader per blob. This reduces memory usage
significantly.

The following actions have been taken to keep the chance of provoking
unreferenced packs due to #358 high:
 * Interweaved processing of subsequent chunks
 * Delaying each goroutine by a few pseudo-randomly chosen nanoseconds
   (depending on the platform this will most probably only make the os
   yield execution to another thread): together with the interweaved
   processing of subsequent chunks, this ensures a minimalistic delay
   between processing of (some) duplicated chunks
 * Repeating the test 5 times with different seeds

On my test machine, the modified test provoked unreferenced packs 60
times in a row.
2015-12-28 18:33:26 +01:00
Philipp Serr
6a548336ec Add a test concurrently saving duplicated chunks
This commit adds an integration test, that calls Archiver.Save from
many goroutines processing several duplicated chunks concurrently.
The test asserts, that after all chunks have been saved, there are no
unreferenced packs in the repository.

The test has been checked to give the expected results:
 1) Running the test with maxParallel=1 (all chunks are processed
    sequentially) has been verified not to produce any unreferenced
    packs. Consequently the test passes.
 2) Running the test with unbounded parallelism (maxParallel=
    math.MaxInt32) has been verified to produce unreferenced packs
    all the time (at least 25 test runs). Consequently the test fails
    due to #358.

references: #358
2015-12-28 18:33:22 +01:00
Alexander Neumann
d3e7766f89 Merge pull request #380 from restic/PKGBUILD-update
Update PKGBUILD to reflect restic official version numbering
2015-12-27 22:07:28 +01:00
Florian Daniel
360193320f Update PKGBUILD to reflect restic official version numbering 2015-12-27 22:05:05 +01:00
Alexander Neumann
1f1b8e16a7 Add Code Quality Badge
Closes #379
2015-12-27 20:35:27 +01:00
Alexander Neumann
3abff7928c Merge pull request #375 from restic/fix-suid
Backup and restore setuid/setgid/sticky bits
2015-12-20 20:45:18 +01:00
Alexander Neumann
2976df2dc6 Call brew update before installing 2015-12-20 20:01:13 +01:00
Alexander Neumann
f49cb62812 Backup and restore setuid/setgid/sticky bits
A user discovered that restic does not restore setuid/setgid/sticky file
attributes. This commit fixes that. The mode is stored in the Go format
as an uint32: https://golang.org/pkg/os/#FileMode
2015-12-20 19:45:36 +01:00
Alexander Neumann
55d9c5f80c Merge pull request #364 from restic/fix-356
Allow reading all data
2015-12-06 21:55:21 +01:00
Alexander Neumann
d1ca986f55 Fix tests 2015-12-06 17:38:51 +01:00
Alexander Neumann
3ac1d0e4d1 add progress 2015-12-06 17:29:31 +01:00
Alexander Neumann
0e66a66bce read packs concurrently 2015-12-06 17:09:06 +01:00
Alexander Neumann
43a23f91a6 checker: add function to read and verify all data 2015-12-02 22:40:36 +01:00
Alexander Neumann
8d229bfd21 Make ReadCloser public 2015-12-02 22:37:58 +01:00
Alexander Neumann
04cd318f6c Merge pull request #362 from restic/improvements
Cleanups
2015-11-30 22:09:24 +01:00
Alexander Neumann
141d400b4a test: improvements 2015-11-29 14:54:20 +01:00
Alexander Neumann
b841eb4c54 crypto: check key for validity 2015-11-29 14:54:20 +01:00
Alexander Neumann
4f6bc754b8 MemBackend: Add Delete() and more debug 2015-11-29 14:53:02 +01:00
Alexander Neumann
26697a0223 Fix MemoryBackend GetReader() method 2015-11-29 14:52:19 +01:00
Alexander Neumann
480054bc3a MemoryBackend: handle config correctly, add tests for that 2015-11-29 14:52:19 +01:00
Alexander Neumann
538e5878a1 add debug logging to MemoryBackend 2015-11-29 14:52:19 +01:00
Alexander Neumann
9cb4e14327 add MemBackend and MockBackend 2015-11-29 14:52:19 +01:00
Alexander Neumann
da71da23d9 Add MockBackend 2015-11-29 14:52:19 +01:00
Alexander Neumann
0d5731383f Remove HashAppendWriter 2015-11-29 14:29:59 +01:00
Alexander Neumann
4fd7676e92 HashingWriter: Add documentation 2015-11-29 14:29:59 +01:00
Alexander Neumann
8209bb309b split out decryptReader and packerManager 2015-11-29 14:29:59 +01:00
Alexander Neumann
d4b873ca76 Repo: add documentation 2015-11-29 14:29:57 +01:00
Alexander Neumann
5b601f00b1 Add error checking 2015-11-29 14:25:57 +01:00
Alexander Neumann
2c95772a6a Update README, instruct users to always open an issue 2015-11-22 19:23:48 +01:00
Alexander Neumann
867fc5bd4b Merge pull request #354 from restic/fix-index
Save new packs in index atomically
2015-11-20 23:20:27 +01:00
Alexander Neumann
567de35df4 Save new packs in index atomically
This commit fixes a situation reported by a user where two indexes
contained information about the same pack without overlap, e.g.:

Index 3e6a32 contained:

    {
      "id": "c02e3b",
      "blobs": [
        {
          "id": "8114b1",
          "type": "data",
          "offset": 0,
          "length": 530107
        }
      ]
    }

And index 62da5f contained:

    {
      "id": "c02e3b",
      "blobs": [
        {
          "id": "e344f8",
          "type": "data",
          "offset": 1975848,
          "length": 3426468
        },
        {
          "id": "939ed9",
          "type": "data",
          "offset": 530107,
          "length": 1445741
        }
      ]
    }

This commit adds all blobs in a pack in one atomic operation so that
intermediate such as these do not happen.
2015-11-20 22:56:56 +01:00
Alexander Neumann
34bf70faea Merge pull request #352 from restic/rework-readme
Rework README
2015-11-13 23:48:17 +01:00
Alexander Neumann
6b9c8ffd14 Merge pull request #350 from restic/allow-read-only-ops
Allow check/restore read-only, without locking
2015-11-13 23:48:05 +01:00
Alexander Neumann
96061d2a2f Fix debug log message 2015-11-13 23:47:53 +01:00
Alexander Neumann
88b167cc10 Add note about unreleased code in the master branch 2015-11-13 12:41:47 +01:00
Alexander Neumann
a79fba13e1 Shorten README, remove options 2015-11-13 12:41:36 +01:00
Alexander Neumann
a1440c819b Make NoLock a global option 2015-11-13 12:33:59 +01:00
Alexander Neumann
6edb7e02d0 Allow restoring and listing without locking 2015-11-10 22:13:53 +01:00
Alexander Neumann
c79dcbd7c4 Allow checking a read-only repo with --no-lock 2015-11-10 21:41:22 +01:00
Alexander Neumann
acba82c8f7 Merge pull request #252 from restic/repack-blobs
WIP: Repack blobs
2015-11-09 20:57:57 +01:00
Alexander Neumann
1f9aea9905 fix test on windows, reset read-only flag 2015-11-08 22:41:45 +01:00
Alexander Neumann
742d69bf4d Add another test for optimizing unused blobs 2015-11-08 22:38:17 +01:00
Alexander Neumann
5776b8f01c remove ConvertIndex 2015-11-08 22:27:13 +01:00
Alexander Neumann
6c54d3fa82 index: also mark old index as final on decode 2015-11-08 22:21:29 +01:00
Alexander Neumann
2e6eee991d Add test for optimize command with old indexes 2015-11-08 22:21:08 +01:00
Alexander Neumann
c59b12c939 Show a hint whech the checker finds an old index 2015-11-08 21:50:48 +01:00
Alexander Neumann
0222b1701e Remvoe automatic index conversion 2015-11-08 21:35:48 +01:00
Alexander Neumann
43e2c9837e check: removing orphaned packs is handled in 'optimize' 2015-11-08 21:24:51 +01:00
Alexander Neumann
c4fc7b52ae Add 'optimize' command that repacks blobs 2015-11-08 21:10:03 +01:00
Alexander Neumann
cd948b56ac cmd_check: Don't display unused blobs by default 2015-11-08 20:46:52 +01:00
Alexander Neumann
19a713970f Merge pull request #344 from restic/fix-338
pipe: propagate errors properly
2015-11-08 17:58:55 +01:00
Alexander Neumann
4484a3ea0d archiver: ignore dir nodes with errors 2015-11-07 11:42:28 +01:00
Alexander Neumann
ea41a1045f Add integration test for error on readdirnames 2015-11-06 23:19:56 +01:00
Alexander Neumann
005c13ff05 pipe: make test platform-independent 2015-11-06 22:38:34 +01:00
Alexander Neumann
1569176e48 pipe: propagate errors properly 2015-11-06 19:41:57 +01:00
Alexander Neumann
7c3d227527 Merge pull request #342 from restic/add-bug-reporting-instructions
Add instructions on reporting bugs
2015-11-06 00:41:22 +01:00
Alexander Neumann
0f92f6319f Add note on determinism 2015-11-05 23:10:35 +01:00
Alexander Neumann
7b9f2fa9ef Add instructions on reporting bugs 2015-11-05 23:09:01 +01:00
Alexander Neumann
8de8ca05f1 Merge pull request #341 from restic/read-password-from-stdin
Read password from stdin if terminal is not a tty
2015-11-04 22:45:54 +01:00
Alexander Neumann
18d7f7f835 Read password from stdin if terminal is not a tty 2015-11-04 22:20:58 +01:00
Alexander Neumann
73e085ae23 Merge pull request #335 from JaCoB1123/windowspathnames
Always use forward slashes in SFTP (Fixes #334)
2015-11-03 21:29:00 +01:00
Jan Bader
af960b9b40 Simplify Implementation of Join 2015-11-03 18:48:51 +01:00
Jan Bader
d09e6d5b0f Fix missing Join calls 2015-11-03 18:47:01 +01:00
Alexander Neumann
db41102bfa Finalize repacker 2015-11-02 19:28:30 +01:00
Alexander Neumann
1fc0d78913 Refactor Index.Store() to take a PackedBlob 2015-11-02 19:07:03 +01:00
Alexander Neumann
f3f84b1544 Add ID handling for index 2015-11-02 18:52:13 +01:00
Alexander Neumann
60a34087c9 Move LoadIndexWithDecoder to index.go 2015-11-02 18:52:13 +01:00
Alexander Neumann
266bc05edc Add mostly ready repacker 2015-11-02 18:52:13 +01:00
Alexander Neumann
51aff3ca57 Add FindBlobsForPacks() 2015-11-02 18:52:13 +01:00
Alexander Neumann
30cf002574 Sort IDSet.List() 2015-11-02 18:52:13 +01:00
Alexander Neumann
89a77ab2f9 Add Index.ListPack() 2015-11-02 18:52:13 +01:00
Alexander Neumann
484331cd8d Add repacker 2015-11-02 18:52:13 +01:00
Alexander Neumann
181963ba08 Fix IDSet.String() 2015-11-02 17:36:05 +01:00
Alexander Neumann
50c2f2e87f cmd_cat: allow dumping raw tree blobs 2015-11-02 17:36:04 +01:00
Alexander Neumann
ba8e6035b0 Merge pull request #336 from restic/refactor-index
Refactor Index.Lookup() to return struct PackedBlob
2015-11-02 17:35:09 +01:00
Jan Bader
81ec7337e0 Always use forward slashes in SFTP (Fixes #334)
Add custom Join func that always uses forward slashes in SFTP
2015-11-02 14:53:42 +01:00
Alexander Neumann
ed9470b19d Remove tempfiles after test 2015-10-31 14:53:03 +01:00
Alexander Neumann
fccde030d5 Refactor Index.Lookup() to return struct PackedBlob 2015-10-31 14:47:42 +01:00
Alexander Neumann
fc0f5d8f72 Merge pull request #331 from tyll/gpgfingerprint
Show full GPG fingerprint in README
2015-10-29 21:55:28 +01:00
Till Maas
34af39667b Show full GPG fingerprint in README 2015-10-29 21:51:21 +01:00
Alexander Neumann
6ffd7da4d7 Merge pull request #328 from restic/improve-backup-speed
Load trees from repository in parallel
2015-10-29 20:03:29 +01:00
Alexander Neumann
5958dc920b Fix walk tree test for windows 2015-10-28 22:02:37 +01:00
Alexander Neumann
5a45d95b80 Fix compatibility with Go < 1.5 2015-10-27 23:06:56 +01:00
Alexander Neumann
23aeca85ff load trees in parallel 2015-10-27 22:44:10 +01:00
Alexander Neumann
b5976474dd backup: add debug output for excluded files/dirs 2015-10-27 22:34:30 +01:00
Alexander Neumann
18478e2d3d walk_test: test correct number of items 2015-10-27 21:41:25 +01:00
Alexander Neumann
ca5c0bf78e Test WalkTree() for correct order 2015-10-27 20:51:55 +01:00
Alexander Neumann
4cc9d946de Add benchmark for WalkTree with high-latency repo 2015-10-27 20:51:55 +01:00
Alexander Neumann
50fd8f6f44 make repo for tree walker mockable 2015-10-27 20:51:55 +01:00
Alexander Neumann
7711fcda69 use new index format for repository tests 2015-10-27 20:51:55 +01:00
Alexander Neumann
7717ea5cca Add benchmark for LoadJSONPack 2015-10-27 20:51:55 +01:00
Alexander Neumann
ae46674cd3 debug: log timing 2015-10-27 20:51:55 +01:00
Alexander Neumann
00e05ae3c9 bugfix: close pack files after reading the header 2015-10-27 20:39:52 +01:00
Alexander Neumann
4bc81c2bd2 Merge pull request #325 from restic/fix-rebuild-index
rebuild-index: Remember already stored blobs
2015-10-25 22:57:12 +01:00
Alexander Neumann
74cd134b54 rebuild index: remember already stored blobs 2015-10-25 22:34:22 +01:00
Alexander Neumann
734ae7fcb8 Add test for corner case
It was observed that a restic repository still contained overlapping
indexes after `rebuild-index` has been called. This is caused by
instantly forgetting that blobs have already been saved once a full
index has been written during index rebuilding.

This commit adds a (failing) test that shows the behaviour.
2015-10-25 21:51:57 +01:00
Alexander Neumann
7b8e42a763 Silence rebuild-index tests 2015-10-25 21:51:46 +01:00
Alexander Neumann
566fb22bcf Merge pull request #324 from restic/fix-index-grow
Fix really large index and memory exhaustion
2015-10-25 18:58:31 +01:00
Alexander Neumann
b88ccb4f1b Fix Unpacker test 2015-10-25 18:21:48 +01:00
Alexander Neumann
efbce9f0fa rebuild-index: handle not yet indexed packs 2015-10-25 18:07:51 +01:00
Alexander Neumann
88849c06a6 rebuild-index: Refactor a bit 2015-10-25 17:53:02 +01:00
Alexander Neumann
5d617edbbf local/sftp backend: Do not seek if offset is 0 2015-10-25 17:51:26 +01:00
Alexander Neumann
6aed9f268b Add command rebuild-index 2015-10-25 17:24:52 +01:00
Alexander Neumann
9074c923ea index: add AddToSupersedes() 2015-10-25 17:06:56 +01:00
Alexander Neumann
1365495599 debug: remove extra space between filename and line 2015-10-25 17:06:20 +01:00
Alexander Neumann
461d54e43c Refactor repository.SaveIndex() 2015-10-25 17:05:54 +01:00
Alexander Neumann
96ecc26507 Let the checker return a list of hints along with errors 2015-10-25 16:26:50 +01:00
Alexander Neumann
91e1929b52 checker: test for packs in multiple indexes 2015-10-25 16:00:06 +01:00
Alexander Neumann
04614c7527 Add test for packs in duplicate indexes 2015-10-25 15:35:33 +01:00
Alexander Neumann
f7ff5b766c Mark written indexes as finalized 2015-10-25 15:35:18 +01:00
Alexander Neumann
d9f9b77d68 Add Index.Packs() and IDSet.Equals() 2015-10-25 15:28:01 +01:00
Alexander Neumann
4b1a2caea7 Allow overwriting the IndexFull function for tests 2015-10-25 15:05:22 +01:00
Alexander Neumann
af0d6f58b9 Remove unneeded pointer to pack id 2015-10-25 14:35:08 +01:00
Alexander Neumann
2710d6399a Cleanup index code
The selectFn wasn't used any more, so remove it from generatePackList().
2015-10-25 14:26:04 +01:00
Alexander Neumann
650eab6a0e Fix typo in dump usage 2015-10-25 13:19:35 +01:00
Alexander Neumann
5de36dfdf0 Merge pull request #310 from restic/resume-backups
resume interrupted backups
2015-10-14 21:53:25 +02:00
Alexander Neumann
1dd731fdb8 Handle concurrent access to the inFlight list 2015-10-14 20:50:54 +02:00
Alexander Neumann
6fa4be5af2 Regularly save intermediate indexes 2015-10-12 23:59:17 +02:00
Alexander Neumann
941b7025b6 Delete Index.Remove() 2015-10-12 22:51:11 +02:00
Alexander Neumann
4b2a4b03ec Remove Index.StoreInProgress() 2015-10-12 22:49:31 +02:00
Alexander Neumann
7ab9915859 Fix 'dump' command 2015-10-12 22:42:31 +02:00
Alexander Neumann
86fcd170f6 Add and use MasterIndex 2015-10-12 22:34:12 +02:00
Alexander Neumann
64fa89d406 Add error checks 2015-10-12 22:07:56 +02:00
Alexander Neumann
eb73182fcf Rework index decode and handling old format 2015-10-12 20:53:07 +02:00
Alexander Neumann
356bb62243 Add CreateEncryptedBlob and GetDecryptReader 2015-10-12 20:53:07 +02:00
Alexander Neumann
de2c20be84 Dump individual indexes 2015-10-12 20:53:07 +02:00
Alexander Neumann
96f2165067 Allow loading index with old format 2015-10-12 20:53:07 +02:00
Alexander Neumann
7944e8e323 Update index format 2015-10-12 20:53:07 +02:00
Alexander Neumann
5c39abfe53 Merge pull request #319 from restic/fix-311
Handle null subtree IDs
2015-10-11 21:48:52 +02:00
Alexander Neumann
1020e9c3af Check for data blobs with null ID, improve errors 2015-10-11 20:58:57 +02:00
Alexander Neumann
cc7acba02b Return the original backend ID on duplicate entries 2015-10-11 20:45:50 +02:00
Alexander Neumann
f188cf81dc Add more panic() calls for invalid conditions 2015-10-11 20:45:42 +02:00
Alexander Neumann
7db2369081 Shorten error message for tree errors 2015-10-11 20:22:52 +02:00
Alexander Neumann
db85ab8aa0 Use the correct channel for sending errors 2015-10-11 19:13:45 +02:00
Alexander Neumann
86c8328f62 Handle null subtree IDs 2015-10-11 19:13:35 +02:00
Alexander Neumann
72fcd00859 Check subtrees with null ID 2015-10-11 18:46:26 +02:00
Alexander Neumann
8a7873ee3a Handle invalid subtree IDs 2015-10-11 18:45:16 +02:00
Alexander Neumann
e738d35c4e Merge pull request #309 from restic/update-ci-go-version
Update Go 1.4 to version 1.4.3
2015-09-27 20:20:35 +02:00
Alexander Neumann
6ddda5fc5e appveyor: remove old Go installation 2015-09-27 18:34:11 +02:00
Alexander Neumann
7291342723 Install current version of Go
This is inspired by the appveyor.yaml from the go-plus project:
https://github.com/joefitzgerald/go-plus/blob/master/appveyor.yml
2015-09-27 18:02:44 +02:00
Alexander Neumann
70a6233b94 Install the 'cover' tool 2015-09-27 17:58:13 +02:00
Alexander Neumann
749ca28534 Update Go 1.4 to version 1.4.3 2015-09-27 17:22:12 +02:00
Alexander Neumann
321c2e6a47 Merge pull request #308 from episource/fix/restic_cr292_unreferenced_pack
fix:restic#292 Prevent concurrent processing of the same blob
2015-09-27 17:18:28 +02:00
Philipp Serr
7b11660f4f Prevent concurrent processing of same blob
... by first adding a preliminary index entry and making this fail if
an index entry for the same blob already exists.

A preliminary index entry is characterized by not yet being associated
with a pack. Until now, these entries where added to the index just
like final index entries using index.Store, which silently overwrites
existing index entries.

This commit adds a new method index.StoreInProgress which refuses to
overwrite existing index entries and allows for creating preliminary
index entries only. The existing method index.Store has not been
changed and continues to silently overwrite existing index entries.
This distinction is important, as otherwise, it would be impossible to
update a preliminary index entry after the blob has been written to a
pack.

Resolves: restic#292
2015-09-27 16:56:49 +02:00
Alexander Neumann
4fb46faae7 Use Go 1.5.1 for travis tests 2015-09-12 22:15:09 +02:00
Alexander Neumann
316f6ed313 Update chunker version 2015-09-12 22:14:58 +02:00
Alexander Neumann
108d28316a Merge pull request #294 from restic/rework-id
Refactor IDs and IDSet
2015-09-08 21:26:07 +02:00
Alexander Neumann
5c46dc41de Add methods to IDSet 2015-09-05 18:49:28 +02:00
Alexander Neumann
d42ff509ba Small refactorings
* use uint instead of uint32 in packs/indexes
 * use ID.Str() for debug messages
 * add ParallelIDWorkFunc
2015-09-05 18:41:58 +02:00
Alexander Neumann
2cb0fbf589 backend: Add String() to IDs 2015-09-05 18:41:58 +02:00
Alexander Neumann
a0bad1695c Remove comment 2015-09-05 18:41:58 +02:00
Alexander Neumann
681d7851aa index: use backend.ID instead of string for maps 2015-09-05 18:41:58 +02:00
Alexander Neumann
3063ad1d05 Split id.go into several files 2015-09-05 18:41:56 +02:00
Alexander Neumann
76b1f017c0 Merge pull request #290 from bchapuis/fix-289
Load the index and search subtree
2015-09-01 21:08:21 +02:00
Chapuis Bertil
c765688779 find command integration tests 2015-08-28 19:31:05 +02:00
Chapuis Bertil
d4686ebcc5 Load the index and search subtree 2015-08-27 23:21:44 +02:00
Alexander Neumann
f653aca0ed Merge pull request #287 from restic/fix-279
Remove tests for directories
2015-08-27 22:07:07 +02:00
Alexander Neumann
0a457eafed Correctly test for config file 2015-08-26 22:06:52 +02:00
Alexander Neumann
b211f834fa Remove tests for directories
For testing whether a repository already exists it is sufficient to
test if the config file (and therefore the master key) exists.

Closes #279
2015-08-26 21:51:40 +02:00
Alexander Neumann
9aefc2b7a6 Merge pull request #281 from restic/version-with-git
build.go: use new combined version string
2015-08-26 20:53:24 +02:00
Alexander Neumann
10f0d7ccac Merge pull request #280 from restic/ldflags-go1.5
build.go: Make `-ldflags` compatible to Go 1.5
2015-08-26 20:33:43 +02:00
Alexander Neumann
cb460b7dec Merge pull request #285 from howeyc/fix-aws-v4
Use new version of s3 library, Fixes #276
2015-08-26 20:20:32 +02:00
Alexander Neumann
39a82d951b Refactor getVersion(), address code review comments 2015-08-26 20:17:51 +02:00
Alexander Neumann
a54f9715b1 Add "build: " prefix to verbose messages 2015-08-26 20:03:26 +02:00
Alexander Neumann
4c47c2b2c9 Address code review comments 2015-08-26 20:03:16 +02:00
Chris Howey
ccb2f00b8a typo 2015-08-26 07:54:39 -05:00
Chris Howey
3bf447b422 Update tests for new s3 lib 2015-08-26 07:44:00 -05:00
Chris Howey
10cd672a92 Use new version of s3 library, Fixes #276 2015-08-26 06:25:05 -05:00
Alexander Neumann
f3c64d0740 build.go: use new combined version string
Previously, when a VERSION file exists it takes precendence over the
git version. This is unfortunate because all restic binaries compiled
from a git checkout will just identify as the latest release (e.g.
'0.1.0'), regardeless of any commits on top of it.

This commit adds a combined version string by using the contents of
the VERSION file, and append the current git version returned by `git
describe` if available, e.g.:

    0.1.0 (v0.1.0-6-gb188217-dirty).
2015-08-25 22:20:53 +02:00
Alexander Neumann
dca200c2e9 build.go: Make -ldflags compatible to Go 1.5
This change uses the old syntax (-ldflags "-X foo bar") for Go <= 1.4
and the new syntax for (-ldflags "-X foo=bar") for Go 1.5 (without a
warning).
2015-08-25 22:07:52 +02:00
Alexander Neumann
b188217e83 Merge pull request #274 from restic/fix-documentation
Documentation fixes
2015-08-22 23:07:31 +02:00
Alexander Neumann
3a50c2bbfb Fix docs 2015-08-22 23:03:25 +02:00
Alexander Neumann
e0e9cd8680 More documentation fixes 2015-08-22 15:09:53 +02:00
Alexander Neumann
b6872fb454 Clarify documentation about MAC key 2015-08-22 15:09:21 +02:00
Florian Daniel
3f3cca8f2a Merge pull request #273 from restic/fix-124
fix typo in Readme
2015-08-22 00:01:35 +02:00
Florian Daniel
647ee5b74a fix typo in Readme 2015-08-21 23:53:59 +02:00
1228 changed files with 95811 additions and 66827 deletions

12
.dockerignore Normal file
View File

@@ -0,0 +1,12 @@
# Folders
.git/
.github/
changelog/
doc/
docker/
helpers/
# Files
.gitignore
.golangci.yml
*.md

93
.github/ISSUE_TEMPLATE/Bug.md vendored Normal file
View File

@@ -0,0 +1,93 @@
---
name: Bug report
about: Report a problem with restic to help us resolve it and improve
---
<!--
Welcome! - We kindly ask that you:
1. Fill out the issue template below - not doing so needs a good reason.
2. Use the forum if you have a question rather than a bug or feature request.
The forum is at: https://forum.restic.net
NOTE: Not filling out the issue template needs a good reason, as otherwise it
may take a lot longer to find the problem, not to mention it can take up a lot
more time which can otherwise be spent on development. Please also take the
time to help us debug the issue by collecting relevant information, even if
it doesn't seem to be relevant to you. Thanks!
The forum is a better place for questions about restic or general suggestions
and topics, e.g. usage or documentation questions! This issue tracker is mainly
for tracking bugs and feature requests directly relating to the development of
the software itself, rather than the project.
Thanks for understanding, and for contributing to the project!
-->
Output of `restic version`
--------------------------
How did you run restic exactly?
-------------------------------
<!--
This section should include at least:
* The complete command line and any environment variables you used to
configure restic's backend access. Make sure to replace sensitive values!
* The output of the commands, what restic prints gives may give us much
information to diagnose the problem!
-->
What backend/server/service did you use to store the repository?
----------------------------------------------------------------
Expected behavior
-----------------
<!--
Describe what you'd like restic to do differently.
-->
Actual behavior
---------------
<!--
In this section, please try to concentrate on observations, so only describe
what you observed directly.
-->
Steps to reproduce the behavior
-------------------------------
<!--
The more time you spend describing an easy way to reproduce the behavior (if
this is possible), the easier it is for the project developers to fix it!
-->
Do you have any idea what may have caused this?
-----------------------------------------------
Do you have an idea how to solve the issue?
-------------------------------------------
Did restic help you today? Did it make you happy in any way?
------------------------------------------------------------
<!--
Answering this question is not required, but if you have anything positive to share, please do so here!
Sometimes we get tired of reading bug reports all day and a little positive end note does wonders.
Idea by Joey Hess, https://joeyh.name/blog/entry/two_holiday_stories/
-->

57
.github/ISSUE_TEMPLATE/Feature.md vendored Normal file
View File

@@ -0,0 +1,57 @@
---
name: Feature request
about: Suggest a new feature or enhancement for restic
---
<!--
Welcome! - We kindly ask that you:
1. Fill out the issue template below - not doing so needs a good reason.
2. Use the forum if you have a question rather than a bug or feature request.
The forum is at: https://forum.restic.net
The forum is a better place for questions about restic or general suggestions
and topics, e.g. usage or documentation questions! This issue tracker is mainly
for tracking bugs and feature requests directly relating to the development of
the software itself, rather than the project.
Thanks for understanding, and for contributing to the project!
-->
Output of `restic version`
--------------------------
<!--
Please add the version of restic you're currently using here, this helps us
later to see what has changed in restic when we revisit this issue after some
time.
-->
What should restic do differently? Which functionality do you think we should add?
----------------------------------------------------------------------------------
<!--
Please describe the feature you'd like us to add here.
-->
What are you trying to do? What problem would this solve?
---------------------------------------------------------
<!--
This section should contain a brief description what you're trying to do, which
would be possible after implementing the new feature.
-->
Did restic help you today? Did it make you happy in any way?
------------------------------------------------------------
<!--
Answering this question is not required, but if you have anything positive to share, please do so here!
Sometimes we get tired of reading bug reports all day and a little positive end note does wonders.
Idea by Joey Hess, https://joeyh.name/blog/entry/two_holiday_stories/
-->

4
.github/ISSUE_TEMPLATE/config.yml vendored Normal file
View File

@@ -0,0 +1,4 @@
contact_links:
- name: restic forum
url: https://forum.restic.net
about: Please ask questions about using restic here, do not open an issue for questions.

40
.github/PULL_REQUEST_TEMPLATE.md vendored Normal file
View File

@@ -0,0 +1,40 @@
<!--
Thank you very much for contributing code or documentation to restic! Please
fill out the following questions to make it easier for us to review your
changes.
-->
What does this PR change? What problem does it solve?
-----------------------------------------------------
<!--
Describe the changes and their purpose here, as detailed as needed.
-->
Was the change previously discussed in an issue or on the forum?
----------------------------------------------------------------
<!--
Link issues and relevant forum posts here.
If this PR resolves an issue on GitHub, use "Closes #1234" so that the issue
is closed automatically when this PR is merged.
-->
Checklist
---------
<!--
You do not need to check all the boxes below all at once. Feel free to take
your time and add more commits. If you're done and ready for review, please
check the last box. Enable a checkbox by replacing [ ] with [x].
-->
- [ ] I have read the [contribution guidelines](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#providing-patches).
- [ ] I have [enabled maintainer edits](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/allowing-changes-to-a-pull-request-branch-created-from-a-fork).
- [ ] I have added tests for all code changes.
- [ ] I have added documentation for relevant changes (in the manual).
- [ ] There's a new file in `changelog/unreleased/` that describes the changes for our users (see [template](https://github.com/restic/restic/blob/master/changelog/TEMPLATE)).
- [ ] I have run `gofmt` on the code in all commits.
- [ ] All commit messages are formatted in the same style as [the other commits in the repo](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#git-commits).
- [ ] I'm done! This pull request is ready for review.

325
.github/workflows/tests.yml vendored Normal file
View File

@@ -0,0 +1,325 @@
name: test
on:
# run tests on push to master, but not when other branches are pushed to
push:
branches:
- master
# run tests for all pull requests
pull_request:
env:
latest_go: "1.18.x"
GO111MODULE: on
jobs:
test:
strategy:
matrix:
# list of jobs to run:
include:
- job_name: Windows
go: 1.18.x
os: windows-latest
install_verb: install
- job_name: macOS
go: 1.18.x
os: macOS-latest
test_fuse: false
install_verb: install
- job_name: Linux
go: 1.18.x
os: ubuntu-latest
test_cloud_backends: true
test_fuse: true
check_changelog: true
install_verb: install
- job_name: Linux
go: 1.17.x
os: ubuntu-latest
test_fuse: true
install_verb: install
- job_name: Linux
go: 1.16.x
os: ubuntu-latest
test_fuse: true
install_verb: get
- job_name: Linux
go: 1.15.x
os: ubuntu-latest
test_fuse: true
install_verb: get
- job_name: Linux
go: 1.14.x
os: ubuntu-latest
test_fuse: true
install_verb: get
name: ${{ matrix.job_name }} Go ${{ matrix.go }}
runs-on: ${{ matrix.os }}
env:
GOPROXY: https://proxy.golang.org
steps:
- name: Set up Go ${{ matrix.go }}
uses: actions/setup-go@v2
with:
go-version: ${{ matrix.go }}
- name: Get programs (Linux/macOS)
run: |
echo "build Go tools"
go ${{ matrix.install_verb }} github.com/restic/rest-server/cmd/rest-server@latest
echo "install minio server"
mkdir $HOME/bin
if [ "$RUNNER_OS" == "macOS" ]; then
wget --no-verbose -O $HOME/bin/minio https://dl.minio.io/server/minio/release/darwin-amd64/minio
else
wget --no-verbose -O $HOME/bin/minio https://dl.minio.io/server/minio/release/linux-amd64/minio
fi
chmod 755 $HOME/bin/minio
echo "install rclone"
if [ "$RUNNER_OS" == "macOS" ]; then
wget --no-verbose -O rclone.zip https://downloads.rclone.org/rclone-current-osx-amd64.zip
else
wget --no-verbose -O rclone.zip https://downloads.rclone.org/rclone-current-linux-amd64.zip
fi
unzip rclone.zip
cp rclone*/rclone $HOME/bin
chmod 755 $HOME/bin/rclone
rm -rf rclone*
# add $HOME/bin to path ($GOBIN was already added to the path by setup-go@v2)
echo $HOME/bin >> $GITHUB_PATH
if: matrix.os == 'ubuntu-latest' || matrix.os == 'macOS-latest'
- name: Get programs (Windows)
shell: powershell
run: |
$ProgressPreference = 'SilentlyContinue'
echo "build Go tools"
go ${{ matrix.install_verb }} github.com/restic/rest-server/...
echo "install minio server"
mkdir $Env:USERPROFILE/bin
Invoke-WebRequest https://dl.minio.io/server/minio/release/windows-amd64/minio.exe -OutFile $Env:USERPROFILE/bin/minio.exe
echo "install rclone"
Invoke-WebRequest https://downloads.rclone.org/rclone-current-windows-amd64.zip -OutFile rclone.zip
unzip rclone.zip
copy rclone*/rclone.exe $Env:USERPROFILE/bin
# add $USERPROFILE/bin to path ($GOBIN was already added to the path by setup-go@v2)
echo $Env:USERPROFILE\bin >> $Env:GITHUB_PATH
echo "install tar"
cd $env:USERPROFILE
mkdir tar
cd tar
# install exactly these versions of tar and the libraries, other combinations might not work!
Invoke-WebRequest https://github.com/restic/test-assets/raw/master/tar-1.13-1-bin.zip -OutFile tar.zip
unzip tar.zip
Invoke-WebRequest https://github.com/restic/test-assets/raw/master/libintl-0.11.5-2-bin.zip -OutFile libintl.zip
unzip libintl.zip
Invoke-WebRequest https://github.com/restic/test-assets/raw/master/libiconv-1.8-1-bin.zip -OutFile libiconv.zip
unzip libiconv.zip
# add $USERPROFILE/tar/bin to path
echo $Env:USERPROFILE\tar\bin >> $Env:GITHUB_PATH
if: matrix.os == 'windows-latest'
- name: Check out code
uses: actions/checkout@v2
- name: Build with build.go
run: |
go run build.go
- name: Run local Tests
env:
RESTIC_TEST_FUSE: ${{ matrix.test_fuse }}
run: |
go test -cover ./...
- name: Test cloud backends
env:
RESTIC_TEST_S3_KEY: ${{ secrets.RESTIC_TEST_S3_KEY }}
RESTIC_TEST_S3_SECRET: ${{ secrets.RESTIC_TEST_S3_SECRET }}
RESTIC_TEST_S3_REPOSITORY: ${{ secrets.RESTIC_TEST_S3_REPOSITORY }}
RESTIC_TEST_AZURE_ACCOUNT_NAME: ${{ secrets.RESTIC_TEST_AZURE_ACCOUNT_NAME }}
RESTIC_TEST_AZURE_ACCOUNT_KEY: ${{ secrets.RESTIC_TEST_AZURE_ACCOUNT_KEY }}
RESTIC_TEST_AZURE_REPOSITORY: ${{ secrets.RESTIC_TEST_AZURE_REPOSITORY }}
RESTIC_TEST_B2_ACCOUNT_ID: ${{ secrets.RESTIC_TEST_B2_ACCOUNT_ID }}
RESTIC_TEST_B2_ACCOUNT_KEY: ${{ secrets.RESTIC_TEST_B2_ACCOUNT_KEY }}
RESTIC_TEST_B2_REPOSITORY: ${{ secrets.RESTIC_TEST_B2_REPOSITORY }}
RESTIC_TEST_GS_REPOSITORY: ${{ secrets.RESTIC_TEST_GS_REPOSITORY }}
RESTIC_TEST_GS_PROJECT_ID: ${{ secrets.RESTIC_TEST_GS_PROJECT_ID }}
GOOGLE_PROJECT_ID: ${{ secrets.RESTIC_TEST_GS_PROJECT_ID }}
RESTIC_TEST_GS_APPLICATION_CREDENTIALS_B64: ${{ secrets.RESTIC_TEST_GS_APPLICATION_CREDENTIALS_B64 }}
RESTIC_TEST_OS_AUTH_URL: ${{ secrets.RESTIC_TEST_OS_AUTH_URL }}
RESTIC_TEST_OS_TENANT_NAME: ${{ secrets.RESTIC_TEST_OS_TENANT_NAME }}
RESTIC_TEST_OS_USERNAME: ${{ secrets.RESTIC_TEST_OS_USERNAME }}
RESTIC_TEST_OS_PASSWORD: ${{ secrets.RESTIC_TEST_OS_PASSWORD }}
RESTIC_TEST_OS_REGION_NAME: ${{ secrets.RESTIC_TEST_OS_REGION_NAME }}
RESTIC_TEST_SWIFT: ${{ secrets.RESTIC_TEST_SWIFT }}
# fail if any of the following tests cannot be run
RESTIC_TEST_DISALLOW_SKIP: "restic/backend/rest.TestBackendREST,\
restic/backend/sftp.TestBackendSFTP,\
restic/backend/s3.TestBackendMinio,\
restic/backend/rclone.TestBackendRclone,\
restic/backend/s3.TestBackendS3,\
restic/backend/swift.TestBackendSwift,\
restic/backend/b2.TestBackendB2,\
restic/backend/gs.TestBackendGS,\
restic/backend/azure.TestBackendAzure"
run: |
# prepare credentials for Google Cloud Storage tests in a temp file
export GOOGLE_APPLICATION_CREDENTIALS=$(mktemp --tmpdir restic-gcs-auth-XXXXXXX)
echo $RESTIC_TEST_GS_APPLICATION_CREDENTIALS_B64 | base64 -d > $GOOGLE_APPLICATION_CREDENTIALS
go test -cover -parallel 4 ./internal/backend/...
# only run cloud backend tests for pull requests from and pushes to our
# own repo, otherwise the secrets are not available
if: (github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository) && matrix.test_cloud_backends
- name: Check changelog files with calens
run: |
echo "install calens"
go install github.com/restic/calens@latest
echo "check changelog files"
calens
if: matrix.check_changelog
cross_compile:
strategy:
# ATTENTION: the list of architectures must be in sync with helpers/build-release-binaries/main.go!
matrix:
# run cross-compile in two batches parallel so the overall tests run faster
targets:
- "linux/386 linux/amd64 linux/arm linux/arm64 linux/ppc64le linux/mips linux/mipsle linux/mips64 linux/mips64le linux/s390x \
openbsd/386 openbsd/amd64"
- "freebsd/386 freebsd/amd64 freebsd/arm \
aix/ppc64 \
darwin/amd64 darwin/arm64 \
netbsd/386 netbsd/amd64 \
windows/386 windows/amd64 \
solaris/amd64"
env:
GOPROXY: https://proxy.golang.org
runs-on: ubuntu-latest
name: Cross Compile for ${{ matrix.targets }}
steps:
- name: Set up Go ${{ env.latest_go }}
uses: actions/setup-go@v2
with:
go-version: ${{ env.latest_go }}
- name: Install gox
run: |
go install github.com/mitchellh/gox@latest
- name: Check out code
uses: actions/checkout@v2
- name: Cross-compile with gox for ${{ matrix.targets }}
env:
GOFLAGS: "-trimpath"
GOX_ARCHS: "${{ matrix.targets }}"
run: |
mkdir build-output
gox -parallel 2 -verbose -osarch "$GOX_ARCHS" -output "build-output/{{.Dir}}_{{.OS}}_{{.Arch}}" ./cmd/restic
gox -parallel 2 -verbose -osarch "$GOX_ARCHS" -tags debug -output "build-output/{{.Dir}}_{{.OS}}_{{.Arch}}_debug" ./cmd/restic
lint:
name: lint
runs-on: ubuntu-latest
steps:
- name: Set up Go ${{ env.latest_go }}
uses: actions/setup-go@v2
with:
go-version: ${{ env.latest_go }}
- name: Check out code
uses: actions/checkout@v2
- name: golangci-lint
uses: golangci/golangci-lint-action@v2
with:
# Required: the version of golangci-lint is required and must be specified without patch version: we always use the latest patch version.
version: v1.45
# Optional: show only new issues if it's a pull request. The default value is `false`.
only-new-issues: true
args: --verbose --timeout 5m
skip-go-installation: true
# only run golangci-lint for pull requests, otherwise ALL hints get
# reported. We need to slowly address all issues until we can enable
# linting the master branch :)
if: github.event_name == 'pull_request'
- name: Check go.mod/go.sum
run: |
echo "check if go.mod and go.sum are up to date"
go mod tidy
git diff --exit-code go.mod go.sum
docker:
name: docker
runs-on: ubuntu-latest
steps:
- name: Check out code
uses: actions/checkout@v2
- name: Docker meta
id: meta
uses: docker/metadata-action@v3
with:
# list of Docker images to use as base name for tags
images: |
restic/restic
# generate Docker tags based on the following events/attributes
tags: |
type=schedule
type=ref,event=branch
type=ref,event=pr
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=semver,pattern={{major}}
type=sha
- name: Set up QEMU
uses: docker/setup-qemu-action@v1
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
- name: Build and push
id: docker_build
uses: docker/build-push-action@v2
with:
push: false
context: .
file: docker/Dockerfile
pull: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}

7
.gitignore vendored
View File

@@ -1,8 +1,3 @@
/.gopath
/restic
/restic.debug
/dirdiff
cmd/dirdiff/dirdiff
cmd/gentestdata/gentestdata
cmd/restic/restic
/.vagrant
/.vscode

57
.golangci.yml Normal file
View File

@@ -0,0 +1,57 @@
# This is the configuration for golangci-lint for the restic project.
#
# A sample config with all settings is here:
# https://github.com/golangci/golangci-lint/blob/master/.golangci.example.yml
linters:
# only enable the linters listed below
disable-all: true
enable:
# make sure all errors returned by functions are handled
- errcheck
# find unused code
- deadcode
# show how code can be simplified
- gosimple
# # make sure code is formatted
- gofmt
# examine code and report suspicious constructs, such as Printf calls whose
# arguments do not align with the format string
- govet
# make sure names and comments are used according to the conventions
- golint
# detect when assignments to existing variables are not used
- ineffassign
# run static analysis and find errors
- staticcheck
# find unused variables, functions, structs, types, etc.
- unused
# find unused struct fields
- structcheck
# find unused global variables
- varcheck
# parse and typecheck code
- typecheck
issues:
# don't use the default exclude rules, this hides (among others) ignored
# errors from Close() calls
exclude-use-default: false
# list of things to not warn about
exclude:
# golint: do not warn about missing comments for exported stuff
- exported (function|method|var|type|const) `.*` should have comment or be unexported
# golint: ignore constants in all caps
- don't use ALL_CAPS in Go names; use CamelCase

View File

@@ -1,32 +0,0 @@
language: go
sudo: false
go:
- 1.3.3
- 1.4.2
- 1.5
os:
- linux
- osx
notifications:
irc:
channels:
- "chat.freenode.net#restic"
on_success: change
on_failure: change
skip_join: true
install:
- go version
- export GOBIN="$GOPATH/bin"
- export PATH="$PATH:$GOBIN"
- export GOPATH="$GOPATH:${TRAVIS_BUILD_DIR}/Godeps/_workspace"
- go env
script:
- go run run_integration_tests.go
after_success:
- goveralls -coverprofile=all.cov -service=travis-ci -repotoken "$COVERALLS_TOKEN" || true

3733
CHANGELOG.md Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -3,17 +3,20 @@ This document describes the way you can contribute to the restic project.
Ways to Help Out
================
Thank you for your contribution!
Thank you for your contribution! Please **open an issue first** (or add a
comment to an existing issue) if you plan to work on any code or add a new
feature. This way, duplicate work is prevented and we can discuss your ideas
and design first.
There are several ways you can help us out. First of all code contributions and
bug fixes are most welcome. However even "minor" details as fixing spelling
errors, improving documentation or pointing out usability issues are a great
help also.
The restic project uses the GitHub infrastructure (see the
[project page](https://github.com/restic/restic)) for all related discussions
as well as the `#restic` channel on `irc.freenode.net`.
as well as the [forum](https://forum.restic.net/) and the `#restic` channel
on [irc.libera.chat](https://kiwiirc.com/nextclient/#ircs://irc.libera.chat:6697/#restic).
If you want to find an area that currently needs improving have a look at the
open issues listed at the
@@ -22,53 +25,63 @@ for discussing enhancement to the restic tools.
If you are unsure what to do, please have a look at the issues, especially
those tagged
[minor complexity](https://github.com/restic/restic/labels/minor%20complexity).
[minor complexity](https://github.com/restic/restic/labels/help%3A%20minor%20complexity)
or [good first issue](https://github.com/restic/restic/labels/help%3A%20good%20first%20issue).
If you are already a bit experienced with the restic internals, take a look
at the issues tagged as [help wanted](https://github.com/restic/restic/labels/help%3A%20wanted).
Reporting Bugs
==============
You've found a bug? Thanks for letting us know so we can fix it! It is a good
idea to describe in detail how to reproduce the bug (when you know how), what
environment was used and so on. Please tell us at least the following things:
* What's the version of restic you used? Please include the output of
`restic version` in your bug report.
* What commands did you execute to get to where the bug occurred?
* What did you expect?
* What happened instead?
* Are you aware of a way to reproduce the bug?
Remember, the easier it is for us to reproduce the bug, the earlier it will be
corrected!
In addition, you can compile restic with debug support by running
`go run build.go -tags debug` and instructing it to create a debug
log by setting the environment variable `DEBUG_LOG` to a file, e.g. like this:
$ export DEBUG_LOG=/tmp/restic-debug.log
$ restic backup ~/work
Please be aware that the debug log file will contain potentially sensitive
things like file and directory names, so please either redact it before
uploading it somewhere or post only the parts that are really relevant.
Development Environment
=======================
For development, it is recommended to check out the restic repository within a
`GOPATH`, an introductory text is
["How to Write Go Code"](https://golang.org/doc/code.html). It is recommended
to have a working directory, we're using `~/work/restic` in the following. This
directory mainly contains the directory `src`, where the source code is stored.
The repository contains the code written for restic in the directories
`cmd/` and `internal/`.
First, create the necessary directory structure and clone the restic repository
to the correct location:
Restic requires Go version 1.14 or later for compiling. Clone the repo (without
having `$GOPATH` set) and `cd` into the directory:
$ mkdir --parents ~/work/restic/src/github.com/restic
$ cd ~/work/restic/src/github.com/restic
$ unset GOPATH
$ git clone https://github.com/restic/restic
$ cd restic
Now we're in the main directory of the restic repository. The last step is to
set the environment variable `$GOPATH` to the correct value:
Then use the `go` tool to build restic:
$ export GOPATH=~/work/restic:~/work/restic/src/github.com/restic/restic/Godeps/_workspace
$ go build ./cmd/restic
$ ./restic version
restic 0.10.0-dev (compiled manually) compiled with go1.15.2 on linux/amd64
The following commands can be used to run all the tests:
You can run all tests with the following command:
$ go test ./...
ok github.com/restic/restic 8.174s
[...]
The restic binary can be built from the directory `cmd/restic` this way:
$ cd cmd/restic
$ go build
$ ./restic version
restic compiled manually on go1.4.2
if you want to run your tests on Linux, OpenBSD or FreeBSD, you can use
[vagrant](https://www.vagrantup.com/) with the proveded `Vagrantfile` to
quickly set up VMs and run the tests, e.g.:
$ vagrant up freebsd
[...]
$ vagrant ssh freebsd -c 'cd restic/restic; go test -v ./...'
[...]
Providing Patches
=================
@@ -78,48 +91,85 @@ get it into the project! The workflow we're using is also described on the
[GitHub Flow](https://guides.github.com/introduction/flow/) website, it boils
down to the following steps:
1. First we would kindly ask you to fork our project on GitHub if you haven't
done so already.
2. Clone the repository locally and create a new branch. If you are working on
the code itself, please set up the development environment as described in
the previous section and instead of cloning add your fork on GitHub as a
remote to the clone of the restic repository.
3. Then commit your changes as fine grained as possible, as smaller patches,
that handle one and only one issue are easier to discuss and merge.
0. If you want to work on something, please add a comment to the issue on
GitHub. For a new feature, please add an issue before starting to work on
it, so that duplicate work is prevented.
1. Next, fork our project on GitHub if you haven't done so already.
2. Clone your fork of the repository locally and **create a new branch** for
your changes. If you are working on the code itself, please set up the
development environment as described in the previous section.
3. Commit your changes to the new branch as fine grained as possible, as
smaller patches, for individual changes, are easier to discuss and merge.
4. Push the new branch with your changes to your fork of the repository.
5. Create a pull request by visiting the GitHub website, it will guide you
through the process.
through the process. Please [allow edits from maintainers](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/allowing-changes-to-a-pull-request-branch-created-from-a-fork).
6. You will receive comments on your code and the feature or bug that they
address. Maybe you need to rework some minor things, in this case push new
commits to the branch you created for the pull request, they will be
commits to the branch you created for the pull request (or amend the
existing commit, use common sense to decide which is better), they will be
automatically added to the pull request.
7. Once your code looks good, we'll merge it. Thanks a low for your
contribution!
7. If your pull request changes anything that users should be aware of
(a bugfix, a new feature, ...) please add an entry as a new file in
`changelog/unreleased` including the issue number in the filename (e.g.
`issue-8756`). Use the template in `changelog/TEMPLATE` for the content.
It will be used in the announcement of the next stable release. While
writing, ask yourself: If I were the user, what would I need to be aware
of with this change?
8. Do not edit the man pages under `doc/man` or `doc/manual_rest.rst` -
these are autogenerated before new releases.
9. Once your code looks good and passes all the tests, we'll merge it. Thanks
a lot for your contribution!
Please provide the patches for each bug or feature in a separate branch and
open up a pull request for each.
open up a pull request for each, as this simplifies discussion and merging.
The restic project uses the `gofmt` tool for Go source indentation, so please
run
gofmt -w **/*.go
in the project root directory before committing. Installing the script
`fmt-check` from https://github.com/edsrzf/gofmt-git-hook locally as a
pre-commit hook checks formatting before committing automatically, just copy
this script to `.git/hooks/pre-commit`.
in the project root directory before committing. For each Pull Request, the
formatting is tested with `gofmt` for the latest stable version of Go.
Installing the script `fmt-check` from https://github.com/edsrzf/gofmt-git-hook
locally as a pre-commit hook checks formatting before committing automatically,
just copy this script to `.git/hooks/pre-commit`.
The project is using the program
[`golangci-lint`](https://github.com/golangci/golangci-lint) to run a list of
linters and checkers. It will be run on the code when you submit a PR. In order
to check your code beforehand, you can run `golangci-lint run` manually.
Eventually, we will enable `golangci-lint` for the whole code base. For now,
you can ignore warnings printed for lines you did not modify, those will be
ignored by the CI.
For each pull request, several different systems run the integration tests on
Linux, macOS and Windows. We won't merge any code that does not pass all tests
for all systems, so when a tests fails, try to find out what's wrong and fix
it. If you need help on this, please leave a comment in the pull request, and
we'll be glad to assist. Having a PR with failing integration tests is nothing
to be ashamed of. In contrast, that happens regularly for all of us. That's
what the tests are there for.
Git Commits
-----------
I would be good if you could follow the same general style regarding Git
It would be good if you could follow the same general style regarding Git
commits as the rest of the project, this makes reviewing code, browsing the
history and triaging bugs much easier.
Git commit messages have a very terse summary in the first line of the commit
message, followed by an empty line, followed by a more verbose description or a
List of changed things. For examples, please refer to the excellent [How to
Write a Git Commit Message](http://chris.beams.io/posts/git-commit/).
Write a Git Commit Message](https://chris.beams.io/posts/git-commit/).
If you change/add multiple different things that aren't related at all, try to
make several smaller commits. This is much easier to review. Using `git add -p`

27
GOVERNANCE.md Normal file
View File

@@ -0,0 +1,27 @@
# restic project governance
## Overview
The restic project uses a governance model commonly described as Benevolent
Dictator For Life (BDFL). This document outlines our understanding of what this
means. It is derived from the [i3 window manager project
governance](https://raw.githubusercontent.com/i3/i3/next/.github/GOVERNANCE.md).
## Roles
* user: anyone who interacts with the restic project
* core contributor: a handful of people who have contributed significantly to
the project by any means (issue triage, support, documentation, code, etc.).
Core contributors are recognizable via GitHubs "Member" badge.
* Benevolent Dictator For Life (BDFL): a single individual who makes decisions
when consensus cannot be reached. restic's current BDFL is [@fd0](https://github.com/fd0).
## Decision making process
In general, we try to reach consensus in discussions. In case consensus cannot
be reached, the BDFL makes a decision.
## Contribution process
The contribution process is described in a separate document called
[CONTRIBUTING](CONTRIBUTING.md).

62
Godeps/Godeps.json generated
View File

@@ -1,62 +0,0 @@
{
"ImportPath": "github.com/restic/restic",
"GoVersion": "go1.4.2",
"Packages": [
"./..."
],
"Deps": [
{
"ImportPath": "bazil.org/fuse",
"Rev": "18419ee53958df28fcfc9490fe6123bd59e237bb"
},
{
"ImportPath": "github.com/jessevdk/go-flags",
"Comment": "v1-297-g1b89bf7",
"Rev": "1b89bf73cd2c3a911d7b2a279ab085c4a18cf539"
},
{
"ImportPath": "github.com/juju/errors",
"Rev": "4567a5e69fd3130ca0d89f69478e7ac025b67452"
},
{
"ImportPath": "github.com/kr/fs",
"Rev": "2788f0dbd16903de03cb8186e5c7d97b69ad387b"
},
{
"ImportPath": "github.com/mitchellh/goamz/aws",
"Rev": "caaaea8b30ee15616494ee68abd5d8ebbbef05cf"
},
{
"ImportPath": "github.com/mitchellh/goamz/s3",
"Rev": "caaaea8b30ee15616494ee68abd5d8ebbbef05cf"
},
{
"ImportPath": "github.com/pkg/sftp",
"Rev": "518aed2757a65cfa64d4b1b2baf08410f8b7a6bc"
},
{
"ImportPath": "github.com/restic/chunker",
"Rev": "e795b80f4c927ebcf2687ce18bcf1a39fee740b1"
},
{
"ImportPath": "github.com/vaughan0/go-ini",
"Rev": "a98ad7ee00ec53921f08832bc06ecf7fd600e6a1"
},
{
"ImportPath": "golang.org/x/crypto/pbkdf2",
"Rev": "cc04154d65fb9296747569b107cfd05380b1ea3e"
},
{
"ImportPath": "golang.org/x/crypto/poly1305",
"Rev": "cc04154d65fb9296747569b107cfd05380b1ea3e"
},
{
"ImportPath": "golang.org/x/crypto/scrypt",
"Rev": "cc04154d65fb9296747569b107cfd05380b1ea3e"
},
{
"ImportPath": "golang.org/x/crypto/ssh",
"Rev": "cc04154d65fb9296747569b107cfd05380b1ea3e"
}
]
}

5
Godeps/Readme generated
View File

@@ -1,5 +0,0 @@
This directory tree is generated automatically by godep.
Please do not edit.
See https://github.com/tools/godep for more information.

2
Godeps/_workspace/.gitignore generated vendored
View File

@@ -1,2 +0,0 @@
/pkg
/bin

View File

@@ -1,2 +0,0 @@
*.go filter=gofmt
*.cgo filter=gofmt

View File

@@ -1,11 +0,0 @@
*~
.#*
## the next line needs to start with a backslash to avoid looking like
## a comment
\#*#
.*.swp
*.test
/clockfs
/hellofs

View File

@@ -1,93 +0,0 @@
Copyright (c) 2013-2015 Tommi Virtanen.
Copyright (c) 2009, 2011, 2012 The Go Authors.
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
The following included software components have additional copyright
notices and license terms that may differ from the above.
File fuse.go:
// Adapted from Plan 9 from User Space's src/cmd/9pfuse/fuse.c,
// which carries this notice:
//
// The files in this directory are subject to the following license.
//
// The author of this software is Russ Cox.
//
// Copyright (c) 2006 Russ Cox
//
// Permission to use, copy, modify, and distribute this software for any
// purpose without fee is hereby granted, provided that this entire notice
// is included in all copies of any software which is or includes a copy
// or modification of this software and in all copies of the supporting
// documentation for such software.
//
// THIS SOFTWARE IS BEING PROVIDED "AS IS", WITHOUT ANY EXPRESS OR IMPLIED
// WARRANTY. IN PARTICULAR, THE AUTHOR MAKES NO REPRESENTATION OR WARRANTY
// OF ANY KIND CONCERNING THE MERCHANTABILITY OF THIS SOFTWARE OR ITS
// FITNESS FOR ANY PARTICULAR PURPOSE.
File fuse_kernel.go:
// Derived from FUSE's fuse_kernel.h
/*
This file defines the kernel interface of FUSE
Copyright (C) 2001-2007 Miklos Szeredi <miklos@szeredi.hu>
This -- and only this -- header file may also be distributed under
the terms of the BSD Licence as follows:
Copyright (C) 2001-2007 Miklos Szeredi. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY AUTHOR AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.
*/

View File

@@ -1,23 +0,0 @@
bazil.org/fuse -- Filesystems in Go
===================================
`bazil.org/fuse` is a Go library for writing FUSE userspace
filesystems.
It is a from-scratch implementation of the kernel-userspace
communication protocol, and does not use the C library from the
project called FUSE. `bazil.org/fuse` embraces Go fully for safety and
ease of programming.
Heres how to get going:
go get bazil.org/fuse
Website: http://bazil.org/fuse/
Github repository: https://github.com/bazil/fuse
API docs: http://godoc.org/bazil.org/fuse
Our thanks to Russ Cox for his fuse library, which this project is
based on.

View File

@@ -1,35 +0,0 @@
package fuse
import "unsafe"
// buffer provides a mechanism for constructing a message from
// multiple segments.
type buffer []byte
// alloc allocates size bytes and returns a pointer to the new
// segment.
func (w *buffer) alloc(size uintptr) unsafe.Pointer {
s := int(size)
if len(*w)+s > cap(*w) {
old := *w
*w = make([]byte, len(*w), 2*cap(*w)+s)
copy(*w, old)
}
l := len(*w)
*w = (*w)[:l+s]
return unsafe.Pointer(&(*w)[l])
}
// reset clears out the contents of the buffer.
func (w *buffer) reset() {
for i := range (*w)[:cap(*w)] {
(*w)[i] = 0
}
*w = (*w)[:0]
}
func newBuffer(extra uintptr) buffer {
const hdrSize = unsafe.Sizeof(outHeader{})
buf := make(buffer, hdrSize, hdrSize+extra)
return buf
}

View File

@@ -1,21 +0,0 @@
package fuse
import (
"runtime"
)
func stack() string {
buf := make([]byte, 1024)
return string(buf[:runtime.Stack(buf, false)])
}
func nop(msg interface{}) {}
// Debug is called to output debug messages, including protocol
// traces. The default behavior is to do nothing.
//
// The messages have human-friendly string representations and are
// safe to marshal to JSON.
//
// Implementations must not retain msg.
var Debug func(msg interface{}) = nop

View File

@@ -1,4 +0,0 @@
/*.seq.svg
# not ignoring *.seq.png; we want those committed to the repo
# for embedding on Github

View File

@@ -1,6 +0,0 @@
# bazil.org/fuse documentation
See also API docs at http://godoc.org/bazil.org/fuse
- [The mount sequence](mount-sequence.md)
- [Writing documentation](writing-docs.md)

View File

@@ -1,32 +0,0 @@
seqdiag {
app;
fuse [label="bazil.org/fuse"];
fusermount;
kernel;
mounts;
app;
fuse [label="bazil.org/fuse"];
fusermount;
kernel;
mounts;
app -> fuse [label="Mount"];
fuse -> fusermount [label="spawn, pass socketpair fd"];
fusermount -> kernel [label="open /dev/fuse"];
fusermount -> kernel [label="mount(2)"];
kernel ->> mounts [label="mount is visible"];
fusermount <-- kernel [label="mount(2) returns"];
fuse <<-- fusermount [diagonal, label="exit, receive /dev/fuse fd", leftnote="on Linux, successful exit here\nmeans the mount has happened,\nthough InitRequest might not have yet"];
app <-- fuse [label="Mount returns\nConn.Ready is already closed"];
app -> fuse [label="fs.Serve"];
fuse => kernel [label="read /dev/fuse fd", note="starts with InitRequest"];
fuse -> app [label="Init"];
fuse <-- app [color=red];
fuse -> kernel [label="write /dev/fuse fd", color=red];
kernel -> kernel [label="set connection\nstate to error", color=red];
fuse <-- kernel;
... conn.MountError == nil, so it is still mounted ...
... call conn.Close to clean up ...
}

Binary file not shown.

Before

Width:  |  Height:  |  Size: 28 KiB

View File

@@ -1,41 +0,0 @@
seqdiag {
// seqdiag -T svg -o doc/mount-osx.svg doc/mount-osx.seq
app;
fuse [label="bazil.org/fuse"];
fusermount;
kernel;
mounts;
app -> fuse [label="Mount"];
fuse -> fusermount [label="spawn, pass socketpair fd"];
fusermount -> kernel [label="open /dev/fuse"];
fusermount -> kernel [label="mount(2)"];
kernel ->> mounts [label="mount is visible"];
fusermount <-- kernel [label="mount(2) returns"];
fuse <<-- fusermount [diagonal, label="exit, receive /dev/fuse fd", leftnote="on Linux, successful exit here\nmeans the mount has happened,\nthough InitRequest might not have yet"];
app <-- fuse [label="Mount returns\nConn.Ready is already closed", rightnote="InitRequest and StatfsRequest\nmay or may not be seen\nbefore Conn.Ready,\ndepending on platform"];
app -> fuse [label="fs.Serve"];
fuse => kernel [label="read /dev/fuse fd", note="starts with InitRequest"];
fuse => app [label="FS/Node/Handle methods"];
fuse => kernel [label="write /dev/fuse fd"];
... repeat ...
... shutting down ...
app -> fuse [label="Unmount"];
fuse -> fusermount [label="fusermount -u"];
fusermount -> kernel;
kernel <<-- mounts;
fusermount <-- kernel;
fuse <<-- fusermount [diagonal];
app <-- fuse [label="Unmount returns"];
// actually triggers before above
fuse <<-- kernel [diagonal, label="/dev/fuse EOF"];
app <-- fuse [label="fs.Serve returns"];
app -> fuse [label="conn.Close"];
fuse -> kernel [label="close /dev/fuse fd"];
fuse <-- kernel;
app <-- fuse;
}

Binary file not shown.

Before

Width:  |  Height:  |  Size: 44 KiB

View File

@@ -1,32 +0,0 @@
seqdiag {
app;
fuse [label="bazil.org/fuse"];
wait [label="callMount\nhelper goroutine"];
mount_osxfusefs;
kernel;
app -> fuse [label="Mount"];
fuse -> kernel [label="open /dev/osxfuseN"];
fuse -> mount_osxfusefs [label="spawn, pass fd"];
fuse -> wait [label="goroutine", note="blocks on cmd.Wait"];
app <-- fuse [label="Mount returns"];
mount_osxfusefs -> kernel [label="mount(2)"];
app -> fuse [label="fs.Serve"];
fuse => kernel [label="read /dev/osxfuseN fd", note="starts with InitRequest,\nalso seen before mount exits:\ntwo StatfsRequest calls"];
fuse -> app [label="Init"];
fuse <-- app [color=red];
fuse -> kernel [label="write /dev/osxfuseN fd", color=red];
fuse <-- kernel;
mount_osxfusefs <-- kernel [label="mount(2) returns", color=red];
wait <<-- mount_osxfusefs [diagonal, label="exit", color=red];
app <<-- wait [diagonal, label="mount has failed,\nclose Conn.Ready", color=red];
// actually triggers before above
fuse <<-- kernel [diagonal, label="/dev/osxfuseN EOF"];
app <-- fuse [label="fs.Serve returns"];
... conn.MountError != nil, so it was was never mounted ...
... call conn.Close to clean up ...
}

Binary file not shown.

Before

Width:  |  Height:  |  Size: 32 KiB

View File

@@ -1,45 +0,0 @@
seqdiag {
// seqdiag -T svg -o doc/mount-osx.svg doc/mount-osx.seq
app;
fuse [label="bazil.org/fuse"];
wait [label="callMount\nhelper goroutine"];
mount_osxfusefs;
kernel;
mounts;
app -> fuse [label="Mount"];
fuse -> kernel [label="open /dev/osxfuseN"];
fuse -> mount_osxfusefs [label="spawn, pass fd"];
fuse -> wait [label="goroutine", note="blocks on cmd.Wait"];
app <-- fuse [label="Mount returns"];
mount_osxfusefs -> kernel [label="mount(2)"];
app -> fuse [label="fs.Serve"];
fuse => kernel [label="read /dev/osxfuseN fd", note="starts with InitRequest,\nalso seen before mount exits:\ntwo StatfsRequest calls"];
fuse => app [label="FS/Node/Handle methods"];
fuse => kernel [label="write /dev/osxfuseN fd"];
... repeat ...
kernel ->> mounts [label="mount is visible"];
mount_osxfusefs <-- kernel [label="mount(2) returns"];
wait <<-- mount_osxfusefs [diagonal, label="exit", leftnote="on OS X, successful exit\nhere means we finally know\nthe mount has happened\n(can't trust InitRequest,\nkernel might have timed out\nwaiting for InitResponse)"];
app <<-- wait [diagonal, label="mount is ready,\nclose Conn.Ready", rightnote="InitRequest and StatfsRequest\nmay or may not be seen\nbefore Conn.Ready,\ndepending on platform"];
... shutting down ...
app -> fuse [label="Unmount"];
fuse -> kernel [label="umount(2)"];
kernel <<-- mounts;
fuse <-- kernel;
app <-- fuse [label="Unmount returns"];
// actually triggers before above
fuse <<-- kernel [diagonal, label="/dev/osxfuseN EOF"];
app <-- fuse [label="fs.Serve returns"];
app -> fuse [label="conn.Close"];
fuse -> kernel [label="close /dev/osxfuseN"];
fuse <-- kernel;
app <-- fuse;
}

Binary file not shown.

Before

Width:  |  Height:  |  Size: 50 KiB

View File

@@ -1,30 +0,0 @@
# The mount sequence
FUSE mounting is a little bit tricky. There's a userspace helper tool
that performs the handshake with the kernel, and then steps out of the
way. This helper behaves differently on different platforms, forcing a
more complex API on us.
## Successful runs
On Linux, the mount is immediate and file system accesses wait until
the requests are served.
![Diagram of Linux FUSE mount sequence](mount-linux.seq.png)
On OS X, the mount becomes visible only after `InitRequest` (and maybe
more) have been served.
![Diagram of OSXFUSE mount sequence](mount-osx.seq.png)
## Errors
Let's see what happens if `InitRequest` gets an error response. On
Linux, the mountpoint is there but all operations will fail:
![Diagram of Linux error handling](mount-linux-error-init.seq.png)
On OS X, the mount never happened:
![Diagram of OS X error handling](mount-osx-error-init.seq.png)

View File

@@ -1,16 +0,0 @@
# Writing documentation
## Sequence diagrams
The sequence diagrams are generated with `seqdiag`:
http://blockdiag.com/en/seqdiag/index.html
An easy way to work on them is to automatically update the generated
files with https://github.com/cespare/reflex :
reflex -g 'doc/[^.]*.seq' -- seqdiag -T svg -o '{}.svg' '{}' &
reflex -g 'doc/[^.]*.seq' -- seqdiag -T png -o '{}.png' '{}' &
The markdown files refer to PNG images because of Github limitations,
but the SVG is generally more pleasant to view.

View File

@@ -1,17 +0,0 @@
package fuse
import (
"syscall"
)
const (
ENOATTR = Errno(syscall.ENOATTR)
)
const (
errNoXattr = ENOATTR
)
func init() {
errnoNames[errNoXattr] = "ENOATTR"
}

View File

@@ -1,15 +0,0 @@
package fuse
import "syscall"
const (
ENOATTR = Errno(syscall.ENOATTR)
)
const (
errNoXattr = ENOATTR
)
func init() {
errnoNames[errNoXattr] = "ENOATTR"
}

View File

@@ -1,17 +0,0 @@
package fuse
import (
"syscall"
)
const (
ENODATA = Errno(syscall.ENODATA)
)
const (
errNoXattr = ENODATA
)
func init() {
errnoNames[errNoXattr] = "ENODATA"
}

View File

@@ -1,31 +0,0 @@
package fuse
// There is very little commonality in extended attribute errors
// across platforms.
//
// getxattr return value for "extended attribute does not exist" is
// ENOATTR on OS X, and ENODATA on Linux and apparently at least
// NetBSD. There may be a #define ENOATTR on Linux too, but the value
// is ENODATA in the actual syscalls. FreeBSD and OpenBSD have no
// ENODATA, only ENOATTR. ENOATTR is not in any of the standards,
// ENODATA exists but is only used for STREAMs.
//
// Each platform will define it a errNoXattr constant, and this file
// will enforce that it implements the right interfaces and hide the
// implementation.
//
// https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man2/getxattr.2.html
// http://mail-index.netbsd.org/tech-kern/2012/04/30/msg013090.html
// http://mail-index.netbsd.org/tech-kern/2012/04/30/msg013097.html
// http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/errno.h.html
// http://www.freebsd.org/cgi/man.cgi?query=extattr_get_file&sektion=2
// http://nixdoc.net/man-pages/openbsd/man2/extattr_get_file.2.html
// ErrNoXattr is a platform-independent error value meaning the
// extended attribute was not found. It can be used to respond to
// GetxattrRequest and such.
const ErrNoXattr = errNoXattr
var _ error = ErrNoXattr
var _ Errno = ErrNoXattr
var _ ErrorNumber = ErrNoXattr

View File

@@ -1,173 +0,0 @@
// Clockfs implements a file system with the current time in a file.
// It was written to demonstrate kernel cache invalidation.
package main
import (
"flag"
"fmt"
"log"
"os"
"sync/atomic"
"syscall"
"time"
"bazil.org/fuse"
"bazil.org/fuse/fs"
_ "bazil.org/fuse/fs/fstestutil"
"bazil.org/fuse/fuseutil"
"golang.org/x/net/context"
)
func usage() {
fmt.Fprintf(os.Stderr, "Usage of %s:\n", os.Args[0])
fmt.Fprintf(os.Stderr, " %s MOUNTPOINT\n", os.Args[0])
flag.PrintDefaults()
}
func main() {
flag.Usage = usage
flag.Parse()
if flag.NArg() != 1 {
usage()
os.Exit(2)
}
mountpoint := flag.Arg(0)
c, err := fuse.Mount(
mountpoint,
fuse.FSName("clock"),
fuse.Subtype("clockfsfs"),
fuse.LocalVolume(),
fuse.VolumeName("Clock filesystem"),
)
if err != nil {
log.Fatal(err)
}
defer c.Close()
srv := fs.New(c, nil)
filesys := &FS{
// We pre-create the clock node so that it's always the same
// object returned from all the Lookups. You could carefully
// track its lifetime between Lookup&Forget, and have the
// ticking & invalidation happen only when active, but let's
// keep this example simple.
clockFile: &File{
fuse: srv,
},
}
filesys.clockFile.tick()
// This goroutine never exits. That's fine for this example.
go filesys.clockFile.update()
if err := srv.Serve(filesys); err != nil {
log.Fatal(err)
}
// Check if the mount process has an error to report.
<-c.Ready
if err := c.MountError; err != nil {
log.Fatal(err)
}
}
type FS struct {
clockFile *File
}
var _ fs.FS = (*FS)(nil)
func (f *FS) Root() (fs.Node, error) {
return &Dir{fs: f}, nil
}
// Dir implements both Node and Handle for the root directory.
type Dir struct {
fs *FS
}
var _ fs.Node = (*Dir)(nil)
func (d *Dir) Attr(ctx context.Context, a *fuse.Attr) error {
a.Inode = 1
a.Mode = os.ModeDir | 0555
return nil
}
var _ fs.NodeStringLookuper = (*Dir)(nil)
func (d *Dir) Lookup(ctx context.Context, name string) (fs.Node, error) {
if name == "clock" {
return d.fs.clockFile, nil
}
return nil, fuse.ENOENT
}
var dirDirs = []fuse.Dirent{
{Inode: 2, Name: "clock", Type: fuse.DT_File},
}
var _ fs.HandleReadDirAller = (*Dir)(nil)
func (d *Dir) ReadDirAll(ctx context.Context) ([]fuse.Dirent, error) {
return dirDirs, nil
}
type File struct {
fs.NodeRef
fuse *fs.Server
content atomic.Value
count uint64
}
var _ fs.Node = (*File)(nil)
func (f *File) Attr(ctx context.Context, a *fuse.Attr) error {
a.Inode = 2
a.Mode = 0444
t := f.content.Load().(string)
a.Size = uint64(len(t))
return nil
}
var _ fs.NodeOpener = (*File)(nil)
func (f *File) Open(ctx context.Context, req *fuse.OpenRequest, resp *fuse.OpenResponse) (fs.Handle, error) {
if !req.Flags.IsReadOnly() {
return nil, fuse.Errno(syscall.EACCES)
}
resp.Flags |= fuse.OpenKeepCache
return f, nil
}
var _ fs.Handle = (*File)(nil)
var _ fs.HandleReader = (*File)(nil)
func (f *File) Read(ctx context.Context, req *fuse.ReadRequest, resp *fuse.ReadResponse) error {
t := f.content.Load().(string)
fuseutil.HandleRead(req, resp, []byte(t))
return nil
}
func (f *File) tick() {
// Intentionally a variable-length format, to demonstrate size changes.
f.count++
s := fmt.Sprintf("%d\t%s\n", f.count, time.Now())
f.content.Store(s)
// For simplicity, this example tries to send invalidate
// notifications even when the kernel does not hold a reference to
// the node, so be extra sure to ignore ErrNotCached.
if err := f.fuse.InvalidateNodeData(f); err != nil && err != fuse.ErrNotCached {
log.Printf("invalidate error: %v", err)
}
}
func (f *File) update() {
tick := time.NewTicker(1 * time.Second)
defer tick.Stop()
for range tick.C {
f.tick()
}
}

View File

@@ -1,101 +0,0 @@
// Hellofs implements a simple "hello world" file system.
package main
import (
"flag"
"fmt"
"log"
"os"
"bazil.org/fuse"
"bazil.org/fuse/fs"
_ "bazil.org/fuse/fs/fstestutil"
"golang.org/x/net/context"
)
var Usage = func() {
fmt.Fprintf(os.Stderr, "Usage of %s:\n", os.Args[0])
fmt.Fprintf(os.Stderr, " %s MOUNTPOINT\n", os.Args[0])
flag.PrintDefaults()
}
func main() {
flag.Usage = Usage
flag.Parse()
if flag.NArg() != 1 {
Usage()
os.Exit(2)
}
mountpoint := flag.Arg(0)
c, err := fuse.Mount(
mountpoint,
fuse.FSName("helloworld"),
fuse.Subtype("hellofs"),
fuse.LocalVolume(),
fuse.VolumeName("Hello world!"),
)
if err != nil {
log.Fatal(err)
}
defer c.Close()
err = fs.Serve(c, FS{})
if err != nil {
log.Fatal(err)
}
// check if the mount process has an error to report
<-c.Ready
if err := c.MountError; err != nil {
log.Fatal(err)
}
}
// FS implements the hello world file system.
type FS struct{}
func (FS) Root() (fs.Node, error) {
return Dir{}, nil
}
// Dir implements both Node and Handle for the root directory.
type Dir struct{}
func (Dir) Attr(ctx context.Context, a *fuse.Attr) error {
a.Inode = 1
a.Mode = os.ModeDir | 0555
return nil
}
func (Dir) Lookup(ctx context.Context, name string) (fs.Node, error) {
if name == "hello" {
return File{}, nil
}
return nil, fuse.ENOENT
}
var dirDirs = []fuse.Dirent{
{Inode: 2, Name: "hello", Type: fuse.DT_File},
}
func (Dir) ReadDirAll(ctx context.Context) ([]fuse.Dirent, error) {
return dirDirs, nil
}
// File implements both Node and Handle for the hello file.
type File struct{}
const greeting = "hello, world\n"
func (File) Attr(ctx context.Context, a *fuse.Attr) error {
a.Inode = 2
a.Mode = 0444
a.Size = uint64(len(greeting))
return nil
}
func (File) ReadAll(ctx context.Context) ([]byte, error) {
return []byte(greeting), nil
}

View File

@@ -1,268 +0,0 @@
package bench_test
import (
"io"
"io/ioutil"
"os"
"path"
"testing"
"bazil.org/fuse"
"bazil.org/fuse/fs"
"bazil.org/fuse/fs/fstestutil"
"golang.org/x/net/context"
)
type benchConfig struct {
directIO bool
}
type benchFS struct {
conf *benchConfig
}
var _ = fs.FS(benchFS{})
func (f benchFS) Root() (fs.Node, error) {
return benchDir{conf: f.conf}, nil
}
type benchDir struct {
conf *benchConfig
}
var _ = fs.Node(benchDir{})
var _ = fs.NodeStringLookuper(benchDir{})
var _ = fs.Handle(benchDir{})
var _ = fs.HandleReadDirAller(benchDir{})
func (benchDir) Attr(ctx context.Context, a *fuse.Attr) error {
a.Inode = 1
a.Mode = os.ModeDir | 0555
return nil
}
func (d benchDir) Lookup(ctx context.Context, name string) (fs.Node, error) {
if name == "bench" {
return benchFile{conf: d.conf}, nil
}
return nil, fuse.ENOENT
}
func (benchDir) ReadDirAll(ctx context.Context) ([]fuse.Dirent, error) {
l := []fuse.Dirent{
{Inode: 2, Name: "bench", Type: fuse.DT_File},
}
return l, nil
}
type benchFile struct {
conf *benchConfig
}
var _ = fs.Node(benchFile{})
var _ = fs.NodeOpener(benchFile{})
var _ = fs.NodeFsyncer(benchFile{})
var _ = fs.Handle(benchFile{})
var _ = fs.HandleReader(benchFile{})
var _ = fs.HandleWriter(benchFile{})
func (benchFile) Attr(ctx context.Context, a *fuse.Attr) error {
a.Inode = 2
a.Mode = 0644
a.Size = 9999999999999999
return nil
}
func (f benchFile) Open(ctx context.Context, req *fuse.OpenRequest, resp *fuse.OpenResponse) (fs.Handle, error) {
if f.conf.directIO {
resp.Flags |= fuse.OpenDirectIO
}
// TODO configurable?
resp.Flags |= fuse.OpenKeepCache
return f, nil
}
func (benchFile) Read(ctx context.Context, req *fuse.ReadRequest, resp *fuse.ReadResponse) error {
resp.Data = resp.Data[:cap(resp.Data)]
return nil
}
func (benchFile) Write(ctx context.Context, req *fuse.WriteRequest, resp *fuse.WriteResponse) error {
resp.Size = len(req.Data)
return nil
}
func (benchFile) Fsync(ctx context.Context, req *fuse.FsyncRequest) error {
return nil
}
func benchmark(b *testing.B, fn func(b *testing.B, mnt string), conf *benchConfig) {
filesys := benchFS{
conf: conf,
}
mnt, err := fstestutil.Mounted(filesys, nil,
fuse.MaxReadahead(64*1024*1024),
fuse.AsyncRead(),
fuse.WritebackCache(),
)
if err != nil {
b.Fatal(err)
}
defer mnt.Close()
fn(b, mnt.Dir)
}
type zero struct{}
func (zero) Read(p []byte) (n int, err error) {
return len(p), nil
}
var Zero io.Reader = zero{}
func doWrites(size int64) func(b *testing.B, mnt string) {
return func(b *testing.B, mnt string) {
p := path.Join(mnt, "bench")
f, err := os.Create(p)
if err != nil {
b.Fatalf("create: %v", err)
}
defer f.Close()
b.ResetTimer()
b.SetBytes(size)
for i := 0; i < b.N; i++ {
_, err = io.CopyN(f, Zero, size)
if err != nil {
b.Fatalf("write: %v", err)
}
}
}
}
func BenchmarkWrite100(b *testing.B) {
benchmark(b, doWrites(100), &benchConfig{})
}
func BenchmarkWrite10MB(b *testing.B) {
benchmark(b, doWrites(10*1024*1024), &benchConfig{})
}
func BenchmarkWrite100MB(b *testing.B) {
benchmark(b, doWrites(100*1024*1024), &benchConfig{})
}
func BenchmarkDirectWrite100(b *testing.B) {
benchmark(b, doWrites(100), &benchConfig{
directIO: true,
})
}
func BenchmarkDirectWrite10MB(b *testing.B) {
benchmark(b, doWrites(10*1024*1024), &benchConfig{
directIO: true,
})
}
func BenchmarkDirectWrite100MB(b *testing.B) {
benchmark(b, doWrites(100*1024*1024), &benchConfig{
directIO: true,
})
}
func doWritesSync(size int64) func(b *testing.B, mnt string) {
return func(b *testing.B, mnt string) {
p := path.Join(mnt, "bench")
f, err := os.Create(p)
if err != nil {
b.Fatalf("create: %v", err)
}
defer f.Close()
b.ResetTimer()
b.SetBytes(size)
for i := 0; i < b.N; i++ {
_, err = io.CopyN(f, Zero, size)
if err != nil {
b.Fatalf("write: %v", err)
}
if err := f.Sync(); err != nil {
b.Fatalf("sync: %v", err)
}
}
}
}
func BenchmarkWriteSync100(b *testing.B) {
benchmark(b, doWritesSync(100), &benchConfig{})
}
func BenchmarkWriteSync10MB(b *testing.B) {
benchmark(b, doWritesSync(10*1024*1024), &benchConfig{})
}
func BenchmarkWriteSync100MB(b *testing.B) {
benchmark(b, doWritesSync(100*1024*1024), &benchConfig{})
}
func doReads(size int64) func(b *testing.B, mnt string) {
return func(b *testing.B, mnt string) {
p := path.Join(mnt, "bench")
f, err := os.Open(p)
if err != nil {
b.Fatalf("close: %v", err)
}
defer f.Close()
b.ResetTimer()
b.SetBytes(size)
for i := 0; i < b.N; i++ {
n, err := io.CopyN(ioutil.Discard, f, size)
if err != nil {
b.Fatalf("read: %v", err)
}
if n != size {
b.Errorf("unexpected size: %d != %d", n, size)
}
}
}
}
func BenchmarkRead100(b *testing.B) {
benchmark(b, doReads(100), &benchConfig{})
}
func BenchmarkRead10MB(b *testing.B) {
benchmark(b, doReads(10*1024*1024), &benchConfig{})
}
func BenchmarkRead100MB(b *testing.B) {
benchmark(b, doReads(100*1024*1024), &benchConfig{})
}
func BenchmarkDirectRead100(b *testing.B) {
benchmark(b, doReads(100), &benchConfig{
directIO: true,
})
}
func BenchmarkDirectRead10MB(b *testing.B) {
benchmark(b, doReads(10*1024*1024), &benchConfig{
directIO: true,
})
}
func BenchmarkDirectRead100MB(b *testing.B) {
benchmark(b, doReads(100*1024*1024), &benchConfig{
directIO: true,
})
}

View File

@@ -1,5 +0,0 @@
// Package bench contains benchmarks.
//
// It is kept in a separate package to avoid conflicting with the
// debug-heavy defaults for the actual tests.
package bench

View File

@@ -1,65 +0,0 @@
package fstestutil
import (
"flag"
"log"
"strconv"
"bazil.org/fuse"
)
type flagDebug bool
var debug flagDebug
var _ = flag.Value(&debug)
func (f *flagDebug) IsBoolFlag() bool {
return true
}
func nop(msg interface{}) {}
func (f *flagDebug) Set(s string) error {
v, err := strconv.ParseBool(s)
if err != nil {
return err
}
*f = flagDebug(v)
if v {
fuse.Debug = logMsg
} else {
fuse.Debug = nop
}
return nil
}
func (f *flagDebug) String() string {
return strconv.FormatBool(bool(*f))
}
func logMsg(msg interface{}) {
log.Printf("FUSE: %s\n", msg)
}
func init() {
flag.Var(&debug, "fuse.debug", "log FUSE processing details")
}
// DebugByDefault changes the default of the `-fuse.debug` flag to
// true.
//
// This package registers a command line flag `-fuse.debug` and when
// run with that flag (and activated inside the tests), logs FUSE
// debug messages.
//
// This is disabled by default, as most callers probably won't care
// about FUSE details. Use DebugByDefault for tests where you'd
// normally be passing `-fuse.debug` all the time anyway.
//
// Call from an init function.
func DebugByDefault() {
f := flag.Lookup("fuse.debug")
f.DefValue = "true"
f.Value.Set(f.DefValue)
}

View File

@@ -1 +0,0 @@
package fstestutil

View File

@@ -1,115 +0,0 @@
package fstestutil
import (
"errors"
"io/ioutil"
"log"
"os"
"testing"
"time"
"bazil.org/fuse"
"bazil.org/fuse/fs"
)
// Mount contains information about the mount for the test to use.
type Mount struct {
// Dir is the temporary directory where the filesystem is mounted.
Dir string
Conn *fuse.Conn
Server *fs.Server
// Error will receive the return value of Serve.
Error <-chan error
done <-chan struct{}
closed bool
}
// Close unmounts the filesystem and waits for fs.Serve to return. Any
// returned error will be stored in Err. It is safe to call Close
// multiple times.
func (mnt *Mount) Close() {
if mnt.closed {
return
}
mnt.closed = true
for tries := 0; tries < 1000; tries++ {
err := fuse.Unmount(mnt.Dir)
if err != nil {
// TODO do more than log?
log.Printf("unmount error: %v", err)
time.Sleep(10 * time.Millisecond)
continue
}
break
}
<-mnt.done
mnt.Conn.Close()
os.Remove(mnt.Dir)
}
// Mounted mounts the fuse.Server at a temporary directory.
//
// It also waits until the filesystem is known to be visible (OS X
// workaround).
//
// After successful return, caller must clean up by calling Close.
func Mounted(filesys fs.FS, conf *fs.Config, options ...fuse.MountOption) (*Mount, error) {
dir, err := ioutil.TempDir("", "fusetest")
if err != nil {
return nil, err
}
c, err := fuse.Mount(dir, options...)
if err != nil {
return nil, err
}
server := fs.New(c, conf)
done := make(chan struct{})
serveErr := make(chan error, 1)
mnt := &Mount{
Dir: dir,
Conn: c,
Server: server,
Error: serveErr,
done: done,
}
go func() {
defer close(done)
serveErr <- server.Serve(filesys)
}()
select {
case <-mnt.Conn.Ready:
if err := mnt.Conn.MountError; err != nil {
return nil, err
}
return mnt, nil
case err = <-mnt.Error:
// Serve quit early
if err != nil {
return nil, err
}
return nil, errors.New("Serve exited early")
}
}
// MountedT mounts the filesystem at a temporary directory,
// directing it's debug log to the testing logger.
//
// See Mounted for usage.
//
// The debug log is not enabled by default. Use `-fuse.debug` or call
// DebugByDefault to enable.
func MountedT(t testing.TB, filesys fs.FS, conf *fs.Config, options ...fuse.MountOption) (*Mount, error) {
if conf == nil {
conf = &fs.Config{}
}
if debug && conf.Debug == nil {
conf.Debug = func(msg interface{}) {
t.Logf("FUSE: %s", msg)
}
}
return Mounted(filesys, conf, options...)
}

View File

@@ -1,26 +0,0 @@
package fstestutil
// MountInfo describes a mounted file system.
type MountInfo struct {
FSName string
Type string
}
// GetMountInfo finds information about the mount at mnt. It is
// intended for use by tests only, and only fetches information
// relevant to the current tests.
func GetMountInfo(mnt string) (*MountInfo, error) {
return getMountInfo(mnt)
}
// cstr converts a nil-terminated C string into a Go string
func cstr(ca []int8) string {
s := make([]byte, 0, len(ca))
for _, c := range ca {
if c == 0x00 {
break
}
s = append(s, byte(c))
}
return string(s)
}

View File

@@ -1,29 +0,0 @@
package fstestutil
import (
"regexp"
"syscall"
)
var re = regexp.MustCompile(`\\(.)`)
// unescape removes backslash-escaping. The escaped characters are not
// mapped in any way; that is, unescape(`\n` ) == `n`.
func unescape(s string) string {
return re.ReplaceAllString(s, `$1`)
}
func getMountInfo(mnt string) (*MountInfo, error) {
var st syscall.Statfs_t
err := syscall.Statfs(mnt, &st)
if err != nil {
return nil, err
}
i := &MountInfo{
// osx getmntent(3) fails to un-escape the data, so we do it..
// this might lead to double-unescaping in the future. fun.
// TestMountOptionFSNameEvilBackslashDouble checks for that.
FSName: unescape(cstr(st.Mntfromname[:])),
}
return i, nil
}

View File

@@ -1,7 +0,0 @@
package fstestutil
import "errors"
func getMountInfo(mnt string) (*MountInfo, error) {
return nil, errors.New("FreeBSD has no useful mount information")
}

View File

@@ -1,51 +0,0 @@
package fstestutil
import (
"errors"
"io/ioutil"
"strings"
)
// Linux /proc/mounts shows current mounts.
// Same format as /etc/fstab. Quoting getmntent(3):
//
// Since fields in the mtab and fstab files are separated by whitespace,
// octal escapes are used to represent the four characters space (\040),
// tab (\011), newline (\012) and backslash (\134) in those files when
// they occur in one of the four strings in a mntent structure.
//
// http://linux.die.net/man/3/getmntent
var fstabUnescape = strings.NewReplacer(
`\040`, "\040",
`\011`, "\011",
`\012`, "\012",
`\134`, "\134",
)
var errNotFound = errors.New("mount not found")
func getMountInfo(mnt string) (*MountInfo, error) {
data, err := ioutil.ReadFile("/proc/mounts")
if err != nil {
return nil, err
}
for _, line := range strings.Split(string(data), "\n") {
fields := strings.Fields(line)
if len(fields) < 3 {
continue
}
// Fields are: fsname dir type opts freq passno
fsname := fstabUnescape.Replace(fields[0])
dir := fstabUnescape.Replace(fields[1])
fstype := fstabUnescape.Replace(fields[2])
if mnt == dir {
info := &MountInfo{
FSName: fsname,
Type: fstype,
}
return info, nil
}
}
return nil, errNotFound
}

View File

@@ -1,28 +0,0 @@
package record
import (
"bytes"
"io"
"sync"
)
// Buffer is like bytes.Buffer but safe to access from multiple
// goroutines.
type Buffer struct {
mu sync.Mutex
buf bytes.Buffer
}
var _ = io.Writer(&Buffer{})
func (b *Buffer) Write(p []byte) (n int, err error) {
b.mu.Lock()
defer b.mu.Unlock()
return b.buf.Write(p)
}
func (b *Buffer) Bytes() []byte {
b.mu.Lock()
defer b.mu.Unlock()
return b.buf.Bytes()
}

View File

@@ -1,384 +0,0 @@
package record
import (
"sync"
"sync/atomic"
"bazil.org/fuse"
"bazil.org/fuse/fs"
"golang.org/x/net/context"
)
// Writes gathers data from FUSE Write calls.
type Writes struct {
buf Buffer
}
var _ = fs.HandleWriter(&Writes{})
func (w *Writes) Write(ctx context.Context, req *fuse.WriteRequest, resp *fuse.WriteResponse) error {
n, err := w.buf.Write(req.Data)
resp.Size = n
if err != nil {
return err
}
return nil
}
func (w *Writes) RecordedWriteData() []byte {
return w.buf.Bytes()
}
// Counter records number of times a thing has occurred.
type Counter struct {
count uint32
}
func (r *Counter) Inc() {
atomic.AddUint32(&r.count, 1)
}
func (r *Counter) Count() uint32 {
return atomic.LoadUint32(&r.count)
}
// MarkRecorder records whether a thing has occurred.
type MarkRecorder struct {
count Counter
}
func (r *MarkRecorder) Mark() {
r.count.Inc()
}
func (r *MarkRecorder) Recorded() bool {
return r.count.Count() > 0
}
// Flushes notes whether a FUSE Flush call has been seen.
type Flushes struct {
rec MarkRecorder
}
var _ = fs.HandleFlusher(&Flushes{})
func (r *Flushes) Flush(ctx context.Context, req *fuse.FlushRequest) error {
r.rec.Mark()
return nil
}
func (r *Flushes) RecordedFlush() bool {
return r.rec.Recorded()
}
type Recorder struct {
mu sync.Mutex
val interface{}
}
// Record that we've seen value. A nil value is indistinguishable from
// no value recorded.
func (r *Recorder) Record(value interface{}) {
r.mu.Lock()
r.val = value
r.mu.Unlock()
}
func (r *Recorder) Recorded() interface{} {
r.mu.Lock()
val := r.val
r.mu.Unlock()
return val
}
type RequestRecorder struct {
rec Recorder
}
// Record a fuse.Request, after zeroing header fields that are hard to
// reproduce.
//
// Make sure to record a copy, not the original request.
func (r *RequestRecorder) RecordRequest(req fuse.Request) {
hdr := req.Hdr()
*hdr = fuse.Header{}
r.rec.Record(req)
}
func (r *RequestRecorder) Recorded() fuse.Request {
val := r.rec.Recorded()
if val == nil {
return nil
}
return val.(fuse.Request)
}
// Setattrs records a Setattr request and its fields.
type Setattrs struct {
rec RequestRecorder
}
var _ = fs.NodeSetattrer(&Setattrs{})
func (r *Setattrs) Setattr(ctx context.Context, req *fuse.SetattrRequest, resp *fuse.SetattrResponse) error {
tmp := *req
r.rec.RecordRequest(&tmp)
return nil
}
func (r *Setattrs) RecordedSetattr() fuse.SetattrRequest {
val := r.rec.Recorded()
if val == nil {
return fuse.SetattrRequest{}
}
return *(val.(*fuse.SetattrRequest))
}
// Fsyncs records an Fsync request and its fields.
type Fsyncs struct {
rec RequestRecorder
}
var _ = fs.NodeFsyncer(&Fsyncs{})
func (r *Fsyncs) Fsync(ctx context.Context, req *fuse.FsyncRequest) error {
tmp := *req
r.rec.RecordRequest(&tmp)
return nil
}
func (r *Fsyncs) RecordedFsync() fuse.FsyncRequest {
val := r.rec.Recorded()
if val == nil {
return fuse.FsyncRequest{}
}
return *(val.(*fuse.FsyncRequest))
}
// Mkdirs records a Mkdir request and its fields.
type Mkdirs struct {
rec RequestRecorder
}
var _ = fs.NodeMkdirer(&Mkdirs{})
// Mkdir records the request and returns an error. Most callers should
// wrap this call in a function that returns a more useful result.
func (r *Mkdirs) Mkdir(ctx context.Context, req *fuse.MkdirRequest) (fs.Node, error) {
tmp := *req
r.rec.RecordRequest(&tmp)
return nil, fuse.EIO
}
// RecordedMkdir returns information about the Mkdir request.
// If no request was seen, returns a zero value.
func (r *Mkdirs) RecordedMkdir() fuse.MkdirRequest {
val := r.rec.Recorded()
if val == nil {
return fuse.MkdirRequest{}
}
return *(val.(*fuse.MkdirRequest))
}
// Symlinks records a Symlink request and its fields.
type Symlinks struct {
rec RequestRecorder
}
var _ = fs.NodeSymlinker(&Symlinks{})
// Symlink records the request and returns an error. Most callers should
// wrap this call in a function that returns a more useful result.
func (r *Symlinks) Symlink(ctx context.Context, req *fuse.SymlinkRequest) (fs.Node, error) {
tmp := *req
r.rec.RecordRequest(&tmp)
return nil, fuse.EIO
}
// RecordedSymlink returns information about the Symlink request.
// If no request was seen, returns a zero value.
func (r *Symlinks) RecordedSymlink() fuse.SymlinkRequest {
val := r.rec.Recorded()
if val == nil {
return fuse.SymlinkRequest{}
}
return *(val.(*fuse.SymlinkRequest))
}
// Links records a Link request and its fields.
type Links struct {
rec RequestRecorder
}
var _ = fs.NodeLinker(&Links{})
// Link records the request and returns an error. Most callers should
// wrap this call in a function that returns a more useful result.
func (r *Links) Link(ctx context.Context, req *fuse.LinkRequest, old fs.Node) (fs.Node, error) {
tmp := *req
r.rec.RecordRequest(&tmp)
return nil, fuse.EIO
}
// RecordedLink returns information about the Link request.
// If no request was seen, returns a zero value.
func (r *Links) RecordedLink() fuse.LinkRequest {
val := r.rec.Recorded()
if val == nil {
return fuse.LinkRequest{}
}
return *(val.(*fuse.LinkRequest))
}
// Mknods records a Mknod request and its fields.
type Mknods struct {
rec RequestRecorder
}
var _ = fs.NodeMknoder(&Mknods{})
// Mknod records the request and returns an error. Most callers should
// wrap this call in a function that returns a more useful result.
func (r *Mknods) Mknod(ctx context.Context, req *fuse.MknodRequest) (fs.Node, error) {
tmp := *req
r.rec.RecordRequest(&tmp)
return nil, fuse.EIO
}
// RecordedMknod returns information about the Mknod request.
// If no request was seen, returns a zero value.
func (r *Mknods) RecordedMknod() fuse.MknodRequest {
val := r.rec.Recorded()
if val == nil {
return fuse.MknodRequest{}
}
return *(val.(*fuse.MknodRequest))
}
// Opens records a Open request and its fields.
type Opens struct {
rec RequestRecorder
}
var _ = fs.NodeOpener(&Opens{})
// Open records the request and returns an error. Most callers should
// wrap this call in a function that returns a more useful result.
func (r *Opens) Open(ctx context.Context, req *fuse.OpenRequest, resp *fuse.OpenResponse) (fs.Handle, error) {
tmp := *req
r.rec.RecordRequest(&tmp)
return nil, fuse.EIO
}
// RecordedOpen returns information about the Open request.
// If no request was seen, returns a zero value.
func (r *Opens) RecordedOpen() fuse.OpenRequest {
val := r.rec.Recorded()
if val == nil {
return fuse.OpenRequest{}
}
return *(val.(*fuse.OpenRequest))
}
// Getxattrs records a Getxattr request and its fields.
type Getxattrs struct {
rec RequestRecorder
}
var _ = fs.NodeGetxattrer(&Getxattrs{})
// Getxattr records the request and returns an error. Most callers should
// wrap this call in a function that returns a more useful result.
func (r *Getxattrs) Getxattr(ctx context.Context, req *fuse.GetxattrRequest, resp *fuse.GetxattrResponse) error {
tmp := *req
r.rec.RecordRequest(&tmp)
return fuse.ErrNoXattr
}
// RecordedGetxattr returns information about the Getxattr request.
// If no request was seen, returns a zero value.
func (r *Getxattrs) RecordedGetxattr() fuse.GetxattrRequest {
val := r.rec.Recorded()
if val == nil {
return fuse.GetxattrRequest{}
}
return *(val.(*fuse.GetxattrRequest))
}
// Listxattrs records a Listxattr request and its fields.
type Listxattrs struct {
rec RequestRecorder
}
var _ = fs.NodeListxattrer(&Listxattrs{})
// Listxattr records the request and returns an error. Most callers should
// wrap this call in a function that returns a more useful result.
func (r *Listxattrs) Listxattr(ctx context.Context, req *fuse.ListxattrRequest, resp *fuse.ListxattrResponse) error {
tmp := *req
r.rec.RecordRequest(&tmp)
return fuse.ErrNoXattr
}
// RecordedListxattr returns information about the Listxattr request.
// If no request was seen, returns a zero value.
func (r *Listxattrs) RecordedListxattr() fuse.ListxattrRequest {
val := r.rec.Recorded()
if val == nil {
return fuse.ListxattrRequest{}
}
return *(val.(*fuse.ListxattrRequest))
}
// Setxattrs records a Setxattr request and its fields.
type Setxattrs struct {
rec RequestRecorder
}
var _ = fs.NodeSetxattrer(&Setxattrs{})
// Setxattr records the request and returns an error. Most callers should
// wrap this call in a function that returns a more useful result.
func (r *Setxattrs) Setxattr(ctx context.Context, req *fuse.SetxattrRequest) error {
tmp := *req
// The byte slice points to memory that will be reused, so make a
// deep copy.
tmp.Xattr = append([]byte(nil), req.Xattr...)
r.rec.RecordRequest(&tmp)
return nil
}
// RecordedSetxattr returns information about the Setxattr request.
// If no request was seen, returns a zero value.
func (r *Setxattrs) RecordedSetxattr() fuse.SetxattrRequest {
val := r.rec.Recorded()
if val == nil {
return fuse.SetxattrRequest{}
}
return *(val.(*fuse.SetxattrRequest))
}
// Removexattrs records a Removexattr request and its fields.
type Removexattrs struct {
rec RequestRecorder
}
var _ = fs.NodeRemovexattrer(&Removexattrs{})
// Removexattr records the request and returns an error. Most callers should
// wrap this call in a function that returns a more useful result.
func (r *Removexattrs) Removexattr(ctx context.Context, req *fuse.RemovexattrRequest) error {
tmp := *req
r.rec.RecordRequest(&tmp)
return nil
}
// RecordedRemovexattr returns information about the Removexattr request.
// If no request was seen, returns a zero value.
func (r *Removexattrs) RecordedRemovexattr() fuse.RemovexattrRequest {
val := r.rec.Recorded()
if val == nil {
return fuse.RemovexattrRequest{}
}
return *(val.(*fuse.RemovexattrRequest))
}

View File

@@ -1,55 +0,0 @@
package record
import (
"sync"
"time"
"bazil.org/fuse"
"bazil.org/fuse/fs"
"golang.org/x/net/context"
)
type nothing struct{}
// ReleaseWaiter notes whether a FUSE Release call has been seen.
//
// Releases are not guaranteed to happen synchronously with any client
// call, so they must be waited for.
type ReleaseWaiter struct {
once sync.Once
seen chan nothing
}
var _ = fs.HandleReleaser(&ReleaseWaiter{})
func (r *ReleaseWaiter) init() {
r.once.Do(func() {
r.seen = make(chan nothing, 1)
})
}
func (r *ReleaseWaiter) Release(ctx context.Context, req *fuse.ReleaseRequest) error {
r.init()
close(r.seen)
return nil
}
// WaitForRelease waits for Release to be called.
//
// With zero duration, wait forever. Otherwise, timeout early
// in a more controller way than `-test.timeout`.
//
// Returns whether a Release was seen. Always true if dur==0.
func (r *ReleaseWaiter) WaitForRelease(dur time.Duration) bool {
r.init()
var timeout <-chan time.Time
if dur > 0 {
timeout = time.After(dur)
}
select {
case <-r.seen:
return true
case <-timeout:
return false
}
}

View File

@@ -1,55 +0,0 @@
package fstestutil
import (
"os"
"bazil.org/fuse"
"bazil.org/fuse/fs"
"golang.org/x/net/context"
)
// SimpleFS is a trivial FS that just implements the Root method.
type SimpleFS struct {
Node fs.Node
}
var _ = fs.FS(SimpleFS{})
func (f SimpleFS) Root() (fs.Node, error) {
return f.Node, nil
}
// File can be embedded in a struct to make it look like a file.
type File struct{}
func (f File) Attr(ctx context.Context, a *fuse.Attr) error {
a.Mode = 0666
return nil
}
// Dir can be embedded in a struct to make it look like a directory.
type Dir struct{}
func (f Dir) Attr(ctx context.Context, a *fuse.Attr) error {
a.Mode = os.ModeDir | 0777
return nil
}
// ChildMap is a directory with child nodes looked up from a map.
type ChildMap map[string]fs.Node
var _ = fs.Node(&ChildMap{})
var _ = fs.NodeStringLookuper(&ChildMap{})
func (f *ChildMap) Attr(ctx context.Context, a *fuse.Attr) error {
a.Mode = os.ModeDir | 0777
return nil
}
func (f *ChildMap) Lookup(ctx context.Context, name string) (fs.Node, error) {
child, ok := (*f)[name]
if !ok {
return nil, fuse.ENOENT
}
return child, nil
}

View File

@@ -1,67 +0,0 @@
package fs_test
import (
"errors"
"flag"
"os"
"os/exec"
"path/filepath"
"testing"
)
var childHelpers = map[string]func(){}
type childProcess struct {
name string
fn func()
}
var _ flag.Value = (*childProcess)(nil)
func (c *childProcess) String() string {
return c.name
}
func (c *childProcess) Set(s string) error {
fn, ok := childHelpers[s]
if !ok {
return errors.New("helper not found")
}
c.name = s
c.fn = fn
return nil
}
var childMode childProcess
func init() {
flag.Var(&childMode, "fuse.internal.child", "internal use only")
}
// childCmd prepares a test function to be run in a subprocess, with
// childMode set to true. Caller must still call Run or Start.
//
// Re-using the test executable as the subprocess is useful because
// now test executables can e.g. be cross-compiled, transferred
// between hosts, and run in settings where the whole Go development
// environment is not installed.
func childCmd(childName string) (*exec.Cmd, error) {
// caller may set cwd, so we can't rely on relative paths
executable, err := filepath.Abs(os.Args[0])
if err != nil {
return nil, err
}
cmd := exec.Command(executable, "-fuse.internal.child="+childName)
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
return cmd, nil
}
func TestMain(m *testing.M) {
flag.Parse()
if childMode.fn != nil {
childMode.fn()
os.Exit(0)
}
os.Exit(m.Run())
}

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -1,99 +0,0 @@
// FUSE directory tree, for servers that wish to use it with the service loop.
package fs
import (
"os"
pathpkg "path"
"strings"
"golang.org/x/net/context"
)
import (
"bazil.org/fuse"
)
// A Tree implements a basic read-only directory tree for FUSE.
// The Nodes contained in it may still be writable.
type Tree struct {
tree
}
func (t *Tree) Root() (Node, error) {
return &t.tree, nil
}
// Add adds the path to the tree, resolving to the given node.
// If path or a prefix of path has already been added to the tree,
// Add panics.
//
// Add is only safe to call before starting to serve requests.
func (t *Tree) Add(path string, node Node) {
path = pathpkg.Clean("/" + path)[1:]
elems := strings.Split(path, "/")
dir := Node(&t.tree)
for i, elem := range elems {
dt, ok := dir.(*tree)
if !ok {
panic("fuse: Tree.Add for " + strings.Join(elems[:i], "/") + " and " + path)
}
n := dt.lookup(elem)
if n != nil {
if i+1 == len(elems) {
panic("fuse: Tree.Add for " + path + " conflicts with " + elem)
}
dir = n
} else {
if i+1 == len(elems) {
dt.add(elem, node)
} else {
dir = &tree{}
dt.add(elem, dir)
}
}
}
}
type treeDir struct {
name string
node Node
}
type tree struct {
dir []treeDir
}
func (t *tree) lookup(name string) Node {
for _, d := range t.dir {
if d.name == name {
return d.node
}
}
return nil
}
func (t *tree) add(name string, n Node) {
t.dir = append(t.dir, treeDir{name, n})
}
func (t *tree) Attr(ctx context.Context, a *fuse.Attr) error {
a.Mode = os.ModeDir | 0555
return nil
}
func (t *tree) Lookup(ctx context.Context, name string) (Node, error) {
n := t.lookup(name)
if n != nil {
return n, nil
}
return nil, fuse.ENOENT
}
func (t *tree) ReadDirAll(ctx context.Context) ([]fuse.Dirent, error) {
var out []fuse.Dirent
for _, d := range t.dir {
out = append(out, fuse.Dirent{Name: d.name})
}
return out, nil
}

File diff suppressed because it is too large Load Diff

View File

@@ -1,766 +0,0 @@
// See the file LICENSE for copyright and licensing information.
// Derived from FUSE's fuse_kernel.h, which carries this notice:
/*
This file defines the kernel interface of FUSE
Copyright (C) 2001-2007 Miklos Szeredi <miklos@szeredi.hu>
This -- and only this -- header file may also be distributed under
the terms of the BSD Licence as follows:
Copyright (C) 2001-2007 Miklos Szeredi. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY AUTHOR AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.
*/
package fuse
import (
"fmt"
"syscall"
"unsafe"
)
// The FUSE version implemented by the package.
const (
protoVersionMinMajor = 7
protoVersionMinMinor = 8
protoVersionMaxMajor = 7
protoVersionMaxMinor = 12
)
const (
rootID = 1
)
type kstatfs struct {
Blocks uint64
Bfree uint64
Bavail uint64
Files uint64
Ffree uint64
Bsize uint32
Namelen uint32
Frsize uint32
Padding uint32
Spare [6]uint32
}
type fileLock struct {
Start uint64
End uint64
Type uint32
Pid uint32
}
// GetattrFlags are bit flags that can be seen in GetattrRequest.
type GetattrFlags uint32
const (
// Indicates the handle is valid.
GetattrFh GetattrFlags = 1 << 0
)
var getattrFlagsNames = []flagName{
{uint32(GetattrFh), "GetattrFh"},
}
func (fl GetattrFlags) String() string {
return flagString(uint32(fl), getattrFlagsNames)
}
// The SetattrValid are bit flags describing which fields in the SetattrRequest
// are included in the change.
type SetattrValid uint32
const (
SetattrMode SetattrValid = 1 << 0
SetattrUid SetattrValid = 1 << 1
SetattrGid SetattrValid = 1 << 2
SetattrSize SetattrValid = 1 << 3
SetattrAtime SetattrValid = 1 << 4
SetattrMtime SetattrValid = 1 << 5
SetattrHandle SetattrValid = 1 << 6
// Linux only(?)
SetattrAtimeNow SetattrValid = 1 << 7
SetattrMtimeNow SetattrValid = 1 << 8
SetattrLockOwner SetattrValid = 1 << 9 // http://www.mail-archive.com/git-commits-head@vger.kernel.org/msg27852.html
// OS X only
SetattrCrtime SetattrValid = 1 << 28
SetattrChgtime SetattrValid = 1 << 29
SetattrBkuptime SetattrValid = 1 << 30
SetattrFlags SetattrValid = 1 << 31
)
func (fl SetattrValid) Mode() bool { return fl&SetattrMode != 0 }
func (fl SetattrValid) Uid() bool { return fl&SetattrUid != 0 }
func (fl SetattrValid) Gid() bool { return fl&SetattrGid != 0 }
func (fl SetattrValid) Size() bool { return fl&SetattrSize != 0 }
func (fl SetattrValid) Atime() bool { return fl&SetattrAtime != 0 }
func (fl SetattrValid) Mtime() bool { return fl&SetattrMtime != 0 }
func (fl SetattrValid) Handle() bool { return fl&SetattrHandle != 0 }
func (fl SetattrValid) AtimeNow() bool { return fl&SetattrAtimeNow != 0 }
func (fl SetattrValid) MtimeNow() bool { return fl&SetattrMtimeNow != 0 }
func (fl SetattrValid) LockOwner() bool { return fl&SetattrLockOwner != 0 }
func (fl SetattrValid) Crtime() bool { return fl&SetattrCrtime != 0 }
func (fl SetattrValid) Chgtime() bool { return fl&SetattrChgtime != 0 }
func (fl SetattrValid) Bkuptime() bool { return fl&SetattrBkuptime != 0 }
func (fl SetattrValid) Flags() bool { return fl&SetattrFlags != 0 }
func (fl SetattrValid) String() string {
return flagString(uint32(fl), setattrValidNames)
}
var setattrValidNames = []flagName{
{uint32(SetattrMode), "SetattrMode"},
{uint32(SetattrUid), "SetattrUid"},
{uint32(SetattrGid), "SetattrGid"},
{uint32(SetattrSize), "SetattrSize"},
{uint32(SetattrAtime), "SetattrAtime"},
{uint32(SetattrMtime), "SetattrMtime"},
{uint32(SetattrHandle), "SetattrHandle"},
{uint32(SetattrAtimeNow), "SetattrAtimeNow"},
{uint32(SetattrMtimeNow), "SetattrMtimeNow"},
{uint32(SetattrLockOwner), "SetattrLockOwner"},
{uint32(SetattrCrtime), "SetattrCrtime"},
{uint32(SetattrChgtime), "SetattrChgtime"},
{uint32(SetattrBkuptime), "SetattrBkuptime"},
{uint32(SetattrFlags), "SetattrFlags"},
}
// Flags that can be seen in OpenRequest.Flags.
const (
// Access modes. These are not 1-bit flags, but alternatives where
// only one can be chosen. See the IsReadOnly etc convenience
// methods.
OpenReadOnly OpenFlags = syscall.O_RDONLY
OpenWriteOnly OpenFlags = syscall.O_WRONLY
OpenReadWrite OpenFlags = syscall.O_RDWR
OpenAppend OpenFlags = syscall.O_APPEND
OpenCreate OpenFlags = syscall.O_CREAT
OpenExclusive OpenFlags = syscall.O_EXCL
OpenSync OpenFlags = syscall.O_SYNC
OpenTruncate OpenFlags = syscall.O_TRUNC
)
// OpenAccessModeMask is a bitmask that separates the access mode
// from the other flags in OpenFlags.
const OpenAccessModeMask OpenFlags = syscall.O_ACCMODE
// OpenFlags are the O_FOO flags passed to open/create/etc calls. For
// example, os.O_WRONLY | os.O_APPEND.
type OpenFlags uint32
func (fl OpenFlags) String() string {
// O_RDONLY, O_RWONLY, O_RDWR are not flags
s := accModeName(fl & OpenAccessModeMask)
flags := uint32(fl &^ OpenAccessModeMask)
if flags != 0 {
s = s + "+" + flagString(flags, openFlagNames)
}
return s
}
// Return true if OpenReadOnly is set.
func (fl OpenFlags) IsReadOnly() bool {
return fl&OpenAccessModeMask == OpenReadOnly
}
// Return true if OpenWriteOnly is set.
func (fl OpenFlags) IsWriteOnly() bool {
return fl&OpenAccessModeMask == OpenWriteOnly
}
// Return true if OpenReadWrite is set.
func (fl OpenFlags) IsReadWrite() bool {
return fl&OpenAccessModeMask == OpenReadWrite
}
func accModeName(flags OpenFlags) string {
switch flags {
case OpenReadOnly:
return "OpenReadOnly"
case OpenWriteOnly:
return "OpenWriteOnly"
case OpenReadWrite:
return "OpenReadWrite"
default:
return ""
}
}
var openFlagNames = []flagName{
{uint32(OpenCreate), "OpenCreate"},
{uint32(OpenExclusive), "OpenExclusive"},
{uint32(OpenTruncate), "OpenTruncate"},
{uint32(OpenAppend), "OpenAppend"},
{uint32(OpenSync), "OpenSync"},
}
// The OpenResponseFlags are returned in the OpenResponse.
type OpenResponseFlags uint32
const (
OpenDirectIO OpenResponseFlags = 1 << 0 // bypass page cache for this open file
OpenKeepCache OpenResponseFlags = 1 << 1 // don't invalidate the data cache on open
OpenNonSeekable OpenResponseFlags = 1 << 2 // mark the file as non-seekable (not supported on OS X)
OpenPurgeAttr OpenResponseFlags = 1 << 30 // OS X
OpenPurgeUBC OpenResponseFlags = 1 << 31 // OS X
)
func (fl OpenResponseFlags) String() string {
return flagString(uint32(fl), openResponseFlagNames)
}
var openResponseFlagNames = []flagName{
{uint32(OpenDirectIO), "OpenDirectIO"},
{uint32(OpenKeepCache), "OpenKeepCache"},
{uint32(OpenNonSeekable), "OpenNonSeekable"},
{uint32(OpenPurgeAttr), "OpenPurgeAttr"},
{uint32(OpenPurgeUBC), "OpenPurgeUBC"},
}
// The InitFlags are used in the Init exchange.
type InitFlags uint32
const (
InitAsyncRead InitFlags = 1 << 0
InitPosixLocks InitFlags = 1 << 1
InitFileOps InitFlags = 1 << 2
InitAtomicTrunc InitFlags = 1 << 3
InitExportSupport InitFlags = 1 << 4
InitBigWrites InitFlags = 1 << 5
InitDontMask InitFlags = 1 << 6
InitSpliceWrite InitFlags = 1 << 7
InitSpliceMove InitFlags = 1 << 8
InitSpliceRead InitFlags = 1 << 9
InitFlockLocks InitFlags = 1 << 10
InitHasIoctlDir InitFlags = 1 << 11
InitAutoInvalData InitFlags = 1 << 12
InitDoReaddirplus InitFlags = 1 << 13
InitReaddirplusAuto InitFlags = 1 << 14
InitAsyncDIO InitFlags = 1 << 15
InitWritebackCache InitFlags = 1 << 16
InitNoOpenSupport InitFlags = 1 << 17
InitCaseSensitive InitFlags = 1 << 29 // OS X only
InitVolRename InitFlags = 1 << 30 // OS X only
InitXtimes InitFlags = 1 << 31 // OS X only
)
type flagName struct {
bit uint32
name string
}
var initFlagNames = []flagName{
{uint32(InitAsyncRead), "InitAsyncRead"},
{uint32(InitPosixLocks), "InitPosixLocks"},
{uint32(InitFileOps), "InitFileOps"},
{uint32(InitAtomicTrunc), "InitAtomicTrunc"},
{uint32(InitExportSupport), "InitExportSupport"},
{uint32(InitBigWrites), "InitBigWrites"},
{uint32(InitDontMask), "InitDontMask"},
{uint32(InitSpliceWrite), "InitSpliceWrite"},
{uint32(InitSpliceMove), "InitSpliceMove"},
{uint32(InitSpliceRead), "InitSpliceRead"},
{uint32(InitFlockLocks), "InitFlockLocks"},
{uint32(InitHasIoctlDir), "InitHasIoctlDir"},
{uint32(InitAutoInvalData), "InitAutoInvalData"},
{uint32(InitDoReaddirplus), "InitDoReaddirplus"},
{uint32(InitReaddirplusAuto), "InitReaddirplusAuto"},
{uint32(InitAsyncDIO), "InitAsyncDIO"},
{uint32(InitWritebackCache), "InitWritebackCache"},
{uint32(InitNoOpenSupport), "InitNoOpenSupport"},
{uint32(InitCaseSensitive), "InitCaseSensitive"},
{uint32(InitVolRename), "InitVolRename"},
{uint32(InitXtimes), "InitXtimes"},
}
func (fl InitFlags) String() string {
return flagString(uint32(fl), initFlagNames)
}
func flagString(f uint32, names []flagName) string {
var s string
if f == 0 {
return "0"
}
for _, n := range names {
if f&n.bit != 0 {
s += "+" + n.name
f &^= n.bit
}
}
if f != 0 {
s += fmt.Sprintf("%+#x", f)
}
return s[1:]
}
// The ReleaseFlags are used in the Release exchange.
type ReleaseFlags uint32
const (
ReleaseFlush ReleaseFlags = 1 << 0
)
func (fl ReleaseFlags) String() string {
return flagString(uint32(fl), releaseFlagNames)
}
var releaseFlagNames = []flagName{
{uint32(ReleaseFlush), "ReleaseFlush"},
}
// Opcodes
const (
opLookup = 1
opForget = 2 // no reply
opGetattr = 3
opSetattr = 4
opReadlink = 5
opSymlink = 6
opMknod = 8
opMkdir = 9
opUnlink = 10
opRmdir = 11
opRename = 12
opLink = 13
opOpen = 14
opRead = 15
opWrite = 16
opStatfs = 17
opRelease = 18
opFsync = 20
opSetxattr = 21
opGetxattr = 22
opListxattr = 23
opRemovexattr = 24
opFlush = 25
opInit = 26
opOpendir = 27
opReaddir = 28
opReleasedir = 29
opFsyncdir = 30
opGetlk = 31
opSetlk = 32
opSetlkw = 33
opAccess = 34
opCreate = 35
opInterrupt = 36
opBmap = 37
opDestroy = 38
opIoctl = 39 // Linux?
opPoll = 40 // Linux?
// OS X
opSetvolname = 61
opGetxtimes = 62
opExchange = 63
)
type entryOut struct {
Nodeid uint64 // Inode ID
Generation uint64 // Inode generation
EntryValid uint64 // Cache timeout for the name
AttrValid uint64 // Cache timeout for the attributes
EntryValidNsec uint32
AttrValidNsec uint32
Attr attr
}
func entryOutSize(p Protocol) uintptr {
switch {
case p.LT(Protocol{7, 9}):
return unsafe.Offsetof(entryOut{}.Attr) + unsafe.Offsetof(entryOut{}.Attr.Blksize)
default:
return unsafe.Sizeof(entryOut{})
}
}
type forgetIn struct {
Nlookup uint64
}
type getattrIn struct {
GetattrFlags uint32
dummy uint32
Fh uint64
}
type attrOut struct {
AttrValid uint64 // Cache timeout for the attributes
AttrValidNsec uint32
Dummy uint32
Attr attr
}
func attrOutSize(p Protocol) uintptr {
switch {
case p.LT(Protocol{7, 9}):
return unsafe.Offsetof(attrOut{}.Attr) + unsafe.Offsetof(attrOut{}.Attr.Blksize)
default:
return unsafe.Sizeof(attrOut{})
}
}
// OS X
type getxtimesOut struct {
Bkuptime uint64
Crtime uint64
BkuptimeNsec uint32
CrtimeNsec uint32
}
type mknodIn struct {
Mode uint32
Rdev uint32
Umask uint32
padding uint32
// "filename\x00" follows.
}
func mknodInSize(p Protocol) uintptr {
switch {
case p.LT(Protocol{7, 12}):
return unsafe.Offsetof(mknodIn{}.Umask)
default:
return unsafe.Sizeof(mknodIn{})
}
}
type mkdirIn struct {
Mode uint32
Umask uint32
// filename follows
}
func mkdirInSize(p Protocol) uintptr {
switch {
case p.LT(Protocol{7, 12}):
return unsafe.Offsetof(mkdirIn{}.Umask) + 4
default:
return unsafe.Sizeof(mkdirIn{})
}
}
type renameIn struct {
Newdir uint64
// "oldname\x00newname\x00" follows
}
// OS X
type exchangeIn struct {
Olddir uint64
Newdir uint64
Options uint64
}
type linkIn struct {
Oldnodeid uint64
}
type setattrInCommon struct {
Valid uint32
Padding uint32
Fh uint64
Size uint64
LockOwner uint64 // unused on OS X?
Atime uint64
Mtime uint64
Unused2 uint64
AtimeNsec uint32
MtimeNsec uint32
Unused3 uint32
Mode uint32
Unused4 uint32
Uid uint32
Gid uint32
Unused5 uint32
}
type openIn struct {
Flags uint32
Unused uint32
}
type openOut struct {
Fh uint64
OpenFlags uint32
Padding uint32
}
type createIn struct {
Flags uint32
Mode uint32
Umask uint32
padding uint32
}
func createInSize(p Protocol) uintptr {
switch {
case p.LT(Protocol{7, 12}):
return unsafe.Offsetof(createIn{}.Umask)
default:
return unsafe.Sizeof(createIn{})
}
}
type releaseIn struct {
Fh uint64
Flags uint32
ReleaseFlags uint32
LockOwner uint32
}
type flushIn struct {
Fh uint64
FlushFlags uint32
Padding uint32
LockOwner uint64
}
type readIn struct {
Fh uint64
Offset uint64
Size uint32
ReadFlags uint32
LockOwner uint64
Flags uint32
padding uint32
}
func readInSize(p Protocol) uintptr {
switch {
case p.LT(Protocol{7, 9}):
return unsafe.Offsetof(readIn{}.ReadFlags) + 4
default:
return unsafe.Sizeof(readIn{})
}
}
// The ReadFlags are passed in ReadRequest.
type ReadFlags uint32
const (
// LockOwner field is valid.
ReadLockOwner ReadFlags = 1 << 1
)
var readFlagNames = []flagName{
{uint32(ReadLockOwner), "ReadLockOwner"},
}
func (fl ReadFlags) String() string {
return flagString(uint32(fl), readFlagNames)
}
type writeIn struct {
Fh uint64
Offset uint64
Size uint32
WriteFlags uint32
LockOwner uint64
Flags uint32
padding uint32
}
func writeInSize(p Protocol) uintptr {
switch {
case p.LT(Protocol{7, 9}):
return unsafe.Offsetof(writeIn{}.LockOwner)
default:
return unsafe.Sizeof(writeIn{})
}
}
type writeOut struct {
Size uint32
Padding uint32
}
// The WriteFlags are passed in WriteRequest.
type WriteFlags uint32
const (
WriteCache WriteFlags = 1 << 0
// LockOwner field is valid.
WriteLockOwner WriteFlags = 1 << 1
)
var writeFlagNames = []flagName{
{uint32(WriteCache), "WriteCache"},
{uint32(WriteLockOwner), "WriteLockOwner"},
}
func (fl WriteFlags) String() string {
return flagString(uint32(fl), writeFlagNames)
}
const compatStatfsSize = 48
type statfsOut struct {
St kstatfs
}
type fsyncIn struct {
Fh uint64
FsyncFlags uint32
Padding uint32
}
type setxattrInCommon struct {
Size uint32
Flags uint32
}
func (setxattrInCommon) position() uint32 {
return 0
}
type getxattrInCommon struct {
Size uint32
Padding uint32
}
func (getxattrInCommon) position() uint32 {
return 0
}
type getxattrOut struct {
Size uint32
Padding uint32
}
type lkIn struct {
Fh uint64
Owner uint64
Lk fileLock
LkFlags uint32
padding uint32
}
func lkInSize(p Protocol) uintptr {
switch {
case p.LT(Protocol{7, 9}):
return unsafe.Offsetof(lkIn{}.LkFlags)
default:
return unsafe.Sizeof(lkIn{})
}
}
type lkOut struct {
Lk fileLock
}
type accessIn struct {
Mask uint32
Padding uint32
}
type initIn struct {
Major uint32
Minor uint32
MaxReadahead uint32
Flags uint32
}
const initInSize = int(unsafe.Sizeof(initIn{}))
type initOut struct {
Major uint32
Minor uint32
MaxReadahead uint32
Flags uint32
Unused uint32
MaxWrite uint32
}
type interruptIn struct {
Unique uint64
}
type bmapIn struct {
Block uint64
BlockSize uint32
Padding uint32
}
type bmapOut struct {
Block uint64
}
type inHeader struct {
Len uint32
Opcode uint32
Unique uint64
Nodeid uint64
Uid uint32
Gid uint32
Pid uint32
Padding uint32
}
const inHeaderSize = int(unsafe.Sizeof(inHeader{}))
type outHeader struct {
Len uint32
Error int32
Unique uint64
}
type dirent struct {
Ino uint64
Off uint64
Namelen uint32
Type uint32
Name [0]byte
}
const direntSize = 8 + 8 + 4 + 4
const (
notifyCodePoll int32 = 1
notifyCodeInvalInode int32 = 2
notifyCodeInvalEntry int32 = 3
)
type notifyInvalInodeOut struct {
Ino uint64
Off int64
Len int64
}
type notifyInvalEntryOut struct {
Parent uint64
Namelen uint32
padding uint32
}

View File

@@ -1,88 +0,0 @@
package fuse
import (
"time"
)
type attr struct {
Ino uint64
Size uint64
Blocks uint64
Atime uint64
Mtime uint64
Ctime uint64
Crtime_ uint64 // OS X only
AtimeNsec uint32
MtimeNsec uint32
CtimeNsec uint32
CrtimeNsec uint32 // OS X only
Mode uint32
Nlink uint32
Uid uint32
Gid uint32
Rdev uint32
Flags_ uint32 // OS X only; see chflags(2)
Blksize uint32
padding uint32
}
func (a *attr) SetCrtime(s uint64, ns uint32) {
a.Crtime_, a.CrtimeNsec = s, ns
}
func (a *attr) SetFlags(f uint32) {
a.Flags_ = f
}
type setattrIn struct {
setattrInCommon
// OS X only
Bkuptime_ uint64
Chgtime_ uint64
Crtime uint64
BkuptimeNsec uint32
ChgtimeNsec uint32
CrtimeNsec uint32
Flags_ uint32 // see chflags(2)
}
func (in *setattrIn) BkupTime() time.Time {
return time.Unix(int64(in.Bkuptime_), int64(in.BkuptimeNsec))
}
func (in *setattrIn) Chgtime() time.Time {
return time.Unix(int64(in.Chgtime_), int64(in.ChgtimeNsec))
}
func (in *setattrIn) Flags() uint32 {
return in.Flags_
}
func openFlags(flags uint32) OpenFlags {
return OpenFlags(flags)
}
type getxattrIn struct {
getxattrInCommon
// OS X only
Position uint32
Padding uint32
}
func (g *getxattrIn) position() uint32 {
return g.Position
}
type setxattrIn struct {
setxattrInCommon
// OS X only
Position uint32
Padding uint32
}
func (s *setxattrIn) position() uint32 {
return s.Position
}

View File

@@ -1,62 +0,0 @@
package fuse
import "time"
type attr struct {
Ino uint64
Size uint64
Blocks uint64
Atime uint64
Mtime uint64
Ctime uint64
AtimeNsec uint32
MtimeNsec uint32
CtimeNsec uint32
Mode uint32
Nlink uint32
Uid uint32
Gid uint32
Rdev uint32
Blksize uint32
padding uint32
}
func (a *attr) Crtime() time.Time {
return time.Time{}
}
func (a *attr) SetCrtime(s uint64, ns uint32) {
// ignored on freebsd
}
func (a *attr) SetFlags(f uint32) {
// ignored on freebsd
}
type setattrIn struct {
setattrInCommon
}
func (in *setattrIn) BkupTime() time.Time {
return time.Time{}
}
func (in *setattrIn) Chgtime() time.Time {
return time.Time{}
}
func (in *setattrIn) Flags() uint32 {
return 0
}
func openFlags(flags uint32) OpenFlags {
return OpenFlags(flags)
}
type getxattrIn struct {
getxattrInCommon
}
type setxattrIn struct {
setxattrInCommon
}

View File

@@ -1,70 +0,0 @@
package fuse
import "time"
type attr struct {
Ino uint64
Size uint64
Blocks uint64
Atime uint64
Mtime uint64
Ctime uint64
AtimeNsec uint32
MtimeNsec uint32
CtimeNsec uint32
Mode uint32
Nlink uint32
Uid uint32
Gid uint32
Rdev uint32
Blksize uint32
padding uint32
}
func (a *attr) Crtime() time.Time {
return time.Time{}
}
func (a *attr) SetCrtime(s uint64, ns uint32) {
// Ignored on Linux.
}
func (a *attr) SetFlags(f uint32) {
// Ignored on Linux.
}
type setattrIn struct {
setattrInCommon
}
func (in *setattrIn) BkupTime() time.Time {
return time.Time{}
}
func (in *setattrIn) Chgtime() time.Time {
return time.Time{}
}
func (in *setattrIn) Flags() uint32 {
return 0
}
func openFlags(flags uint32) OpenFlags {
// on amd64, the 32-bit O_LARGEFILE flag is always seen;
// on i386, the flag probably depends on the app
// requesting, but in any case should be utterly
// uninteresting to us here; our kernel protocol messages
// are not directly related to the client app's kernel
// API/ABI
flags &^= 0x8000
return OpenFlags(flags)
}
type getxattrIn struct {
getxattrInCommon
}
type setxattrIn struct {
setxattrInCommon
}

View File

@@ -1 +0,0 @@
package fuse

View File

@@ -1,31 +0,0 @@
package fuse_test
import (
"os"
"testing"
"bazil.org/fuse"
)
func TestOpenFlagsAccmodeMask(t *testing.T) {
var f = fuse.OpenFlags(os.O_RDWR | os.O_SYNC)
if g, e := f&fuse.OpenAccessModeMask, fuse.OpenReadWrite; g != e {
t.Fatalf("OpenAccessModeMask behaves wrong: %v: %o != %o", f, g, e)
}
if f.IsReadOnly() {
t.Fatalf("IsReadOnly is wrong: %v", f)
}
if f.IsWriteOnly() {
t.Fatalf("IsWriteOnly is wrong: %v", f)
}
if !f.IsReadWrite() {
t.Fatalf("IsReadWrite is wrong: %v", f)
}
}
func TestOpenFlagsString(t *testing.T) {
var f = fuse.OpenFlags(os.O_RDWR | os.O_SYNC | os.O_APPEND)
if g, e := f.String(), "OpenReadWrite+OpenAppend+OpenSync"; g != e {
t.Fatalf("OpenFlags.String: %q != %q", g, e)
}
}

View File

@@ -1,20 +0,0 @@
package fuseutil
import (
"bazil.org/fuse"
)
// HandleRead handles a read request assuming that data is the entire file content.
// It adjusts the amount returned in resp according to req.Offset and req.Size.
func HandleRead(req *fuse.ReadRequest, resp *fuse.ReadResponse, data []byte) {
if req.Offset >= int64(len(data)) {
data = nil
} else {
data = data[req.Offset:]
}
if len(data) > req.Size {
data = data[:req.Size]
}
n := copy(resp.Data[:req.Size], data)
resp.Data = resp.Data[:n]
}

View File

@@ -1,126 +0,0 @@
package fuse
import (
"bytes"
"errors"
"fmt"
"os"
"os/exec"
"strconv"
"strings"
"syscall"
)
var errNoAvail = errors.New("no available fuse devices")
var errNotLoaded = errors.New("osxfusefs is not loaded")
func loadOSXFUSE() error {
cmd := exec.Command("/Library/Filesystems/osxfusefs.fs/Support/load_osxfusefs")
cmd.Dir = "/"
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
err := cmd.Run()
return err
}
func openOSXFUSEDev() (*os.File, error) {
var f *os.File
var err error
for i := uint64(0); ; i++ {
path := "/dev/osxfuse" + strconv.FormatUint(i, 10)
f, err = os.OpenFile(path, os.O_RDWR, 0000)
if os.IsNotExist(err) {
if i == 0 {
// not even the first device was found -> fuse is not loaded
return nil, errNotLoaded
}
// we've run out of kernel-provided devices
return nil, errNoAvail
}
if err2, ok := err.(*os.PathError); ok && err2.Err == syscall.EBUSY {
// try the next one
continue
}
if err != nil {
return nil, err
}
return f, nil
}
}
func callMount(dir string, conf *mountConfig, f *os.File, ready chan<- struct{}, errp *error) error {
bin := "/Library/Filesystems/osxfusefs.fs/Support/mount_osxfusefs"
for k, v := range conf.options {
if strings.Contains(k, ",") || strings.Contains(v, ",") {
// Silly limitation but the mount helper does not
// understand any escaping. See TestMountOptionCommaError.
return fmt.Errorf("mount options cannot contain commas on darwin: %q=%q", k, v)
}
}
cmd := exec.Command(
bin,
"-o", conf.getOptions(),
// Tell osxfuse-kext how large our buffer is. It must split
// writes larger than this into multiple writes.
//
// OSXFUSE seems to ignore InitResponse.MaxWrite, and uses
// this instead.
"-o", "iosize="+strconv.FormatUint(maxWrite, 10),
// refers to fd passed in cmd.ExtraFiles
"3",
dir,
)
cmd.ExtraFiles = []*os.File{f}
cmd.Env = os.Environ()
cmd.Env = append(cmd.Env, "MOUNT_FUSEFS_CALL_BY_LIB=")
// TODO this is used for fs typenames etc, let app influence it
cmd.Env = append(cmd.Env, "MOUNT_FUSEFS_DAEMON_PATH="+bin)
var buf bytes.Buffer
cmd.Stdout = &buf
cmd.Stderr = &buf
err := cmd.Start()
if err != nil {
return err
}
go func() {
err := cmd.Wait()
if err != nil {
if buf.Len() > 0 {
output := buf.Bytes()
output = bytes.TrimRight(output, "\n")
msg := err.Error() + ": " + string(output)
err = errors.New(msg)
}
}
*errp = err
close(ready)
}()
return err
}
func mount(dir string, conf *mountConfig, ready chan<- struct{}, errp *error) (*os.File, error) {
f, err := openOSXFUSEDev()
if err == errNotLoaded {
err = loadOSXFUSE()
if err != nil {
return nil, err
}
// try again
f, err = openOSXFUSEDev()
}
if err != nil {
return nil, err
}
err = callMount(dir, conf, f, ready, errp)
if err != nil {
f.Close()
return nil, err
}
return f, nil
}

View File

@@ -1,41 +0,0 @@
package fuse
import (
"fmt"
"os"
"os/exec"
"strings"
)
func mount(dir string, conf *mountConfig, ready chan<- struct{}, errp *error) (*os.File, error) {
for k, v := range conf.options {
if strings.Contains(k, ",") || strings.Contains(v, ",") {
// Silly limitation but the mount helper does not
// understand any escaping. See TestMountOptionCommaError.
return nil, fmt.Errorf("mount options cannot contain commas on FreeBSD: %q=%q", k, v)
}
}
f, err := os.OpenFile("/dev/fuse", os.O_RDWR, 0000)
if err != nil {
*errp = err
return nil, err
}
cmd := exec.Command(
"/sbin/mount_fusefs",
"--safe",
"-o", conf.getOptions(),
"3",
dir,
)
cmd.ExtraFiles = []*os.File{f}
out, err := cmd.CombinedOutput()
if err != nil {
return nil, fmt.Errorf("mount_fusefs: %q, %v", out, err)
}
close(ready)
return f, nil
}

View File

@@ -1,112 +0,0 @@
package fuse
import (
"bufio"
"fmt"
"io"
"log"
"net"
"os"
"os/exec"
"sync"
"syscall"
)
func lineLogger(wg *sync.WaitGroup, prefix string, r io.ReadCloser) {
defer wg.Done()
scanner := bufio.NewScanner(r)
for scanner.Scan() {
switch line := scanner.Text(); line {
case `fusermount: failed to open /etc/fuse.conf: Permission denied`:
// Silence this particular message, it occurs way too
// commonly and isn't very relevant to whether the mount
// succeeds or not.
continue
default:
log.Printf("%s: %s", prefix, line)
}
}
if err := scanner.Err(); err != nil {
log.Printf("%s, error reading: %v", prefix, err)
}
}
func mount(dir string, conf *mountConfig, ready chan<- struct{}, errp *error) (fusefd *os.File, err error) {
// linux mount is never delayed
close(ready)
fds, err := syscall.Socketpair(syscall.AF_FILE, syscall.SOCK_STREAM, 0)
if err != nil {
return nil, fmt.Errorf("socketpair error: %v", err)
}
writeFile := os.NewFile(uintptr(fds[0]), "fusermount-child-writes")
defer writeFile.Close()
readFile := os.NewFile(uintptr(fds[1]), "fusermount-parent-reads")
defer readFile.Close()
cmd := exec.Command(
"fusermount",
"-o", conf.getOptions(),
"--",
dir,
)
cmd.Env = append(os.Environ(), "_FUSE_COMMFD=3")
cmd.ExtraFiles = []*os.File{writeFile}
var wg sync.WaitGroup
stdout, err := cmd.StdoutPipe()
if err != nil {
return nil, fmt.Errorf("setting up fusermount stderr: %v", err)
}
stderr, err := cmd.StderrPipe()
if err != nil {
return nil, fmt.Errorf("setting up fusermount stderr: %v", err)
}
if err := cmd.Start(); err != nil {
return nil, fmt.Errorf("fusermount: %v", err)
}
wg.Add(2)
go lineLogger(&wg, "mount helper output", stdout)
go lineLogger(&wg, "mount helper error", stderr)
wg.Wait()
if err := cmd.Wait(); err != nil {
return nil, fmt.Errorf("fusermount: %v", err)
}
c, err := net.FileConn(readFile)
if err != nil {
return nil, fmt.Errorf("FileConn from fusermount socket: %v", err)
}
defer c.Close()
uc, ok := c.(*net.UnixConn)
if !ok {
return nil, fmt.Errorf("unexpected FileConn type; expected UnixConn, got %T", c)
}
buf := make([]byte, 32) // expect 1 byte
oob := make([]byte, 32) // expect 24 bytes
_, oobn, _, _, err := uc.ReadMsgUnix(buf, oob)
scms, err := syscall.ParseSocketControlMessage(oob[:oobn])
if err != nil {
return nil, fmt.Errorf("ParseSocketControlMessage: %v", err)
}
if len(scms) != 1 {
return nil, fmt.Errorf("expected 1 SocketControlMessage; got scms = %#v", scms)
}
scm := scms[0]
gotFds, err := syscall.ParseUnixRights(&scm)
if err != nil {
return nil, fmt.Errorf("syscall.ParseUnixRights: %v", err)
}
if len(gotFds) != 1 {
return nil, fmt.Errorf("wanted 1 fd; got %#v", gotFds)
}
f := os.NewFile(uintptr(gotFds[0]), "/dev/fuse")
return f, nil
}

View File

@@ -1,170 +0,0 @@
package fuse
import (
"errors"
"strings"
)
func dummyOption(conf *mountConfig) error {
return nil
}
// mountConfig holds the configuration for a mount operation.
// Use it by passing MountOption values to Mount.
type mountConfig struct {
options map[string]string
maxReadahead uint32
initFlags InitFlags
}
func escapeComma(s string) string {
s = strings.Replace(s, `\`, `\\`, -1)
s = strings.Replace(s, `,`, `\,`, -1)
return s
}
// getOptions makes a string of options suitable for passing to FUSE
// mount flag `-o`. Returns an empty string if no options were set.
// Any platform specific adjustments should happen before the call.
func (m *mountConfig) getOptions() string {
var opts []string
for k, v := range m.options {
k = escapeComma(k)
if v != "" {
k += "=" + escapeComma(v)
}
opts = append(opts, k)
}
return strings.Join(opts, ",")
}
type mountOption func(*mountConfig) error
// MountOption is passed to Mount to change the behavior of the mount.
type MountOption mountOption
// FSName sets the file system name (also called source) that is
// visible in the list of mounted file systems.
//
// FreeBSD ignores this option.
func FSName(name string) MountOption {
return func(conf *mountConfig) error {
conf.options["fsname"] = name
return nil
}
}
// Subtype sets the subtype of the mount. The main type is always
// `fuse`. The type in a list of mounted file systems will look like
// `fuse.foo`.
//
// OS X ignores this option.
// FreeBSD ignores this option.
func Subtype(fstype string) MountOption {
return func(conf *mountConfig) error {
conf.options["subtype"] = fstype
return nil
}
}
// LocalVolume sets the volume to be local (instead of network),
// changing the behavior of Finder, Spotlight, and such.
//
// OS X only. Others ignore this option.
func LocalVolume() MountOption {
return localVolume
}
// VolumeName sets the volume name shown in Finder.
//
// OS X only. Others ignore this option.
func VolumeName(name string) MountOption {
return volumeName(name)
}
var ErrCannotCombineAllowOtherAndAllowRoot = errors.New("cannot combine AllowOther and AllowRoot")
// AllowOther allows other users to access the file system.
//
// Only one of AllowOther or AllowRoot can be used.
func AllowOther() MountOption {
return func(conf *mountConfig) error {
if _, ok := conf.options["allow_root"]; ok {
return ErrCannotCombineAllowOtherAndAllowRoot
}
conf.options["allow_other"] = ""
return nil
}
}
// AllowRoot allows other users to access the file system.
//
// Only one of AllowOther or AllowRoot can be used.
//
// FreeBSD ignores this option.
func AllowRoot() MountOption {
return func(conf *mountConfig) error {
if _, ok := conf.options["allow_other"]; ok {
return ErrCannotCombineAllowOtherAndAllowRoot
}
conf.options["allow_root"] = ""
return nil
}
}
// DefaultPermissions makes the kernel enforce access control based on
// the file mode (as in chmod).
//
// Without this option, the Node itself decides what is and is not
// allowed. This is normally ok because FUSE file systems cannot be
// accessed by other users without AllowOther/AllowRoot.
//
// FreeBSD ignores this option.
func DefaultPermissions() MountOption {
return func(conf *mountConfig) error {
conf.options["default_permissions"] = ""
return nil
}
}
// ReadOnly makes the mount read-only.
func ReadOnly() MountOption {
return func(conf *mountConfig) error {
conf.options["ro"] = ""
return nil
}
}
// MaxReadahead sets the number of bytes that can be prefetched for
// sequential reads. The kernel can enforce a maximum value lower than
// this.
//
// This setting makes the kernel perform speculative reads that do not
// originate from any client process. This usually tremendously
// improves read performance.
func MaxReadahead(n uint32) MountOption {
return func(conf *mountConfig) error {
conf.maxReadahead = n
return nil
}
}
// AsyncRead enables multiple outstanding read requests for the same
// handle. Without this, there is at most one request in flight at a
// time.
func AsyncRead() MountOption {
return func(conf *mountConfig) error {
conf.initFlags |= InitAsyncRead
return nil
}
}
// WritebackCache enables the kernel to buffer writes before sending
// them to the FUSE server. Without this, writethrough caching is
// used.
func WritebackCache() MountOption {
return func(conf *mountConfig) error {
conf.initFlags |= InitWritebackCache
return nil
}
}

View File

@@ -1,13 +0,0 @@
package fuse
func localVolume(conf *mountConfig) error {
conf.options["local"] = ""
return nil
}
func volumeName(name string) MountOption {
return func(conf *mountConfig) error {
conf.options["volname"] = name
return nil
}
}

View File

@@ -1,9 +0,0 @@
package fuse
func localVolume(conf *mountConfig) error {
return nil
}
func volumeName(name string) MountOption {
return dummyOption
}

View File

@@ -1,10 +0,0 @@
package fuse
// for TestMountOptionCommaError
func ForTestSetMountOption(k, v string) MountOption {
fn := func(conf *mountConfig) error {
conf.options[k] = v
return nil
}
return fn
}

View File

@@ -1,9 +0,0 @@
package fuse
func localVolume(conf *mountConfig) error {
return nil
}
func volumeName(name string) MountOption {
return dummyOption
}

View File

@@ -1,31 +0,0 @@
// This file contains tests for platforms that have no escape
// mechanism for including commas in mount options.
//
// +build darwin
package fuse_test
import (
"runtime"
"testing"
"bazil.org/fuse"
"bazil.org/fuse/fs/fstestutil"
)
func TestMountOptionCommaError(t *testing.T) {
t.Parallel()
// this test is not tied to any specific option, it just needs
// some string content
var evil = "FuseTest,Marker"
mnt, err := fstestutil.MountedT(t, fstestutil.SimpleFS{fstestutil.Dir{}}, nil,
fuse.ForTestSetMountOption("fusetest", evil),
)
if err == nil {
mnt.Close()
t.Fatal("expected an error about commas")
}
if g, e := err.Error(), `mount options cannot contain commas on `+runtime.GOOS+`: "fusetest"="FuseTest,Marker"`; g != e {
t.Fatalf("wrong error: %q != %q", g, e)
}
}

View File

@@ -1,231 +0,0 @@
package fuse_test
import (
"os"
"runtime"
"syscall"
"testing"
"bazil.org/fuse"
"bazil.org/fuse/fs"
"bazil.org/fuse/fs/fstestutil"
"golang.org/x/net/context"
)
func init() {
fstestutil.DebugByDefault()
}
func TestMountOptionFSName(t *testing.T) {
if runtime.GOOS == "freebsd" {
t.Skip("FreeBSD does not support FSName")
}
t.Parallel()
const name = "FuseTestMarker"
mnt, err := fstestutil.MountedT(t, fstestutil.SimpleFS{fstestutil.Dir{}}, nil,
fuse.FSName(name),
)
if err != nil {
t.Fatal(err)
}
defer mnt.Close()
info, err := fstestutil.GetMountInfo(mnt.Dir)
if err != nil {
t.Fatal(err)
}
if g, e := info.FSName, name; g != e {
t.Errorf("wrong FSName: %q != %q", g, e)
}
}
func testMountOptionFSNameEvil(t *testing.T, evil string) {
if runtime.GOOS == "freebsd" {
t.Skip("FreeBSD does not support FSName")
}
t.Parallel()
var name = "FuseTest" + evil + "Marker"
mnt, err := fstestutil.MountedT(t, fstestutil.SimpleFS{fstestutil.Dir{}}, nil,
fuse.FSName(name),
)
if err != nil {
t.Fatal(err)
}
defer mnt.Close()
info, err := fstestutil.GetMountInfo(mnt.Dir)
if err != nil {
t.Fatal(err)
}
if g, e := info.FSName, name; g != e {
t.Errorf("wrong FSName: %q != %q", g, e)
}
}
func TestMountOptionFSNameEvilComma(t *testing.T) {
if runtime.GOOS == "darwin" {
// see TestMountOptionCommaError for a test that enforces we
// at least give a nice error, instead of corrupting the mount
// options
t.Skip("TODO: OS X gets this wrong, commas in mount options cannot be escaped at all")
}
testMountOptionFSNameEvil(t, ",")
}
func TestMountOptionFSNameEvilSpace(t *testing.T) {
testMountOptionFSNameEvil(t, " ")
}
func TestMountOptionFSNameEvilTab(t *testing.T) {
testMountOptionFSNameEvil(t, "\t")
}
func TestMountOptionFSNameEvilNewline(t *testing.T) {
testMountOptionFSNameEvil(t, "\n")
}
func TestMountOptionFSNameEvilBackslash(t *testing.T) {
testMountOptionFSNameEvil(t, `\`)
}
func TestMountOptionFSNameEvilBackslashDouble(t *testing.T) {
// catch double-unescaping, if it were to happen
testMountOptionFSNameEvil(t, `\\`)
}
func TestMountOptionSubtype(t *testing.T) {
if runtime.GOOS == "darwin" {
t.Skip("OS X does not support Subtype")
}
if runtime.GOOS == "freebsd" {
t.Skip("FreeBSD does not support Subtype")
}
t.Parallel()
const name = "FuseTestMarker"
mnt, err := fstestutil.MountedT(t, fstestutil.SimpleFS{fstestutil.Dir{}}, nil,
fuse.Subtype(name),
)
if err != nil {
t.Fatal(err)
}
defer mnt.Close()
info, err := fstestutil.GetMountInfo(mnt.Dir)
if err != nil {
t.Fatal(err)
}
if g, e := info.Type, "fuse."+name; g != e {
t.Errorf("wrong Subtype: %q != %q", g, e)
}
}
// TODO test LocalVolume
// TODO test AllowOther; hard because needs system-level authorization
func TestMountOptionAllowOtherThenAllowRoot(t *testing.T) {
t.Parallel()
mnt, err := fstestutil.MountedT(t, fstestutil.SimpleFS{fstestutil.Dir{}}, nil,
fuse.AllowOther(),
fuse.AllowRoot(),
)
if err == nil {
mnt.Close()
}
if g, e := err, fuse.ErrCannotCombineAllowOtherAndAllowRoot; g != e {
t.Fatalf("wrong error: %v != %v", g, e)
}
}
// TODO test AllowRoot; hard because needs system-level authorization
func TestMountOptionAllowRootThenAllowOther(t *testing.T) {
t.Parallel()
mnt, err := fstestutil.MountedT(t, fstestutil.SimpleFS{fstestutil.Dir{}}, nil,
fuse.AllowRoot(),
fuse.AllowOther(),
)
if err == nil {
mnt.Close()
}
if g, e := err, fuse.ErrCannotCombineAllowOtherAndAllowRoot; g != e {
t.Fatalf("wrong error: %v != %v", g, e)
}
}
type unwritableFile struct{}
func (f unwritableFile) Attr(ctx context.Context, a *fuse.Attr) error {
a.Mode = 0000
return nil
}
func TestMountOptionDefaultPermissions(t *testing.T) {
if runtime.GOOS == "freebsd" {
t.Skip("FreeBSD does not support DefaultPermissions")
}
t.Parallel()
mnt, err := fstestutil.MountedT(t,
fstestutil.SimpleFS{
&fstestutil.ChildMap{"child": unwritableFile{}},
},
nil,
fuse.DefaultPermissions(),
)
if err != nil {
t.Fatal(err)
}
defer mnt.Close()
// This will be prevented by kernel-level access checking when
// DefaultPermissions is used.
f, err := os.OpenFile(mnt.Dir+"/child", os.O_WRONLY, 0000)
if err == nil {
f.Close()
t.Fatal("expected an error")
}
if !os.IsPermission(err) {
t.Fatalf("expected a permission error, got %T: %v", err, err)
}
}
type createrDir struct {
fstestutil.Dir
}
var _ fs.NodeCreater = createrDir{}
func (createrDir) Create(ctx context.Context, req *fuse.CreateRequest, resp *fuse.CreateResponse) (fs.Node, fs.Handle, error) {
// pick a really distinct error, to identify it later
return nil, nil, fuse.Errno(syscall.ENAMETOOLONG)
}
func TestMountOptionReadOnly(t *testing.T) {
t.Parallel()
mnt, err := fstestutil.MountedT(t,
fstestutil.SimpleFS{createrDir{}},
nil,
fuse.ReadOnly(),
)
if err != nil {
t.Fatal(err)
}
defer mnt.Close()
// This will be prevented by kernel-level access checking when
// ReadOnly is used.
f, err := os.Create(mnt.Dir + "/child")
if err == nil {
f.Close()
t.Fatal("expected an error")
}
perr, ok := err.(*os.PathError)
if !ok {
t.Fatalf("expected PathError, got %T: %v", err, err)
}
if perr.Err != syscall.EROFS {
t.Fatalf("expected EROFS, got %T: %v", err, err)
}
}

View File

@@ -1,75 +0,0 @@
package fuse
import (
"fmt"
)
// Protocol is a FUSE protocol version number.
type Protocol struct {
Major uint32
Minor uint32
}
func (p Protocol) String() string {
return fmt.Sprintf("%d.%d", p.Major, p.Minor)
}
// LT returns whether a is less than b.
func (a Protocol) LT(b Protocol) bool {
return a.Major < b.Major ||
(a.Major == b.Major && a.Minor < b.Minor)
}
// GE returns whether a is greater than or equal to b.
func (a Protocol) GE(b Protocol) bool {
return a.Major > b.Major ||
(a.Major == b.Major && a.Minor >= b.Minor)
}
func (a Protocol) is79() bool {
return a.GE(Protocol{7, 9})
}
// HasAttrBlockSize returns whether Attr.BlockSize is respected by the
// kernel.
func (a Protocol) HasAttrBlockSize() bool {
return a.is79()
}
// HasReadWriteFlags returns whether ReadRequest/WriteRequest
// fields Flags and FileFlags are valid.
func (a Protocol) HasReadWriteFlags() bool {
return a.is79()
}
// HasGetattrFlags returns whether GetattrRequest field Flags is
// valid.
func (a Protocol) HasGetattrFlags() bool {
return a.is79()
}
func (a Protocol) is710() bool {
return a.GE(Protocol{7, 10})
}
// HasOpenNonSeekable returns whether OpenResponse field Flags flag
// OpenNonSeekable is supported.
func (a Protocol) HasOpenNonSeekable() bool {
return a.is710()
}
func (a Protocol) is712() bool {
return a.GE(Protocol{7, 12})
}
// HasUmask returns whether CreateRequest/MkdirRequest/MknodRequest
// field Umask is valid.
func (a Protocol) HasUmask() bool {
return a.is712()
}
// HasInvalidate returns whether InvalidateNode/InvalidateEntry are
// supported.
func (a Protocol) HasInvalidate() bool {
return a.is712()
}

View File

@@ -1,13 +0,0 @@
// Package syscallx provides wrappers that make syscalls on various
// platforms more interoperable.
//
// The API intentionally omits the OS X-specific position and option
// arguments for extended attribute calls.
//
// Not having position means it might not be useful for accessing the
// resource fork. If that's needed by code inside fuse, a function
// with a different name may be added on the side.
//
// Options can be implemented with separate wrappers, in the style of
// Linux getxattr/lgetxattr/fgetxattr.
package syscallx

View File

@@ -1,34 +0,0 @@
#!/bin/sh
set -e
mksys="$(go env GOROOT)/src/pkg/syscall/mksyscall.pl"
fix() {
sed 's,^package syscall$,&x\nimport "syscall",' \
| gofmt -r='BytePtrFromString -> syscall.BytePtrFromString' \
| gofmt -r='Syscall6 -> syscall.Syscall6' \
| gofmt -r='Syscall -> syscall.Syscall' \
| gofmt -r='SYS_GETXATTR -> syscall.SYS_GETXATTR' \
| gofmt -r='SYS_LISTXATTR -> syscall.SYS_LISTXATTR' \
| gofmt -r='SYS_SETXATTR -> syscall.SYS_SETXATTR' \
| gofmt -r='SYS_REMOVEXATTR -> syscall.SYS_REMOVEXATTR' \
| gofmt -r='SYS_MSYNC -> syscall.SYS_MSYNC'
}
cd "$(dirname "$0")"
$mksys xattr_darwin.go \
| fix \
>xattr_darwin_amd64.go
$mksys -l32 xattr_darwin.go \
| fix \
>xattr_darwin_386.go
$mksys msync.go \
| fix \
>msync_amd64.go
$mksys -l32 msync.go \
| fix \
>msync_386.go

View File

@@ -1,9 +0,0 @@
package syscallx
/* This is the source file for msync_*.go, to regenerate run
./generate
*/
//sys Msync(b []byte, flags int) (err error)

View File

@@ -1,24 +0,0 @@
// mksyscall.pl -l32 msync.go
// MACHINE GENERATED BY THE COMMAND ABOVE; DO NOT EDIT
package syscallx
import "syscall"
import "unsafe"
// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT
func Msync(b []byte, flags int) (err error) {
var _p0 unsafe.Pointer
if len(b) > 0 {
_p0 = unsafe.Pointer(&b[0])
} else {
_p0 = unsafe.Pointer(&_zero)
}
_, _, e1 := syscall.Syscall(syscall.SYS_MSYNC, uintptr(_p0), uintptr(len(b)), uintptr(flags))
if e1 != 0 {
err = e1
}
return
}

View File

@@ -1,24 +0,0 @@
// mksyscall.pl msync.go
// MACHINE GENERATED BY THE COMMAND ABOVE; DO NOT EDIT
package syscallx
import "syscall"
import "unsafe"
// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT
func Msync(b []byte, flags int) (err error) {
var _p0 unsafe.Pointer
if len(b) > 0 {
_p0 = unsafe.Pointer(&b[0])
} else {
_p0 = unsafe.Pointer(&_zero)
}
_, _, e1 := syscall.Syscall(syscall.SYS_MSYNC, uintptr(_p0), uintptr(len(b)), uintptr(flags))
if e1 != 0 {
err = e1
}
return
}

View File

@@ -1,4 +0,0 @@
package syscallx
// make us look more like package syscall, so mksyscall.pl output works
var _zero uintptr

View File

@@ -1,26 +0,0 @@
// +build !darwin
package syscallx
// This file just contains wrappers for platforms that already have
// the right stuff in golang.org/x/sys/unix.
import (
"golang.org/x/sys/unix"
)
func Getxattr(path string, attr string, dest []byte) (sz int, err error) {
return unix.Getxattr(path, attr, dest)
}
func Listxattr(path string, dest []byte) (sz int, err error) {
return unix.Listxattr(path, dest)
}
func Setxattr(path string, attr string, data []byte, flags int) (err error) {
return unix.Setxattr(path, attr, data, flags)
}
func Removexattr(path string, attr string) (err error) {
return unix.Removexattr(path, attr)
}

View File

@@ -1,38 +0,0 @@
package syscallx
/* This is the source file for syscallx_darwin_*.go, to regenerate run
./generate
*/
// cannot use dest []byte here because OS X getxattr really wants a
// NULL to trigger size probing, size==0 is not enough
//
//sys getxattr(path string, attr string, dest *byte, size int, position uint32, options int) (sz int, err error)
func Getxattr(path string, attr string, dest []byte) (sz int, err error) {
var destp *byte
if len(dest) > 0 {
destp = &dest[0]
}
return getxattr(path, attr, destp, len(dest), 0, 0)
}
//sys listxattr(path string, dest []byte, options int) (sz int, err error)
func Listxattr(path string, dest []byte) (sz int, err error) {
return listxattr(path, dest, 0)
}
//sys setxattr(path string, attr string, data []byte, position uint32, flags int) (err error)
func Setxattr(path string, attr string, data []byte, flags int) (err error) {
return setxattr(path, attr, data, 0, flags)
}
//sys removexattr(path string, attr string, options int) (err error)
func Removexattr(path string, attr string) (err error) {
return removexattr(path, attr, 0)
}

View File

@@ -1,97 +0,0 @@
// mksyscall.pl -l32 xattr_darwin.go
// MACHINE GENERATED BY THE COMMAND ABOVE; DO NOT EDIT
package syscallx
import "syscall"
import "unsafe"
// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT
func getxattr(path string, attr string, dest *byte, size int, position uint32, options int) (sz int, err error) {
var _p0 *byte
_p0, err = syscall.BytePtrFromString(path)
if err != nil {
return
}
var _p1 *byte
_p1, err = syscall.BytePtrFromString(attr)
if err != nil {
return
}
r0, _, e1 := syscall.Syscall6(syscall.SYS_GETXATTR, uintptr(unsafe.Pointer(_p0)), uintptr(unsafe.Pointer(_p1)), uintptr(unsafe.Pointer(dest)), uintptr(size), uintptr(position), uintptr(options))
sz = int(r0)
if e1 != 0 {
err = e1
}
return
}
// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT
func listxattr(path string, dest []byte, options int) (sz int, err error) {
var _p0 *byte
_p0, err = syscall.BytePtrFromString(path)
if err != nil {
return
}
var _p1 unsafe.Pointer
if len(dest) > 0 {
_p1 = unsafe.Pointer(&dest[0])
} else {
_p1 = unsafe.Pointer(&_zero)
}
r0, _, e1 := syscall.Syscall6(syscall.SYS_LISTXATTR, uintptr(unsafe.Pointer(_p0)), uintptr(_p1), uintptr(len(dest)), uintptr(options), 0, 0)
sz = int(r0)
if e1 != 0 {
err = e1
}
return
}
// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT
func setxattr(path string, attr string, data []byte, position uint32, flags int) (err error) {
var _p0 *byte
_p0, err = syscall.BytePtrFromString(path)
if err != nil {
return
}
var _p1 *byte
_p1, err = syscall.BytePtrFromString(attr)
if err != nil {
return
}
var _p2 unsafe.Pointer
if len(data) > 0 {
_p2 = unsafe.Pointer(&data[0])
} else {
_p2 = unsafe.Pointer(&_zero)
}
_, _, e1 := syscall.Syscall6(syscall.SYS_SETXATTR, uintptr(unsafe.Pointer(_p0)), uintptr(unsafe.Pointer(_p1)), uintptr(_p2), uintptr(len(data)), uintptr(position), uintptr(flags))
if e1 != 0 {
err = e1
}
return
}
// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT
func removexattr(path string, attr string, options int) (err error) {
var _p0 *byte
_p0, err = syscall.BytePtrFromString(path)
if err != nil {
return
}
var _p1 *byte
_p1, err = syscall.BytePtrFromString(attr)
if err != nil {
return
}
_, _, e1 := syscall.Syscall(syscall.SYS_REMOVEXATTR, uintptr(unsafe.Pointer(_p0)), uintptr(unsafe.Pointer(_p1)), uintptr(options))
if e1 != 0 {
err = e1
}
return
}

View File

@@ -1,97 +0,0 @@
// mksyscall.pl xattr_darwin.go
// MACHINE GENERATED BY THE COMMAND ABOVE; DO NOT EDIT
package syscallx
import "syscall"
import "unsafe"
// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT
func getxattr(path string, attr string, dest *byte, size int, position uint32, options int) (sz int, err error) {
var _p0 *byte
_p0, err = syscall.BytePtrFromString(path)
if err != nil {
return
}
var _p1 *byte
_p1, err = syscall.BytePtrFromString(attr)
if err != nil {
return
}
r0, _, e1 := syscall.Syscall6(syscall.SYS_GETXATTR, uintptr(unsafe.Pointer(_p0)), uintptr(unsafe.Pointer(_p1)), uintptr(unsafe.Pointer(dest)), uintptr(size), uintptr(position), uintptr(options))
sz = int(r0)
if e1 != 0 {
err = e1
}
return
}
// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT
func listxattr(path string, dest []byte, options int) (sz int, err error) {
var _p0 *byte
_p0, err = syscall.BytePtrFromString(path)
if err != nil {
return
}
var _p1 unsafe.Pointer
if len(dest) > 0 {
_p1 = unsafe.Pointer(&dest[0])
} else {
_p1 = unsafe.Pointer(&_zero)
}
r0, _, e1 := syscall.Syscall6(syscall.SYS_LISTXATTR, uintptr(unsafe.Pointer(_p0)), uintptr(_p1), uintptr(len(dest)), uintptr(options), 0, 0)
sz = int(r0)
if e1 != 0 {
err = e1
}
return
}
// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT
func setxattr(path string, attr string, data []byte, position uint32, flags int) (err error) {
var _p0 *byte
_p0, err = syscall.BytePtrFromString(path)
if err != nil {
return
}
var _p1 *byte
_p1, err = syscall.BytePtrFromString(attr)
if err != nil {
return
}
var _p2 unsafe.Pointer
if len(data) > 0 {
_p2 = unsafe.Pointer(&data[0])
} else {
_p2 = unsafe.Pointer(&_zero)
}
_, _, e1 := syscall.Syscall6(syscall.SYS_SETXATTR, uintptr(unsafe.Pointer(_p0)), uintptr(unsafe.Pointer(_p1)), uintptr(_p2), uintptr(len(data)), uintptr(position), uintptr(flags))
if e1 != 0 {
err = e1
}
return
}
// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT
func removexattr(path string, attr string, options int) (err error) {
var _p0 *byte
_p0, err = syscall.BytePtrFromString(path)
if err != nil {
return
}
var _p1 *byte
_p1, err = syscall.BytePtrFromString(attr)
if err != nil {
return
}
_, _, e1 := syscall.Syscall(syscall.SYS_REMOVEXATTR, uintptr(unsafe.Pointer(_p0)), uintptr(unsafe.Pointer(_p1)), uintptr(options))
if e1 != 0 {
err = e1
}
return
}

View File

@@ -1,6 +0,0 @@
package fuse
// Unmount tries to unmount the filesystem mounted at dir.
func Unmount(dir string) error {
return unmount(dir)
}

View File

@@ -1,21 +0,0 @@
package fuse
import (
"bytes"
"errors"
"os/exec"
)
func unmount(dir string) error {
cmd := exec.Command("fusermount", "-u", dir)
output, err := cmd.CombinedOutput()
if err != nil {
if len(output) > 0 {
output = bytes.TrimRight(output, "\n")
msg := err.Error() + ": " + string(output)
err = errors.New(msg)
}
return err
}
return nil
}

View File

@@ -1,17 +0,0 @@
// +build !linux
package fuse
import (
"os"
"syscall"
)
func unmount(dir string) error {
err := syscall.Unmount(dir, 0)
if err != nil {
err = &os.PathError{Op: "unmount", Path: dir, Err: err}
return err
}
return nil
}

View File

@@ -1,35 +0,0 @@
language: go
install:
# go-flags
- go get -d -v ./...
- go build -v ./...
# linting
- go get golang.org/x/tools/cmd/vet
- go get github.com/golang/lint
- go install github.com/golang/lint/golint
# code coverage
- go get golang.org/x/tools/cmd/cover
- go get github.com/onsi/ginkgo/ginkgo
- go get github.com/modocache/gover
- if [ "$TRAVIS_SECURE_ENV_VARS" = "true" ]; then go get github.com/mattn/goveralls; fi
script:
# go-flags
- $(exit $(gofmt -l . | wc -l))
- go test -v ./...
# linting
- go tool vet -all=true -v=true . || true
- $(go env GOPATH | awk 'BEGIN{FS=":"} {print $1}')/bin/golint ./...
# code coverage
- $(go env GOPATH | awk 'BEGIN{FS=":"} {print $1}')/bin/ginkgo -r -cover
- $(go env GOPATH | awk 'BEGIN{FS=":"} {print $1}')/bin/gover
- if [ "$TRAVIS_SECURE_ENV_VARS" = "true" ]; then $(go env GOPATH | awk 'BEGIN{FS=":"} {print $1}')/bin/goveralls -coverprofile=gover.coverprofile -service=travis-ci -repotoken $COVERALLS_TOKEN; fi
env:
# coveralls.io
secure: "RCYbiB4P0RjQRIoUx/vG/AjP3mmYCbzOmr86DCww1Z88yNcy3hYr3Cq8rpPtYU5v0g7wTpu4adaKIcqRE9xknYGbqj3YWZiCoBP1/n4Z+9sHW3Dsd9D/GRGeHUus0laJUGARjWoCTvoEtOgTdGQDoX7mH+pUUY0FBltNYUdOiiU="

View File

@@ -1,26 +0,0 @@
Copyright (c) 2012 Jesse van den Kieboom. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

View File

@@ -1,131 +0,0 @@
go-flags: a go library for parsing command line arguments
=========================================================
[![GoDoc](https://godoc.org/github.com/jessevdk/go-flags?status.png)](https://godoc.org/github.com/jessevdk/go-flags) [![Build Status](https://travis-ci.org/jessevdk/go-flags.svg?branch=master)](https://travis-ci.org/jessevdk/go-flags) [![Coverage Status](https://img.shields.io/coveralls/jessevdk/go-flags.svg)](https://coveralls.io/r/jessevdk/go-flags?branch=master)
This library provides similar functionality to the builtin flag library of
go, but provides much more functionality and nicer formatting. From the
documentation:
Package flags provides an extensive command line option parser.
The flags package is similar in functionality to the go builtin flag package
but provides more options and uses reflection to provide a convenient and
succinct way of specifying command line options.
Supported features:
* Options with short names (-v)
* Options with long names (--verbose)
* Options with and without arguments (bool v.s. other type)
* Options with optional arguments and default values
* Multiple option groups each containing a set of options
* Generate and print well-formatted help message
* Passing remaining command line arguments after -- (optional)
* Ignoring unknown command line options (optional)
* Supports -I/usr/include -I=/usr/include -I /usr/include option argument specification
* Supports multiple short options -aux
* Supports all primitive go types (string, int{8..64}, uint{8..64}, float)
* Supports same option multiple times (can store in slice or last option counts)
* Supports maps
* Supports function callbacks
* Supports namespaces for (nested) option groups
The flags package uses structs, reflection and struct field tags
to allow users to specify command line options. This results in very simple
and concise specification of your application options. For example:
type Options struct {
Verbose []bool `short:"v" long:"verbose" description:"Show verbose debug information"`
}
This specifies one option with a short name -v and a long name --verbose.
When either -v or --verbose is found on the command line, a 'true' value
will be appended to the Verbose field. e.g. when specifying -vvv, the
resulting value of Verbose will be {[true, true, true]}.
Example:
--------
var opts struct {
// Slice of bool will append 'true' each time the option
// is encountered (can be set multiple times, like -vvv)
Verbose []bool `short:"v" long:"verbose" description:"Show verbose debug information"`
// Example of automatic marshalling to desired type (uint)
Offset uint `long:"offset" description:"Offset"`
// Example of a callback, called each time the option is found.
Call func(string) `short:"c" description:"Call phone number"`
// Example of a required flag
Name string `short:"n" long:"name" description:"A name" required:"true"`
// Example of a value name
File string `short:"f" long:"file" description:"A file" value-name:"FILE"`
// Example of a pointer
Ptr *int `short:"p" description:"A pointer to an integer"`
// Example of a slice of strings
StringSlice []string `short:"s" description:"A slice of strings"`
// Example of a slice of pointers
PtrSlice []*string `long:"ptrslice" description:"A slice of pointers to string"`
// Example of a map
IntMap map[string]int `long:"intmap" description:"A map from string to int"`
}
// Callback which will invoke callto:<argument> to call a number.
// Note that this works just on OS X (and probably only with
// Skype) but it shows the idea.
opts.Call = func(num string) {
cmd := exec.Command("open", "callto:"+num)
cmd.Start()
cmd.Process.Release()
}
// Make some fake arguments to parse.
args := []string{
"-vv",
"--offset=5",
"-n", "Me",
"-p", "3",
"-s", "hello",
"-s", "world",
"--ptrslice", "hello",
"--ptrslice", "world",
"--intmap", "a:1",
"--intmap", "b:5",
"arg1",
"arg2",
"arg3",
}
// Parse flags from `args'. Note that here we use flags.ParseArgs for
// the sake of making a working example. Normally, you would simply use
// flags.Parse(&opts) which uses os.Args
args, err := flags.ParseArgs(&opts, args)
if err != nil {
panic(err)
os.Exit(1)
}
fmt.Printf("Verbosity: %v\n", opts.Verbose)
fmt.Printf("Offset: %d\n", opts.Offset)
fmt.Printf("Name: %s\n", opts.Name)
fmt.Printf("Ptr: %d\n", *opts.Ptr)
fmt.Printf("StringSlice: %v\n", opts.StringSlice)
fmt.Printf("PtrSlice: [%v %v]\n", *opts.PtrSlice[0], *opts.PtrSlice[1])
fmt.Printf("IntMap: [a:%v b:%v]\n", opts.IntMap["a"], opts.IntMap["b"])
fmt.Printf("Remaining args: %s\n", strings.Join(args, " "))
// Output: Verbosity: [true true]
// Offset: 5
// Name: Me
// Ptr: 3
// StringSlice: [hello world]
// PtrSlice: [hello world]
// IntMap: [a:1 b:5]
// Remaining args: arg1 arg2 arg3
More information can be found in the godocs: <http://godoc.org/github.com/jessevdk/go-flags>

View File

@@ -1,21 +0,0 @@
package flags
import (
"reflect"
)
// Arg represents a positional argument on the command line.
type Arg struct {
// The name of the positional argument (used in the help)
Name string
// A description of the positional argument (used in the help)
Description string
value reflect.Value
tag multiTag
}
func (a *Arg) isRemaining() bool {
return a.value.Type().Kind() == reflect.Slice
}

View File

@@ -1,53 +0,0 @@
package flags
import (
"testing"
)
func TestPositional(t *testing.T) {
var opts = struct {
Value bool `short:"v"`
Positional struct {
Command int
Filename string
Rest []string
} `positional-args:"yes" required:"yes"`
}{}
p := NewParser(&opts, Default)
ret, err := p.ParseArgs([]string{"10", "arg_test.go", "a", "b"})
if err != nil {
t.Fatalf("Unexpected error: %v", err)
return
}
if opts.Positional.Command != 10 {
t.Fatalf("Expected opts.Positional.Command to be 10, but got %v", opts.Positional.Command)
}
if opts.Positional.Filename != "arg_test.go" {
t.Fatalf("Expected opts.Positional.Filename to be \"arg_test.go\", but got %v", opts.Positional.Filename)
}
assertStringArray(t, opts.Positional.Rest, []string{"a", "b"})
assertStringArray(t, ret, []string{})
}
func TestPositionalRequired(t *testing.T) {
var opts = struct {
Value bool `short:"v"`
Positional struct {
Command int
Filename string
Rest []string
} `positional-args:"yes" required:"yes"`
}{}
p := NewParser(&opts, None)
_, err := p.ParseArgs([]string{"10"})
assertError(t, err, ErrRequired, "the required argument `Filename` was not provided")
}

View File

@@ -1,177 +0,0 @@
package flags
import (
"fmt"
"io"
"io/ioutil"
"os"
"os/exec"
"path"
"runtime"
"testing"
)
func assertCallerInfo() (string, int) {
ptr := make([]uintptr, 15)
n := runtime.Callers(1, ptr)
if n == 0 {
return "", 0
}
mef := runtime.FuncForPC(ptr[0])
mefile, meline := mef.FileLine(ptr[0])
for i := 2; i < n; i++ {
f := runtime.FuncForPC(ptr[i])
file, line := f.FileLine(ptr[i])
if file != mefile {
return file, line
}
}
return mefile, meline
}
func assertErrorf(t *testing.T, format string, args ...interface{}) {
msg := fmt.Sprintf(format, args...)
file, line := assertCallerInfo()
t.Errorf("%s:%d: %s", path.Base(file), line, msg)
}
func assertFatalf(t *testing.T, format string, args ...interface{}) {
msg := fmt.Sprintf(format, args...)
file, line := assertCallerInfo()
t.Fatalf("%s:%d: %s", path.Base(file), line, msg)
}
func assertString(t *testing.T, a string, b string) {
if a != b {
assertErrorf(t, "Expected %#v, but got %#v", b, a)
}
}
func assertStringArray(t *testing.T, a []string, b []string) {
if len(a) != len(b) {
assertErrorf(t, "Expected %#v, but got %#v", b, a)
return
}
for i, v := range a {
if b[i] != v {
assertErrorf(t, "Expected %#v, but got %#v", b, a)
return
}
}
}
func assertBoolArray(t *testing.T, a []bool, b []bool) {
if len(a) != len(b) {
assertErrorf(t, "Expected %#v, but got %#v", b, a)
return
}
for i, v := range a {
if b[i] != v {
assertErrorf(t, "Expected %#v, but got %#v", b, a)
return
}
}
}
func assertParserSuccess(t *testing.T, data interface{}, args ...string) (*Parser, []string) {
parser := NewParser(data, Default&^PrintErrors)
ret, err := parser.ParseArgs(args)
if err != nil {
t.Fatalf("Unexpected parse error: %s", err)
return nil, nil
}
return parser, ret
}
func assertParseSuccess(t *testing.T, data interface{}, args ...string) []string {
_, ret := assertParserSuccess(t, data, args...)
return ret
}
func assertError(t *testing.T, err error, typ ErrorType, msg string) {
if err == nil {
assertFatalf(t, "Expected error: %s", msg)
return
}
if e, ok := err.(*Error); !ok {
assertFatalf(t, "Expected Error type, but got %#v", err)
} else {
if e.Type != typ {
assertErrorf(t, "Expected error type {%s}, but got {%s}", typ, e.Type)
}
if e.Message != msg {
assertErrorf(t, "Expected error message %#v, but got %#v", msg, e.Message)
}
}
}
func assertParseFail(t *testing.T, typ ErrorType, msg string, data interface{}, args ...string) []string {
parser := NewParser(data, Default&^PrintErrors)
ret, err := parser.ParseArgs(args)
assertError(t, err, typ, msg)
return ret
}
func diff(a, b string) (string, error) {
atmp, err := ioutil.TempFile("", "help-diff")
if err != nil {
return "", err
}
btmp, err := ioutil.TempFile("", "help-diff")
if err != nil {
return "", err
}
if _, err := io.WriteString(atmp, a); err != nil {
return "", err
}
if _, err := io.WriteString(btmp, b); err != nil {
return "", err
}
ret, err := exec.Command("diff", "-u", "-d", "--label", "got", atmp.Name(), "--label", "expected", btmp.Name()).Output()
os.Remove(atmp.Name())
os.Remove(btmp.Name())
if err.Error() == "exit status 1" {
return string(ret), nil
}
return string(ret), err
}
func assertDiff(t *testing.T, actual, expected, msg string) {
if actual == expected {
return
}
ret, err := diff(actual, expected)
if err != nil {
assertErrorf(t, "Unexpected diff error: %s", err)
assertErrorf(t, "Unexpected %s, expected:\n\n%s\n\nbut got\n\n%s", msg, expected, actual)
} else {
assertErrorf(t, "Unexpected %s:\n\n%s", msg, ret)
}
}

View File

@@ -1,16 +0,0 @@
#!/bin/bash
set -e
echo '# linux arm7'
GOARM=7 GOARCH=arm GOOS=linux go build
echo '# linux arm5'
GOARM=5 GOARCH=arm GOOS=linux go build
echo '# windows 386'
GOARCH=386 GOOS=windows go build
echo '# windows amd64'
GOARCH=amd64 GOOS=windows go build
echo '# darwin'
GOARCH=amd64 GOOS=darwin go build
echo '# freebsd'
GOARCH=amd64 GOOS=freebsd go build

View File

@@ -1,59 +0,0 @@
package flags
func levenshtein(s string, t string) int {
if len(s) == 0 {
return len(t)
}
if len(t) == 0 {
return len(s)
}
dists := make([][]int, len(s)+1)
for i := range dists {
dists[i] = make([]int, len(t)+1)
dists[i][0] = i
}
for j := range t {
dists[0][j] = j
}
for i, sc := range s {
for j, tc := range t {
if sc == tc {
dists[i+1][j+1] = dists[i][j]
} else {
dists[i+1][j+1] = dists[i][j] + 1
if dists[i+1][j] < dists[i+1][j+1] {
dists[i+1][j+1] = dists[i+1][j] + 1
}
if dists[i][j+1] < dists[i+1][j+1] {
dists[i+1][j+1] = dists[i][j+1] + 1
}
}
}
}
return dists[len(s)][len(t)]
}
func closestChoice(cmd string, choices []string) (string, int) {
if len(choices) == 0 {
return "", 0
}
mincmd := -1
mindist := -1
for i, c := range choices {
l := levenshtein(cmd, c)
if mincmd < 0 || l < mindist {
mindist = l
mincmd = i
}
}
return choices[mincmd], mindist
}

View File

@@ -1,106 +0,0 @@
package flags
// Command represents an application command. Commands can be added to the
// parser (which itself is a command) and are selected/executed when its name
// is specified on the command line. The Command type embeds a Group and
// therefore also carries a set of command specific options.
type Command struct {
// Embedded, see Group for more information
*Group
// The name by which the command can be invoked
Name string
// The active sub command (set by parsing) or nil
Active *Command
// Whether subcommands are optional
SubcommandsOptional bool
// Aliases for the command
Aliases []string
// Whether positional arguments are required
ArgsRequired bool
commands []*Command
hasBuiltinHelpGroup bool
args []*Arg
}
// Commander is an interface which can be implemented by any command added in
// the options. When implemented, the Execute method will be called for the last
// specified (sub)command providing the remaining command line arguments.
type Commander interface {
// Execute will be called for the last active (sub)command. The
// args argument contains the remaining command line arguments. The
// error that Execute returns will be eventually passed out of the
// Parse method of the Parser.
Execute(args []string) error
}
// Usage is an interface which can be implemented to show a custom usage string
// in the help message shown for a command.
type Usage interface {
// Usage is called for commands to allow customized printing of command
// usage in the generated help message.
Usage() string
}
// AddCommand adds a new command to the parser with the given name and data. The
// data needs to be a pointer to a struct from which the fields indicate which
// options are in the command. The provided data can implement the Command and
// Usage interfaces.
func (c *Command) AddCommand(command string, shortDescription string, longDescription string, data interface{}) (*Command, error) {
cmd := newCommand(command, shortDescription, longDescription, data)
cmd.parent = c
if err := cmd.scan(); err != nil {
return nil, err
}
c.commands = append(c.commands, cmd)
return cmd, nil
}
// AddGroup adds a new group to the command with the given name and data. The
// data needs to be a pointer to a struct from which the fields indicate which
// options are in the group.
func (c *Command) AddGroup(shortDescription string, longDescription string, data interface{}) (*Group, error) {
group := newGroup(shortDescription, longDescription, data)
group.parent = c
if err := group.scanType(c.scanSubcommandHandler(group)); err != nil {
return nil, err
}
c.groups = append(c.groups, group)
return group, nil
}
// Commands returns a list of subcommands of this command.
func (c *Command) Commands() []*Command {
return c.commands
}
// Find locates the subcommand with the given name and returns it. If no such
// command can be found Find will return nil.
func (c *Command) Find(name string) *Command {
for _, cc := range c.commands {
if cc.match(name) {
return cc
}
}
return nil
}
// Args returns a list of positional arguments associated with this command.
func (c *Command) Args() []*Arg {
ret := make([]*Arg, len(c.args))
copy(ret, c.args)
return ret
}

View File

@@ -1,271 +0,0 @@
package flags
import (
"reflect"
"sort"
"strings"
"unsafe"
)
type lookup struct {
shortNames map[string]*Option
longNames map[string]*Option
commands map[string]*Command
}
func newCommand(name string, shortDescription string, longDescription string, data interface{}) *Command {
return &Command{
Group: newGroup(shortDescription, longDescription, data),
Name: name,
}
}
func (c *Command) scanSubcommandHandler(parentg *Group) scanHandler {
f := func(realval reflect.Value, sfield *reflect.StructField) (bool, error) {
mtag := newMultiTag(string(sfield.Tag))
if err := mtag.Parse(); err != nil {
return true, err
}
positional := mtag.Get("positional-args")
if len(positional) != 0 {
stype := realval.Type()
for i := 0; i < stype.NumField(); i++ {
field := stype.Field(i)
m := newMultiTag((string(field.Tag)))
if err := m.Parse(); err != nil {
return true, err
}
name := m.Get("positional-arg-name")
if len(name) == 0 {
name = field.Name
}
arg := &Arg{
Name: name,
Description: m.Get("description"),
value: realval.Field(i),
tag: m,
}
c.args = append(c.args, arg)
if len(mtag.Get("required")) != 0 {
c.ArgsRequired = true
}
}
return true, nil
}
subcommand := mtag.Get("command")
if len(subcommand) != 0 {
ptrval := reflect.NewAt(realval.Type(), unsafe.Pointer(realval.UnsafeAddr()))
shortDescription := mtag.Get("description")
longDescription := mtag.Get("long-description")
subcommandsOptional := mtag.Get("subcommands-optional")
aliases := mtag.GetMany("alias")
subc, err := c.AddCommand(subcommand, shortDescription, longDescription, ptrval.Interface())
if err != nil {
return true, err
}
if len(subcommandsOptional) > 0 {
subc.SubcommandsOptional = true
}
if len(aliases) > 0 {
subc.Aliases = aliases
}
return true, nil
}
return parentg.scanSubGroupHandler(realval, sfield)
}
return f
}
func (c *Command) scan() error {
return c.scanType(c.scanSubcommandHandler(c.Group))
}
func (c *Command) eachCommand(f func(*Command), recurse bool) {
f(c)
for _, cc := range c.commands {
if recurse {
cc.eachCommand(f, true)
} else {
f(cc)
}
}
}
func (c *Command) eachActiveGroup(f func(cc *Command, g *Group)) {
c.eachGroup(func(g *Group) {
f(c, g)
})
if c.Active != nil {
c.Active.eachActiveGroup(f)
}
}
func (c *Command) addHelpGroups(showHelp func() error) {
if !c.hasBuiltinHelpGroup {
c.addHelpGroup(showHelp)
c.hasBuiltinHelpGroup = true
}
for _, cc := range c.commands {
cc.addHelpGroups(showHelp)
}
}
func (c *Command) makeLookup() lookup {
ret := lookup{
shortNames: make(map[string]*Option),
longNames: make(map[string]*Option),
commands: make(map[string]*Command),
}
parent := c.parent
for parent != nil {
if cmd, ok := parent.(*Command); ok {
cmd.fillLookup(&ret, true)
}
if grp, ok := parent.(*Group); ok {
parent = grp
} else {
parent = nil
}
}
c.fillLookup(&ret, false)
return ret
}
func (c *Command) fillLookup(ret *lookup, onlyOptions bool) {
c.eachGroup(func(g *Group) {
for _, option := range g.options {
if option.ShortName != 0 {
ret.shortNames[string(option.ShortName)] = option
}
if len(option.LongName) > 0 {
ret.longNames[option.LongNameWithNamespace()] = option
}
}
})
if onlyOptions {
return
}
for _, subcommand := range c.commands {
ret.commands[subcommand.Name] = subcommand
for _, a := range subcommand.Aliases {
ret.commands[a] = subcommand
}
}
}
func (c *Command) groupByName(name string) *Group {
if grp := c.Group.groupByName(name); grp != nil {
return grp
}
for _, subc := range c.commands {
prefix := subc.Name + "."
if strings.HasPrefix(name, prefix) {
if grp := subc.groupByName(name[len(prefix):]); grp != nil {
return grp
}
} else if name == subc.Name {
return subc.Group
}
}
return nil
}
type commandList []*Command
func (c commandList) Less(i, j int) bool {
return c[i].Name < c[j].Name
}
func (c commandList) Len() int {
return len(c)
}
func (c commandList) Swap(i, j int) {
c[i], c[j] = c[j], c[i]
}
func (c *Command) sortedCommands() []*Command {
ret := make(commandList, len(c.commands))
copy(ret, c.commands)
sort.Sort(ret)
return []*Command(ret)
}
func (c *Command) match(name string) bool {
if c.Name == name {
return true
}
for _, v := range c.Aliases {
if v == name {
return true
}
}
return false
}
func (c *Command) hasCliOptions() bool {
ret := false
c.eachGroup(func(g *Group) {
if g.isBuiltinHelp {
return
}
for _, opt := range g.options {
if opt.canCli() {
ret = true
}
}
})
return ret
}
func (c *Command) fillParseState(s *parseState) {
s.positional = make([]*Arg, len(c.args))
copy(s.positional, c.args)
s.lookup = c.makeLookup()
s.command = c
}

View File

@@ -1,402 +0,0 @@
package flags
import (
"fmt"
"testing"
)
func TestCommandInline(t *testing.T) {
var opts = struct {
Value bool `short:"v"`
Command struct {
G bool `short:"g"`
} `command:"cmd"`
}{}
p, ret := assertParserSuccess(t, &opts, "-v", "cmd", "-g")
assertStringArray(t, ret, []string{})
if p.Active == nil {
t.Errorf("Expected active command")
}
if !opts.Value {
t.Errorf("Expected Value to be true")
}
if !opts.Command.G {
t.Errorf("Expected Command.G to be true")
}
if p.Command.Find("cmd") != p.Active {
t.Errorf("Expected to find command `cmd' to be active")
}
}
func TestCommandInlineMulti(t *testing.T) {
var opts = struct {
Value bool `short:"v"`
C1 struct {
} `command:"c1"`
C2 struct {
G bool `short:"g"`
} `command:"c2"`
}{}
p, ret := assertParserSuccess(t, &opts, "-v", "c2", "-g")
assertStringArray(t, ret, []string{})
if p.Active == nil {
t.Errorf("Expected active command")
}
if !opts.Value {
t.Errorf("Expected Value to be true")
}
if !opts.C2.G {
t.Errorf("Expected C2.G to be true")
}
if p.Command.Find("c1") == nil {
t.Errorf("Expected to find command `c1'")
}
if c2 := p.Command.Find("c2"); c2 == nil {
t.Errorf("Expected to find command `c2'")
} else if c2 != p.Active {
t.Errorf("Expected to find command `c2' to be active")
}
}
func TestCommandFlagOrder1(t *testing.T) {
var opts = struct {
Value bool `short:"v"`
Command struct {
G bool `short:"g"`
} `command:"cmd"`
}{}
assertParseFail(t, ErrUnknownFlag, "unknown flag `g'", &opts, "-v", "-g", "cmd")
}
func TestCommandFlagOrder2(t *testing.T) {
var opts = struct {
Value bool `short:"v"`
Command struct {
G bool `short:"g"`
} `command:"cmd"`
}{}
assertParseSuccess(t, &opts, "cmd", "-v", "-g")
if !opts.Value {
t.Errorf("Expected Value to be true")
}
if !opts.Command.G {
t.Errorf("Expected Command.G to be true")
}
}
func TestCommandFlagOverride1(t *testing.T) {
var opts = struct {
Value bool `short:"v"`
Command struct {
Value bool `short:"v"`
} `command:"cmd"`
}{}
assertParseSuccess(t, &opts, "-v", "cmd")
if !opts.Value {
t.Errorf("Expected Value to be true")
}
if opts.Command.Value {
t.Errorf("Expected Command.Value to be false")
}
}
func TestCommandFlagOverride2(t *testing.T) {
var opts = struct {
Value bool `short:"v"`
Command struct {
Value bool `short:"v"`
} `command:"cmd"`
}{}
assertParseSuccess(t, &opts, "cmd", "-v")
if opts.Value {
t.Errorf("Expected Value to be false")
}
if !opts.Command.Value {
t.Errorf("Expected Command.Value to be true")
}
}
func TestCommandEstimate(t *testing.T) {
var opts = struct {
Value bool `short:"v"`
Cmd1 struct {
} `command:"remove"`
Cmd2 struct {
} `command:"add"`
}{}
p := NewParser(&opts, None)
_, err := p.ParseArgs([]string{})
assertError(t, err, ErrCommandRequired, "Please specify one command of: add or remove")
}
func TestCommandEstimate2(t *testing.T) {
var opts = struct {
Value bool `short:"v"`
Cmd1 struct {
} `command:"remove"`
Cmd2 struct {
} `command:"add"`
}{}
p := NewParser(&opts, None)
_, err := p.ParseArgs([]string{"rmive"})
assertError(t, err, ErrUnknownCommand, "Unknown command `rmive', did you mean `remove'?")
}
type testCommand struct {
G bool `short:"g"`
Executed bool
EArgs []string
}
func (c *testCommand) Execute(args []string) error {
c.Executed = true
c.EArgs = args
return nil
}
func TestCommandExecute(t *testing.T) {
var opts = struct {
Value bool `short:"v"`
Command testCommand `command:"cmd"`
}{}
assertParseSuccess(t, &opts, "-v", "cmd", "-g", "a", "b")
if !opts.Value {
t.Errorf("Expected Value to be true")
}
if !opts.Command.Executed {
t.Errorf("Did not execute command")
}
if !opts.Command.G {
t.Errorf("Expected Command.C to be true")
}
assertStringArray(t, opts.Command.EArgs, []string{"a", "b"})
}
func TestCommandClosest(t *testing.T) {
var opts = struct {
Value bool `short:"v"`
Cmd1 struct {
} `command:"remove"`
Cmd2 struct {
} `command:"add"`
}{}
args := assertParseFail(t, ErrUnknownCommand, "Unknown command `addd', did you mean `add'?", &opts, "-v", "addd")
assertStringArray(t, args, []string{"addd"})
}
func TestCommandAdd(t *testing.T) {
var opts = struct {
Value bool `short:"v"`
}{}
var cmd = struct {
G bool `short:"g"`
}{}
p := NewParser(&opts, Default)
c, err := p.AddCommand("cmd", "", "", &cmd)
if err != nil {
t.Fatalf("Unexpected error: %v", err)
return
}
ret, err := p.ParseArgs([]string{"-v", "cmd", "-g", "rest"})
if err != nil {
t.Fatalf("Unexpected error: %v", err)
return
}
assertStringArray(t, ret, []string{"rest"})
if !opts.Value {
t.Errorf("Expected Value to be true")
}
if !cmd.G {
t.Errorf("Expected Command.G to be true")
}
if p.Command.Find("cmd") != c {
t.Errorf("Expected to find command `cmd'")
}
if p.Commands()[0] != c {
t.Errorf("Expected command %#v, but got %#v", c, p.Commands()[0])
}
if c.Options()[0].ShortName != 'g' {
t.Errorf("Expected short name `g' but got %v", c.Options()[0].ShortName)
}
}
func TestCommandNestedInline(t *testing.T) {
var opts = struct {
Value bool `short:"v"`
Command struct {
G bool `short:"g"`
Nested struct {
N string `long:"n"`
} `command:"nested"`
} `command:"cmd"`
}{}
p, ret := assertParserSuccess(t, &opts, "-v", "cmd", "-g", "nested", "--n", "n", "rest")
assertStringArray(t, ret, []string{"rest"})
if !opts.Value {
t.Errorf("Expected Value to be true")
}
if !opts.Command.G {
t.Errorf("Expected Command.G to be true")
}
assertString(t, opts.Command.Nested.N, "n")
if c := p.Command.Find("cmd"); c == nil {
t.Errorf("Expected to find command `cmd'")
} else {
if c != p.Active {
t.Errorf("Expected `cmd' to be the active parser command")
}
if nested := c.Find("nested"); nested == nil {
t.Errorf("Expected to find command `nested'")
} else if nested != c.Active {
t.Errorf("Expected to find command `nested' to be the active `cmd' command")
}
}
}
func TestRequiredOnCommand(t *testing.T) {
var opts = struct {
Value bool `short:"v" required:"true"`
Command struct {
G bool `short:"g"`
} `command:"cmd"`
}{}
assertParseFail(t, ErrRequired, fmt.Sprintf("the required flag `%cv' was not specified", defaultShortOptDelimiter), &opts, "cmd")
}
func TestRequiredAllOnCommand(t *testing.T) {
var opts = struct {
Value bool `short:"v" required:"true"`
Missing bool `long:"missing" required:"true"`
Command struct {
G bool `short:"g"`
} `command:"cmd"`
}{}
assertParseFail(t, ErrRequired, fmt.Sprintf("the required flags `%smissing' and `%cv' were not specified", defaultLongOptDelimiter, defaultShortOptDelimiter), &opts, "cmd")
}
func TestDefaultOnCommand(t *testing.T) {
var opts = struct {
Command struct {
G bool `short:"g" default:"true"`
} `command:"cmd"`
}{}
assertParseSuccess(t, &opts, "cmd")
if !opts.Command.G {
t.Errorf("Expected G to be true")
}
}
func TestSubcommandsOptional(t *testing.T) {
var opts = struct {
Value bool `short:"v"`
Cmd1 struct {
} `command:"remove"`
Cmd2 struct {
} `command:"add"`
}{}
p := NewParser(&opts, None)
p.SubcommandsOptional = true
_, err := p.ParseArgs([]string{"-v"})
if err != nil {
t.Fatalf("Unexpected error: %v", err)
return
}
if !opts.Value {
t.Errorf("Expected Value to be true")
}
}
func TestCommandAlias(t *testing.T) {
var opts = struct {
Command struct {
G bool `short:"g" default:"true"`
} `command:"cmd" alias:"cm"`
}{}
assertParseSuccess(t, &opts, "cm")
if !opts.Command.G {
t.Errorf("Expected G to be true")
}
}

View File

@@ -1,304 +0,0 @@
package flags
import (
"fmt"
"path/filepath"
"reflect"
"sort"
"strings"
"unicode/utf8"
)
// Completion is a type containing information of a completion.
type Completion struct {
// The completed item
Item string
// A description of the completed item (optional)
Description string
}
type completions []Completion
func (c completions) Len() int {
return len(c)
}
func (c completions) Less(i, j int) bool {
return c[i].Item < c[j].Item
}
func (c completions) Swap(i, j int) {
c[i], c[j] = c[j], c[i]
}
// Completer is an interface which can be implemented by types
// to provide custom command line argument completion.
type Completer interface {
// Complete receives a prefix representing a (partial) value
// for its type and should provide a list of possible valid
// completions.
Complete(match string) []Completion
}
type completion struct {
parser *Parser
ShowDescriptions bool
}
// Filename is a string alias which provides filename completion.
type Filename string
func completionsWithoutDescriptions(items []string) []Completion {
ret := make([]Completion, len(items))
for i, v := range items {
ret[i].Item = v
}
return ret
}
// Complete returns a list of existing files with the given
// prefix.
func (f *Filename) Complete(match string) []Completion {
ret, _ := filepath.Glob(match + "*")
return completionsWithoutDescriptions(ret)
}
func (c *completion) skipPositional(s *parseState, n int) {
if n >= len(s.positional) {
s.positional = nil
} else {
s.positional = s.positional[n:]
}
}
func (c *completion) completeOptionNames(names map[string]*Option, prefix string, match string) []Completion {
n := make([]Completion, 0, len(names))
for k, opt := range names {
if strings.HasPrefix(k, match) {
n = append(n, Completion{
Item: prefix + k,
Description: opt.Description,
})
}
}
return n
}
func (c *completion) completeLongNames(s *parseState, prefix string, match string) []Completion {
return c.completeOptionNames(s.lookup.longNames, prefix, match)
}
func (c *completion) completeShortNames(s *parseState, prefix string, match string) []Completion {
if len(match) != 0 {
return []Completion{
Completion{
Item: prefix + match,
},
}
}
return c.completeOptionNames(s.lookup.shortNames, prefix, match)
}
func (c *completion) completeCommands(s *parseState, match string) []Completion {
n := make([]Completion, 0, len(s.command.commands))
for _, cmd := range s.command.commands {
if cmd.data != c && strings.HasPrefix(cmd.Name, match) {
n = append(n, Completion{
Item: cmd.Name,
Description: cmd.ShortDescription,
})
}
}
return n
}
func (c *completion) completeValue(value reflect.Value, prefix string, match string) []Completion {
i := value.Interface()
var ret []Completion
if cmp, ok := i.(Completer); ok {
ret = cmp.Complete(match)
} else if value.CanAddr() {
if cmp, ok = value.Addr().Interface().(Completer); ok {
ret = cmp.Complete(match)
}
}
for i, v := range ret {
ret[i].Item = prefix + v.Item
}
return ret
}
func (c *completion) completeArg(arg *Arg, prefix string, match string) []Completion {
if arg.isRemaining() {
// For remaining positional args (that are parsed into a slice), complete
// based on the element type.
return c.completeValue(reflect.New(arg.value.Type().Elem()), prefix, match)
}
return c.completeValue(arg.value, prefix, match)
}
func (c *completion) complete(args []string) []Completion {
if len(args) == 0 {
args = []string{""}
}
s := &parseState{
args: args,
}
c.parser.fillParseState(s)
var opt *Option
for len(s.args) > 1 {
arg := s.pop()
if (c.parser.Options&PassDoubleDash) != None && arg == "--" {
opt = nil
c.skipPositional(s, len(s.args)-1)
break
}
if argumentIsOption(arg) {
prefix, optname, islong := stripOptionPrefix(arg)
optname, _, argument := splitOption(prefix, optname, islong)
if argument == nil {
var o *Option
canarg := true
if islong {
o = s.lookup.longNames[optname]
} else {
for i, r := range optname {
sname := string(r)
o = s.lookup.shortNames[sname]
if o == nil {
break
}
if i == 0 && o.canArgument() && len(optname) != len(sname) {
canarg = false
break
}
}
}
if o == nil && (c.parser.Options&PassAfterNonOption) != None {
opt = nil
c.skipPositional(s, len(s.args)-1)
break
} else if o != nil && o.canArgument() && !o.OptionalArgument && canarg {
if len(s.args) > 1 {
s.pop()
} else {
opt = o
}
}
}
} else {
if len(s.positional) > 0 {
if !s.positional[0].isRemaining() {
// Don't advance beyond a remaining positional arg (because
// it consumes all subsequent args).
s.positional = s.positional[1:]
}
} else if cmd, ok := s.lookup.commands[arg]; ok {
cmd.fillParseState(s)
}
opt = nil
}
}
lastarg := s.args[len(s.args)-1]
var ret []Completion
if opt != nil {
// Completion for the argument of 'opt'
ret = c.completeValue(opt.value, "", lastarg)
} else if argumentStartsOption(lastarg) {
// Complete the option
prefix, optname, islong := stripOptionPrefix(lastarg)
optname, split, argument := splitOption(prefix, optname, islong)
if argument == nil && !islong {
rname, n := utf8.DecodeRuneInString(optname)
sname := string(rname)
if opt := s.lookup.shortNames[sname]; opt != nil && opt.canArgument() {
ret = c.completeValue(opt.value, prefix+sname, optname[n:])
} else {
ret = c.completeShortNames(s, prefix, optname)
}
} else if argument != nil {
if islong {
opt = s.lookup.longNames[optname]
} else {
opt = s.lookup.shortNames[optname]
}
if opt != nil {
ret = c.completeValue(opt.value, prefix+optname+split, *argument)
}
} else if islong {
ret = c.completeLongNames(s, prefix, optname)
} else {
ret = c.completeShortNames(s, prefix, optname)
}
} else if len(s.positional) > 0 {
// Complete for positional argument
ret = c.completeArg(s.positional[0], "", lastarg)
} else if len(s.command.commands) > 0 {
// Complete for command
ret = c.completeCommands(s, lastarg)
}
sort.Sort(completions(ret))
return ret
}
func (c *completion) execute(args []string) {
ret := c.complete(args)
if c.ShowDescriptions && len(ret) > 1 {
maxl := 0
for _, v := range ret {
if len(v.Item) > maxl {
maxl = len(v.Item)
}
}
for _, v := range ret {
fmt.Printf("%s", v.Item)
if len(v.Description) > 0 {
fmt.Printf("%s # %s", strings.Repeat(" ", maxl-len(v.Item)), v.Description)
}
fmt.Printf("\n")
}
} else {
for _, v := range ret {
fmt.Println(v.Item)
}
}
}

Some files were not shown because too many files have changed in this diff Show More