amorfo77/restic
1
0
Fork 0
mirror of https://github.com/restic/restic.git synced 2026-07-14 19:24:53 +00:00
Code Issues Projects Releases Wiki Activity
5,058 Commits 11 Branches 53 Tags
4a0b7328eccfcd4aa6a36dd32a8c4fd312fbb2ac
Commit Graph
216 Commits
Author SHA1 Message Date
Alexander Neumann 4a0b7328ec s3: Remove dots for config description 2020-11-11 20:20:35 +01:00
Nick Douma 829959390a Provide UseV1 parameter to minio.ListObjectsOptions based on s3.list-objects-v1 2020-11-11 11:54:38 +01:00
Nick Douma ccd55d529d Add s3.list-objects-v1 extended option and default to false 2020-11-11 11:54:36 +01:00
Alexander Weiss fef408a8bd Return context error in mem backend 2020-11-08 00:05:53 +01:00
Ivan Andreev ab2790d9de Fix http2 stream reset between restic and rest backends #3014 2020-11-05 15:57:40 +03:00
Nick Craig-WoodandAlexander Neumann 86b5d8ffaa s3: add bucket-lookup parameter to select path or dns style bucket lookup 
This is to enable restic working with Alibaba cloud

Fixes #2528
 2020-11-05 12:20:10 +01:00
Alexander NeumannandGitHub 9a88fb253b Merge pull request #3051 from greatroar/sanitize-env 
Sanitize environment before starting backend processes (rclone, ssh)
 2020-11-02 21:18:57 +01:00
greatroar 11fbaaae9a Sanitize environment before starting backend processes (rclone, ssh) 
The restic security model includes full trust of the local machine, so
this should not fix any actual security problems, but it's better to be
safe than sorry.

Fixes #2192.
 2020-11-02 16:41:23 +01:00
Alexander NeumannandGitHub 3ff37215df Merge pull request #2935 from MichaelEischer/upgrade-minio 
Upgrade minio SDK to version 7
 2020-11-02 09:09:10 +01:00
Michael Eischer 37a5e2d681 rest: use global context on repository creation 2020-10-09 22:39:06 +02:00
Michael Eischer 45e9a55c62 Wire context into backend layout detection 2020-10-09 22:37:24 +02:00
Michael Eischer 307a6ba3a3 Upgrade minio sdk to v7 
This changes are primarily straightforward modifications to pass the
parameters in the now expected way.
 2020-10-09 22:37:24 +02:00
Alexander NeumannandGitHub 30cb553c8d Merge pull request #2932 from MichaelEischer/proper-rclone-create 
Call rclone.Create to create a new repository for the rclone backend
 2020-10-09 21:29:15 +02:00
MichaelEischerandGitHub 88cc444779 Merge pull request #2934 from MichaelEischer/upgrade-backoff 
Upgrade github.com/cenkalti/backoff module
 2020-10-08 20:33:30 +02:00
Michael Eischer b79f18209f Upgrade github.com/cenkalti/backoff module 
We now use v4 of the module. `backoff.WithMaxRetries` no longer repeats
an operation endlessly when a retry count of 0 is specified. This
required a few fixes for the tests.
 2020-10-07 22:04:59 +02:00
Michael Eischer f4282aa6fd local: mark repository files as read-only 
This is intended to prevent accidental modifications of data files.
Marking the files as read-only was accidentally removed in #1258.
 2020-10-07 12:29:37 +02:00
Michael Eischer 40ee17167e local: Ignore permission errors on chmod call in Save/Remove operation 
The file is already created with the proper permissions, thus the chmod
call is not critical. However, some file systems have don't allow
modifications of the file permissions. Similarly the chmod call in the Remove
operation should not prevent it from working.
 2020-10-07 12:29:37 +02:00
Ingo Gottwald 8b8e230771 Swap deprecated GCS lib with replacement 2020-10-03 18:55:56 +02:00
Ingo Gottwald 00cedd22aa Replace deprecated method in gs backend 2020-10-01 10:02:42 +02:00
MichaelEischerandGitHub fd02407863 Merge pull request #2849 from classmarkets/gcs-access-token 
gs: support authentication with access token
 2020-09-30 17:42:56 +02:00
FredandMichael Eischer 206cadfab4 Hide password from repository URLs 2020-09-22 22:00:51 +02:00
Michael Eischer 8c36317b71 rclone: use configured number of connections during create 2020-09-19 19:11:43 +02:00
Michael Eischer 60a5c35de9 rclone: close connection to rclone if open fails 2020-09-19 19:11:43 +02:00
Michael Eischer b4a7ce86cf uint cannot be less than zero 2020-09-05 10:07:16 +02:00
Hristo TrendevandLeo R. Lundgren 51c22f4223 local backend: ignore not supported error on sync() 
Closes #2395
 2020-08-22 01:27:07 +02:00
aawsomeandGitHub 0fed6a8dfc Use "pack file" instead of "data file" (#2885) 
- changed variable names, especially changed DataFile into PackFile
- changed in some comments
- always use "pack file" in docu
 2020-08-16 11:16:38 +02:00
Michael Eischer c81b122374 rclone: Don't panic after unexpected subprocess exit 
As the connection to the rclone child process is now closed after an
unexpected subprocess exit, later requests will cause the http2
transport to try to reestablish a new connection. As previously this never
should have happened, the connection called panic in that case. This
panic is now replaced with a simple error message, as it no longer
indicates an internal problem.
 2020-08-01 12:17:40 +02:00
Michael Eischer ea97ff1ba4 rclone: Skip crash test when rclone is not found 2020-07-26 12:06:18 +02:00
Michael Eischer 01b9581453 rclone: Better field names for stdio conn 2020-07-26 00:29:25 +02:00
Michael Eischer 3cd927d180 rclone: Give rclone time to finish before closing stdin pipe 
Calling `Close()` on the rclone backend sometimes failed during test
execution with 'signal: Broken pipe'. The stdio connection closed both
the stdin and stdout file descriptors at the same moment, therefore
giving rclone no chance to properly send any final http2 data frames.

Now the stdin connection to rclone is closed first and will only be
forcefully closed after a timeout. In case rclone exits before the
timeout then the stdio connection will be closed normally.
 2020-07-26 00:29:25 +02:00
greatroarandMichael Eischer bf7b1f12ea rclone: Add test for pipe handling when rclone exits 2020-07-26 00:29:25 +02:00
Michael Eischer 8554332894 rclone: Close rclone side of stdio_conn pipes 
restic did not notice when the rclone subprocess exited unexpectedly.

restic manually created pipes for stdin and stdout and used these for the
connection to the rclone subprocess. The process creating a pipe gets
file descriptors for the sender and receiver side of a pipe and passes
them on to the subprocess. The expected behavior would be that reads or
writes in the parent process fail / return once the child process dies
as a pipe would now just have a reader or writer but not both.

However, this never happened as restic kept the reader and writer
file descriptors of the pipes. `cmd.StdinPipe` and `cmd.StdoutPipe`
close the subprocess side of pipes once the child process was started
and close the parent process side of pipes once wait has finished. We
can't use these functions as we need access to the raw `os.File` so just
replicate that behavior.
 2020-07-26 00:29:25 +02:00
greatroarandMichael Eischer 3e93b36ca4 Make rclone.New private 2020-07-26 00:28:45 +02:00
Peter Schultz 758b44b9c0 gs: support authentication with access token 
In the Google Cloud Storage backend, support specifying access tokens
directly, as an alternative to a credentials file. This is useful when
restic is used non-interactively by some other program that is already
authenticated and eliminates the need to store long lived credentials.

The access token is specified in the GOOGLE_ACCESS_TOKEN environment
variable and takes precedence over GOOGLE_APPLICATION_CREDENTIALS.
 2020-07-22 16:23:03 +02:00
Alexander Neumann 8074879c5f Remove 'go generate' 2020-07-19 17:28:42 +02:00
greatroar 190d8e2f51 Flatten backend.LimitedReadCloser structure 
This inlines the io.LimitedReader into the LimitedReadCloser body to
achieve fewer allocations. Results on linux/amd64:

name                                      old time/op    new time/op    delta
Backend/BenchmarkLoadPartialFile-8           412µs ± 4%     413µs ± 4%    ~     (p=0.634 n=17+17)
Backend/BenchmarkLoadPartialFileOffset-8     455µs ±13%     441µs ±10%    ~     (p=0.072 n=20+18)

name                                      old speed      new speed      delta
Backend/BenchmarkLoadPartialFile-8        10.2GB/s ± 3%  10.2GB/s ± 4%    ~     (p=0.817 n=16+17)
Backend/BenchmarkLoadPartialFileOffset-8  9.25GB/s ±12%  9.54GB/s ± 9%    ~     (p=0.072 n=20+18)

name                                      old alloc/op   new alloc/op   delta
Backend/BenchmarkLoadPartialFile-8            888B ± 0%      872B ± 0%  -1.80%  (p=0.000 n=15+15)
Backend/BenchmarkLoadPartialFileOffset-8      888B ± 0%      872B ± 0%  -1.80%  (p=0.000 n=15+15)

name                                      old allocs/op  new allocs/op  delta
Backend/BenchmarkLoadPartialFile-8            18.0 ± 0%      17.0 ± 0%  -5.56%  (p=0.000 n=15+15)
Backend/BenchmarkLoadPartialFileOffset-8      18.0 ± 0%      17.0 ± 0%  -5.56%  (p=0.000 n=15+15)
 2020-06-17 13:11:45 +02:00
greatroar f4cd2a7120 Make backend benchmarks fairer by removing checks 
Checking whether the right data is returned takes up half the time in
some benchmarks. Results for local backend benchmarks on linux/amd64:

name                                      old time/op    new time/op    delta
Backend/BenchmarkLoadFile-8                 4.89ms ± 0%    2.72ms ± 1%   -44.26%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFile-8           936µs ± 6%     439µs ±15%   -53.07%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFileOffset-8     940µs ± 1%     456µs ±10%   -51.50%  (p=0.008 n=5+5)
Backend/BenchmarkSave-8                     23.9ms ±14%    24.8ms ±41%      ~     (p=0.690 n=5+5)

name                                      old speed      new speed      delta
Backend/BenchmarkLoadFile-8               3.43GB/s ± 0%  6.16GB/s ± 1%   +79.40%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFile-8        4.48GB/s ± 6%  9.63GB/s ±14%  +114.78%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFileOffset-8  4.46GB/s ± 1%  9.22GB/s ±10%  +106.74%  (p=0.008 n=5+5)
Backend/BenchmarkSave-8                    706MB/s ±13%   698MB/s ±31%      ~     (p=0.690 n=5+5)
 2020-06-17 13:11:45 +02:00
greatroar df66daa5c9 Fix context usage in backend tests 
Found by go vet. This is also the only complaint is has.
 2020-04-18 17:39:06 +02:00
greatroar 8cf3bb8737 Revert "Put host last in SSH command line" 
This reverts commit e1969d1e33.
 2020-03-08 16:45:33 +01:00
Alexander NeumannandGitHub 0c03a80fc4 Merge pull request #2592 from greatroar/sftp-ipv6 
Support IPv6 in SFTP backend
 2020-03-01 19:38:44 +01:00
Alexander Neumann 99fd80a585 Remove all workarounds for Go < 1.11 2020-02-26 20:35:13 +01:00
greatroar e1969d1e33 Put host last in SSH command line 
This is how the SSH manpage says the command line should look, and the
"--" prevents mistakes in hostnames from being interpreted as options.
 2020-02-19 15:53:20 +01:00
greatroar 6ac6bca7a1 Support IPv6 in SFTP backend 
The previous code was doing its own hostname:port splitting, which
caused IPv6 addresses to be misinterpreted.
 2020-02-19 15:42:12 +01:00
rawtazandGitHub 680a14afa1 Merge pull request #2530 from restic/fix-sftp-mkdirall 
sftp: Use MkdirAll provided by the client
 2020-02-13 01:12:59 +01:00
Lars Lehtonen 3ed54e762e internal/backend/sftp: fix dropped test error 2020-02-12 13:36:21 -08:00
Alexander Neumann 2cd9c7ef16 sftp: Use MkdirAll provided by the client 
Closes #2518
 2020-01-01 17:26:38 +01:00
streambinder 97e5ce4344 internal: backend: sftp: support user@domain parsing as user 2019-12-19 13:15:37 +01:00
rawtazandGitHub e14c4b1737 Merge pull request #2484 from restic/add-s3-region 
s3: Allow specifying region
 2019-11-22 15:51:17 +01:00
Alexander Neumann 409909a7f5 Add option description for Region 2019-11-22 15:09:09 +01:00
mdauphinandAlexander Neumann df500a372d Add AWS_REGION env var to specify s3 region 2019-11-22 15:04:48 +01:00