Compare commits

...

3247 Commits

Author SHA1 Message Date
Leo R. Lundgren
6d39410958 doc: Reword parts of the text, replace rclone with rest-server
The intent here is to make the text more consistent in its use of different
concepts involved in explaining the idea and setup that is explained, and
to make it easier to follow.

We're also replacing rclone with rest-server, not because we dislike rclone
but in order to keep the text to the basic tooling and the main restic eco-
system.

Finally we also remove the previous tip at the end about keeping the SSH tunnel
up, as it will be during the time the SSH session is running (in which the user
is expected to run the restic commands).
2026-02-18 22:19:56 +01:00
JL710
5c3116901e use rclone for rest server instead of docker 2026-01-17 16:29:26 +01:00
JL710
8943ca15ed apply suggestions from Michael Eischer 2025-09-25 13:35:40 +02:00
JL710
a9d51db68d add example for "Pulling a Backup with HTTP over a ssh tunnel" 2025-09-11 16:07:50 +02:00
Michael Eischer
cc1fe6c111 Merge pull request #5493 from greatroar/ioctl
backend,termstatus: Unify foreground/background detection
2025-09-08 10:40:36 +02:00
greatroar
1ed93bd54d backend,termstatus: Unify foreground/background detection
PR #5358 reintroduced a version of the TIOCGPGRP ioctl call that works
on all Unix platforms except Linux, due to a bug/inconsistency in
x/sys/unix. This commit fixes that by introducing termstatus.Tcgetpgrp.

It also introduces termstatus.Getpgrp and termstatus.Tcsetpgrp to deal
with the different signature of unix.Getpgrp in Solaris vs. all other
Unix platforms and an int-overflowing constant on AIX, so that some
AIX/Solaris-specific code can be removed elsewhere and
foreground/background detection is done the same everywhere except on
Windows.
2025-09-08 09:33:44 +02:00
Michael Eischer
c55df65bb6 Merge pull request #5490 from MichaelEischer/sync-compatibility-section
docs: sync compatibility section with website
2025-09-07 11:35:48 +02:00
Michael Eischer
289adebbb7 Merge pull request #5492 from MichaelEischer/update-dependencies
update dependencies
2025-09-07 11:34:37 +02:00
Michael Eischer
eef047cfa4 update dependencies 2025-09-06 22:58:30 +02:00
Michael Eischer
7f1ac5764a Merge pull request #5358 from MichaelEischer/fix-tty-capture-in-background
Fix tty capture when started in background
2025-09-06 21:01:03 +02:00
Michael Eischer
52aa1cd17f docs: sync compatibility section with website
This is no change in policy, just a more precise description of the
status quo.
2025-09-06 20:55:48 +02:00
Michael Eischer
42f690dbab add changelog 2025-09-06 20:50:06 +02:00
Michael Eischer
914bd699be backend: always start backend process in separate process group
The process group is necessary to properly handle ctrl-c.
2025-09-06 19:51:10 +02:00
Michael Eischer
4c19d6410f backend: only move child process to foreground if already in foreground
The rclone and sftp backends require starting a child process. The are
first moved into the foreground and back into the background after the
initial startup is done.

However, this behavior is also active if restic itself is started in the
background. In this case, restic changing the foreground process may
confuse the shell and in case of bash causes it to exit. Thus, disable
modification of the controlling process group of the tty if restic is
run in the background.
2025-09-06 19:51:10 +02:00
Michael Eischer
2ad703bfd8 Merge pull request #5486 from greatroar/duration-panic
Avoid panic in duration parsing
2025-09-06 18:32:36 +02:00
greatroar
0864d04c5c internal/restic: Fix panic in ParseDuration
Fixes #5485. Includes test case by @MichaelEischer.
2025-09-06 10:30:34 +02:00
gregoster
839c38b4c4 EOPNOTSUPP can be returned if the filesystem does not support xattrs (#5344)
---------

Co-authored-by: Greg Oster <oster@netbsd.org>
2025-09-05 19:09:27 +00:00
Michael Eischer
e98c44baaf Merge pull request #5421 from MichaelEischer/fix-backup-metadata-crash
backup: fix possible crash if directory disappears before metadata collection
2025-09-05 19:53:16 +02:00
Michael Eischer
01bc60e96f backup: fix test on windows 2025-09-05 19:35:25 +02:00
Michael Eischer
484b706dd8 add changelog 2025-09-05 19:28:18 +02:00
Michael Eischer
350f6452e7 backup: test that parent directory errors can be correctly filtered 2025-09-05 19:28:18 +02:00
Michael Eischer
484cdf12e5 backup: test that missing parent directory is correctly handled 2025-09-05 19:28:18 +02:00
Michael Eischer
c8bb7bd312 backup: do not fail backup is some parent folder is inaccessible
Handle errors for parent directories of backup directories in the same
way as all other file access errors during a backup.
2025-09-05 19:28:18 +02:00
Michael Eischer
a8ce2e45cc backup: do not crash if nodeFromFileInfo fails
this could crash in two cases:
- if a directory is deleted between restic stating it and trying to list
  its directory content.
- when restic tries to list the parent directory of a backup target, but
  the parent directory has been deleted.

return an error in this case instead.
2025-09-05 19:28:18 +02:00
Michael Eischer
275507fb3e Merge pull request #5452 from dominikschulz/fix/storage
Mark HTTP Error 507 as permanent
2025-09-05 19:17:15 +02:00
Michael Eischer
391c27975a Merge pull request #5460 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/storage/azblob-1.6.2
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from 1.6.1 to 1.6.2
2025-09-05 19:16:54 +02:00
Michael Eischer
7d47e60e27 Merge pull request #5481 from restic/dependabot/github_actions/actions/checkout-5
build(deps): bump actions/checkout from 4 to 5
2025-09-05 19:07:58 +02:00
Michael Eischer
5c17c277f3 add changelog 2025-09-05 19:07:09 +02:00
Dominik Schulz
48e5c0984e Mark HTTP Error 507 as permanent
This change classifies HTTP error 507 (Insufficient Storage) as a
permanent error that should not be retried. I keep running into
this once in a while and there is literally no point in retrying when
the server is full.

Fixes #5429

Signed-off-by: Dominik Schulz <dominik.schulz@gauner.org>
2025-09-05 19:04:16 +02:00
dependabot[bot]
2414771a59 build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
Bumps [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) from 1.6.1 to 1.6.2.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/go-mgmt-sdk-release-guideline.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.6.1...sdk/storage/azblob/v1.6.2)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
  dependency-version: 1.6.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-05 16:58:21 +00:00
Michael Eischer
0e381bdbf1 Merge pull request #5483 from restic/dependabot/go_modules/google.golang.org/api-0.248.0
build(deps): bump google.golang.org/api from 0.228.0 to 0.248.0
2025-09-05 18:47:58 +02:00
dependabot[bot]
11cd4a0a88 build(deps): bump actions/checkout from 4 to 5
Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 5.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-05 16:38:46 +00:00
dependabot[bot]
f54989f634 build(deps): bump google.golang.org/api from 0.228.0 to 0.248.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.228.0 to 0.248.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.228.0...v0.248.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-version: 0.248.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-05 16:36:17 +00:00
Michael Eischer
af7f10d16b Merge pull request #5482 from restic/dependabot/github_actions/docker/login-action-3.5.0
build(deps): bump docker/login-action from 3.4.0 to 3.5.0
2025-09-05 18:35:59 +02:00
Michael Eischer
6de95a3d58 Merge pull request #5484 from restic/dependabot/go_modules/golang.org/x/crypto-0.41.0
build(deps): bump golang.org/x/crypto from 0.39.0 to 0.41.0
2025-09-05 18:35:24 +02:00
dependabot[bot]
93f436e999 build(deps): bump golang.org/x/crypto from 0.39.0 to 0.41.0
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.39.0 to 0.41.0.
- [Commits](https://github.com/golang/crypto/compare/v0.39.0...v0.41.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.41.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-01 02:10:03 +00:00
dependabot[bot]
6edb199efd build(deps): bump docker/login-action from 3.4.0 to 3.5.0
Bumps [docker/login-action](https://github.com/docker/login-action) from 3.4.0 to 3.5.0.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](74a5d14239...184bdaa072)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-version: 3.5.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-01 01:57:28 +00:00
rawtaz
9b2c0a0c54 Merge pull request #5466 from y0n3d4/gentoo-install-info
Added Gentoo install info
2025-08-07 13:28:17 +02:00
y0n3d4
64273ea027 Update 020_installation.rst removing command options
Removed command options: their use is a user choice
2025-08-06 13:57:29 +02:00
Michele Testa
5a00d26431 Update 020_installation.rst adding instruction for Gentoo Linux 2025-08-05 15:16:37 +02:00
Michele Testa
3faad5751d Revert "Update 020_installation.rst adding install command for Gentoo Linux"
This reverts commit f487eb1c66.
2025-08-05 14:12:04 +02:00
y0n3d4
f487eb1c66 Update 020_installation.rst adding install command for Gentoo Linux
Added basic instructions for restic installation on Gentoo using the official package manager (Portage)
2025-08-05 14:01:10 +02:00
Michael Eischer
72636238d0 Merge pull request #5400 from rhhub/patch-2
docs: clarify ** wildcard must me between path separators
2025-08-04 20:56:07 +02:00
rawtaz
51098157e2 Merge pull request #5462 from TheAlchemistOf42/wasabi-link
Correct Wasabi link
2025-08-02 22:12:57 +02:00
A Crutcher
0b080c44d7 doc: Correct Wasabi link 2025-08-02 13:03:43 -05:00
Michael Eischer
b71fe91643 Merge pull request #5345 from mikix/chmod-enotsup
backend/local: ignore chmod "not supported" errors
2025-07-21 22:28:53 +02:00
Michael Eischer
9c3b8d171a Merge pull request #5434 from restic/dependabot/go_modules/golang.org/x/time-0.12.0
build(deps): bump golang.org/x/time from 0.11.0 to 0.12.0
2025-07-21 22:17:40 +02:00
Michael Eischer
ddb7fb837b Merge pull request #5435 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/azidentity-1.10.1
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azidentity from 1.10.0 to 1.10.1
2025-07-21 22:14:29 +02:00
Michael Eischer
3433c5abac Merge pull request #5408 from MichaelEischer/fix-walker-crash
walker: fix error handling if tree cannot be loaded
2025-07-21 21:46:59 +02:00
dependabot[bot]
09bc58c950 build(deps): bump golang.org/x/time from 0.11.0 to 0.12.0
Bumps [golang.org/x/time](https://github.com/golang/time) from 0.11.0 to 0.12.0.
- [Commits](https://github.com/golang/time/compare/v0.11.0...v0.12.0)

---
updated-dependencies:
- dependency-name: golang.org/x/time
  dependency-version: 0.12.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-07-21 19:46:35 +00:00
Michael Eischer
20eb9018a0 Merge pull request #5409 from MichaelEischer/fix-release-notes-typos
Fix release note typos
2025-07-21 21:46:05 +02:00
Michael Eischer
651f553530 Merge pull request #5436 from restic/dependabot/go_modules/github.com/peterbourgon/unixtransport-0.0.6
build(deps): bump github.com/peterbourgon/unixtransport from 0.0.4 to 0.0.6
2025-07-21 21:45:06 +02:00
Michael Eischer
aad4b53ead Merge pull request #5438 from restic/dependabot/go_modules/golang.org/x/crypto-0.39.0
build(deps): bump golang.org/x/crypto from 0.38.0 to 0.39.0
2025-07-21 21:42:06 +02:00
dependabot[bot]
e467496ace build(deps): bump golang.org/x/crypto from 0.38.0 to 0.39.0
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.38.0 to 0.39.0.
- [Commits](https://github.com/golang/crypto/compare/v0.38.0...v0.39.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.39.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-07-01 05:46:21 +00:00
dependabot[bot]
f2de260524 build(deps): bump github.com/peterbourgon/unixtransport
Bumps [github.com/peterbourgon/unixtransport](https://github.com/peterbourgon/unixtransport) from 0.0.4 to 0.0.6.
- [Release notes](https://github.com/peterbourgon/unixtransport/releases)
- [Commits](https://github.com/peterbourgon/unixtransport/compare/v0.0.4...v0.0.6)

---
updated-dependencies:
- dependency-name: github.com/peterbourgon/unixtransport
  dependency-version: 0.0.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-07-01 05:07:50 +00:00
dependabot[bot]
c17d5ab2e1 build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azidentity
Bumps [github.com/Azure/azure-sdk-for-go/sdk/azidentity](https://github.com/Azure/azure-sdk-for-go) from 1.10.0 to 1.10.1.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/go-mgmt-sdk-release-guideline.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.10.0...sdk/azidentity/v1.10.1)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azidentity
  dependency-version: 1.10.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-07-01 05:07:41 +00:00
Michael Terry
a8535aba58 backend/local: ignore chmod "not supported" errors 2025-06-02 17:44:30 -04:00
rhhub
521fbad701 docs: clarify ** must me between path separators 2025-06-02 13:46:06 -07:00
Michael Eischer
15b7d7c3fc Fix release note typos 2025-06-02 21:12:37 +02:00
Patrick Wolf
7d39b1bfe8 Update 047_tuning_backup_parameters.rst - local backend (#5355)
users would find it helpful to know how to adjust the "local" backend and they might not get the idea that the local backend is just called local... which in turn leads them to think restic is slow as they can't adjust away  from 2 threads for restore and backup.
2025-06-02 18:40:04 +00:00
Michael Eischer
e4a7f4aadf Merge pull request #5356 from MichaelEischer/fix-backup-stdin-filename
backup: Fix `--stdin-filename` with directory
2025-06-02 20:27:26 +02:00
Michael Eischer
10cfe96cd4 walker: fix error handling if tree cannot be loaded
A tree that cannot be loaded is a fatal error when walking the tree.
Thus, return the error and exit the tree walk.
2025-06-02 20:04:26 +02:00
Michael Eischer
2eaa79d33f Merge pull request #5374 from ilyagr/docprofile
docs: document profiling options a bit better
2025-06-02 19:57:33 +02:00
Ilya Grigoriev
99ee5696f3 bugfix: have --{cpu,mem,...}-profile work even if Restic exits with error code (#5373)
* bugfix: write pprof file for `--{cpu,mem,...}-profile` even on error code

Before this, if `restic backup --cpu-profile dir/ backup-dir/` couldn't
read some of the input files (e.g. they weren't readable by the user
restic was running under), the `cpu.pprof` file it outputs would be
empty.

https://github.com/spf13/cobra/issues/1893

* drop changelog as it's not relevant for end users

---------

Co-authored-by: Michael Eischer <michael.eischer@fau.de>
2025-06-02 17:57:07 +00:00
Ilya Grigoriev
e8dbb69a94 docs: when describing profiling, briefly explain .pprof files 2025-06-02 19:49:19 +02:00
Ilya Grigoriev
f4e21cdb75 docs: document profiling options a bit better
Previously, the docs were a bit mysterious about what "enables profiling
support" means or how one could take advantage of it.
2025-06-02 19:49:19 +02:00
Michael Eischer
e5bdc3c74f Merge pull request #5382 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/storage/azblob-1.6.1
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from 1.6.0 to 1.6.1
2025-06-02 19:40:33 +02:00
Michael Eischer
7e51c928c4 Merge pull request #5384 from zmanda/feat-gh-5377-check-add-percentage-for-read-data-subset
check: add percentage of repository checked
2025-06-02 19:37:08 +02:00
dependabot[bot]
21e87851aa build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
Bumps [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) from 1.6.0 to 1.6.1.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.6.0...sdk/azcore/v1.6.1)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
  dependency-version: 1.6.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-06-02 17:32:18 +00:00
Michael Eischer
2bc1bf2702 Merge pull request #5402 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/azidentity-1.10.0
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azidentity from 1.8.2 to 1.10.0
2025-06-02 19:19:10 +02:00
dependabot[bot]
df110060d1 build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azidentity
Bumps [github.com/Azure/azure-sdk-for-go/sdk/azidentity](https://github.com/Azure/azure-sdk-for-go) from 1.8.2 to 1.10.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/go-mgmt-sdk-release-guideline.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azidentity/v1.8.2...sdk/azcore/v1.10.0)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azidentity
  dependency-version: 1.10.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-06-02 16:45:38 +00:00
Michael Eischer
337a7d1205 Merge pull request #5385 from Silvenga/windows-docs
docs: updated installation docs for Windows
2025-06-02 18:44:43 +02:00
Michael Eischer
322e271dd2 Merge pull request #5404 from restic/dependabot/go_modules/golang.org/x/sys-0.33.0
build(deps): bump golang.org/x/sys from 0.31.0 to 0.33.0
2025-06-02 18:35:34 +02:00
dependabot[bot]
1ac224458f build(deps): bump golang.org/x/sys from 0.31.0 to 0.33.0
Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.31.0 to 0.33.0.
- [Commits](https://github.com/golang/sys/compare/v0.31.0...v0.33.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sys
  dependency-version: 0.33.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-06-01 01:25:20 +00:00
rawtaz
126ad04568 Merge pull request #5389 from fronesis47/patch-3
Update passwords section of FAQ
2025-05-11 20:35:20 +02:00
Samuel Chambers
e732bdbfb8 updated doc/faq.rst_commitsSquashed 2025-05-11 14:02:07 -04:00
Mark Lopez
2db08fd749 docs: updated installation docs for Windows 2025-05-03 10:27:50 -05:00
Srigovind Nayak
debb110a7c check: add percentage of repository checked 2025-05-03 18:08:12 +05:30
rawtaz
5eb4f5af61 Merge pull request #5360 from makuhama/master
doc: typo & minor rewording in 'Removing files from snapshots'
2025-05-02 01:05:08 +02:00
Markus Hansmair
287b601f01 doc: typo & minor rewording in 'Removing files from snapshots' 2025-04-16 09:37:26 +02:00
Michael Eischer
64c82a5d9c Merge pull request #5357 from Hello71/patch-1
doc: add fastest, better compression
2025-04-14 20:02:41 +02:00
Alex Xu
12f36ebf07 doc: add fastest, better compression
Follow-up for #5321
2025-04-13 19:33:13 -04:00
Michael Eischer
45e09dca2a add changelog for --stdin-filename with/directory 2025-04-11 22:29:18 +02:00
Michael Eischer
5bb9d0d996 backup: test subdirectories in stdin filenames work 2025-04-11 22:14:32 +02:00
Michael Eischer
9f39e8a1d3 fs/reader: return proper error on invalid filename 2025-04-11 22:07:31 +02:00
Michael Eischer
ddd48f1e98 fs/reader: test file not exist case 2025-04-11 21:57:45 +02:00
Michael Eischer
6e91ea3397 fs/reader: use test helpers 2025-04-11 21:54:15 +02:00
Michael Eischer
e7c1e4f1ff fs/reader: deduplicate test code 2025-04-11 21:50:47 +02:00
Michael Eischer
70e1037a49 fs/reader: fix open+stat handling 2025-04-11 21:49:25 +02:00
Michael Eischer
19f48084ea fs/reader: use modification time for file and directories
This ensures that a fixed input generates a fully deterministic output
file structure.
2025-04-11 21:46:24 +02:00
Michael Eischer
3a995172b7 fs: rewrite Reader to build fs tree up front
This adds proper support for filenames that include directories. For
example, `/foo/bar` would result in an error when trying to open `/foo`.

The directory tree is now build upfront. This ensures let's the
directory tree construction be handled only once. All accessors then
only have to look up the constructed directory entries.
2025-04-11 21:37:40 +02:00
Michael Eischer
0dffa1208d Merge pull request #5243 from zmanda/feat-gh-4868-show-repo-id-in-df-and-mount
mount: append the repository ID to the name of the FUSE mount
2025-04-02 21:16:11 +02:00
Michael Eischer
6fbcce1d1a docs: fix typos in developer information (#5329) 2025-04-02 21:12:43 +02:00
Michael Eischer
e8d458be7e update direct dependencies (#5340) 2025-04-02 21:10:40 +02:00
dependabot[bot]
4471c7847b build(deps): bump docker/login-action from 3.3.0 to 3.4.0 (#5333)
Bumps [docker/login-action](https://github.com/docker/login-action) from 3.3.0 to 3.4.0.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](9780b0c442...74a5d14239)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-version: 3.4.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-04-02 20:54:17 +02:00
Srigovind Nayak
f13e9c10a4 Add support for additional compression levels fastest and better (#5321)
* repository: expose addtional compression levels

* adding better and fastest compression levels for zstd

* repository: add changelog entry for issue-4728

* chore: fix golint issues

* chore: sort compression modes in the help text

* updating review comments
2025-03-31 21:21:12 +02:00
Michael Eischer
f768683162 Merge pull request #5322 from zmanda/fix-gh-5233-forget-failure-exit-codes
forget: return exit code 3 on partial removal of snapshots
2025-03-31 20:08:33 +02:00
Michael Eischer
0b7bdfed7e Merge pull request #5320 from mjnaderi/patch-1
doc: fix typos
2025-03-31 20:07:51 +02:00
Michael Eischer
a4fe94ec82 Merge pull request #5317 from gilbsgilbs/fix-s3-restore-timeout-unit
docs: fix unit for S3 restore timeout
2025-03-31 20:01:22 +02:00
Michael Eischer
6684d1d2f5 Merge pull request #5327 from MichaelEischer/fix-forget-hostname-default
forget: fix ignored RESTIC_HOST environment variable
2025-03-31 19:21:13 +02:00
Michael Eischer
e1f7522174 forget: fix ignored RESTIC_HOST environment variable 2025-03-31 18:10:17 +02:00
Srigovind Nayak
d1649affb2 chore: update changelog for issue-5233 2025-03-30 14:12:06 +05:30
Srigovind Nayak
936c783c0f forget: exit code 3 for snapshot removal failures 2025-03-30 14:11:32 +05:30
Mohammad Javad Naderi
5614cf4758 doc: fix typos 2025-03-29 12:40:47 +03:30
Srigovind Nayak
6db0d84ab0 changelog: add changelog entry for issue-4868 2025-03-29 13:27:50 +05:30
Srigovind Nayak
88b599c4f3 mount: append repository ID to FS name of FUSE mount
* update review comments

mount: append repository ID to FS name of the FUSE mount
2025-03-29 13:22:10 +05:30
Gilbert Gilb's
eefff0d793 docs: fix unit for S3 restore timeout
"d" is not a valid unit.
2025-03-27 21:20:50 +01:00
Alexander Neumann
3d14e92905 Set development version for 0.18.0 2025-03-27 20:17:36 +01:00
Alexander Neumann
d401ad6c1e Add version for 0.18.0 2025-03-27 20:16:56 +01:00
Alexander Neumann
ab024e6a51 Update manpages and auto-completion 2025-03-27 20:16:56 +01:00
Alexander Neumann
0e5f41c842 Generate CHANGELOG.md for 0.18.0 2025-03-27 20:16:25 +01:00
Alexander Neumann
321ac6c1c9 Prepare changelog for 0.18.0 2025-03-27 20:16:25 +01:00
Michael Eischer
94b1af580b Merge pull request #5316 from MichaelEischer/fix-docs
docs: SLSA is only setup for GHCR
2025-03-26 19:18:15 +01:00
Michael Eischer
cc6fbbe6ad Merge pull request #5315 from MichaelEischer/proper-feature-flag-deprecation
Readd feature flags removed too soon
2025-03-26 19:17:17 +01:00
Michael Eischer
3f70485671 docs: SLSA is only setup for GHCR 2025-03-26 18:46:52 +01:00
Michael Eischer
d4772aa469 readd feature flags removed too soon 2025-03-26 18:38:30 +01:00
Michael Eischer
13cb90b83a Merge pull request #5295 from MichaelEischer/randomize-pack-order
Randomize blob to pack file assignment
2025-03-25 18:13:49 +01:00
Michael Eischer
823cc3d93a Polish changelogs (#5308)
* polish changelogs

* Additional changelog polishing

* fix test failure

* Correct changelog for recover command

---------

Co-authored-by: Leo R. Lundgren <leo@finalresort.org>
2025-03-25 18:12:51 +01:00
Michael Eischer
9eee32131a Merge pull request #5307 from Martin2112/dial_tls
Replace deprecated DialTLS with DialTLSContext.
2025-03-25 18:12:10 +01:00
Michael Eischer
5e519a25f7 tweak changelog
Co-authored-by: rawtaz <rawtaz@users.noreply.github.com>
2025-03-24 19:44:13 +01:00
Michael Eischer
c4eb2be31f Merge pull request #5304 from restic/disable-gs-grpc-api
backend/gs: disable GRPC API to reduce binary size bloat
2025-03-24 19:03:09 +01:00
Michael Eischer
0b22d8dc64 Merge pull request #5306 from MichaelEischer/sftp-better-errors
sftp: improve error messages
2025-03-24 18:43:47 +01:00
Michael Eischer
2b65ef5710 backend/gs: disable GRPC API to reduce binary size bloat
Since cloud.google.com/go/storage v1.44.0 the GRPC API is enabled by
default. However, this causes the restic binary size to explode by 20MB.
So just disable it again.
2025-03-24 18:41:45 +01:00
Michael Eischer
ccb92f5bf0 repository/packer: add unit test for Merge method 2025-03-24 17:04:02 +01:00
Michael Eischer
37aa4f824f add changelog and update threat model 2025-03-24 17:03:43 +01:00
Martin Smith
47b048f437 Rename param as it looks like context isn't used. 2025-03-24 15:01:47 +00:00
Martin Smith
cd7f384d77 Replace deprecated DialTLS with DialTLSContext. 2025-03-24 14:05:13 +00:00
Michael Eischer
9d58a27428 Merge pull request #5298 from Martin2112/lock_by_value
Fix lock pass by value and handle error from Release().
2025-03-24 14:42:38 +01:00
Michael Eischer
9aad8e9ea5 Merge pull request #5299 from Martin2112/go_cleanup
A few more small cleanups that should not change behaviour.
2025-03-24 13:59:39 +01:00
Michael Eischer
3adf7d4efb backend/sftp: wrap further errors 2025-03-24 12:45:15 +01:00
Michael Eischer
66ec735ac2 backend/sftp: include file path in error messages 2025-03-24 12:44:42 +01:00
Michael Eischer
63a71f70e3 backend/sftp: ensure all errors are wrapped with the method name 2025-03-24 12:32:52 +01:00
Michael Eischer
e3ddc8a463 Merge pull request #5305 from restic/dependabot/go_modules/github.com/golang-jwt/jwt/v5-5.2.2
build(deps): bump github.com/golang-jwt/jwt/v5 from 5.2.1 to 5.2.2
2025-03-24 11:23:42 +01:00
Michael Eischer
66a8e897a9 Merge pull request #5300 from MichaelEischer/fix-output-race
ui/termstatus: fix race condition in StdioWrapper
2025-03-24 11:17:14 +01:00
Michael Eischer
ffd63f893a Merge pull request #5296 from MichaelEischer/reindex-before-recover
recover: reindex before reassembling snapshot
2025-03-24 11:16:38 +01:00
Michael Eischer
ec19d67512 ui/termstatus: fix race condition in StdioWrapper 2025-03-24 11:07:15 +01:00
dependabot[bot]
ef18feaeeb build(deps): bump github.com/golang-jwt/jwt/v5 from 5.2.1 to 5.2.2
Bumps [github.com/golang-jwt/jwt/v5](https://github.com/golang-jwt/jwt) from 5.2.1 to 5.2.2.
- [Release notes](https://github.com/golang-jwt/jwt/releases)
- [Changelog](https://github.com/golang-jwt/jwt/blob/main/VERSION_HISTORY.md)
- [Commits](https://github.com/golang-jwt/jwt/compare/v5.2.1...v5.2.2)

---
updated-dependencies:
- dependency-name: github.com/golang-jwt/jwt/v5
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-24 10:00:55 +00:00
Michael Eischer
171f303399 Merge pull request #5302 from restic/update-dependencies
update all direct dependencies
2025-03-24 10:51:59 +01:00
Michael Eischer
dda652614e update all direct dependencies 2025-03-23 21:53:27 +01:00
Michael Eischer
784097a4f8 Merge pull request #5297 from MichaelEischer/fix-overwrite-behavior-help
restore: fix help message on invalid OverwriteBehavior
2025-03-23 20:06:10 +01:00
Michael Eischer
f5989964ed restore: fix redundant default value for --overwrite option 2025-03-23 19:54:22 +01:00
Martin Smith
cfa3c5884d Fix lock pass by value and handle error from Release(). 2025-03-23 18:53:21 +00:00
Michael Eischer
d60acc5697 restore: fix help message on invalid OverwriteBehavior 2025-03-23 19:51:37 +01:00
Michael Eischer
2240d1801c add changelog recover enhancement 2025-03-23 18:17:33 +01:00
Michael Eischer
99fdb00d39 recover: add minimal integration test 2025-03-23 18:07:41 +01:00
Michael Eischer
2409078d55 recover: automatically run repair index before recovering snapshots 2025-03-23 17:55:33 +01:00
Michael Eischer
0b6c355678 recover: refactor to use termstatus 2025-03-23 17:46:49 +01:00
Michael Eischer
f7f48b3026 ui/progress: extend Printer interface with print to stdout method 2025-03-23 17:46:04 +01:00
Michael Eischer
1221453d08 Merge pull request #5264 from restic/dependabot/go_modules/github.com/minio/minio-go/v7-7.0.87
build(deps): bump github.com/minio/minio-go/v7 from 7.0.77 to 7.0.87
2025-03-23 17:38:30 +01:00
Michael Eischer
4b975bda37 backend/s3: increase timeout for test initialization 2025-03-23 17:28:08 +01:00
Michael Eischer
f8b481fd9b backend/s3: resolve credential retrieval deprecation 2025-03-23 17:28:08 +01:00
dependabot[bot]
f88d5adaa2 build(deps): bump github.com/minio/minio-go/v7 from 7.0.77 to 7.0.87
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.77 to 7.0.87.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.77...v7.0.87)

---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-23 17:28:08 +01:00
Michael Eischer
89909d41aa Merge pull request #5292 from restic/go-1.23
Add Go 1.24 and drop Go 1.22 support
2025-03-23 17:26:31 +01:00
Michael Eischer
06535e62c1 CI: increase timeout to work around slow cloud backend 2025-03-23 16:36:31 +01:00
Michael Eischer
c99c76ada8 backend/test: increase parallelism to run all TestBackend tests in parallel 2025-03-23 16:36:31 +01:00
Michael Eischer
4350b95d27 backend/test: fix delayedRemoval timeout handling
The timeout for all blobs starts to run after the delete calls have been
issue. Thus, use the same start time for all blobs instead of individual
timeouts.
2025-03-23 16:36:31 +01:00
Michael Eischer
2e58561ad6 backend/test: remove redundant test call to the backend 2025-03-23 16:36:31 +01:00
Michael Eischer
17b585f7c7 backend/test: partially parallelize delayedRemove 2025-03-23 16:36:31 +01:00
Michael Eischer
4640b3c41a backend/test: parallelize slow tests 2025-03-23 16:36:31 +01:00
Michael Eischer
c36970074d CI: bump golangci version 2025-03-23 16:36:31 +01:00
Michael Eischer
15e90b7a4c test go 1.24 and drop support for 1.22 2025-03-23 16:36:31 +01:00
Michael Eischer
8d2d50d095 repository: merge small pack files before flushing
This prevents chunk size leaks when a backup only consists of a small
file which is split in two parts, which end up in two individual pack
files.
2025-03-23 12:29:16 +01:00
Michael Eischer
62453f9356 repository: randomly distribute blobs over two pack files 2025-03-23 12:29:16 +01:00
Martin Smith
6caad10840 Remove extra brackets. 2025-03-23 10:11:43 +00:00
Martin Smith
4420fde378 Remove deprecated HTTP option that is now the default. 2025-03-23 10:10:54 +00:00
Martin Smith
a389977bd7 Remove redudnant error check, handled above. 2025-03-23 10:05:13 +00:00
Martin Smith
6e45c51509 Fix name including package name and variable shadowing package. 2025-03-23 10:01:19 +00:00
Martin Smith
5e7333d28d Unify repository receiver name. 2025-03-23 09:57:59 +00:00
Michael Eischer
c617364d15 Merge pull request #5262 from Martin2112/go_cleanup
A set of mostly automated Go cleanups for the code
2025-03-22 23:55:28 +01:00
Martin Smith
e2ccb18e22 Fix lint for missing const, after fixing godoc for the outer type. 2025-03-22 18:42:12 +00:00
Martin Smith
d2c5241961 Revert a fix that broke compile of sd_windows.go. 2025-03-22 18:27:09 +00:00
Martin Smith
f238f81ba6 Renames to fix clashes with reserved words. 2025-03-22 18:20:30 +00:00
Martin Smith
3788605127 Rename unused parameters to '_'. 2025-03-22 18:20:30 +00:00
Martin Smith
29b4680873 Remove unnecessary type conversions, second set. 2025-03-22 18:20:30 +00:00
Martin Smith
092899df8b Remove unnecessary type conversions. 2025-03-22 18:20:30 +00:00
Martin Smith
2099ec1cd6 Remove import aliases that match package name. 2025-03-22 18:20:30 +00:00
Martin Smith
1daf5317f8 Fix import ordering. 2025-03-22 18:20:30 +00:00
Martin Smith
db8daeb192 Fix godoc comments. 2025-03-22 18:20:30 +00:00
Michael Eischer
ef692991a4 Merge pull request #5183 from wplapper/cmd_prune
restic prune: selection of packs to repack based on size
2025-03-22 15:43:32 +01:00
Michael Eischer
062cfc549d prune: fix not working option 2025-03-22 15:34:40 +01:00
Michael Eischer
3e58b15ace prune: allow --repack-smaller-than independently of --repack-small 2025-03-22 15:08:36 +01:00
Michael Eischer
69249372bf Merge pull request #5249 from MichaelEischer/fix-repair-index
Prevent creation of oversized indexes and automatically rewrite them.
2025-03-22 14:29:39 +01:00
Michael Eischer
445477312c Merge pull request #5251 from MichaelEischer/rclone-retries
Retry temporary rclone backend errors
2025-03-22 14:24:47 +01:00
Winfried Plappert
cc4712f8e9 add a test to cmd_ls_integration_test.go: test rest ls --json (#5255)
* cmd_ls: one more test: ls --json to check the JSON lines

validate that the individual JSON lines are valid JSON statements.
Check for snap ID and the path names in the backup.
2025-03-22 14:20:19 +01:00
Michael Eischer
c405e9e748 Merge pull request #5270 from prajwalbharadwajbm/master
build: improve GoVersion comparison logic
2025-03-22 12:05:40 +01:00
Michael Eischer
5f40e4b7c5 Merge pull request #5274 from luzpaz/typos
doc: fix various typos
2025-03-22 11:56:52 +01:00
Michael Eischer
0b0987233f Merge pull request #5263 from restic/dependabot/github_actions/slsa-framework/slsa-github-generator-2.1.0
build(deps): bump slsa-framework/slsa-github-generator from 2.0.0 to 2.1.0
2025-03-22 11:53:09 +01:00
Michael Eischer
d66e9cfff5 Merge pull request #5267 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/azidentity-1.8.2
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azidentity from 1.8.1 to 1.8.2
2025-03-22 11:47:36 +01:00
Michael Eischer
e40996f0f1 Merge pull request #5265 from restic/dependabot/go_modules/github.com/klauspost/compress-1.18.0
build(deps): bump github.com/klauspost/compress from 1.17.11 to 1.18.0
2025-03-22 11:46:18 +01:00
dependabot[bot]
818cb386a5 build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azidentity
Bumps [github.com/Azure/azure-sdk-for-go/sdk/azidentity](https://github.com/Azure/azure-sdk-for-go) from 1.8.1 to 1.8.2.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azidentity/v1.8.1...sdk/azidentity/v1.8.2)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azidentity
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-22 10:39:29 +00:00
Michael Eischer
9f724f7dc5 Merge pull request #5268 from restic/dependabot/go_modules/golang.org/x/sync-0.11.0
build(deps): bump golang.org/x/sync from 0.10.0 to 0.11.0
2025-03-22 11:38:35 +01:00
Winfried Plappert
3f42c0ad96 restic prune changelog/unreleased - corrext typo
changed option from `--small-pack-size` to `--repack-smaller-than`
2025-03-15 12:51:29 +00:00
Luz Paz
794341a494 doc: fix various typos
Found via `codespell -q 3 -L atleast,iinclude,ist,programm,reenable,ser,uptodate`
2025-03-05 20:47:08 -05:00
Prajwal Bharadwaj BM
74b76ca0df build: improve GoVersion comparison logic
Refactor AtLeast method to correctly handle version comparisons by:
- Checking major version first
- Handling minor version comparisons
- Ensuring correct comparison of patch versions
2025-03-01 12:51:11 +05:30
dependabot[bot]
3b21c7da3d build(deps): bump golang.org/x/sync from 0.10.0 to 0.11.0
Bumps [golang.org/x/sync](https://github.com/golang/sync) from 0.10.0 to 0.11.0.
- [Commits](https://github.com/golang/sync/compare/v0.10.0...v0.11.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sync
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-01 01:53:26 +00:00
dependabot[bot]
f838bf1056 build(deps): bump github.com/klauspost/compress from 1.17.11 to 1.18.0
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.17.11 to 1.18.0.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml)
- [Commits](https://github.com/klauspost/compress/compare/v1.17.11...v1.18.0)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-01 01:53:16 +00:00
dependabot[bot]
664971eb1d build(deps): bump slsa-framework/slsa-github-generator
Bumps [slsa-framework/slsa-github-generator](https://github.com/slsa-framework/slsa-github-generator) from 2.0.0 to 2.1.0.
- [Release notes](https://github.com/slsa-framework/slsa-github-generator/releases)
- [Changelog](https://github.com/slsa-framework/slsa-github-generator/blob/main/CHANGELOG.md)
- [Commits](https://github.com/slsa-framework/slsa-github-generator/compare/v2.0.0...v2.1.0)

---
updated-dependencies:
- dependency-name: slsa-framework/slsa-github-generator
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-01 01:11:52 +00:00
Michael Eischer
de9a040d27 Merge pull request #5256 from abaumg/fix/links-to-backblaze-documentation
Fix links to Backblaze documentation
2025-02-23 21:06:28 +01:00
Andreas Baumgartner
89826ef5ce doc: fix links to Backblaze documentation 2025-02-21 23:43:02 +01:00
Winfried Plappert
a2a1309fd9 prune: make small pack size configureable for prune all changes together
cmd_prune.go: added option `--repack-smaller-than`
prune.go: added field `SmallPackBytes` to `PruneOptions`, including checking and processing
prune_test.go: added test `TestPruneSmall`
doc/060_forget.rst: added description of enhancement
changelog/unreleased/issue-5109: description of enhancement
2025-02-18 16:54:44 +00:00
Michael Eischer
6309952a82 add changelog for rclone retries 2025-02-17 21:33:35 +01:00
Michael Eischer
5e7ce45ede retry: test error retries for flaky backends 2025-02-16 22:56:43 +01:00
Michael Eischer
cb8575f001 rclone: remove redundant Warmup methods 2025-02-16 22:41:22 +01:00
Michael Eischer
8d1185b3b8 retry/rclone: retry errors up to 5 times 2025-02-16 22:40:55 +01:00
Michael Eischer
c970e58739 backend: refactor backend Connections and HasAtomicReplace into Properties 2025-02-16 22:27:58 +01:00
Michael Eischer
5ddda7f5e9 Merge pull request #5242 from MichaelEischer/fix-read-stdin-msg
print password from stdin message only to terminal
2025-02-16 18:29:34 +01:00
Michael Eischer
8c12291f56 Merge pull request #5241 from MichaelEischer/cleanup-cli
Refactor CLI command initialization to use less global state
2025-02-16 18:28:48 +01:00
Michael Eischer
5190933561 Merge pull request #5240 from MichaelEischer/better-json-docs
Improve JSON output type documentation
2025-02-16 18:28:29 +01:00
Michael Eischer
00e69f242e docs: fix datatypes 2025-02-16 18:17:22 +01:00
Michael Eischer
00628e952f add changelog for oversized indexes 2025-02-16 17:58:36 +01:00
Michael Eischer
39e63ee4e3 index: add tests for oversized index handling 2025-02-16 17:42:00 +01:00
Michael Eischer
3b8d15d651 index: rewrite oversized indexes 2025-02-16 17:03:14 +01:00
Michael Eischer
2fd8a3865c index: automatically write full indexes in StorePack 2025-02-16 16:39:38 +01:00
Michael Eischer
0c4e65228a refactor secondary options 2025-02-07 21:29:33 +01:00
Michael Eischer
120bd08c0d move globalOptions initialization into method 2025-02-07 21:29:33 +01:00
Michael Eischer
d378a171c8 cleanup backend initialization 2025-02-07 21:29:33 +01:00
Michael Eischer
c752867f0a fix linter errors 2025-02-07 21:29:33 +01:00
Michael Eischer
412d6d9ec5 Create root command via function 2025-02-07 21:29:33 +01:00
Michael Eischer
5497217018 print password from stdin message only to terminal 2025-02-07 20:54:18 +01:00
Michael Eischer
aa9cdf93cf refactor persistent options to be applied via functions 2025-02-07 19:03:46 +01:00
Michael Eischer
aacd6a47e3 refactor to use constructor functions to create cobra commands
This allows getting rid of the global options variables
2025-02-07 18:56:45 +01:00
Michael Eischer
dc9b6378f3 move cli flags into AddFlags on option structs 2025-02-06 22:10:41 +01:00
Michael Eischer
4e58902de6 doc: fix broken links 2025-02-06 20:59:36 +01:00
Michael Eischer
39823c5f6c doc: deprecate short_id and add some missing fields 2025-02-06 20:53:01 +01:00
Michael Eischer
421842f41f doc: add datatypes to JSON outputs 2025-02-06 20:10:42 +01:00
Michael Eischer
59b7007534 doc: reformat scripting tables 2025-02-06 19:46:41 +01:00
Michael Eischer
da47967316 Merge pull request #5194 from darkdragon-001/json-check
Json check
2025-02-05 22:15:10 +01:00
Dark Dragon
49a411f7ac Print JSON summary in all error cases 2025-02-05 22:08:06 +01:00
Dark Dragon
7cc1aa0cd4 Add check summary 2025-02-05 22:08:06 +01:00
Dark Dragon
a58a8f2ce0 Add JSON output to check command 2025-02-05 22:08:05 +01:00
Dark Dragon
79d435efb1 Use printer.NewCounter() instead of newTerminalProgressMax()
where possible (max is unknown).
2025-02-05 22:07:47 +01:00
Michael Eischer
9cdf91b406 Merge pull request #5235 from MichaelEischer/refactor-ls-sorting
Refactor ls sorting
2025-02-05 20:44:08 +01:00
Winfried Plappert
4104a8e6a5 Issue: 4942: cmd_rewrite: add snapshot summary data to an existing snapshot. (#5185)
Co-authored-by: Michael Eischer <michael.eischer@fau.de>
2025-02-05 20:40:20 +01:00
Michael Eischer
6cc06e0812 ls: add missing error handling 2025-02-03 22:15:59 +01:00
Michael Eischer
c32613a624 ls: extract comparator 2025-02-03 22:15:59 +01:00
Michael Eischer
1807627dda ls: refactor sorting into sortedPrinter struct 2025-02-03 22:15:59 +01:00
Michael Eischer
993eb112cd ls: deduplicate sorting test 2025-02-03 22:15:54 +01:00
Michael Eischer
36d8916354 ls: use numeric based enum for SortMode 2025-02-03 22:11:46 +01:00
Winfried Plappert
060a44202f ls: sort output by size, atime, ctime, mtime, time(=mtime), extension (#5182)
Enhancement: create ability to sort output of restic ls -l by
name, size, atime, ctime, mtime, time(=mtime), X(=extension), extension

---------

Co-authored-by: Michael Eischer <michael.eischer@fau.de>
2025-02-03 22:07:04 +01:00
Michael Eischer
d79681b987 Merge pull request #5223 from restic/dependabot/go_modules/google.golang.org/api-0.219.0
build(deps): bump google.golang.org/api from 0.204.0 to 0.219.0
2025-02-03 21:32:15 +01:00
dependabot[bot]
90e2c419e4 build(deps): bump google.golang.org/api from 0.204.0 to 0.219.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.204.0 to 0.219.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.204.0...v0.219.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-02-03 20:16:08 +00:00
Michael Eischer
7ab5bb6df4 Merge pull request #5232 from MichaelEischer/bump-go-version
Bump minimum go version to 1.22
2025-02-03 21:14:57 +01:00
Michael Eischer
efd2ec086f Merge pull request #5179 from zmanda/fix-gh-5140-forget-reports-incorrect-number-of-files-deleted
forget: report count of deleted files correctly
2025-02-02 20:14:15 +01:00
Srigovind Nayak
8d970e36cf tests: add unit test to check the progress counter for forget/prune 2025-02-02 20:18:56 +05:30
Srigovind Nayak
58f58a995d parallel: increment progress bar before report function which may absorb the error
* sometimes, the report function may absorb the error and return nil, in those cases the bar.Add(1) method would execute even if the file deletion had failed
2025-02-02 19:45:36 +05:30
Michael Eischer
d71ddfb89b bump minimum go version to 1.22 2025-02-02 15:05:47 +01:00
Gilbert Gilb's
536ebefff4 feat(backends/s3): add warmup support before repacks and restores (#5173)
* feat(backends/s3): add warmup support before repacks and restores

This commit introduces basic support for transitioning pack files stored
in cold storage to hot storage on S3 and S3-compatible providers.

To prevent unexpected behavior for existing users, the feature is gated
behind new flags:

- `s3.enable-restore`: opt-in flag (defaults to false)
- `s3.restore-days`: number of days for the restored objects to remain
  in hot storage (defaults to `7`)
- `s3.restore-timeout`: maximum time to wait for a single restoration
  (default to `1 day`)
- `s3.restore-tier`: retrieval tier at which the restore will be
  processed. (default to `Standard`)

As restoration times can be lengthy, this implementation preemptively
restores selected packs to prevent incessant restore-delays during
downloads. This is slightly sub-optimal as we could process packs
out-of-order (as soon as they're transitioned), but this would really
add too much complexity for a marginal gain in speed.

To maintain simplicity and prevent resources exhautions with lots of
packs, no new concurrency mechanisms or goroutines were added. This just
hooks gracefully into the existing routines.

**Limitations:**

- Tests against the backend were not written due to the lack of cold
  storage class support in MinIO. Testing was done manually on
  Scaleway's S3-compatible object storage. If necessary, we could
  explore testing with LocalStack or mocks, though this requires further
  discussion.
- Currently, this feature only warms up before restores and repacks
  (prune/copy), as those are the two main use-cases I came across.
  Support for other commands may be added in future iterations, as long
  as affected packs can be calculated in advance.
- The feature is gated behind a new alpha `s3-restore` feature flag to
  make it explicit that the feature is still wet behind the ears.
- There is no explicit user notification for ongoing pack restorations.
  While I think it is not necessary because of the opt-in flag, showing
  some notice may improve usability (but would probably require major
  refactoring in the progress bar which I didn't want to start). Another
  possibility would be to add a flag to send restores requests and fail
  early.

See https://github.com/restic/restic/issues/3202

* ui: warn user when files are warming up from cold storage

* refactor: remove the PacksWarmer struct

It's easier to handle multiple handles in the backend directly, and it
may open the door to reducing the number of requests made to the backend
in the future.
2025-02-01 18:26:27 +00:00
Michael Eischer
9566e2db4a Merge pull request #5222 from MichaelEischer/docs-edit-links
doc: add edit on github link
2025-02-01 18:37:05 +01:00
Michael Eischer
7829728182 Merge pull request #5225 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/storage/azblob-1.6.0
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from 1.5.0 to 1.6.0
2025-02-01 17:03:33 +01:00
Michael Eischer
72b343fe5a Merge pull request #5228 from greatroar/cleanup
ui/termstatus: Remove unused bytes.Buffer
2025-02-01 16:58:54 +01:00
Michael Eischer
9c8c59c889 Merge pull request #5226 from restic/dependabot/go_modules/github.com/spf13/pflag-1.0.6
build(deps): bump github.com/spf13/pflag from 1.0.5 to 1.0.6
2025-02-01 16:57:10 +01:00
dependabot[bot]
c4d988faf8 build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
Bumps [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) from 1.5.0 to 1.6.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.5.0...sdk/azcore/v1.6.0)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-02-01 15:56:10 +00:00
Michael Eischer
080c8de1a9 Merge pull request #5227 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/azidentity-1.8.1
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azidentity from 1.8.0 to 1.8.1
2025-02-01 16:55:04 +01:00
greatroar
c1781e0abb ui/termstatus: Remove unused bytes.Buffer 2025-02-01 08:21:40 +01:00
dependabot[bot]
2b9113721c build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azidentity
Bumps [github.com/Azure/azure-sdk-for-go/sdk/azidentity](https://github.com/Azure/azure-sdk-for-go) from 1.8.0 to 1.8.1.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.8.0...sdk/azidentity/v1.8.1)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azidentity
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-02-01 01:43:20 +00:00
dependabot[bot]
afe4fcc0d9 build(deps): bump github.com/spf13/pflag from 1.0.5 to 1.0.6
Bumps [github.com/spf13/pflag](https://github.com/spf13/pflag) from 1.0.5 to 1.0.6.
- [Release notes](https://github.com/spf13/pflag/releases)
- [Commits](https://github.com/spf13/pflag/compare/v1.0.5...v1.0.6)

---
updated-dependencies:
- dependency-name: github.com/spf13/pflag
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-02-01 01:43:14 +00:00
Michael Eischer
c2e404a0ee doc: add edit on github link 2025-01-31 18:53:03 +01:00
Winfried Plappert
c4be05dbc2 Issue 4433: Ability to define sort order for output of find command (#5184)
The old sorting behaviour was to sort snapshots from oldest to newest.
The new sorting order is from newest to oldest. If one wants to revert to the
old behaviour, use the option --reverse.

---------

Co-authored-by: Michael Eischer <michael.eischer@fau.de>
2025-01-29 20:44:16 +00:00
rawtaz
d0d887138c Merge pull request #5219 from MichaelEischer/ci-silence-shadow-builtin
CI: bump golangci-lint and silence warnings about shadowed builtins
2025-01-29 01:30:40 +01:00
Michael Eischer
8eaa4b6602 CI: bump golangci-lint to v1.63.4 2025-01-28 19:55:45 +01:00
Michael Eischer
e77681f2cd remove unnecessary min function 2025-01-28 19:52:22 +01:00
Michael Eischer
a63500663a CI: disable shadow builtin rule
Removing the shadowing cases leads to weird workarounds but doesn't help
much with code clarity.
2025-01-28 19:51:14 +01:00
Michael Eischer
fde64133df Merge pull request #5212 from MichaelEischer/prune-fix-unused-size-duplicates
prune: correctly account for duplicates in max-unused check
2025-01-26 22:07:47 +01:00
Snshadow
6301250d83 fix: Windows VSS Event ID 8194 (#5170) 2025-01-26 15:25:38 +00:00
Michael Eischer
9331461a13 prune: correctly account for duplicates in max-unused check
The size comparison for `--max-unused` only accounted for unused but not
for duplicate data. For repositories with a large amount of duplicates
this can result in a situation where no data gets pruned even though
the amount of unused data is much higher than specified.
2025-01-19 17:47:49 +01:00
rawtaz
ed3922ac82 Merge pull request #5211 from MichaelEischer/bump-dockerfile-go-version
Bump dockerfile to go 1.23
2025-01-19 08:32:42 +01:00
Michael Eischer
8b63e1cd72 Merge pull request #5129 from tesshuflower/5089_exclude_xattrs_on_restore
Allow excluding xattrs at restore time
2025-01-18 23:15:11 +01:00
Michael Eischer
5e8654c71d restore: fix xattr filter test on windows 2025-01-18 23:07:39 +01:00
Michael Eischer
d5a94583ed bump dockerfile to go 1.23 2025-01-18 18:27:43 +01:00
Srigovind Nayak
115ecb3c92 tag: output the original ID and new snapshotID (#5144)
* tag: output the original ID and new snapshotID

tag: print changed snapshot information immediately

* print changed snapshot immediately after it has been saved
* add message type to the changedSnapshot
* add a summary type which will share the JSON output of the numer of changed snapshots
* updated verbosity of the changed snapshot in text mode to only work when verbosity > 2
* also use the terminal status printer for a standard handling for stdout messages
2025-01-14 18:57:47 +01:00
Michael Eischer
e6f9cfb8c8 Merge pull request #5168 from MichaelEischer/restrict-repository-unpacked
repository: restrict SaveUnpacked and RemoveUnpacked
2025-01-13 22:49:44 +01:00
Michael Eischer
b7ff8ea9cd repository: expose cache via method 2025-01-13 22:40:18 +01:00
Michael Eischer
99e105eeb6 repository: restrict SaveUnpacked and RemoveUnpacked
Those methods now only allow modifying snapshots. Internal data types
used by the repository are now read-only. The repository-internal code
can bypass the restrictions by wrapping the repository in an
`internalRepository` type.

The restriction itself is implemented by using a new datatype
WriteableFileType in the SaveUnpacked and RemoveUnpacked methods. This
statically ensures that code cannot bypass the access restrictions.

The test changes are somewhat noisy as some of them modify repository
internals and therefore require some way to bypass the access
restrictions. This works by capturing an `internalRepository` or
`Backend` when creating the Repository using a test helper function.
2025-01-13 22:39:57 +01:00
Albin Vass
5bf0204caf Do not skip root tree when searching for trees (#5153)
This fixes an issue where restic cannot find the tree when trying to find the
tree id of a snapshot.

---------

Co-authored-by: Albin Vass <albinvass@gmail.com>
Co-authored-by: Michael Eischer <michael.eischer@fau.de>
2025-01-13 21:08:38 +00:00
Michael Eischer
14d02df8bb Merge pull request #5162 from MichaelEischer/promote-feature-gates
Stabilize `explicit-s3-anonymous-auth` and `safe-forget-keep-tags` feature flags
2025-01-13 22:03:06 +01:00
Michael Eischer
bd4ce8aac1 Merge pull request #4990 from m-wild/exclude-cloud-files
backup: allow excluding online-only cloud files
2025-01-13 21:24:53 +01:00
Michael Wildman
da71e77b28 backup: allow excluding online-only cloud files 2025-01-13 21:11:23 +01:00
Michael Eischer
27189e03ee Merge pull request #4999 from konidev20/fix-gh-4983-slsa-provenance-for-ghcr-container-images
docker: sign container images pushed to GHCR with GitHub OIDC tokens
2025-01-12 22:38:33 +01:00
Michael Eischer
4e1eeeb721 Merge pull request #5207 from restic/dependabot/go_modules/golang.org/x/net-0.34.0
build(deps): bump golang.org/x/net from 0.30.0 to 0.34.0
2025-01-11 19:41:07 +01:00
Michael Eischer
3b37983a60 Merge pull request #5196 from restic/dependabot/go_modules/github.com/klauspost/compress-1.17.11
build(deps): bump github.com/klauspost/compress from 1.17.9 to 1.17.11
2025-01-11 19:39:25 +01:00
dependabot[bot]
99646fdf62 build(deps): bump golang.org/x/net from 0.30.0 to 0.34.0
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.30.0 to 0.34.0.
- [Commits](https://github.com/golang/net/compare/v0.30.0...v0.34.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-01-11 18:31:06 +00:00
Michael Eischer
0331891545 Merge pull request #5202 from knbr13/remove-duplicate-imports
remove duplicate imports
2025-01-11 19:28:07 +01:00
Michael Eischer
2b45c004be Merge pull request #5200 from restic/dependabot/go_modules/golang.org/x/sys-0.28.0
build(deps): bump golang.org/x/sys from 0.27.0 to 0.28.0
2025-01-11 19:27:52 +01:00
Tesshu Flower
44cef25077 remove bad test xattr
Signed-off-by: Tesshu Flower <tflower@redhat.com>
2025-01-10 21:12:03 -05:00
Tesshu Flower
cd84fe0853 xattrs - restore all by default, doc/chglog update
Signed-off-by: Tesshu Flower <tflower@redhat.com>
2025-01-10 15:25:09 -05:00
Tesshu Flower
3ac697d03d linux default restore only user xattrs, doc update
* On Linux restore only user.* xattrs by default
* restore all for other OSs
* Update docs and changelog about the new restore
flags --exclude-xattr and --include-xattr

Signed-off-by: Tesshu Flower <tflower@redhat.com>
2025-01-10 15:13:45 -05:00
Tesshu Flower
24422e20a6 restore: xattr restore filter tests
Signed-off-by: Tesshu Flower <tflower@redhat.com>
2025-01-10 15:13:44 -05:00
Tesshu Flower
f457b16b23 update nodeRestoreExtendedAttributes() for win
- also other platforms
- move xattr include/exclude filter parsing into
  separate func

Signed-off-by: Tesshu Flower <tflower@redhat.com>
2025-01-10 15:13:44 -05:00
Tesshu Flower
af839f9548 restore: exclude/include xattrs
For: https://github.com/restic/restic/issues/5089

Signed-off-by: Tesshu Flower <tflower@redhat.com>
2025-01-10 15:13:40 -05:00
knbr13
bbb492ee65 remove duplicate imports 2025-01-05 13:53:20 +02:00
dependabot[bot]
01405f1e1b build(deps): bump golang.org/x/sys from 0.27.0 to 0.28.0
Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.27.0 to 0.28.0.
- [Commits](https://github.com/golang/sys/compare/v0.27.0...v0.28.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sys
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-01-01 01:54:17 +00:00
dependabot[bot]
caa59bb81b build(deps): bump github.com/klauspost/compress from 1.17.9 to 1.17.11
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.17.9 to 1.17.11.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml)
- [Commits](https://github.com/klauspost/compress/compare/v1.17.9...v1.17.11)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-01-01 01:54:05 +00:00
rawtaz
de3acd7937 Merge pull request #5180 from vmlemon/master
Enable xattr support, on NetBSD 10+
2024-12-23 20:05:06 +01:00
Tyson Key
9e85119d73 Update changelog, for issue 5174 (Enable xattr support, on NetBSD 10+) 2024-12-19 14:32:16 +00:00
Tyson Key
37969ae8e3 Enable xattr support, on NetBSD 10+ 2024-12-18 16:52:44 +00:00
Aneesh N
6808004ad1 Refactor extended attributes and security descriptor helpers to use go-winio (#5040)
* Refactor ea and sd helpers to use go-winio

Import go-winio and instead of copying the functions to encode/decode extended attributes and enable process privileges for security descriptors, call the functions defined in go-winio.
2024-12-09 21:48:38 +01:00
Srigovind Nayak
8d45a4b283 changelog: update to indicate change applies only for GHCR images. 2024-12-02 01:11:26 +05:30
Srigovind Nayak
4fb9aa4351 docker: fix typos and permissions for jobs 2024-12-02 00:16:19 +05:30
Srigovind Nayak
d422e75e08 docs: add instructions for verifying SLSA provenance of Docker images
docs: update the documentation
2024-12-02 00:14:39 +05:30
Srigovind Nayak
144221b430 docker: add SLSA provenance to .github workflow
* the id-token of the GitHub Actions workflow will be used for image signing
* replace branch-based tagging with SHA-based tagging since, branch names are mutable, SLSA provenance requires immutable tagging
* use official SLSA framework Github Reusable workflow

docker: fix incorrect registry name in image output step

* use REGISTRY environment variable instead of IMAGE_REGISTRY

docker: revert change to remove branch tag
2024-12-02 00:14:38 +05:30
Srigovind Nayak
d7d9af4c9f ui: restore --delete indicates number of deleted files (#5100)
* ui: restore --delete indicates number of deleted files

* adds new field `FilesDeleted` to the State struct, JSON and text progress updaters
* increment FilesDeleted count when ReportedDeletedFile

* ui: collect the files to be deleted, delete, then update the count post deletion

* docs: update scripting output fields for restore command

ui: report deleted directories and refactor function name to ReportDeletion
2024-12-01 15:29:11 +01:00
Michael Eischer
2f0049cd6c Merge pull request #5141 from richgrov/missing-azure-env-error
Return error if AZURE_ACCOUNT_NAME not set
2024-12-01 14:01:56 +01:00
Michael Eischer
72c02fa759 Merge pull request #5167 from restic/dependabot/go_modules/github.com/pkg/sftp-1.13.7
build(deps): bump github.com/pkg/sftp from 1.13.6 to 1.13.7
2024-12-01 13:14:03 +01:00
dependabot[bot]
770841f95d build(deps): bump github.com/pkg/sftp from 1.13.6 to 1.13.7
Bumps [github.com/pkg/sftp](https://github.com/pkg/sftp) from 1.13.6 to 1.13.7.
- [Release notes](https://github.com/pkg/sftp/releases)
- [Commits](https://github.com/pkg/sftp/compare/v1.13.6...v1.13.7)

---
updated-dependencies:
- dependency-name: github.com/pkg/sftp
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-12-01 12:02:01 +00:00
Michael Eischer
5e0a045481 Merge pull request #5163 from restic/dependabot/go_modules/golang.org/x/sys-0.27.0
build(deps): bump golang.org/x/sys from 0.26.0 to 0.27.0
2024-12-01 13:00:28 +01:00
Michael Eischer
3fecddafe8 Merge pull request #5165 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/storage/azblob-1.5.0
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from 1.4.0 to 1.5.0
2024-12-01 12:58:24 +01:00
dependabot[bot]
40987a5f80 build(deps): bump golang.org/x/sys from 0.26.0 to 0.27.0
Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.26.0 to 0.27.0.
- [Commits](https://github.com/golang/sys/compare/v0.26.0...v0.27.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sys
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-12-01 11:48:44 +00:00
Michael Eischer
875976f4a8 Merge pull request #5166 from restic/dependabot/go_modules/golang.org/x/text-0.20.0
build(deps): bump golang.org/x/text from 0.19.0 to 0.20.0
2024-12-01 12:47:55 +01:00
dependabot[bot]
2dc00cfd36 build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
Bumps [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) from 1.4.0 to 1.5.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.4.0...sdk/azcore/v1.5.0)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-12-01 11:45:54 +00:00
Michael Eischer
45d2b4cd3c Merge pull request #5161 from restic/bump-backblaze-library
bump backblaze/blazer to v0.7.1
2024-12-01 12:45:00 +01:00
dependabot[bot]
a4d776ec8f build(deps): bump golang.org/x/text from 0.19.0 to 0.20.0
Bumps [golang.org/x/text](https://github.com/golang/text) from 0.19.0 to 0.20.0.
- [Release notes](https://github.com/golang/text/releases)
- [Commits](https://github.com/golang/text/compare/v0.19.0...v0.20.0)

---
updated-dependencies:
- dependency-name: golang.org/x/text
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-12-01 01:41:13 +00:00
Michael Eischer
098db935f7 Stabilize explicit-s3-anonymous-auth and safe-forget-keep-tags flags
The features can no longer be disabled.
2024-11-30 21:22:51 +01:00
Michael Eischer
ead57ec501 bump backblaze/blazer to v0.7.1 2024-11-30 21:17:06 +01:00
Michael Eischer
8f9d755b44 Merge pull request #5158 from dnnr/clarify-max-repack-size
Reword description of --max-repack-size for clarity
2024-11-30 19:19:01 +01:00
Daniel Danner
1062546563 Mention size 2024-11-30 17:52:29 +01:00
Michael Eischer
0bf8af7188 Merge pull request #5138 from vmlemon/issue-5131
Implement basic DragonFlyBSD support
2024-11-30 17:32:59 +01:00
Michael Eischer
9a674ecc34 Merge pull request #5146 from MichaelEischer/inline-extended-stat
fs: Inline ExtendedFileInfo
2024-11-30 17:23:34 +01:00
Michael Eischer
9a99141a5f fs: remove os.FileInfo from fs.ExtendedFileInfo
Only the `Sys()` value from os.FileInfo is kept as field `sys` to
support Windows. The os.FileInfo removal ensures that for values like
`ModTime` that existed in both data structures there's no more confusion
which value is actually used.
2024-11-30 17:07:36 +01:00
Michael Eischer
847b2efba2 archiver: remove fs parameter from fileChanged function 2024-11-30 16:19:16 +01:00
Michael Eischer
641390103d fs: inline ExtendedStat 2024-11-30 16:19:16 +01:00
Michael Eischer
806fa534ce Merge pull request #5145 from MichaelEischer/ignore-disappeared-files
backup: Ignore disappeared files
2024-11-30 16:15:31 +01:00
Michael Eischer
5df6bf80b1 fs: retry vss creation on VSS_E_SNAPSHOT_SET_IN_PROGRESS error
Depending on the change packages, the VSS tests from ./cmd/restic and
the fs package may overlap in time. This causes the snapshot creation to
fail. Add retries in that case.
2024-11-30 16:07:18 +01:00
Michael Eischer
dc89aad722 build dragonflybsd binaries 2024-11-30 15:47:39 +01:00
Tyson Key
3c0ceda536 Add basic support for DragonFlyBSD 2024-11-30 15:42:15 +01:00
Michael Eischer
c5fb46da53 archiver: ignore files removed in the meantime 2024-11-30 15:30:42 +01:00
Michael Eischer
8642049532 Merge pull request #5143 from MichaelEischer/fs-handle-interface
fs: rework FS interface to be handle based
2024-11-30 15:29:31 +01:00
Michael Eischer
8644bb145b Merge pull request #5134 from MichaelEischer/better-time-restore-error
restore: improve error if timestamp fails to restore
2024-11-30 13:09:33 +01:00
Daniel Danner
0997f26461 Reword description --max-repack-size for clarity 2024-11-29 23:29:43 +01:00
Michael Eischer
a5c49e5340 Merge pull request #5142 from MichaelEischer/fix-not-ordered-error-message
restic: add missing space in error message
2024-11-29 22:48:16 +01:00
Michael Eischer
b51bf0c0c4 fs: test File implementation of Local FS 2024-11-16 16:09:17 +01:00
Michael Eischer
6cb19e0190 archiver: fix file type change test
The test did not test the case that the type of a file changed
unexpectedly.
2024-11-16 16:09:17 +01:00
Michael Eischer
d7f4b9db60 fs: deduplicate placeholders for generic and xattrs 2024-11-16 16:09:17 +01:00
Michael Eischer
087f95a298 fs: make generic and extended attrs independent of each other 2024-11-16 15:38:56 +01:00
Michael Eischer
6084848e5a fs: fix O_NOFOLLOW for metadata handles on Windows 2024-11-16 15:38:56 +01:00
Michael Eischer
48dbefc37e fs / archiver: convert to handle based interface
The actual implementation still relies on file paths, but with the
abstraction layer in place, an FS implementation can ensure atomic file
accesses in the future.
2024-11-16 12:56:23 +01:00
Michael Eischer
2f2ce9add2 fs: remove Stat from FS interface 2024-11-16 12:56:23 +01:00
Michael Eischer
623ba92b98 fs: drop unused permission parameter from OpenFile 2024-11-16 12:56:23 +01:00
Michael Eischer
b402e8a6fc fs: stricter enforcement to only call readdir on a directory
Use O_DIRECTORY to prevent opening any other than a directory in
readdirnames.
2024-11-16 12:56:23 +01:00
Richard Grover
548fa07577 Add changelog info 2024-11-15 14:46:34 -07:00
Michael Eischer
f8031561f2 archiver: deduplicate error filtering 2024-11-15 17:58:06 +01:00
Michael Eischer
49ef3ebec3 restic: add missing space in error message 2024-11-15 17:52:09 +01:00
Richard Grover
dfbd4fb983 Error if AZURE_ACCOUNT_NAME not set 2024-11-13 08:02:22 -07:00
Michael Eischer
1133498ef8 Merge pull request #5046 from konidev20/fix-gh-4521-azure-blob-storage-add-support-for-access-tiers
azure: add support for access tiers hot, cool and cold
2024-11-11 22:01:52 +01:00
Michael Eischer
9c758313e3 Merge pull request #5119 from MichaelEischer/backup-json-start-end-time
backup: include start and end time in json output
2024-11-11 21:50:30 +01:00
Michael Eischer
82c5043fc9 Reduce checkboxes in PR checklist (#5120)
The basics around how to format commits and PR settings are primarily
relevant when opening a PR for the first time. But for repeated
contributors it is tedious to always tick those checkboxes.

Co-authored-by: rawtaz <rawtaz@users.noreply.github.com>
2024-11-11 21:49:26 +01:00
Michael Eischer
a73ae7ba1a restore: improve error if timestamp fails to restore 2024-11-11 21:37:28 +01:00
Michael Eischer
bd16804812 Merge branch 'patch-release' 2024-11-09 11:43:01 +01:00
Alexander Neumann
e2a98aa955 Set development version for 0.17.3 2024-11-08 20:36:48 +01:00
Alexander Neumann
bc64921a8e Add version for 0.17.3 2024-11-08 20:36:36 +01:00
Alexander Neumann
633883bdb6 Generate CHANGELOG.md for 0.17.3 2024-11-08 20:36:25 +01:00
Alexander Neumann
8348024664 Prepare changelog for 0.17.3 2024-11-08 20:36:25 +01:00
Michael Eischer
c3f5748e5b Merge pull request #5126 from restic/polish-changelogs
doc: Polish changelogs before release
2024-11-04 19:32:03 +01:00
Leo R. Lundgren
06ba4af436 doc: Polish changelogs before release 2024-11-03 22:55:06 +01:00
Michael Eischer
fb4d9b3232 Merge pull request #5125 from restic/patch-release-cherrypicks
Prepare patch release
2024-11-03 22:24:49 +01:00
Michael Eischer
7bfe3d99ae fs: fallback to low privilege security descriptors on access denied 2024-11-03 21:45:52 +01:00
Michael Eischer
d46525a51b fix double printf usage 2024-11-03 21:44:45 +01:00
Michael Eischer
3800eac54b prepare-release: improve handling of release from non-master branch
The final push command now states the correct branch to push.
2024-11-03 21:44:45 +01:00
Michael Eischer
75f317eaf1 sftp: check for broken connection in Load/List operation 2024-11-03 21:44:45 +01:00
Michael Eischer
b8527f4b38 prune: allow dry-run without taking a lock 2024-11-03 21:44:45 +01:00
Joram Berger
b8b7896d4c doc: Clarify number of blobs are added
The numbers reported as `data_blobs` and `tree_blobs` are not total numbers of blobs but numbers of blobs added with the given snapshot.
2024-11-03 21:42:58 +01:00
Michael Eischer
d0c5b5a9b7 add changelog for fuse fix 2024-11-03 21:42:20 +01:00
Michael Eischer
8aebea7ba2 fuse: test that the same fs.Node is used for the same file 2024-11-03 21:42:20 +01:00
Michael Eischer
0e9716a6e6 fuse: forget fs.Node instances on request by the kernel
Forget fs.Node instances once the kernel frees the corresponding nodeId.
This ensures that restic does not run out of memory on large snapshots.
2024-11-03 21:42:19 +01:00
Michael Eischer
de4f8b344e fuse: add missing type assertion for optional interfaces 2024-11-03 21:41:22 +01:00
Michael Eischer
75ec7d3269 fuse: cache fs.Node instances
A particular node should always be represented by a single instance.
This is necessary to allow the fuse library to assign a stable nodeId to
a node. macOS Sonoma trips over the previous, unstable behavior when
using fuse-t.
2024-11-03 21:41:13 +01:00
Michael Eischer
d8e0384940 doc: document safety feature for --target / --delete 2024-11-03 21:38:01 +01:00
Michael Eischer
408ec41a1d Merge pull request #5123 from MichaelEischer/fix-removable-media-handling
fs: fallback to low privilege security descriptors on access denied
2024-11-03 21:35:38 +01:00
Michael Eischer
270e7b7679 Merge pull request #5122 from restic/bump-golangci-lint
Bump go and golangci lint version
2024-11-03 21:34:25 +01:00
Michael Eischer
97f3e15039 Merge pull request #5121 from MichaelEischer/improve-release-helper
prepare-release: improve handling of release from non-master branch
2024-11-03 21:31:33 +01:00
Michael Eischer
d5bd3fcda5 Merge pull request #5112 from MichaelEischer/fix-vss-root-volume
Fix VSS metadata error (master)
2024-11-03 21:30:39 +01:00
Michael Eischer
62222edc4a Merge pull request #5110 from MichaelEischer/fix-vss-root-volume-patch
Fix VSS metadata error (v0.17.2)
2024-11-03 21:28:34 +01:00
Michael Eischer
f9a90aae89 fs: fallback to low privilege security descriptors on access denied 2024-11-01 19:10:52 +01:00
Michael Eischer
289159beaf fs: remove redundant fixpath in vss code 2024-11-01 19:03:45 +01:00
Michael Eischer
4052a5927c fs: move getVolumePathName function 2024-11-01 19:03:45 +01:00
Michael Eischer
d3c3390a51 ls: proper error handling if output is not possible 2024-11-01 17:07:43 +01:00
Michael Eischer
569a117a1d improve fprintf related error handling 2024-11-01 17:07:43 +01:00
Michael Eischer
41fa41b28b fix double printf usage 2024-11-01 16:36:23 +01:00
Michael Eischer
3eb9556f6a CI: add go 1.23 2024-11-01 16:34:00 +01:00
Michael Eischer
f5b1f9c8b1 CI: bump golangci-lint to latest version 2024-11-01 16:33:47 +01:00
Michael Eischer
e65f4e2231 backup: include start and end time in json output
The timestamps were already stored in the created snapshot.
2024-11-01 16:31:34 +01:00
Michael Eischer
bcf5fbe498 prepare-release: improve handling of release from non-master branch
The final push command now states the correct branch to push.
2024-11-01 16:22:32 +01:00
Michael Eischer
ded9fc7690 Merge pull request #5101 from MichaelEischer/sftp-load-error
sftp: check for broken connection in Load/List operation
2024-11-01 16:05:29 +01:00
Michael Eischer
b3b173a47c fs: use non existing vss path to avoid flaky test
The test used \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1 , which if
it exists and supports extended attributes can cause the test to fail.
2024-11-01 15:38:05 +01:00
Michael Eischer
e18a2a0072 Merge pull request #5096 from MichaelEischer/prune-allow-dry-run
prune: allow dry-run without taking a lock
2024-11-01 15:34:15 +01:00
Michael Eischer
1eea41c49e Merge pull request #5095 from MichaelEischer/retry-load-config
Retry loading or creating repository config
2024-11-01 15:33:45 +01:00
Michael Eischer
71c185313e sftp: check for broken connection in Load/List operation 2024-11-01 15:33:27 +01:00
Michael Eischer
868efe4968 prune: allow dry-run without taking a lock 2024-11-01 15:27:25 +01:00
Michael Eischer
3be2b8a54b add config retry changelog 2024-11-01 15:22:55 +01:00
Michael Eischer
b5bc76cdc7 test retry on repo opening 2024-11-01 15:17:54 +01:00
Michael Eischer
58dc4a6892 backend/retry: hide final log for stat() method
stat is only used to check the config file's existence. We don't want
log output in this case.
2024-11-01 15:17:54 +01:00
Michael Eischer
74c783b850 retry load or creating repository config
By now missing files are not endlessly retried by the retry backend such
that it can be enabled right from the start.

In addition, this change also enables the retry backend for the `init`
command.
2024-11-01 15:17:54 +01:00
Michael Eischer
fc92a04284 Merge pull request #5116 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/azidentity-1.8.0
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azidentity from 1.7.0 to 1.8.0
2024-11-01 15:07:23 +01:00
Michael Eischer
2f698d1cff Merge pull request #5117 from restic/dependabot/go_modules/google.golang.org/api-0.204.0
build(deps): bump google.golang.org/api from 0.199.0 to 0.204.0
2024-11-01 15:01:10 +01:00
dependabot[bot]
d8bf327d8b build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azidentity
Bumps [github.com/Azure/azure-sdk-for-go/sdk/azidentity](https://github.com/Azure/azure-sdk-for-go) from 1.7.0 to 1.8.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.7.0...sdk/azcore/v1.8.0)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azidentity
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-11-01 13:54:05 +00:00
Michael Eischer
2b3672198c Merge pull request #5115 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/azcore-1.16.0
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.14.0 to 1.16.0
2024-11-01 14:53:13 +01:00
dependabot[bot]
de847a48bf build(deps): bump google.golang.org/api from 0.199.0 to 0.204.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.199.0 to 0.204.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.199.0...v0.204.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-11-01 13:52:51 +00:00
Michael Eischer
d1d8ae7368 Merge pull request #5113 from restic/dependabot/go_modules/golang.org/x/time-0.7.0
build(deps): bump golang.org/x/time from 0.6.0 to 0.7.0
2024-11-01 14:52:18 +01:00
Michael Eischer
a32c98a39c Merge pull request #5114 from restic/dependabot/go_modules/golang.org/x/sys-0.26.0
build(deps): bump golang.org/x/sys from 0.25.0 to 0.26.0
2024-11-01 14:51:58 +01:00
dependabot[bot]
53cb6200fa build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore
Bumps [github.com/Azure/azure-sdk-for-go/sdk/azcore](https://github.com/Azure/azure-sdk-for-go) from 1.14.0 to 1.16.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.14.0...sdk/azcore/v1.16.0)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azcore
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-11-01 01:43:23 +00:00
dependabot[bot]
ae9268dadf build(deps): bump golang.org/x/sys from 0.25.0 to 0.26.0
Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.25.0 to 0.26.0.
- [Commits](https://github.com/golang/sys/compare/v0.25.0...v0.26.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sys
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-11-01 01:43:12 +00:00
dependabot[bot]
a494bf661d build(deps): bump golang.org/x/time from 0.6.0 to 0.7.0
Bumps [golang.org/x/time](https://github.com/golang/time) from 0.6.0 to 0.7.0.
- [Commits](https://github.com/golang/time/compare/v0.6.0...v0.7.0)

---
updated-dependencies:
- dependency-name: golang.org/x/time
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-11-01 01:43:08 +00:00
Michael Eischer
962279479d add vss metadata changelog 2024-10-31 22:07:15 +01:00
Michael Eischer
0aee70b496 restic: test path handling of volume shadow copy root path 2024-10-31 22:07:15 +01:00
Michael Eischer
4380627cb7 backup: run test with absolute path 2024-10-31 22:07:15 +01:00
Michael Eischer
e38f6794cd restic: fix error in fillGenericAttributes for vss volumes
Extended attributes and security descriptors apparently cannot be
retrieved from a vss volume. Fix the volume check to correctly detect
vss volumes and just completely disable extended attributes for volumes.
2024-10-31 22:07:15 +01:00
Michael Eischer
f77e67086c fs: add correct vss support to fixpath
Paths that only contain the volume shadow copy snapshot name require
special treatment. These paths must end with a slash for regular file
operations to work.
2024-10-31 22:07:15 +01:00
Michael Eischer
51cd1c847b backup: log error if test backup fails 2024-10-31 22:06:50 +01:00
Michael Eischer
14370fbf9e add vss metadata changelog 2024-10-31 22:06:50 +01:00
Michael Eischer
62af5f0b4a restic: test path handling of volume shadow copy root path 2024-10-31 22:06:50 +01:00
Michael Eischer
cb9247530e backup: run test with absolute path 2024-10-31 22:06:50 +01:00
Michael Eischer
1d0d5d87bc fs: fix error in fillGenericAttributes for vss volumes
Extended attributes and security descriptors apparently cannot be
retrieved from a vss volume. Fix the volume check to correctly detect
vss volumes and just completely disable extended attributes for volumes.
2024-10-31 22:06:50 +01:00
Michael Eischer
03aad742d3 fs: add correct vss support to fixpath
Paths that only contain the volume shadow copy snapshot name require
special treatment. These paths must end with a slash for regular file
operations to work.
2024-10-31 22:06:50 +01:00
Michael Eischer
15b7fb784f fs: cleanup fixpath 2024-10-31 21:49:03 +01:00
rawtaz
33da501c35 Merge pull request #5105 from joram-berger/patch-2
doc: Clarify number of blobs are added
2024-10-27 19:11:56 +00:00
Joram Berger
cd44b2bf8b doc: Clarify number of blobs are added
The numbers reported as `data_blobs` and `tree_blobs` are not total numbers of blobs but numbers of blobs added with the given snapshot.
2024-10-27 19:58:21 +01:00
Michael Eischer
1f0f6ad63d Merge branch 'patch-release' 2024-10-27 18:35:32 +01:00
Michael Eischer
ca4bd1b8ca Merge pull request #5094 from MichaelEischer/document-restore-delete-safety
doc: document safety feature for --target / --delete
2024-10-27 18:21:47 +01:00
Alexander Neumann
7eec85b4eb Set development version for 0.17.2 2024-10-27 16:37:28 +01:00
Alexander Neumann
2fb07dcdb1 Add version for 0.17.2 2024-10-27 16:37:19 +01:00
Alexander Neumann
5dcee7f0a3 Update manpages and auto-completion 2024-10-27 16:37:19 +01:00
Alexander Neumann
44968c7d43 Generate CHANGELOG.md for 0.17.2 2024-10-27 16:37:08 +01:00
Alexander Neumann
dbb5fb9fbd Prepare changelog for 0.17.2 2024-10-27 16:37:08 +01:00
Michael Eischer
e320edd416 Merge pull request #5048 from MichaelEischer/fix-macos-fuse
Fix unusable `mount` on macOS Sonoma
2024-10-23 22:51:00 +02:00
Michael Eischer
3a4a5a8215 Merge pull request #5102 from MichaelEischer/polish-changelogs
Polish patch release changelogs
2024-10-23 18:52:40 +02:00
Michael Eischer
d8d955e0aa Tweak wording
Co-authored-by: rawtaz <rawtaz@users.noreply.github.com>
2024-10-22 20:00:39 +02:00
Michael Eischer
2ce485063f polish changelogs 2024-10-22 19:48:59 +02:00
Michael Eischer
f72febb34f Merge pull request #5099 from MichaelEischer/hackport-fix-vss-metadata
Hackport "backup: read extended metadata from snapshot"
2024-10-22 19:24:08 +02:00
Michael Eischer
821000cb68 Merge pull request #5097 from MichaelEischer/fix-vss-metadata
backup: read extended metadata from snapshot
2024-10-22 19:23:06 +02:00
Srigovind Nayak
db686592a1 debug: azure add debug log to show access-tier 2024-10-20 20:24:49 +05:30
Srigovind Nayak
bff3341d10 azure: add support for hot, cool, or cool access tiers 2024-10-20 15:27:21 +05:30
Michael Eischer
5fe6607127 Merge pull request #5084 from greatroar/utimesnano
Simplify and refactor restoring of timestamps
2024-10-19 12:47:13 +00:00
greatroar
8f20d5dcd5 fs: Refactor UtimesNano replacements
Previously, nodeRestoreTimestamps would do something like

	if node.Type == restic.NodeTypeSymlink {
	    return nodeRestoreSymlinkTimestamps(...)
	}
	return syscall.UtimesNano(...)

where nodeRestoreSymlinkTimestamps was either a no-op or a
reimplementation of syscall.UtimesNano that handles symlinks, with some
repeated converting between timestamp types. The Linux implementation
was a bit clumsy, requiring three syscalls to set the timestamps.

In this new setup, there is a function utimesNano that has three
implementations:

* on Linux, it's a modified syscall.UtimesNano that uses
  AT_SYMLINK_NOFOLLOW and AT_FDCWD so it can handle any type in a single
  call;
* on other Unix platforms, it just calls the syscall function after
  skipping symlinks;
* on Windows, it's the modified UtimesNano that was previously called
  nodeRestoreSymlinkTimestamps, except with different arguments.
2024-10-19 12:04:09 +02:00
greatroar
f967a33ccc fs: Use AT_FDCWD in Linux nodeRestoreSymlinkTimestamps
There's no need to open the containing directory. This is exactly what
syscall.UtimesNano does, except for the AT_SYMLINK_NOFOLLOW flag.
2024-10-19 11:29:35 +02:00
Michael Eischer
ee9a5cdf70 add vss metadata changelog 2024-10-18 22:51:55 +02:00
Michael Eischer
46dce1f4fa backup: work around file deletion error in test 2024-10-18 22:51:55 +02:00
Michael Eischer
841f8bfef0 redirect test log output to t.Log() 2024-10-18 22:51:55 +02:00
Michael Eischer
1f5791222a backup: test that vss backups work if underlying data was removed 2024-10-18 22:51:55 +02:00
Michael Eischer
ec43594003 add vss metadata changelog 2024-10-18 22:36:03 +02:00
Michael Eischer
a7b13bd603 fs: remove file.Name() from interface
The only user was archiver.fileSaver.
2024-10-18 22:29:03 +02:00
Michael Eischer
0c711f5605 archiver: use correct filepath in fileSaver for vss
When using the VSS FS, then `f.Name()` contained the filename in the
snapshot. This caused a double mapping when calling NodeFromFileInfo.
2024-10-18 22:29:03 +02:00
Michael Eischer
4df2e33568 archiver: properly create node for vss backups
Previously, NodeFromFileInfo used the original file path to create the
node, which also meant that extended metadata was read from there
instead of within the vss snapshot.

This change is a temporary solution for restic 0.17.2 and will be
replaced with a clean fix in restic 0.18.0.
2024-10-18 22:26:18 +02:00
Michael Eischer
11c1fbce20 Merge pull request #5098 from MichaelEischer/prepare-patch-release
Prepare patch release
2024-10-18 22:20:27 +02:00
Michael Eischer
e1faf7b18c backup: work around file deletion error in test 2024-10-18 22:08:10 +02:00
Connor Findlay
9553d873ff backend/azure: Add tests for both token types
Add two new test cases, TestBackendAzureAccountToken and
TestBackendAzureContainerToken, that ensure that the authorization using
both types of token works.

This introduces two new environment variables,
RESTIC_TEST_AZURE_ACCOUNT_SAS and RESTIC_TEST_AZURE_CONTAINER_SAS, that
contain the tokens to use when testing restic. If an environment
variable is missing, the related test is skipped.
2024-10-18 21:59:03 +02:00
Connor Findlay
048c3bb240 changelog: Add changes in issue-4004
Add changelog entry in the 'unreleased' sub-folder for changes
introduced when fixing issue #4004.
2024-10-18 21:59:03 +02:00
Connor Findlay
d6e76a22a8 backend/azure: Handle Container SAS/SAT
Ignore AuthorizationFailure caused by using a container level SAS/SAT
token when calling GetProperties during the Create() call. This is because the
GetProperties call expects an Account Level token, and the container
level token simply lacks the appropriate permissions. Supressing the
Authorization Failure is OK, because if the token is actually invalid,
this is caught elsewhere when we try to actually use the token to do
work.
2024-10-18 21:59:03 +02:00
Michael Eischer
e3a022f9b5 add irregular files bug changelog 2024-10-18 21:58:04 +02:00
Michael Eischer
fe269c752a repair snapshots: remove irregular files 2024-10-18 21:57:52 +02:00
Michael Eischer
fc1fc00aa4 backup: exclude irregular files from backup
restic cannot backup irregular files as those don't behave like normal
files. Thus skip them with an error.
2024-10-18 21:56:41 +02:00
greatroar
3c82fe6ef5 fs: Include filename in mknod errors 2024-10-18 21:53:15 +02:00
Michael Eischer
986d981bf6 tag: fix swallowed error if repository cannot be opened 2024-10-18 21:50:29 +02:00
Michael Eischer
0df2fa8135 fs: retry preallocate on Linux if interrupted by signal 2024-10-18 21:47:59 +02:00
Roman Inflianskas
49ccb7734c list: validate subcommand 2024-10-18 21:47:59 +02:00
Roman Inflianskas
491cc65e3a list: add subcommand completion 2024-10-18 21:47:59 +02:00
Damien Clark
8c1d6a50c1 cache: fix race condition in cache cleanup
Fix multiple restic processes executing concurrently and racing to remove obsolete snapshots.

Co-authored-by: Michael Eischer <michael.eischer@fau.de>
2024-10-18 21:47:59 +02:00
Michael Eischer
9386acc4a6 Fix indentation of blockquotes in github release notes 2024-10-18 21:47:59 +02:00
Git'Fellow
5b60d49654 fix: shorten sentence 2024-10-18 21:47:59 +02:00
Git'Fellow
8056181301 docs: Recommend to setup B2 versions lifecycle rules 2024-10-18 21:46:58 +02:00
Michael Eischer
fc6f1b4b06 redirect test log output to t.Log() 2024-10-18 21:43:46 +02:00
Michael Eischer
9f206601af backup: test that vss backups work if underlying data was removed 2024-10-18 21:43:46 +02:00
Michael Eischer
ca79cb92e3 fs/vss: test that vss functions actually read from snapshot 2024-10-18 21:43:46 +02:00
Michael Eischer
352605d9f0 fs: remove file.Name() from interface
The only user was archiver.fileSaver.
2024-10-18 21:43:23 +02:00
Michael Eischer
26b77a543d archiver: use correct filepath in fileSaver for vss
When using the VSS FS, then `f.Name()` contained the filename in the
snapshot. This caused a double mapping when calling NodeFromFileInfo.
2024-10-18 21:41:02 +02:00
Michael Eischer
b988754a6d fs/vss: reuse functions from underlying FS
OpenFile, Stat and Lstat should reuse the underlying FS implementation
to avoid diverging behavior.
2024-10-18 19:30:05 +02:00
Michael Eischer
60960d2405 fs/vss: properly create node from vss path
Previously, NodeFromFileInfo used the original file path to create the
node, which also meant that extended metadata was read from there
instead of within the vss snapshot.
2024-10-18 19:27:44 +02:00
Michael Eischer
7c02141548 Merge pull request #5093 from Seefin/fix-containerSAS
Fix Azure Container Token Auth
2024-10-17 18:45:06 +00:00
Connor Findlay
b434f560cc backend/azure: Add tests for both token types
Add two new test cases, TestBackendAzureAccountToken and
TestBackendAzureContainerToken, that ensure that the authorization using
both types of token works.

This introduces two new environment variables,
RESTIC_TEST_AZURE_ACCOUNT_SAS and RESTIC_TEST_AZURE_CONTAINER_SAS, that
contain the tokens to use when testing restic. If an environment
variable is missing, the related test is skipped.
2024-10-17 20:38:03 +02:00
Connor Findlay
7bdfcf13fb changelog: Add changes in issue-4004
Add changelog entry in the 'unreleased' sub-folder for changes
introduced when fixing issue #4004.
2024-10-17 20:38:03 +02:00
Connor Findlay
2e704c69ac backend/azure: Handle Container SAS/SAT
Ignore AuthorizationFailure caused by using a container level SAS/SAT
token when calling GetProperties during the Create() call. This is because the
GetProperties call expects an Account Level token, and the container
level token simply lacks the appropriate permissions. Supressing the
Authorization Failure is OK, because if the token is actually invalid,
this is caught elsewhere when we try to actually use the token to do
work.
2024-10-17 20:38:03 +02:00
Michael Eischer
5838896962 doc: document safety feature for --target / --delete 2024-10-17 19:45:03 +02:00
Michael Eischer
bcd5ac34bb Merge pull request #5060 from MichaelEischer/proper-nodefromfileinfo
fs: move NodeFromFileInfo into FS interface
2024-10-16 21:34:37 +02:00
Michael Eischer
618f306f13 Merge pull request #5054 from phillipp/dump-compress-zip
dump: add --compress flag to compress archives
2024-10-16 19:17:47 +00:00
Michael Eischer
75711446e1 fs: move NodeFromFileInfo into FS interface 2024-10-16 21:17:21 +02:00
Michael Eischer
c3b3120e10 Merge pull request #5057 from MichaelEischer/fix-backup-irregular
backup: fix handling of files with type irregular
2024-10-16 21:13:08 +02:00
Michael Eischer
e29d38f8bf dump/zip: test that files are compressed 2024-10-16 21:11:24 +02:00
Michael Eischer
da3c02405b dump/zip: only compress regular files 2024-10-16 21:09:05 +02:00
Michael Eischer
55c150054d add irregular files bug changelog 2024-10-16 20:54:08 +02:00
Michael Eischer
012cb06fe9 repair snapshots: remove irregular files 2024-10-16 20:54:08 +02:00
Michael Eischer
f44b7cdf8c backup: exclude irregular files from backup
restic cannot backup irregular files as those don't behave like normal
files. Thus skip them with an error.
2024-10-16 20:54:08 +02:00
Michael Eischer
e91a456656 Merge pull request #5061 from MichaelEischer/fix-timestamp-restore-windows
fs: fix restoring timestamps on older Windows versions for long paths
2024-10-16 20:47:17 +02:00
Michael Eischer
e21496f217 Merge pull request #5074 from greatroar/dump
dump: Simplify writeNode and use fewer goroutines
2024-10-16 18:33:35 +00:00
Michael Eischer
0c0d8b8cfd Merge pull request #5083 from greatroar/errors
Some error handling patches
2024-10-16 18:22:49 +00:00
Michael Eischer
60cba55647 Merge pull request #5079 from restic/dependabot/go_modules/google.golang.org/api-0.199.0
build(deps): bump google.golang.org/api from 0.195.0 to 0.199.0
2024-10-09 20:35:03 +00:00
dependabot[bot]
221fa0fa7c build(deps): bump google.golang.org/api from 0.195.0 to 0.199.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.195.0 to 0.199.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.195.0...v0.199.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-09 20:26:34 +00:00
Michael Eischer
7cfd8a6715 Merge pull request #5080 from restic/dependabot/go_modules/golang.org/x/oauth2-0.23.0
build(deps): bump golang.org/x/oauth2 from 0.22.0 to 0.23.0
2024-10-09 20:15:43 +00:00
Michael Eischer
0ada0b56b6 Merge pull request #5078 from restic/dependabot/go_modules/github.com/minio/minio-go/v7-7.0.77
build(deps): bump github.com/minio/minio-go/v7 from 7.0.76 to 7.0.77
2024-10-09 20:09:05 +00:00
Michael Eischer
7c12bd59a0 Merge pull request #5053 from rominf/rominf-generate-stdout
generate: allow passing `-` for stdout output
2024-10-09 20:06:54 +00:00
Michael Eischer
888abff7e0 Merge pull request #5058 from MichaelEischer/clarify-changelog
Changelogs should omit problem if its description duplicates the new behavior
2024-10-09 22:06:41 +02:00
Michael Eischer
783901726e Merge pull request #5056 from MichaelEischer/fix-tag-error-handling
tag: fix swallowed error if repository cannot be opened
2024-10-09 22:06:26 +02:00
dependabot[bot]
eac00eb933 build(deps): bump golang.org/x/oauth2 from 0.22.0 to 0.23.0
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.22.0 to 0.23.0.
- [Commits](https://github.com/golang/oauth2/compare/v0.22.0...v0.23.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-09 19:58:42 +00:00
Michael Eischer
96c1c1a0fc Merge pull request #5075 from greatroar/idset
internal/restic: Use IDSet.Clone + use maps package
2024-10-09 19:55:26 +00:00
Michael Eischer
8d7f4574b4 Merge pull request #5077 from restic/dependabot/go_modules/go.uber.org/automaxprocs-1.6.0
build(deps): bump go.uber.org/automaxprocs from 1.5.3 to 1.6.0
2024-10-09 19:51:15 +00:00
Michael Eischer
ddf65b04f3 Merge pull request #5076 from restic/dependabot/go_modules/golang.org/x/sys-0.25.0
build(deps): bump golang.org/x/sys from 0.24.0 to 0.25.0
2024-10-09 19:50:45 +00:00
greatroar
2b609d3e77 errors, fs: Replace CombineErrors with stdlib Join
This does not produce exactly the same messages, as it inserts newlines
instead of "; ". But given how long our error messages can be, that
might be a good thing.
2024-10-05 10:56:40 +02:00
greatroar
19653f9e06 fs: Simplify NodeCreateAt 2024-10-05 10:56:39 +02:00
greatroar
e10e2bb50f fs: Include filename in mknod errors 2024-10-05 10:56:39 +02:00
greatroar
b5c28a7ba2 internal/restic: Use IDSet.Clone + use maps package
One place where IDSet.Clone is useful was reinventing it, using a
conversion to list, a sort, and a conversion back to map.

Also, use the stdlib "maps" package to implement as much of IDSet as
possible. This requires changing one caller, which assumed that cloning
nil would return a non-nil IDSet.
2024-10-03 21:14:29 +02:00
dependabot[bot]
f3f629bb69 build(deps): bump github.com/minio/minio-go/v7 from 7.0.76 to 7.0.77
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.76 to 7.0.77.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.76...v7.0.77)

---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-01 01:49:46 +00:00
dependabot[bot]
e90085b375 build(deps): bump go.uber.org/automaxprocs from 1.5.3 to 1.6.0
Bumps [go.uber.org/automaxprocs](https://github.com/uber-go/automaxprocs) from 1.5.3 to 1.6.0.
- [Release notes](https://github.com/uber-go/automaxprocs/releases)
- [Changelog](https://github.com/uber-go/automaxprocs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/uber-go/automaxprocs/compare/v1.5.3...v1.6.0)

---
updated-dependencies:
- dependency-name: go.uber.org/automaxprocs
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-01 01:49:41 +00:00
dependabot[bot]
3f08dee685 build(deps): bump golang.org/x/sys from 0.24.0 to 0.25.0
Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.24.0 to 0.25.0.
- [Commits](https://github.com/golang/sys/compare/v0.24.0...v0.25.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sys
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-01 01:49:38 +00:00
greatroar
8c7a6daa47 dump: Simplify writeNode and use fewer goroutines
This changes Dumper.writeNode to spawn loader goroutines as needed
instead of as a pool. The code is shorter, fewer goroutines are spawned
for small files, and crash dumps (also for unrelated errors) should be
smaller.
2024-09-30 17:24:05 +02:00
Roman Inflianskas
3d976562fa generate: allow passing - for stdout output
Since generating completions to stdout for multiple shells does not make
sense, enforce `-` is supplied only once.
2024-09-16 10:54:00 +03:00
Phillipp Röll
1a7fafc7eb dump: compress zip archives 2024-09-15 21:04:54 +02:00
Michael Eischer
4469fe1575 fs: fix restoring timestamps on Windows for long paths 2024-09-15 18:28:11 +02:00
Phillipp Röll
bad6c54a33 dump: add --compress-zip flag to compress zip archives 2024-09-15 14:25:02 +02:00
Michael Eischer
7680f48258 Changelogs should omit problem if it duplicates the new behavior
When adding a new feature, the problem description often just says that
feature Y was missing, followed by saying that feature Y is now
supported.

This duplication just makes the changelog entries unnecessarily verbose.
2024-09-14 20:54:27 +02:00
Michael Eischer
efec1a5e96 Merge pull request #5045 from MichaelEischer/fix-preallocate-eintr
Linux: retry preallocate if interrutped by signal
2024-09-14 19:17:51 +02:00
Michael Eischer
bd2c986592 Merge pull request #5051 from rominf/rominf-list-subcommands
list: complete and validate subcommand
2024-09-14 16:43:04 +00:00
Michael Eischer
cab6b15603 tag: fix swallowed error if repository cannot be opened 2024-09-14 18:38:48 +02:00
Michael Eischer
4105e4a356 Merge pull request #5047 from damoclark/patch-1
cache: fix race condition in cache cleanup or similar.
2024-09-14 16:14:48 +00:00
Michael Eischer
ccf5be235a add changelog for fuse fix 2024-09-14 18:11:44 +02:00
Michael Eischer
5ce6ca2219 fuse: test that the same fs.Node is used for the same file 2024-09-14 18:11:44 +02:00
Michael Eischer
51173c5003 fuse: forget fs.Node instances on request by the kernel
Forget fs.Node instances once the kernel frees the corresponding nodeId.
This ensures that restic does not run out of memory on large snapshots.
2024-09-14 18:11:44 +02:00
Michael Eischer
e9940f39dc fuse: add missing type assertion for optional interfaces 2024-09-14 18:11:44 +02:00
Michael Eischer
6ec2b62ec5 fuse: cache fs.Node instances
A particular node should always be represented by a single instance.
This is necessary to allow the fuse library to assign a stable nodeId to
a node. macOS Sonoma trips over the previous, unstable behavior when
using fuse-t.
2024-09-14 18:11:44 +02:00
Damien Clark
4795143d6d cache: fix race condition in cache cleanup
Fix multiple restic processes executing concurrently and racing to remove obsolete snapshots.

Co-authored-by: Michael Eischer <michael.eischer@fau.de>
2024-09-14 18:07:46 +02:00
Roman Inflianskas
a84e65b7f9 list: validate subcommand 2024-09-13 12:23:26 +03:00
Roman Inflianskas
6f08dbb2d7 list: add subcommand completion 2024-09-13 12:22:53 +03:00
Michael Eischer
c1532179d4 Merge pull request #5043 from MichaelEischer/fix-github-release-note-formatting
Fix indentation of blockquotes in github release notes
2024-09-07 17:11:22 +02:00
Michael Eischer
34fe73ea42 fs: retry preallocate on Linux if interrupted by signal 2024-09-07 16:39:40 +02:00
Michael Eischer
37d5bd61a0 Merge pull request #5042 from solracsf/patch-1
docs: Recommend to setup B2 versions lifecycle rules
2024-09-07 14:36:29 +00:00
Michael Eischer
7b1a15916d Merge pull request #5039 from konidev20/fix-gh-4806-forget-add-reason-for-oldest-snapshot-retained
forget: indicate why the oldest snapshot in a group is kept
2024-09-07 14:31:47 +00:00
Git'Fellow
113439c69b fix: shorten sentence 2024-09-07 15:27:15 +02:00
Srigovind Nayak
5468e85222 docs: mention that the oldest snapshot is marked oldest in the reasons of the forget comman 2024-09-07 15:07:23 +05:30
Srigovind Nayak
b69c6408a6 forget: make oldest snapshot marker more strict
Now, a snapshot is only marked as oldest if it's the last in the list AND its values matches the last seen value for that bucket.

Also, updated the corresponding golden files for the tests.
2024-09-07 15:07:23 +05:30
Srigovind Nayak
d656a50852 forget: update tests to reflect specific reasons for keeping oldest snapshots in a group 2024-09-07 15:07:23 +05:30
Srigovind Nayak
87f30bc787 forget: indicate why the oldest snapshot in a group is kept
When the oldest snapshot in the
list is retained, the reason is now prefixed with "oldest" to clearly
indicate why it's being kept.
2024-09-07 15:07:23 +05:30
Michael Eischer
4f0affd4f7 Merge branch 'patch-release' 2024-09-06 22:32:22 +02:00
Michael Eischer
3df8337d63 Fix indentation of blockquotes in github release notes 2024-09-05 22:33:57 +02:00
Alexander Neumann
76a647febf Set development version for 0.17.1 2024-09-05 21:25:24 +02:00
Alexander Neumann
975aa41e1e Add version for 0.17.1 2024-09-05 21:25:21 +02:00
Alexander Neumann
a98370cc9e Update manpages and auto-completion 2024-09-05 21:25:21 +02:00
Alexander Neumann
d8870a2f73 Generate CHANGELOG.md for 0.17.1 2024-09-05 21:25:20 +02:00
Alexander Neumann
17e54b04ab Prepare changelog for 0.17.1 2024-09-05 21:25:19 +02:00
Git'Fellow
00ca0b371b docs: Recommend to setup B2 versions lifecycle rules 2024-09-04 13:21:37 +02:00
Michael Eischer
8a0edde407 Merge pull request #5038 from restic/dependabot/go_modules/google.golang.org/api-0.195.0
build(deps): bump google.golang.org/api from 0.191.0 to 0.195.0
2024-09-01 22:36:39 +00:00
Michael Eischer
0a225049d8 Merge pull request #5035 from restic/dependabot/go_modules/github.com/minio/minio-go/v7-7.0.76
build(deps): bump github.com/minio/minio-go/v7 from 7.0.74 to 7.0.76
2024-09-01 22:14:47 +00:00
Michael Eischer
3023b2f566 Merge pull request #5033 from MichaelEischer/s3-clarify-docs
docs: make s3-compatible section standalone
2024-09-02 00:14:31 +02:00
dependabot[bot]
a6490feab2 build(deps): bump github.com/minio/minio-go/v7 from 7.0.74 to 7.0.76
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.74 to 7.0.76.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.74...v7.0.76)

---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-09-01 22:00:55 +00:00
Michael Eischer
daa6448a77 Merge pull request #5034 from restic/dependabot/go_modules/golang.org/x/sys-0.24.0
build(deps): bump golang.org/x/sys from 0.23.0 to 0.24.0
2024-09-01 21:52:56 +00:00
Michael Eischer
07a8b73f25 Merge pull request #5037 from restic/dependabot/go_modules/github.com/ncw/swift/v2-2.0.3
build(deps): bump github.com/ncw/swift/v2 from 2.0.2 to 2.0.3
2024-09-01 21:52:41 +00:00
Michael Eischer
9a6059eb71 Merge pull request #5032 from dropbigfish/master
chore: fix some function name comments
2024-09-01 21:52:26 +00:00
Michael Eischer
05a8b05773 Merge pull request #5030 from restic/changelogs
doc: Polish unreleased changelogs
2024-09-01 23:45:10 +02:00
dependabot[bot]
790dbd442b build(deps): bump google.golang.org/api from 0.191.0 to 0.195.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.191.0 to 0.195.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.191.0...v0.195.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-09-01 01:33:45 +00:00
dependabot[bot]
daf156a76a build(deps): bump github.com/ncw/swift/v2 from 2.0.2 to 2.0.3
Bumps [github.com/ncw/swift/v2](https://github.com/ncw/swift) from 2.0.2 to 2.0.3.
- [Release notes](https://github.com/ncw/swift/releases)
- [Changelog](https://github.com/ncw/swift/blob/master/RELEASE.md)
- [Commits](https://github.com/ncw/swift/compare/v2.0.2...v2.0.3)

---
updated-dependencies:
- dependency-name: github.com/ncw/swift/v2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-09-01 01:33:35 +00:00
dependabot[bot]
154ca4d9e8 build(deps): bump golang.org/x/sys from 0.23.0 to 0.24.0
Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.23.0 to 0.24.0.
- [Commits](https://github.com/golang/sys/compare/v0.23.0...v0.24.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sys
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-09-01 01:33:18 +00:00
Michael Eischer
ebd8f0c74a docs: make s3-compatible section standalone 2024-08-31 19:39:30 +02:00
Leo R. Lundgren
5d658f216c doc: Polish unreleased changelogs 2024-08-31 19:10:36 +02:00
dropbigfish
6f9513d88c chore: fix some function names
Signed-off-by: dropbigfish <fillfish@foxmail.com>
2024-09-01 00:54:39 +08:00
Michael Eischer
d8be8f1e06 Merge pull request #5024 from MichaelEischer/move-node-to-fs
Cleanup FS package
2024-08-31 18:47:11 +02:00
Michael Eischer
b91ef3f1ff fs: remove dead code 2024-08-31 18:40:36 +02:00
Michael Eischer
e2bce1b9ee fs: move WindowsAttributes definition back to restic package 2024-08-31 18:40:36 +02:00
Michael Eischer
ebdd946ac1 fs: unexport nodeRestoreTimestamps 2024-08-31 18:40:36 +02:00
Michael Eischer
2aa1e2615b fs: fix comments 2024-08-31 18:40:36 +02:00
Michael Eischer
6c16733dfd fs: remove unused methods from File interface 2024-08-31 18:40:36 +02:00
Michael Eischer
f0329bb4e6 fs: replace statT with ExtendedFileInfo 2024-08-31 18:40:36 +02:00
Michael Eischer
6d3a5260d3 fs: unexport a several windows functions 2024-08-31 18:40:36 +02:00
Michael Eischer
cf051e777a fs: remove Readdir method from File interface 2024-08-31 18:20:41 +02:00
Michael Eischer
cc7f99125a minimize usage of internal/fs in tests 2024-08-31 18:20:41 +02:00
Michael Eischer
65a7157383 mount: use os instead of fs package 2024-08-31 18:20:41 +02:00
Michael Eischer
24f4e780f1 backend: consistently use os package for filesystem access
The go std library should be good enough to manage the files in the
backend and cache folders.
2024-08-31 18:20:40 +02:00
Michael Eischer
ca1e5e10b6 add proper constants for node type 2024-08-31 18:20:01 +02:00
Michael Eischer
3b438e5c7c Merge pull request #5023 from MichaelEischer/cleanup-archiver
archiver: use FS interface nearly everywhere and cleanup exports
2024-08-31 18:14:47 +02:00
Michael Eischer
7bb92dc7bd archiver: use ExtendedStat from FS interface
With this change, NodeFromFileInfo is the last function that bypasses
the FS interface in the archiver.
2024-08-31 18:05:09 +02:00
Michael Eischer
e79dca644e fs: unexport DeviceID 2024-08-31 18:04:53 +02:00
Michael Eischer
70fbad6623 archiver: minimize imports 2024-08-31 18:04:37 +02:00
Michael Eischer
6fd5d5f2d5 archiver: move helper functions to combine rejects 2024-08-31 18:04:22 +02:00
Michael Eischer
f1585af0f2 move include/exclude options to filter package 2024-08-31 18:04:07 +02:00
Michael Eischer
5d58945718 cleanup include / exclude option setup 2024-08-31 18:03:53 +02:00
Michael Eischer
41c031a19e backup: move RejectFuncs to archiver package 2024-08-31 18:03:35 +02:00
Michael Eischer
f9dbcd2531 backup: convert reject funcs to use FS interface
Depending on parameters the paths in a snapshot do not directly
correspond to real paths on the filesystem. Therefore, reject funcs must
use the FS interface to work correctly.
2024-08-31 18:03:02 +02:00
Michael Eischer
c6fae0320e archiver: hide implementation details 2024-08-31 17:52:45 +02:00
Michael Eischer
e5cdae9c84 Merge pull request #5022 from MichaelEischer/extract-fs-code
Extract filesystem code from restic.Node
2024-08-31 17:52:11 +02:00
Michael Eischer
507842b614 fs: remove Open method from FS interface 2024-08-31 17:37:25 +02:00
Michael Eischer
263709da8c fs: unexport isListxattrPermissionError 2024-08-31 17:37:25 +02:00
Michael Eischer
80ed863aab repository: remove redundant cleanup code
The temp files used by the packer manager are either delete after
creation (unix) or marked as delete on close (windows). Thus, no
explicit cleanup is necessary.
2024-08-31 17:37:25 +02:00
Michael Eischer
0ddb4441d7 fs: clean up helper functions 2024-08-31 17:37:25 +02:00
Michael Eischer
fc549c9462 cleanup imports 2024-08-31 17:37:25 +02:00
Michael Eischer
b9b32e5647 restic: extract Node filesystem code to fs package 2024-08-31 17:37:25 +02:00
Michael Eischer
a2e54eac64 restic: simplify nodeCreateFileAt
The code to write the file content is never used.
2024-08-31 17:37:25 +02:00
Michael Eischer
5644079707 restic: prepare extraction of fs code from Node 2024-08-31 17:37:25 +02:00
Michael Eischer
3e0c081bed Merge pull request #5020 from MichaelEischer/remove-legacy-formats
Remove support for legacy index format and s3 layout
2024-08-31 17:37:09 +02:00
Michael Eischer
97f696b937 backend: remove dead code 2024-08-31 17:25:24 +02:00
Michael Eischer
af989aab4e backend/layout: unexport fields and simplify rest layout 2024-08-31 17:25:24 +02:00
Michael Eischer
6024597028 drop support for s3legacy layout 2024-08-31 17:25:24 +02:00
Michael Eischer
943b6ccfba index: remove support for legacy index format 2024-08-31 17:12:43 +02:00
Michael Eischer
7b0b9539b1 Merge pull request #5029 from MichaelEischer/cleanup-changelogs
cleanup changelogs
2024-08-31 17:04:31 +02:00
Michael Eischer
259caf942d cleanup changelogs 2024-08-31 16:55:38 +02:00
Michael Eischer
ba71141f0a backup: support specifying volume instead of path on Windows
"C:" (volume name) versus "C:\" (path)
2024-08-31 16:50:06 +02:00
Michael Eischer
174f20dc4a use OrderedListOnceBackend where possible 2024-08-31 16:50:06 +02:00
Michael Eischer
361fbbf58f Add temporary files repositories in integration tests
This is intended to catch problems with temporary files stored in the
backend, even if the responsible component forgets to test for those.
2024-08-31 16:50:06 +02:00
Michael Eischer
1f4c9d2806 cache: correctly ignore files whose filename is no ID
this can for example be the case for temporary files created by the
backend implementation.
2024-08-31 16:50:06 +02:00
Michael Eischer
a5533344f9 Merge pull request #5028 from MichaelEischer/windows-allow-specifying-volumes
backup: support specifying volume instead of path on Windows
2024-08-31 16:43:20 +02:00
Michael Eischer
ddf35a60ad Merge pull request #5026 from MichaelEischer/fix-handling-invalid-filenames
cache: Fix handling of invalid filenames
2024-08-31 16:42:13 +02:00
Michael Eischer
a12a6edfd1 add changelog for security descriptor race condition 2024-08-30 12:46:07 +02:00
Michael Eischer
ac5bc7c2f9 fs: fix error handling for retried get/set of security descriptor
The retry code path did not filter `ERROR_NOT_SUPPORTED`. Just call the
original function a second time to correctly follow the low privilege
code path.
2024-08-30 12:46:07 +02:00
Michael Eischer
3e4c1ea196 fs: fix race condition in get/set security descriptor
Calling `Load()` twice for an atomic variable can return different
values each time. This resulted in trying to read the security
descriptor with high privileges, but then not entering the code path to
switch to low privileges when another thread has already done so
concurrently.
2024-08-30 12:46:07 +02:00
Michael Eischer
8828c76f92 rest: improve handling of HTTP2 goaway
The HTTP client can only retry HTTP2 requests after receiving a GOAWAY
response if it can rewind the body. As we use a custom data type,
explicitly provide an implementation of `GetBody`.
2024-08-30 12:46:07 +02:00
Michael Eischer
55ff4e046e doc: full tree blob data structure is in the code 2024-08-30 12:46:07 +02:00
Michael Eischer
7ea558db99 doc: JSON encoder must be deterministic 2024-08-30 12:46:07 +02:00
Michael Eischer
71e8068d86 doc: mark S3 layout as deprecated 2024-08-30 12:46:07 +02:00
Michael Eischer
a45d21e2b9 doc: describe how to handle rewrite encoding error 2024-08-30 12:46:07 +02:00
Michael Eischer
97eb81564a doc: fix typos 2024-08-30 12:45:20 +02:00
Michael Eischer
262e85c37f doc: shrink wasabi / alibaba cloud example
Remove descriptions for both providers and shorten the example to the
minimum.
2024-08-30 12:45:20 +02:00
Michael Eischer
f451001f75 doc: use regional urls for Amazon S3 and add generic s3 provider section
Split description for non-Amazon S3 providers into separate section. The
section now also includes the `s3.bucket-lookup` extended option. Switch
to using regional URLs for Amazon S3 to replace the need for setting the
region.
2024-08-30 12:45:20 +02:00
Michael Eischer
5980daea64 doc/backup: move exit status codes section up 2024-08-30 12:45:20 +02:00
Michael Eischer
6b4f16f77b doc/backup: link to exit code for scripting section 2024-08-30 12:45:20 +02:00
Michael Eischer
64d628bd75 make timeout for slow requests configurable 2024-08-30 12:45:20 +02:00
Michael Eischer
6eece31dc3 lock: introduce short delay between failed locking retries
Failed locking attempts were immediately retried up to three times
without any delay between the retries. If a lock file is not found while
checking for other locks, with the reworked backend retries there is no
delay between those retries. This is a problem if a backend requires a
few seconds to reflect file deletions in the file listings. To work
around this problem, introduce a short exponentially increasing delay
between the retries. The number of retries is now increased to 4. This
results in delays of 5, 10 and 20 seconds between the retries.
2024-08-30 12:45:20 +02:00
Michael Eischer
8206cd19c8 backend/retry: don't trip circuit breaker if context is canceled
When the context used for a load operation is canceled, then the result
is always an error independent of whether the file could be retrieved
from the backend. Do not false positively trip the circuit breaker in
this case.

The old behavior was problematic when trying to lock a repository. When
`Lock.checkForOtherLocks` listed multiple lock files in parallel and one
of them fails to load, then all other loads were canceled. This
cancelation was remembered by the circuit breaker, such that locking
retries would fail.
2024-08-30 12:45:20 +02:00
Michael Eischer
a99b824508 update docs 2024-08-30 12:45:20 +02:00
Michael Eischer
424740f62c Add missing DisableAutoGenTag flag for commands 2024-08-30 12:45:20 +02:00
Michael Eischer
e5a08e6808 group commands and make features/options visible 2024-08-30 12:45:20 +02:00
Michael Eischer
cb16add8c8 restic: cleanup redundant code in test case 2024-08-30 12:45:20 +02:00
Michael Eischer
bc1aecfb15 restore: test timestamps for macOS resource forks are restored correctly 2024-08-30 12:45:20 +02:00
Michael Eischer
61aaddac28 restic: restore timestamps after extended attributes
restoring the xattr containing resource forks on macOS apparently
modifies the file modification timestamps. Thus, restore the timestamp
after xattrs.
2024-08-30 12:45:20 +02:00
Michael Eischer
a5f2d0cf56 Improve description for no password on secondary repo 2024-08-30 12:45:20 +02:00
Michael Eischer
00f63d72fa Mention RESTIC_HOST environment variable in docs 2024-08-30 12:45:20 +02:00
Andreas Deininger
12089054d8 GitHub test actions: fix warnings 'Restore cache failed' 2024-08-30 12:45:20 +02:00
Srigovind Nayak
f6e8d92590 changelog: update changelog 2024-08-30 12:44:54 +02:00
Srigovind Nayak
a8032c932c cache: remove redundant index file cleanup
addressing code review comments
2024-08-30 12:43:13 +02:00
Srigovind Nayak
8a7ae17d4d Revert "repository: removed redundant prepareCache method from Repository"
This reverts commit 720609f8ba.
2024-08-30 12:43:13 +02:00
Srigovind Nayak
0ca9355bc0 cache: add test for the automated cache clear to cache backend 2024-08-30 12:43:13 +02:00
Srigovind Nayak
b10d7ccdda changelog: add unrelease changelog 2024-08-30 12:43:13 +02:00
Srigovind Nayak
1e68fbca90 repository: removed redundant prepareCache method from Repository
* remove the prepareCache method from the Repository
* changed the signature of the SetIndex function to no longer return an error
2024-08-30 12:43:13 +02:00
Srigovind Nayak
0cf1737289 cache: check for context cancellation before clearing cache 2024-08-30 12:43:13 +02:00
Srigovind Nayak
fac1d9fea1 cache: backend add List method and a cache clear functionality
* removes files which are no longer in the repository, including index files, snapshot files and pack files from the cache.

cache: fix ids set initialisation with NewIDSet()
2024-08-30 12:43:13 +02:00
Michael Terry
48e3832322 main: return an exit code (12) for "bad password" errors 2024-08-30 12:39:22 +02:00
Michael Eischer
61e1f4a916 backend: return correct error on upload/request timeout 2024-08-30 12:37:10 +02:00
aneesh-n
7642e05eed Add test cases and handle volume GUID paths
Gracefully handle errors while checking for EA and add debug logs.
2024-08-30 12:37:10 +02:00
aneesh-n
51fad2eecb Gracefully handle invalid prepared volume names 2024-08-30 12:37:10 +02:00
aneesh-n
111490b8be Add changelog 2024-08-30 12:37:10 +02:00
aneesh-n
8861421cd6 Fix extended attributes handling for VSS snapshots 2024-08-30 12:37:10 +02:00
Andreas Deininger
c83b529c47 Fix typos 2024-08-30 12:37:10 +02:00
Shivashis Padhi
d15e693045 restore: Add progress bar to 'restore --verify' 2024-08-30 12:37:10 +02:00
Michael Eischer
4fcedb4bae backup: support specifying volume instead of path on Windows
"C:" (volume name) versus "C:\" (path)
2024-08-30 11:35:43 +02:00
Michael Eischer
a0f2dfbc19 Merge pull request #5019 from MichaelEischer/fix-windows-sd-race
backup: Fix spurious "A Required Privilege Is Not Held by the Client" error
2024-08-29 16:59:06 +02:00
Michael Eischer
0aadfe32bb Merge pull request #5018 from MichaelEischer/rest-retry-http2-goaway
rest: improve handling of HTTP2 goaway
2024-08-29 16:58:04 +02:00
Michael Eischer
dab3e549af Merge pull request #5017 from MichaelEischer/rewrite-data-loss
rewrite: Document handling of "cannot encode tree" errors
2024-08-29 16:57:13 +02:00
Michael Eischer
5c238ea359 Merge pull request #5016 from MichaelEischer/s3-doc-rework
Rework documentation for s3-compatible storages
2024-08-29 16:55:40 +02:00
Michael Eischer
2c85d2468a Merge pull request #5015 from MichaelEischer/update-exit-code-docs
Update exit code docs
2024-08-29 16:53:14 +02:00
Michael Eischer
7bbf75237d Merge pull request #5014 from MichaelEischer/configurable-slow-request-timeout
Make timeout for slow requests configurable
2024-08-29 16:52:24 +02:00
Michael Eischer
dd90e1926b use OrderedListOnceBackend where possible 2024-08-29 16:35:48 +02:00
Michael Eischer
d19f706d50 Add temporary files repositories in integration tests
This is intended to catch problems with temporary files stored in the
backend, even if the responsible component forgets to test for those.
2024-08-29 16:33:18 +02:00
Michael Eischer
8eff4e0e5c cache: correctly ignore files whose filename is no ID
this can for example be the case for temporary files created by the
backend implementation.
2024-08-29 16:32:15 +02:00
Michael Eischer
45d05eb691 add changelog for security descriptor race condition 2024-08-26 19:43:18 +02:00
Michael Eischer
9c70794886 fs: fix error handling for retried get/set of security descriptor
The retry code path did not filter `ERROR_NOT_SUPPORTED`. Just call the
original function a second time to correctly follow the low privilege
code path.
2024-08-26 19:36:43 +02:00
Michael Eischer
6fbfccc2d3 fs: fix race condition in get/set security descriptor
Calling `Load()` twice for an atomic variable can return different
values each time. This resulted in trying to read the security
descriptor with high privileges, but then not entering the code path to
switch to low privileges when another thread has already done so
concurrently.
2024-08-26 19:31:21 +02:00
Michael Eischer
1931beab8e Merge pull request #5012 from MichaelEischer/fix-lock-retries
lock: introduce short delay between failed locking retries
2024-08-26 18:10:30 +02:00
Michael Eischer
2296fdf668 lock: introduce short delay between failed locking retries
Failed locking attempts were immediately retried up to three times
without any delay between the retries. If a lock file is not found while
checking for other locks, with the reworked backend retries there is no
delay between those retries. This is a problem if a backend requires a
few seconds to reflect file deletions in the file listings. To work
around this problem, introduce a short exponentially increasing delay
between the retries. The number of retries is now increased to 4. This
results in delays of 5, 10 and 20 seconds between the retries.
2024-08-26 16:31:42 +02:00
Michael Eischer
89d216ca76 Merge pull request #5011 from MichaelEischer/fix-canceled-retry
backend/retry: don't trip circuit breaker if context is canceled
2024-08-26 16:30:03 +02:00
Michael Eischer
5cffd40002 Merge pull request #5013 from MichaelEischer/group-cli-commands
Group CLI commands and show features/options
2024-08-26 16:23:39 +02:00
Michael Eischer
e24dd5a162 backend/retry: don't trip circuit breaker if context is canceled
When the context used for a load operation is canceled, then the result
is always an error independent of whether the file could be retrieved
from the backend. Do not false positively trip the circuit breaker in
this case.

The old behavior was problematic when trying to lock a repository. When
`Lock.checkForOtherLocks` listed multiple lock files in parallel and one
of them fails to load, then all other loads were canceled. This
cancelation was remembered by the circuit breaker, such that locking
retries would fail.
2024-08-26 16:22:21 +02:00
Michael Eischer
2063bf5de4 Merge pull request #5006 from MichaelEischer/restore-time-last
restic: restore timestamps after extended attributes
2024-08-26 16:21:02 +02:00
Michael Eischer
36c4475ad9 rest: improve handling of HTTP2 goaway
The HTTP client can only retry HTTP2 requests after receiving a GOAWAY
response if it can rewind the body. As we use a custom data type,
explicitly provide an implementation of `GetBody`.
2024-08-26 15:44:17 +02:00
Michael Eischer
dc5d3fc473 doc: full tree blob data structure is in the code 2024-08-26 14:41:09 +02:00
Michael Eischer
05077eaa20 doc: JSON encoder must be deterministic 2024-08-26 14:41:09 +02:00
Michael Eischer
908d097904 doc: mark S3 layout as deprecated 2024-08-26 14:41:09 +02:00
Michael Eischer
828c8bc1e8 doc: describe how to handle rewrite encoding error 2024-08-26 14:41:09 +02:00
Michael Eischer
b8f409723d make timeout for slow requests configurable 2024-08-26 14:14:43 +02:00
Michael Eischer
8a8f5f3986 doc: fix typos 2024-08-26 12:24:02 +02:00
Michael Eischer
7de53a51b8 doc: shrink wasabi / alibaba cloud example
Remove descriptions for both providers and shorten the example to the
minimum.
2024-08-26 12:21:13 +02:00
Michael Eischer
9649a9c62b doc: use regional urls for Amazon S3 and add generic s3 provider section
Split description for non-Amazon S3 providers into separate section. The
section now also includes the `s3.bucket-lookup` extended option. Switch
to using regional URLs for Amazon S3 to replace the need for setting the
region.
2024-08-26 12:17:43 +02:00
Michael Eischer
354c2c38cc doc/backup: move exit status codes section up 2024-08-25 23:53:12 +02:00
Michael Eischer
ff9ef08f65 doc/backup: link to exit code for scripting section 2024-08-25 23:52:33 +02:00
Michael Eischer
311b27ced8 restic: cleanup redundant code in test case 2024-08-25 23:18:55 +02:00
Michael Eischer
43b36ad2b0 restore: test timestamps for macOS resource forks are restored correctly 2024-08-25 23:18:55 +02:00
Michael Eischer
2e55209b34 restic: restore timestamps after extended attributes
restoring the xattr containing resource forks on macOS apparently
modifies the file modification timestamps. Thus, restore the timestamp
after xattrs.
2024-08-25 23:18:55 +02:00
Michael Eischer
e7db5febcf update docs 2024-08-23 23:52:21 +02:00
Michael Eischer
7739aa685c Add missing DisableAutoGenTag flag for commands 2024-08-23 23:49:20 +02:00
Michael Eischer
5988d825b7 group commands and make features/options visible 2024-08-23 23:48:45 +02:00
Michael Eischer
a8efaee03c Merge pull request #5010 from MichaelEischer/cleanup-cli-help
Improve description for  --from-insecure-no-password option
2024-08-23 23:41:08 +02:00
Michael Eischer
8672cef972 Merge pull request #5009 from restic/document-restic-host
Mention RESTIC_HOST environment variable in docs
2024-08-23 23:40:48 +02:00
Michael Eischer
551dfee707 Improve description for no password on secondary repo 2024-08-18 19:45:54 +02:00
Michael Eischer
1b8ca32e7d Mention RESTIC_HOST environment variable in docs 2024-08-18 19:41:58 +02:00
Michael Eischer
489af2a670 Merge pull request #5008 from mikix/doc-typo
docs: correct wrong exit_error message field name
2024-08-18 17:38:31 +00:00
Michael Terry
97df01b9b8 docs: correct wrong exit_error message field name 2024-08-17 15:00:39 -04:00
Michael Eischer
68f7abcff1 Merge pull request #5007 from deining/fix-warnings
GitHub test actions: fix warnings 'Restore cache failed'
2024-08-17 14:31:41 +00:00
Andreas Deininger
ceb45d9816 GitHub test actions: fix warnings 'Restore cache failed' 2024-08-17 12:39:41 +02:00
Michael Eischer
5cca6e66be Merge pull request #4981 from konidev20/fix-gh-4934-cleanup-removed-snaphots-from-cache
cache: clear snapshot files from cache during load index
2024-08-16 19:04:59 +00:00
Srigovind Nayak
c9097994b9 changelog: update changelog 2024-08-17 00:24:19 +05:30
Michael Eischer
c636ad51a8 Merge pull request #4959 from mikix/fatal-wrap
main: return an exit code (12) for "bad password" errors
2024-08-16 18:52:36 +00:00
Srigovind Nayak
88174cd0a4 cache: remove redundant index file cleanup
addressing code review comments
2024-08-17 00:21:49 +05:30
Srigovind Nayak
b7d014b685 Revert "repository: removed redundant prepareCache method from Repository"
This reverts commit 720609f8ba.
2024-08-17 00:18:13 +05:30
Michael Terry
56f28c9bd5 main: return an exit code (12) for "bad password" errors 2024-08-15 16:55:45 -04:00
Michael Eischer
7462471c6b Merge pull request #4952 from mikix/json-exit
Format exit errors as JSON if requested
2024-08-15 20:19:38 +00:00
Michael Eischer
74d3f92cc7 Merge pull request #4993 from MichaelEischer/fix-timeout-error
backend: return correct error on upload/request timeout
2024-08-15 22:07:37 +02:00
Michael Eischer
80f24584a5 Merge pull request #4998 from zmanda/ea_vss_fix
Fix extended attributes handling for VSS snapshots
2024-08-15 19:51:35 +00:00
Michael Eischer
8e00158c34 Merge pull request #5000 from deining/fix-typo
Fix typos
2024-08-15 19:42:14 +00:00
Michael Eischer
36b5580c1c Merge pull request #4989 from plant99/progress-bar-for-restore-verify
restore: Add progress bar to 'restore --verify'
2024-08-15 19:34:05 +00:00
aneesh-n
19f487750e Add test cases and handle volume GUID paths
Gracefully handle errors while checking for EA and add debug logs.
2024-08-11 19:25:58 -06:00
Shivashis Padhi
f1407afd1f restore: Add progress bar to 'restore --verify' 2024-08-11 22:25:21 +02:00
Andreas Deininger
4401265e36 Fix typos 2024-08-11 21:38:15 +02:00
Srigovind Nayak
5fd984ba6f cache: add test for the automated cache clear to cache backend 2024-08-11 23:41:07 +05:30
Srigovind Nayak
506e07127f changelog: add unrelease changelog 2024-08-11 23:41:07 +05:30
Srigovind Nayak
720609f8ba repository: removed redundant prepareCache method from Repository
* remove the prepareCache method from the Repository
* changed the signature of the SetIndex function to no longer return an error
2024-08-11 23:41:07 +05:30
Srigovind Nayak
a23e7bfb82 cache: check for context cancellation before clearing cache 2024-08-11 23:41:07 +05:30
Srigovind Nayak
f66624f5bf cache: backend add List method and a cache clear functionality
* removes files which are no longer in the repository, including index files, snapshot files and pack files from the cache.

cache: fix ids set initialisation with NewIDSet()
2024-08-11 23:40:52 +05:30
Michael Terry
d3f9c05312 docs: update scripting documentation 2024-08-11 12:52:54 -04:00
Michael Terry
6283915f86 main: format exit errors as JSON when using --json 2024-08-11 12:52:50 -04:00
Michael Terry
2d250a9135 version: add message_type in --json mode 2024-08-11 12:51:15 -04:00
Michael Eischer
33c670dd7a Merge pull request #4996 from restic/dependabot/go_modules/google.golang.org/api-0.191.0
build(deps): bump google.golang.org/api from 0.189.0 to 0.191.0
2024-08-11 09:25:19 +00:00
aneesh-n
849c441455 Gracefully handle invalid prepared volume names 2024-08-11 01:48:25 -06:00
aneesh-n
b5b5c1fe8e Add changelog 2024-08-11 01:32:55 -06:00
aneesh-n
1d392a36f9 Fix extended attributes handling for VSS snapshots 2024-08-11 01:23:47 -06:00
dependabot[bot]
049186371f build(deps): bump google.golang.org/api from 0.189.0 to 0.191.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.189.0 to 0.191.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.189.0...v0.191.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-08-10 18:17:20 +00:00
Michael Eischer
910f64ce47 Merge pull request #4997 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/azcore-1.14.0
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.13.0 to 1.14.0
2024-08-10 18:11:28 +00:00
Michael Eischer
b3b71e78cd Merge pull request #4995 from restic/dependabot/go_modules/golang.org/x/crypto-0.26.0
build(deps): bump golang.org/x/crypto from 0.25.0 to 0.26.0
2024-08-10 18:08:20 +00:00
dependabot[bot]
f2e2e5f5ab build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore
Bumps [github.com/Azure/azure-sdk-for-go/sdk/azcore](https://github.com/Azure/azure-sdk-for-go) from 1.13.0 to 1.14.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.13.0...sdk/azcore/v1.14.0)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azcore
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-08-10 17:58:58 +00:00
dependabot[bot]
ecd03b4fc6 build(deps): bump golang.org/x/crypto from 0.25.0 to 0.26.0
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.25.0 to 0.26.0.
- [Commits](https://github.com/golang/crypto/compare/v0.25.0...v0.26.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-08-10 17:58:50 +00:00
Michael Eischer
3f5e2160de Merge pull request #4938 from MichaelEischer/bump-go-version
Bump go version to 1.21
2024-08-10 19:57:59 +02:00
Michael Eischer
283225f15f Merge pull request #4994 from MichaelEischer/less-flaky-watchdog-test
backend: tweak timeouts to make watchdog timeout test less flaky
2024-08-10 19:35:38 +02:00
Michael Eischer
400ae55940 replace deprecated usages of math/rand 2024-08-10 19:34:49 +02:00
Michael Eischer
84c79f1456 bump required go version to 1.21 2024-08-10 19:16:10 +02:00
Michael Eischer
0b19f6cf5a Switch back to sha256 from the std library
The std library now also supports the sha assembly instructions on
ARM64. Thus, sha256-simd no longer provides a performance benefit.
2024-08-10 19:16:10 +02:00
Michael Eischer
fbecc9db66 upgrade all direct dependencies 2024-08-10 19:16:10 +02:00
Michael Eischer
ad48751adb bump required go version to 1.21 2024-08-10 19:16:10 +02:00
Michael Eischer
86390b453d Merge pull request #4980 from zmanda/unsupported-ea-handling
Skip ExtendedAttribute processing in Windows for volumes that do not support EA
2024-08-10 17:11:12 +00:00
Michael Eischer
fa35e72214 backend: tweak timeouts to make watchdog timeout test less flaky 2024-08-10 19:08:03 +02:00
Michael Eischer
05571286b2 Merge pull request #4946 from mikix/json-errors
Improve error handling in --json mode
2024-08-10 17:04:56 +00:00
Michael Eischer
4ee3c9c8b9 Merge pull request #4951 from MichaelEischer/better-completions-poc
Improve CLI completions POC
2024-08-10 18:40:57 +02:00
aneesh-n
18e9d71d7a Fix review comments 2024-08-10 10:38:04 -06:00
Michael Eischer
853a686994 backend: return correct error on upload/request timeout 2024-08-10 18:06:24 +02:00
Michael Eischer
a164789321 Merge pull request #4977 from MichaelEischer/warn-on-incomplete-metadata
backup: warn but store store item if extended metadata is incomplete
2024-08-10 17:59:25 +02:00
Michael Eischer
09fd599057 Merge pull request #4973 from MichaelEischer/err-on-unknown-migration
migrate: show warning if migration is unknown
2024-08-10 15:43:58 +02:00
Michael Eischer
fb815abca5 Merge pull request #4979 from MichaelEischer/debug-log-flaky-mount-test
mount: enable debug log for TestMountSameTimestamps
2024-08-10 15:43:31 +02:00
aneesh-n
71632a8197 Handle extended length paths 2024-08-05 16:03:43 -06:00
aneesh-n
85639f5159 Add handling for relative paths, vss paths, UNC paths 2024-08-04 13:19:13 -06:00
aneesh-n
c13725b5d0 Check EA support only for volumes, files and dirs 2024-08-04 11:05:40 -06:00
aneesh-n
89712f6640 Formatted 2024-08-04 10:36:13 -06:00
aneesh-n
9dedba6dfc Address review comments 2024-08-04 10:23:39 -06:00
aneesh-n
8c8a066c0e Correct the bitmask for fileSupportsExtendedAttributes and add link 2024-08-03 18:06:47 -06:00
aneesh-n
041c0705e4 Add changelog 2024-08-03 16:19:59 -06:00
aneesh-n
56113a8da7 Skip EA processing for volumes that do not support EA 2024-08-03 16:03:30 -06:00
Michael Eischer
73c9780321 backup: store but warn if extended metadata for item is incomplete
Files were not included in the backup if the extended metadata for the
file could not be read. This is rather drastic. Instead settle on
returning a warning but still including the file in the backup.
2024-08-03 22:25:29 +02:00
Michael Terry
88f59fc2d6 json: switch backup and restore errors from string to struct types
This keeps backwards compatibility with the previous empty structs.
And maybe we'd want to put other fields into the inner struct later,
rather than the outer message.
2024-08-03 16:21:44 -04:00
Michael Eischer
03be64a094 mount: enable debug log for TestMountSameTimestamps
The test has started to become flaky.
2024-08-03 22:17:07 +02:00
Michael Eischer
a48baf6f3a Merge pull request #4976 from textaligncenter/backup-panic
fix panic in fs_reader
2024-08-03 19:56:15 +00:00
Michael Terry
a376323331 restore: print JSON versions of errors in --json mode
Previously, they were printed as freeform text.

This also adds a ui.Terminal interface to make writing
tests easier and also adds a few tests.
2024-08-03 15:18:46 -04:00
Michael Eischer
e622135e7e Merge pull request #4974 from MichaelEischer/fix-cache-debug-log
cache: fix confusing debug log
2024-08-03 21:12:32 +02:00
textaligncenter
d8ea178e69 fix panic in fs_reader 2024-08-03 19:08:04 +00:00
Michael Terry
ad2585af67 backup: show actual error strings in --json mode
Previously, an error JSON fragment would look like:
{"message_type": "error", "error": {}}

This is because encoding/json cannot marshal an error interface.
Instead, we now call .Error() to get the string value.
2024-08-03 15:07:52 -04:00
Michael Terry
f4bdfea1c9 backup: print scanner errors to stderr, not stdout 2024-08-03 15:07:52 -04:00
Michael Eischer
d2f7c5a9c6 Merge pull request #4978 from konidev20/fix-gh-4949-repair-index-spurious-index
rewrite: skip saving empty indexes during MasterIndex.Rewrite
2024-08-03 18:53:57 +00:00
Srigovind Nayak
068d5b95c3 rewrite: skip saving empty indexes during MasterIndex.Rewrite 2024-08-03 23:34:59 +05:30
Michael Eischer
d4db5a364e Merge pull request #4958 from mikix/restore-errors
restore: clean up error handling when restoring metadata
2024-08-03 17:58:54 +00:00
Michael Eischer
f3af264674 Merge pull request #4967 from restic/dependabot/github_actions/docker/login-action-3.3.0
build(deps): bump docker/login-action from 3.2.0 to 3.3.0
2024-08-03 17:58:52 +00:00
Michael Eischer
4266dca1b6 cache: fix confusing debug log 2024-08-03 18:51:38 +02:00
Michael Eischer
d407abb50f Merge pull request #4960 from MichaelEischer/reduce-cancelation-delays
Add more checks for canceled contexts
2024-08-03 18:23:20 +02:00
Michael Eischer
3faeddcd5f Merge pull request #4954 from MichaelEischer/improve-windows-path-handling
Improve windows path handling
2024-08-03 18:22:39 +02:00
Michael Eischer
1c775feecc add changelog for cancelation delay 2024-08-03 18:13:24 +02:00
Michael Eischer
b3bfb5ed44 add changelog for long paths on windows 2024-08-03 18:09:28 +02:00
Michael Eischer
db77919550 migrate: show warning if migration is unknown 2024-08-03 17:22:56 +02:00
dependabot[bot]
7b423a0915 build(deps): bump docker/login-action from 3.2.0 to 3.3.0
Bumps [docker/login-action](https://github.com/docker/login-action) from 3.2.0 to 3.3.0.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](0d4c9c5ea7...9780b0c442)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-08-01 01:11:48 +00:00
Michael Eischer
a639454f28 diff: improve cancelation handling 2024-07-31 19:45:33 +02:00
Michael Eischer
ae1cb889dd Add more checks for canceled contexts 2024-07-31 19:30:47 +02:00
Michael Terry
6a97833337 restore: clean up error handling when restoring metadata
- Fix a logic error that instead of reporting the *first*
  metadata-setting error that appears, we were instead reporting the
  *last* error (and only if the lchown call failed!).
- Don't show any errors when setting metadata for files in non-root
  mode (things like timestamps, attributes). Previously, only lchown
  errors were skipped. But other kinds of attribute errors make sense
  to skip as well. The code path happened to work correctly before
  because of the above logic error. But once that was fixed, this
  change needed to happen too.
2024-07-30 19:27:34 -04:00
rawtaz
8d5e188218 Merge pull request #4956 from northben/patch-2
Update faq.rst
2024-07-30 15:27:24 +00:00
Ben Northway
98c73eeca9 Update faq.rst
the correct output of `echo "~/documents"` is `~/documents`
2024-07-30 09:11:18 -06:00
Michael Eischer
a9be986782 restorer: add minimal long path handling test 2024-07-29 21:11:47 +02:00
Michael Eischer
62c4a5e9a0 fs: fix long path handling for ResetPermissions, ClearAttributes and security descriptors 2024-07-29 20:10:58 +02:00
Michael Eischer
7448a15f72 stats: add --mode completions 2024-07-28 12:21:42 +02:00
Michael Eischer
bb50d86e68 cat: add subcommand completion 2024-07-28 12:21:30 +02:00
Michael Eischer
76d56e24d6 Merge pull request #4939 from MichaelEischer/fix-stdin-command-docs
docs: clarify how to pass arguments using backup --stdin-from-command
2024-07-26 20:17:00 +00:00
Michael Eischer
d4b28cea6c docs: clarify how to pass arguments using backup --stdin-from-command 2024-07-26 20:12:26 +02:00
Alexander Neumann
ebc15b8680 Set development version for 0.17.0 2024-07-26 11:30:50 +02:00
Alexander Neumann
277c8f5029 Add version for 0.17.0 2024-07-26 11:30:37 +02:00
Alexander Neumann
65748fcd6b Update manpages and auto-completion 2024-07-26 11:30:37 +02:00
Alexander Neumann
905da91e61 Generate CHANGELOG.md for 0.17.0 2024-07-26 11:29:39 +02:00
Alexander Neumann
ad09d21aaf Prepare changelog for 0.17.0 2024-07-26 11:28:55 +02:00
Michael Eischer
aa69b15f3e Merge pull request #4936 from restic/828-fix-changelog
doc: Remove unrelated PR URL from changelog entry
2024-07-25 16:48:31 +00:00
Leo R. Lundgren
5a8c915443 doc: Remove unrelated PR URL from changelog entry 2024-07-25 14:27:43 +02:00
Michael Eischer
2be8793de2 Merge pull request #4933 from MichaelEischer/backup-source
docs: Replace all uses of target in relation to backup
2024-07-24 21:08:39 +02:00
Michael Eischer
fe2d1dfd84 Merge pull request #4932 from restic/polish-changelogs
doc: Polish unreleased changelogs
2024-07-24 18:47:03 +00:00
Michael Eischer
636bf2915e minor tweaks 2024-07-24 20:40:15 +02:00
Michael Eischer
729e009485 replace all uses of target in relation to backup 2024-07-24 19:40:53 +02:00
Michael Eischer
0ddff92323 Merge pull request #4930 from MichaelEischer/update-xattr-lib
update xattr library to 0.4.10
2024-07-24 19:06:56 +02:00
Michael Eischer
4d6cd8f069 Merge pull request #4931 from restic/doc-vss-options
doc: Clarify VSS extended options
2024-07-23 20:54:38 +00:00
Leo R. Lundgren
77ec8ae091 doc: Polish unreleased changelogs 2024-07-23 00:51:57 +02:00
Leo R. Lundgren
c54d5a8eca doc: Clarify VSS extended options 2024-07-23 00:37:56 +02:00
Michael Eischer
948c6f1da3 update xattr library to 0.4.10 2024-07-22 22:16:26 +02:00
Michael Eischer
fa5ff0873a Merge pull request #4925 from MichaelEischer/fix-restore-truncation
restore: Fix truncation of uptodate files
2024-07-21 12:28:12 +02:00
Michael Eischer
b5d0586b26 Merge pull request #4926 from MichaelEischer/restore-size-only-files
restorer: only show size in text output for files
2024-07-21 12:06:15 +02:00
Michael Eischer
0dcac90bea restorer: don't track already uptodate blobs 2024-07-21 12:03:56 +02:00
Michael Eischer
10efa77103 restorer: add test for file truncation case 2024-07-21 12:03:56 +02:00
Michael Eischer
6f8e17a463 restorer: minor code cleanups 2024-07-21 12:03:56 +02:00
Michael Eischer
5d6ce59a8d restorer: also truncate files if their content is already uptodate
Files for which no blobs have to be restored, still have to be truncated
to the correct size. Take a file with content "foobar" that should be
replaced by restore with content "foo". The first three bytes are
already uptodate, such that no data has to be written. As file
truncation normally happens when writing data, a special case is
necessary.

This no blobs written special case is unified with the empty file
special case.
2024-07-21 12:03:56 +02:00
Michael Eischer
59c880fce3 restorer: only show size in text output for files 2024-07-20 12:46:28 +02:00
Michael Eischer
4f4598a1e3 Merge pull request #4905 from alex-kattathra-johnson/issue-4902
Update snapshot summary on rewrite
2024-07-16 18:06:15 +00:00
Alex Johnson
3bf2927006 Update snapshot summary on rewrite
Signed-off-by: Alex Johnson <hello@alex-johnson.net>
2024-07-16 12:06:50 -05:00
Michael Eischer
1a45f05e19 Merge pull request #4921 from MichaelEischer/restorer-bugs
restore: fix cancelation and partial updates of large files
2024-07-14 11:38:58 +02:00
Michael Eischer
dcfffd7779 restore: extend overwrite test for small files 2024-07-14 11:30:41 +02:00
Michael Eischer
44e3610b32 restore: progress bar total on windows 2024-07-14 11:21:11 +02:00
Michael Eischer
4a9536b829 amend restore overwrite changelog 2024-07-14 11:21:11 +02:00
Michael Eischer
98cfb2c4c8 restore: test progress reporting for partially up to date files 2024-07-14 11:21:11 +02:00
Michael Eischer
26aa65e0d4 restore: add regression test for corrupt in-place restore of large file 2024-07-14 11:21:11 +02:00
Michael Eischer
2833b2f699 restore: fix progress bar for partially up to date files 2024-07-14 11:21:11 +02:00
Michael Eischer
c77b2d5ca2 restore: avoid long cancelation delay for frequently used blobs 2024-07-14 11:21:11 +02:00
Michael Eischer
f64191da9c restore: improve reporting of cancelation errors 2024-07-14 11:21:08 +02:00
Michael Eischer
2976a666c3 Merge pull request #4920 from MichaelEischer/fix-ls-ncdu-toplevel
ls: fix handling of toplevel directories in ncdu output
2024-07-14 11:09:57 +02:00
Michael Eischer
bdf24dc8f9 ls: extend test sample with a second toplevel file 2024-07-13 10:22:01 +02:00
Michael Eischer
5671dfb481 ls: fix ncdu output for file with timestamp before 1970 2024-07-13 10:21:02 +02:00
Michael Eischer
ff9238ebf1 ls: use / as top-level dir name in ncdu output 2024-07-12 23:46:50 +02:00
Michael Eischer
2971a769da restore: fix corrupt restore of partially up to date files 2024-07-12 22:27:00 +02:00
Michael Eischer
40e5163114 restore: properly cancel file verification 2024-07-12 22:26:22 +02:00
Michael Eischer
31887ea9aa restore: fix hang on command cancelation 2024-07-12 22:24:35 +02:00
Michael Eischer
9f66065237 ls: fix handling of toplevel directories in ncdu output 2024-07-12 21:49:35 +02:00
Michael Eischer
375c572c4f Merge pull request #4884 from MichaelEischer/lock-and-not-exist-error-codes
Return different exit code if repo is locked or does not exist
2024-07-12 21:05:52 +02:00
Michael Eischer
6010c98b3f Merge pull request #4918 from MichaelEischer/restore-dont-delete-target-if-file
restore: do not delete target if it is a file
2024-07-12 21:05:04 +02:00
Michael Eischer
36cc62075c add exit code changelog 2024-07-10 22:13:37 +02:00
Michael Eischer
54ca27a6f2 doc: update init check with exit code 10 2024-07-10 22:13:37 +02:00
Michael Eischer
1328cdd002 Merge pull request #4917 from MichaelEischer/dry-run-speed-up-hint
restore: document how to speed up dry-run
2024-07-10 20:12:39 +00:00
Michael Eischer
8731667156 restore: do not delete target if it is a file
This is implicitly achieved by creating the target directory as the
first step, which will fail if a file already exist in its place.
2024-07-10 22:08:49 +02:00
Michael Eischer
294c8524ed doc: document exit codes 2024-07-10 21:47:24 +02:00
Michael Eischer
94fdca08c4 return exit code 10 if repository does not exist 2024-07-10 21:46:26 +02:00
Michael Eischer
d76a78aa99 return exit code 11 if the repository could not be locked 2024-07-10 21:37:32 +02:00
Michael Eischer
3b7228c8c1 explicitly document exit code 1 on error 2024-07-10 21:35:46 +02:00
Michael Eischer
f560152627 restore: document how to speed up dry-run 2024-07-10 21:34:10 +02:00
Michael Eischer
36a2857df6 Merge pull request #4916 from restic/dependabot/go_modules/google.golang.org/grpc-1.64.1
build(deps): bump google.golang.org/grpc from 1.64.0 to 1.64.1
2024-07-10 20:28:11 +02:00
Michael Eischer
5e1d0ffd99 Merge pull request #4911 from MichaelEischer/fix-filtered-ls-ncdu
ls: Fix --ncdu output with path filters and fix disk size
2024-07-10 20:27:42 +02:00
dependabot[bot]
8c7a2f1792 build(deps): bump google.golang.org/grpc from 1.64.0 to 1.64.1
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.64.0 to 1.64.1.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.64.0...v1.64.1)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-07-10 18:23:27 +00:00
Michael Eischer
d8dbc71deb Merge pull request #4908 from MichaelEischer/improve-anonymous-s3
S3: Improve anonymous access
2024-07-10 20:19:23 +02:00
Michael Eischer
5067a40bd8 update ncdu changelog 2024-07-10 20:18:53 +02:00
Michael Eischer
79c9fc45a9 ls: fix disk size in --ncdu output
Ncdu expects a size in disk blocks. Round up to the next full block.
Otherwise, small files with size below 255bytes would be rounded down to
0B by ncdu.
2024-07-10 20:18:53 +02:00
Michael Eischer
15419d603d ls: add missing intermediate directories to --ncdu output 2024-07-10 20:18:53 +02:00
Michael Eischer
dc0db4eda4 add s3 anonymous authentication changelog entry 2024-07-10 20:10:27 +02:00
Michael Eischer
f74e70cc36 s3: forbid anonymous authentication unless explicitly requested 2024-07-10 20:10:27 +02:00
Michael Eischer
7cea5402ff Merge pull request #4913 from MichaelEischer/always-show-snapshot-size
snapshots: also show snapshot size in compact view
2024-07-10 20:05:41 +02:00
Michael Eischer
a8fc67d68b Merge pull request #4914 from drdo/minor-correction-design-doc
doc: Update index explanation to match the example
2024-07-09 18:46:47 +00:00
Daniel Oliveira
94c3d3f097 doc: Update index explanation to match the example
In commit 00575ec the example was changed to three data blobs due to the deprecation of mixed pack files
but the following description was not updated to reflect this.
2024-07-09 00:18:09 +01:00
Michael Eischer
f4ecb9d615 Merge pull request #4909 from MichaelEischer/properly-set-dev-version
Fix VERSION file and dev version in binaries
2024-07-08 20:15:20 +02:00
Michael Eischer
c309ffb35b Merge pull request #4912 from MichaelEischer/restore-delete-preserve-root
restore: forbid --target / --delete unless include/exclude is specified
2024-07-08 20:14:44 +02:00
Michael Eischer
f980f5647e snapshots: also show snapshot size in compact view 2024-07-08 20:00:26 +02:00
Michael Eischer
b71a885b92 Merge pull request #4907 from MichaelEischer/bump-dependencies
Bump some dependencies
2024-07-08 19:32:25 +02:00
Michael Eischer
24a7ff45f2 Merge pull request #4904 from MichaelEischer/fix-status-cleanup
ui/termstatus: fix clearing status lines
2024-07-08 19:31:55 +02:00
Michael Eischer
00d9ce7bee fix setting version in build script 2024-07-08 19:30:17 +02:00
Michael Eischer
a0cac7fcd1 helpers/prepare-release: write dev version to VERSION file 2024-07-07 14:04:09 +02:00
Michael Eischer
1856f4d87a restore: forbid --target / --delete unless include/exclude is specified 2024-07-07 13:54:45 +02:00
Michael Eischer
894ec9d05d ls: fix broken folder if --ncdu an file filters are combined 2024-07-07 13:08:41 +02:00
Michael Eischer
4b364940aa s3: use http client with configured timeouts for s3 IAM communication
The default client has no timeouts configured opening network
connections. Thus, if 169.254.169.254 is inaccessible, then the client
would wait for until the operating system gives up, which will take
several minutes.
2024-07-07 11:32:40 +02:00
Michael Eischer
a2a2401a68 s3: prevent repeated credential queries with anonymous authentication 2024-07-07 11:31:04 +02:00
Michael Eischer
be98402ac6 Merge pull request #4906 from joram-berger/patch-1
Add explanation to restic diff symbols
2024-07-06 19:07:08 +00:00
Michael Eischer
10765151d3 doc: fine tune description of diff command 2024-07-06 21:01:58 +02:00
Michael Eischer
f3a3c46153 bump some dependencies 2024-07-06 17:34:28 +02:00
Joram Berger
33de00bba0 Add explanation to restic diff symbols
Explain what the characters for each file in a restic diff output mean.
2024-07-06 16:47:04 +02:00
Michael Eischer
aedead2823 ui/termstatus: fix clearing status lines
To clear the status lines, they should be set to an empty array to
prevent future updates of those lines. Setting the status lines to an
array containing an empty string is wrong as this causes the output to
continuously add that empty status line after each message.
2024-07-06 11:27:35 +02:00
Michael Eischer
8e27a934de Merge pull request #4881 from MichaelEischer/restore-delete-actual
restore: add `--delete` option
2024-07-05 22:52:54 +02:00
Michael Eischer
54316978cd add restore --delete changelog 2024-07-05 22:38:39 +02:00
Michael Eischer
569f111cb1 restore: document --delete option 2024-07-05 22:38:39 +02:00
Michael Eischer
868219aad1 restore: test --dry-run plus --delete 2024-07-05 22:38:39 +02:00
Michael Eischer
aa8e18cf32 restore: add deletions to progress output 2024-07-05 22:38:39 +02:00
Michael Eischer
f4b15fdd96 restore: allow deleting a directory to replace it with a file
When the `--delete` option is specified, recursively delete directories
that should be replaced with a file.
2024-07-05 22:38:39 +02:00
Michael Eischer
168fc09d5f restore: use case insensitive file name comparison on windows 2024-07-05 22:38:39 +02:00
Michael Eischer
013a6156bd restore: remove unused parameter from SelectFilter 2024-07-05 22:38:39 +02:00
Michael Eischer
ac44bdf6dd restore: add --delete option to remove files that are not in snapshot 2024-07-05 22:38:39 +02:00
Michael Eischer
144e2a451f restore: track expected filenames in a folder 2024-07-05 21:03:35 +02:00
Michael Eischer
d762f4ee64 restore: simplfy selectFilter arguments 2024-07-05 21:03:35 +02:00
Michael Eischer
a9a60f77ce restore: optimize memory usage 2024-07-05 21:03:35 +02:00
Michael Eischer
1369658a32 archiver: extract Readdirnames to fs package 2024-07-05 21:03:34 +02:00
Michael Eischer
83fdcf21fe Merge pull request #4839 from MichaelEischer/restore-dry-run
restore: add `--dry-run` and extended progress output
2024-07-05 21:02:13 +02:00
Michael Eischer
a90665028f doc: fix syntax error 2024-07-05 20:41:27 +02:00
Michael Eischer
ae978d60cc restore: pass action enum to restore progress 2024-07-05 20:41:27 +02:00
Michael Eischer
798256ec52 restore: add dry-run docs 2024-07-05 20:41:27 +02:00
Michael Eischer
83351f42e3 restore: add dry-run support 2024-07-05 20:41:27 +02:00
Michael Eischer
c47bf33884 restore: print processed files at maximum verbosity 2024-07-05 20:41:27 +02:00
Michael Eischer
2a7d257036 Merge pull request #4882 from MichaelEischer/improve-check-output
check: improve output if repository is damaged
2024-07-05 20:16:45 +02:00
Michael Eischer
283038056e doc: suggest to follow troubleshooting steps if check reports error 2024-07-05 20:04:25 +02:00
Michael Eischer
fdc7349aa4 check: improve error on damaged index
Always return the `repository contains errors` message if a repository
is damaged and must be repaired. Also provide specific instructions how
to repair the index.
2024-07-05 20:04:25 +02:00
Michael Eischer
a03e00373c update repair packs changelog 2024-07-05 20:04:25 +02:00
Michael Eischer
879ba07a87 check: only show additional files if verbose output is enabled
Additional files are nearly always caused by interrupted backup runs.
This is unproblematic, thus don't pollute the check output with it.
2024-07-05 20:04:25 +02:00
Michael Eischer
6d9dfff1cb check: point users towards the troubleshooting guide if repo has errors 2024-07-05 20:04:25 +02:00
Michael Eischer
c9a4a95848 check: suggest using repair packs to repair truncated pack files
Previously, that help message was only shown for running `check
--read-data`.
2024-07-05 20:04:25 +02:00
Michael Eischer
da338d5aa8 check: tweak wording of repair packs message
By now, the message is also shown for truncated or otherwise damaged pack
files, not just those with corrupted blobs.
2024-07-05 20:04:25 +02:00
Michael Eischer
1671a3fe2e check: hide message about additional files if error in repo
The message says "[...] addition files were found [...]. This is
non-critical [...]". Unless users are highly experienced with restic,
it's hard to correctly interpret what "This" refers to. Thus, just hide
the message if there is a real problem.
2024-07-05 20:04:25 +02:00
Michael Eischer
b1250eead9 check: mark s3legacy layout and legacy indexes are error 2024-07-05 20:04:25 +02:00
Michael Eischer
66a67d568d Merge pull request #4885 from MichaelEischer/clarify-path-option
Clarify semantics of `--path` option and update manual_rest.rst
2024-07-05 19:53:15 +02:00
Michael Eischer
737389ea68 Merge pull request #4896 from MichaelEischer/fix-subfolder-help-text
man: work around broken rendering of <snapshotID>:<subfolder> help text
2024-07-04 22:06:51 +02:00
Michael Eischer
6fadc0131b Merge pull request #4883 from MichaelEischer/fix-check-cache
check: fix cachedir creation when using default location
2024-07-04 22:06:28 +02:00
Michael Eischer
a44e4bdd40 man: work around broken rendering of <snapshotID>:<subfolder> help text 2024-07-03 20:39:59 +02:00
Michael Eischer
639326903c Merge pull request #4893 from szepeviktor/typos
Fix typos
2024-07-03 18:09:40 +00:00
Viktor Szépe
ac00229386 Fix typos 2024-07-03 20:02:06 +02:00
Michael Eischer
278ec55d3f Merge pull request #4894 from eltociear/patch-1
docs: update 045_working_with_repos.rst
2024-07-03 17:26:39 +00:00
Ikko Eltociear Ashimine
a60643de9c docs: update 045_working_with_repos.rst
colums -> columns
2024-07-03 14:31:11 +09:00
Michael Eischer
a72b418319 Merge pull request #4889 from restic/dependabot/go_modules/github.com/spf13/cobra-1.8.1
build(deps): bump github.com/spf13/cobra from 1.8.0 to 1.8.1
2024-07-01 21:02:47 +00:00
Michael Eischer
9e8d7c3ad4 Merge pull request #4887 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/azidentity-1.7.0
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azidentity from 1.6.0 to 1.7.0
2024-07-01 21:00:58 +00:00
dependabot[bot]
f02cb8eb62 build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azidentity
Bumps [github.com/Azure/azure-sdk-for-go/sdk/azidentity](https://github.com/Azure/azure-sdk-for-go) from 1.6.0 to 1.7.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.6.0...sdk/azcore/v1.7.0)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azidentity
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-07-01 20:50:05 +00:00
Michael Eischer
7a21924d72 Merge pull request #4891 from restic/dependabot/go_modules/golang.org/x/oauth2-0.21.0
build(deps): bump golang.org/x/oauth2 from 0.20.0 to 0.21.0
2024-07-01 20:49:49 +00:00
Michael Eischer
cef24bdcf4 Merge pull request #4890 from restic/dependabot/go_modules/github.com/klauspost/compress-1.17.9
build(deps): bump github.com/klauspost/compress from 1.17.8 to 1.17.9
2024-07-01 20:45:57 +00:00
Michael Eischer
97969e515d Merge pull request #4886 from restic/dependabot/github_actions/docker/build-push-action-6
build(deps): bump docker/build-push-action from 5 to 6
2024-07-01 20:39:44 +00:00
Michael Eischer
74ef20b406 Merge pull request #4888 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/azcore-1.12.0
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.11.1 to 1.12.0
2024-07-01 20:36:04 +00:00
Michael Eischer
c3ad56474d Merge branch 'patch-release' 2024-07-01 21:49:39 +02:00
Alexander Neumann
5820dccaef Set development version for 0.16.5 2024-07-01 21:25:34 +02:00
Alexander Neumann
fe9f142b52 Add version for 0.16.5 2024-07-01 21:25:34 +02:00
Alexander Neumann
6ae760751a Generate CHANGELOG.md for 0.16.5 2024-07-01 21:25:33 +02:00
Alexander Neumann
2fa1b42706 Prepare changelog for 0.16.5 2024-07-01 21:25:33 +02:00
dependabot[bot]
e939c9fd6c build(deps): bump golang.org/x/oauth2 from 0.20.0 to 0.21.0
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.20.0 to 0.21.0.
- [Commits](https://github.com/golang/oauth2/compare/v0.20.0...v0.21.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-07-01 01:41:05 +00:00
dependabot[bot]
59b5eacd0f build(deps): bump github.com/klauspost/compress from 1.17.8 to 1.17.9
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.17.8 to 1.17.9.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml)
- [Commits](https://github.com/klauspost/compress/compare/v1.17.8...v1.17.9)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-07-01 01:40:59 +00:00
dependabot[bot]
3de98f11f0 build(deps): bump github.com/spf13/cobra from 1.8.0 to 1.8.1
Bumps [github.com/spf13/cobra](https://github.com/spf13/cobra) from 1.8.0 to 1.8.1.
- [Release notes](https://github.com/spf13/cobra/releases)
- [Commits](https://github.com/spf13/cobra/compare/v1.8.0...v1.8.1)

---
updated-dependencies:
- dependency-name: github.com/spf13/cobra
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-07-01 01:40:50 +00:00
dependabot[bot]
6171389803 build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore
Bumps [github.com/Azure/azure-sdk-for-go/sdk/azcore](https://github.com/Azure/azure-sdk-for-go) from 1.11.1 to 1.12.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.11.1...sdk/azcore/v1.12.0)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azcore
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-07-01 01:40:44 +00:00
dependabot[bot]
bda7aba97d build(deps): bump docker/build-push-action from 5 to 6
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 5 to 6.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v5...v6)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-07-01 01:15:18 +00:00
Michael Eischer
aa33af3e75 update command help docs 2024-06-30 20:34:36 +02:00
Michael Eischer
ed904dfac0 clarify that all paths specified using --path option must match 2024-06-30 20:32:36 +02:00
Michael Eischer
ce0cbc7a36 check: fix cachedir creation when using default location 2024-06-30 12:04:57 +02:00
Michael Eischer
132965fd5c Merge pull request #4864 from MichaelEischer/restore-delete
restore: improve file type mismatch handling
2024-06-26 21:27:57 +02:00
Michael Eischer
9817126341 Merge pull request #4863 from MichaelEischer/safer-repo-init2
repository: prevent initialization if a snapshot exists
2024-06-26 21:11:11 +02:00
Michael Eischer
ca04a88e65 Merge pull request #4879 from restic/backport-azure-cli-option
Backport azure cli option
2024-06-26 21:07:16 +02:00
Michael Eischer
12e858b7af azure: deduplicate cli and default credentials case 2024-06-26 20:59:51 +02:00
Maik Riechert
834f08fe2d Azure: add option to force use of CLI credential 2024-06-26 20:59:51 +02:00
Michael Eischer
814ef4901f Merge pull request #4878 from restic/patch-deps-bump
Bump dependencies
2024-06-26 20:58:29 +02:00
Michael Eischer
84bc9432de update release verification script for latest docker 2024-06-26 20:49:23 +02:00
Michael Eischer
e9d711422a bump azure, golang and gcs dependencies 2024-06-26 20:49:23 +02:00
Michael Eischer
e37c11979a Merge pull request #4866 from MichaelEischer/fix-ea-windows
Fix xattr backup on windows
2024-06-26 20:46:24 +02:00
Michael Eischer
c01b655229 backup: fix reading xattrs on Windows for root disks as regular user
Unprivileged users cannot open the root disks with write permissions for
xattrs. Thus, only request read permissions for reading the xattrs.
2024-06-26 20:13:50 +02:00
Michael Eischer
20a6c526e5 restic: also fix file path when reading windows xattrs 2024-06-14 22:40:25 +02:00
Michael Eischer
b80aa7b1cc repository: prevent initialization if a snapshot exists 2024-06-14 20:37:01 +02:00
Michael Eischer
deca7d08ac restorer: cleanup unexpected xattrs on windows 2024-06-14 20:17:06 +02:00
Michael Eischer
9572b7224f restorer: windows test fixes 2024-06-13 23:20:09 +02:00
Michael Eischer
ca41c8fd11 restore: use fs function wrappers
This ensures proper path handling on Windows.
2024-06-13 22:40:37 +02:00
Michael Eischer
3d73ae9988 update restore changelog 2024-06-13 22:32:53 +02:00
Michael Eischer
4d6042fe95 restore: remove unexpected xattrs from files 2024-06-13 22:21:00 +02:00
Michael Eischer
ebbd4e26d7 restorer: allow directory to replace existing file 2024-06-13 21:57:48 +02:00
Michael Eischer
ac729db3ce restorer: fix overwriting of special file types
An attempt to replace an existing file with a hardlink previously ended
with a missing file.

Remove an existing file before trying to restore a special node. This
generalizes the existing behavior for symlinks to all special node
types.
2024-06-13 21:56:18 +02:00
Michael Eischer
c7902b7724 restorer: cleanup overwrite tests 2024-06-13 21:55:33 +02:00
Michael Eischer
c598a751c2 restore: fine-grained sparse support for windows 2024-06-13 21:55:33 +02:00
Michael Eischer
d265ec64f2 restore: correctly handle existing hardlinks
With hardlinks there's no efficient way to detect which files are linked
with each other. Thus, just start from scratch when restore has to
modify a hardlinked file.
2024-06-13 21:55:33 +02:00
Michael Eischer
f19b69af25 restore: allow overwrite to replace empty directories and symlinks
With an already existing file tree an old directory or symlink may exist
in a place where restore wants to create a new file. Thus, check for
unexpected file types and clean up if necessary.
2024-06-13 21:55:33 +02:00
Michael Eischer
267cd62ae4 Merge pull request #4838 from MichaelEischer/restore-skip-unchanged
restore: skip unchanged files and add `--overwrite if-changed` option
2024-06-13 21:26:04 +02:00
Michael Eischer
d890bfff7c doc: suggest backup before in-place restore 2024-06-13 21:17:30 +02:00
Michael Eischer
f1c76a8286 restore: fix corrupted sparse files 2024-06-13 21:17:30 +02:00
Michael Eischer
1eccd6504b restore: test restore only changed parts functionality 2024-06-13 21:17:30 +02:00
Michael Eischer
3ec28ff853 update restore --overwrite documentation 2024-06-13 21:17:30 +02:00
Michael Eischer
5c3709e17a restore: add --overwrite=if-changed to skip files if their mtime&size matches
--overwrite=always still checks the file content
2024-06-13 21:17:30 +02:00
Michael Eischer
a66658b4c9 restore: only restore changed file parts
For files that should be overwritten, first check whether their content
matches the snapshot and only restore diverging parts.
2024-06-13 21:17:30 +02:00
Michael Eischer
63ea017e67 Merge pull request #4861 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/azidentity-1.6.0
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azidentity from 1.5.2 to 1.6.0
2024-06-13 17:50:10 +00:00
Michael Eischer
62e0e5e8d8 restore: cleanup options handling 2024-06-12 22:53:53 +02:00
Michael Eischer
8733cd21b9 restore: fix file encryption on windows
The assumption that a file is not readonly is no longer valid.
2024-06-12 22:53:53 +02:00
Michael Eischer
8aa3ab6ef8 restore: check file type in verify 2024-06-12 22:53:53 +02:00
Michael Eischer
663151db57 Merge pull request #4837 from MichaelEischer/restore-options
Make restore overwrite behavior configurable
2024-06-12 22:52:55 +02:00
Michael Eischer
7f7c995977 fix linter warnings 2024-06-12 22:36:52 +02:00
Michael Eischer
105261e12e add changelog for restore --overwrite 2024-06-12 22:36:52 +02:00
Michael Eischer
e1ec60c2ee document restore --overwrite 2024-06-12 22:36:52 +02:00
Michael Eischer
ba53a2abb5 test overwrite behavior 2024-06-12 22:36:52 +02:00
Michael Eischer
e47e08a688 restorer: separately track skipped files 2024-06-12 22:36:52 +02:00
Michael Eischer
64b7b6b975 restore/ui: refactor for extensibility 2024-06-12 22:36:52 +02:00
Michael Eischer
6a4ae9d6b1 restore: configurable overwrite behavior 2024-06-12 22:36:52 +02:00
Michael Eischer
a23cb3a428 restore: reduce memory usage 2024-06-12 22:36:52 +02:00
Michael Eischer
2b50c2606c restorer: use options struct 2024-06-12 22:36:52 +02:00
Michael Eischer
0fcd89f892 restorer: remove special case for blobs with many occurrences
Loading blobs by now is no longer prone to timeouts when processing
takes a long time.
2024-06-12 22:36:52 +02:00
Michael Eischer
fd2ff464a2 restorer: remove stale comment 2024-06-12 22:36:52 +02:00
Michael Eischer
607daeed4f restore: move nil pointer check into restoreui 2024-06-12 22:36:52 +02:00
Michael Eischer
30320a249a restore: let filerestorer also handle empty files
This get's rid of the corresponding special cases.
2024-06-12 22:36:52 +02:00
Michael Eischer
c166ad7daf restore: factor out file creation helper 2024-06-12 22:36:52 +02:00
dependabot[bot]
1e7bc6dab8 build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azidentity
Bumps [github.com/Azure/azure-sdk-for-go/sdk/azidentity](https://github.com/Azure/azure-sdk-for-go) from 1.5.2 to 1.6.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/internal/v1.5.2...sdk/azcore/v1.6.0)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azidentity
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-06-12 20:09:17 +00:00
Michael Eischer
f834c1f08a Merge pull request #4825 from MichaelEischer/bump-fuse
mount: support fuse-t on macOS
2024-06-12 20:33:05 +02:00
Michael Eischer
058292700c mount: support fuse-t on macOS 2024-06-11 22:52:08 +02:00
Michael Eischer
be05a17e15 Merge pull request #4811 from konidev20/fix-gh-4781-restore-include-and-exclude-file-switches
Restore flags to include from file and exclude from file
2024-06-10 19:36:07 +00:00
Srigovind Nayak
fe412e2553 fix: restore inclusion logic and restore tests
doc: update exclude and include docs
2024-06-10 01:55:39 +05:30
Srigovind Nayak
188684ee9e fix: include and exclude logic, add tests for include file and exclude file 2024-06-09 19:07:23 +05:30
Michael Eischer
fc3841e7cc Merge pull request #4851 from MichaelEischer/fix-utf-16-key
key add/passwd: handle UTF-16 encoding correctly
2024-06-09 11:27:58 +02:00
Michael Eischer
da4512738a Merge pull request #4853 from MichaelEischer/safer-repo-init
init: double check that no repository exists yet
2024-06-09 11:27:45 +02:00
Michael Eischer
7861fc1942 Merge pull request #4856 from MichaelEischer/fix-verify-script
update release verification script for latest docker
2024-06-09 11:27:35 +02:00
Michael Eischer
8440b94159 Merge pull request #4807 from zmanda/windows-extendedattribs
Back up and restore Extended Attributes on Windows NTFS
2024-06-09 09:18:04 +00:00
Michael Eischer
bab760369f repository: double check that there is not repository before init
Apparently, calling `Stat` on the config file can be unreliable for some
backends.
2024-06-09 00:05:32 +02:00
Srigovind Nayak
24a247a0dc test: update test case name 2024-06-08 13:23:35 +05:30
Srigovind Nayak
3a52176121 restore: accumulate results of multiple pattern checks
addressing review comments
2024-06-08 13:23:35 +05:30
Srigovind Nayak
e579dfe72a doc: update documentation for restore command 2024-06-08 13:23:35 +05:30
Srigovind Nayak
2a2c09e666 changelog: update changelog for gh-4781 2024-06-08 13:23:35 +05:30
Srigovind Nayak
1a7574e4b4 test: add tests for include By pattern 2024-06-08 13:23:35 +05:30
Srigovind Nayak
4e449ffaff test: add tests for reading patterns from file 2024-06-08 13:23:35 +05:30
Srigovind Nayak
14d2799b44 fix: move include and exclude pattern validations to top 2024-06-08 13:23:35 +05:30
Srigovind Nayak
fdf2e4ed0e restore: refactor include and exclude
- added includePatternOptions similar to excludePatternOptions
- followed similar approach to backup for selecting files for restore
2024-06-08 13:23:35 +05:30
Srigovind Nayak
7d5dd6db66 fix: add string.Lower for insenstive includes and excludes read from file 2024-06-08 13:23:35 +05:30
Srigovind Nayak
b02117ef0b restore: read includes, insensitive includes, excludes and insensitive excludes from a file
feature for gh-4781
2024-06-08 13:23:35 +05:30
Michael Eischer
78485160fc Merge pull request #4849 from fthoma/table-tcwidth
Use character display width for table padding
2024-06-07 19:53:14 +00:00
Michael Eischer
7cd324fe26 ui/table: avoid duplicate table cell width calculation 2024-06-07 21:45:40 +02:00
Michael Eischer
edd3e214c2 ui/table: fix width calculation of multi-line column headers 2024-06-07 21:44:49 +02:00
Michael Eischer
08c43d2c7e update release verification script for latest docker 2024-06-07 21:00:10 +02:00
Michael Eischer
3f878aa8e7 Merge pull request #4845 from greatroar/errors
Fix error handling bug + clean up error messages
2024-06-07 17:07:07 +00:00
Michael Eischer
112de8526d Merge pull request #4852 from MichaelEischer/master
crypto: cleanup ciphertext verification error
2024-06-07 19:06:08 +02:00
Michael Eischer
0a70bbcea5 Merge pull request #4844 from MichaelEischer/improve-timeout-error
backend: Improve timeout error message
2024-06-07 19:05:39 +02:00
aneesh-n
4f053da06a Fix test case by correcting cleanup order 2024-06-05 23:10:29 -06:00
aneesh-n
2101dfe448 Add missing return to fix failing test 2024-06-05 22:40:21 -06:00
aneesh-n
7a48c9ebd7 Fix review comments 2024-06-05 16:06:57 -06:00
Michael Eischer
b82ecc6425 crypto: cleanup ciphertext verification error 2024-06-05 22:33:20 +02:00
Michael Eischer
c55665be2c key add/passwd: handle UTF-16 encoding correctly
Just use the exact some function for load a password from a file
everywhere.
2024-06-05 22:19:26 +02:00
Michael Eischer
ef85a81e83 Merge pull request #4847 from rawtaz/azure-error-msg
azure: Improve error message in azure.Create()
2024-06-05 20:05:57 +00:00
Florian Thoma
e9de9684f4 Use character display width for table padding
Using len(...) for table cell padding produced wrong results for unicode
chracters leading to misaligned tables. Implementation changed to take
the actual terminal display width into consideration.
2024-06-05 09:33:15 +02:00
Leo R. Lundgren
b2bbbe805f azure: Improve error message in azure.Create() 2024-06-03 23:37:17 +02:00
greatroar
10fdb914df cmd: Return error in readPassword
The returned error was always nil.

Replaced Wrap by WithStack because the function name was stale.
2024-06-02 10:11:27 +02:00
Michael Eischer
660679c2f6 Merge pull request #4835 from MichaelEischer/fix-list-cancel
backend/retry: do not log final error if context was canceled
2024-06-01 19:17:30 +02:00
Michael Eischer
db2398f35b backend: increase request progress timeout to 5 minutes
Apparently, 2 minutes are too short in some cases and can result in
canceled List requests.
2024-06-01 19:01:51 +02:00
Michael Eischer
6bf3d4859f backend: improve error on http request timeout
Now yields a "request timeout" error instead of "context canceled".
2024-06-01 18:52:39 +02:00
greatroar
4a874000b7 gs: Replace some errors.Wrap calls
The first one in Create is already a WithStack error. The rest were
referencing code that hasn't existed for quite some time. Note that
errors from Google SDKs tends to start with "google:" or "googleapi:".

Also, use restic/internal/errors.
2024-06-01 15:11:06 +02:00
Michael Eischer
456ebfb0c7 Merge pull request #4840 from restic/dependabot/go_modules/google.golang.org/api-0.182.0
build(deps): bump google.golang.org/api from 0.181.0 to 0.182.0
2024-06-01 13:06:49 +00:00
Michael Eischer
830e87059a Merge pull request #4842 from restic/dependabot/github_actions/docker/login-action-3.2.0
build(deps): bump docker/login-action from 3.1.0 to 3.2.0
2024-06-01 13:06:46 +00:00
Michael Eischer
a26beb9c1c Merge pull request #4843 from restic/dependabot/github_actions/golangci/golangci-lint-action-6
build(deps): bump golangci/golangci-lint-action from 5 to 6
2024-06-01 13:06:43 +00:00
dependabot[bot]
46b0fac11e build(deps): bump golangci/golangci-lint-action from 5 to 6
Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 5 to 6.
- [Release notes](https://github.com/golangci/golangci-lint-action/releases)
- [Commits](https://github.com/golangci/golangci-lint-action/compare/v5...v6)

---
updated-dependencies:
- dependency-name: golangci/golangci-lint-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-06-01 01:48:46 +00:00
dependabot[bot]
fced29b479 build(deps): bump docker/login-action from 3.1.0 to 3.2.0
Bumps [docker/login-action](https://github.com/docker/login-action) from 3.1.0 to 3.2.0.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](e92390c5fb...0d4c9c5ea7)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-06-01 01:48:43 +00:00
dependabot[bot]
05d7885a87 build(deps): bump google.golang.org/api from 0.181.0 to 0.182.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.181.0 to 0.182.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.181.0...v0.182.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-06-01 01:02:01 +00:00
Michael Eischer
38654a3bd7 backend/retry: do not log final error if context was canceled
Calls to `List(ctx, ...)` are usually stopped by canceling the context
once no further entries are required by the caller. Thus, don't log the
final error if the used context was canceled.
2024-05-30 18:48:52 +02:00
Michael Eischer
2280fbfd2e Merge pull request #4810 from konidev20/fix-gh-4768-add-custom-user-agent-for-http-client
Allow custom User-Agent to be specified for outgoing requests
2024-05-30 16:34:52 +00:00
Michael Eischer
7e0ee5974f Merge pull request #4816 from MichaelEischer/skip-if-unchanged
backup: add support for `--skip-if-unchanged`
2024-05-30 15:39:08 +02:00
Srigovind Nayak
de7b418bbe http: allow custom User-Agent for outgoing HTTP requests 2024-05-30 15:38:06 +02:00
Michael Eischer
c103c0830e add changelog for skip-if-unchanged 2024-05-30 15:25:44 +02:00
Michael Eischer
513135b238 doc: slightly modernize backup examples 2024-05-30 15:25:44 +02:00
Michael Eischer
cdd210185d Merge pull request #4823 from MichaelEischer/fix-check-output
Fix mangled check output
2024-05-30 15:17:27 +02:00
Michael Eischer
5649334099 backup: document --skip-if-unchanged 2024-05-30 15:14:32 +02:00
Michael Eischer
6869bdaaa8 backup: implement --skip-if-unchanged 2024-05-30 15:14:31 +02:00
Michael Eischer
13127f3f6b check: remove output mangling workaround 2024-05-30 15:09:27 +02:00
Michael Eischer
939b537c80 check/migrate: convert to use termstatus to prevent mangled output
Errors reported by check would result in corrupted output.
2024-05-30 15:09:27 +02:00
Michael Eischer
7b4f81d964 Merge pull request #4829 from greatroar/termstatus
termstatus: Do formatting in Message, not Terminal
2024-05-30 11:57:44 +00:00
Michael Eischer
a5c7a83470 Merge pull request #4819 from restic/update-dependencies
Update dependencies
2024-05-30 13:55:46 +02:00
Michael Eischer
52b3fee42b Merge pull request #4818 from MichaelEischer/move-code
Move cache / index / hashing / pack packages
2024-05-30 13:54:31 +02:00
Michael Eischer
e95cfff923 Merge pull request #4826 from MichaelEischer/fix-docs-warninigs
doc: fix warnings
2024-05-30 13:53:49 +02:00
Michael Eischer
fe8fbcc5ed Merge pull request #4822 from MichaelEischer/consistent-backup-source-name
backup: refer to backed up data as backup source not target
2024-05-30 13:53:25 +02:00
Michael Eischer
3d4a620089 Merge pull request #4824 from MichaelEischer/fix-cache-race
bloblru: Fix flaky test due to race condition
2024-05-30 13:52:34 +02:00
greatroar
46f04bc5f0 termstatus: Do formatting in Message, not Terminal 2024-05-27 19:05:39 +02:00
Michael Eischer
a651af70d6 doc: fix warings 2024-05-26 14:32:09 +02:00
Michael Eischer
1c6067d93d bloblru: variable name cleanup 2024-05-26 12:42:46 +02:00
Michael Eischer
21ce03cff2 bloblru: move defer outside critical section 2024-05-26 12:38:20 +02:00
Michael Eischer
0ad0b7ca7c bloblru: fix race condition that can compute value multiple times 2024-05-26 12:37:24 +02:00
Michael Eischer
619ce8bfa2 Merge pull request #4820 from MichaelEischer/forget-extend-descritpion
forget: explain oldest snapshot special case in command help
2024-05-26 10:04:43 +00:00
Michael Eischer
107f18849c doc: update help output in manual_rest.rst 2024-05-25 23:16:22 +02:00
Michael Eischer
18990dbed8 backup: refer to backed up data as backup source not target
A backup reads from the source and writes it to the backup target
(repository). The name "target" is kept within the source code as it's
used there excessively.
2024-05-25 23:13:37 +02:00
Michael Eischer
3828313974 Merge pull request #4821 from MichaelEischer/master
update `repair packs` changelog
2024-05-25 17:02:21 +02:00
Michael Eischer
04c181dbd0 update-repair-packs-changelog 2024-05-25 16:42:55 +02:00
Michael Eischer
21528c9f02 Merge pull request #4805 from plant99/mkdirall-cachedir
prepareCheckCache function: Add MkdirAll() to restic-cache directory if it doesn't exist
2024-05-25 14:30:21 +00:00
Shivashis Padhi
0271bb97f8 check: enhance check command to create cache directory if it doesn't exist 2024-05-25 19:15:42 +05:30
Michael Eischer
2c7ebf4c7f forget: explain oldest snapshot special case in command help 2024-05-25 15:10:21 +02:00
Michael Eischer
7ea508e7b8 Merge pull request #4815 from greatroar/termstatus
Termstatus refactoring
2024-05-25 12:08:53 +00:00
Michael Eischer
496e57f956 hashing: move to repository package 2024-05-25 13:13:03 +02:00
Michael Eischer
5e0ea8fcfa pack: move to repository package 2024-05-25 13:13:03 +02:00
Michael Eischer
50ec408302 index: move to repository package 2024-05-25 13:13:03 +02:00
Michael Eischer
58bbb27b4c downgrade minio to keep go 1.19 support 2024-05-25 12:52:08 +02:00
Michael Eischer
c5260373ca update backoff/xxhash libraries 2024-05-25 12:37:14 +02:00
Michael Eischer
b2f7a878cd update backend libraries
minio-go is kept at 7.0.67 as later versions require Go 1.21.
2024-05-25 12:36:16 +02:00
Michael Eischer
f20bf5eeba update golang.org/x packages 2024-05-25 12:35:28 +02:00
Michael Eischer
8e5d7d719c cache: move to backend package 2024-05-24 23:04:06 +02:00
Michael Eischer
80132e71d8 Merge pull request #4808 from MichaelEischer/insecure-no-password
Implement `--insecure-no-password` option.
2024-05-24 22:58:25 +02:00
Michael Eischer
130506250f document insecure-no-password 2024-05-24 22:38:20 +02:00
Michael Eischer
1d2277b4c3 Add --insecure-no-password option
This also includes two derived options `--from-insecure-no-password`
used for commands that require specifying a source repository. And
`--new-insecure-no-password` for the `key add` and `key passwd`
commands.

Specifying `--insecure-no-password` disabled the password prompt and
immediately uses an empty password. Passing a password via CLI option or
environment variable at the same time is an error.
2024-05-24 22:38:20 +02:00
Michael Eischer
d4b0d21199 key add/passwd: deduplicate options setup and remove globals
The current pattern of using a global options variable is problematic.
2024-05-24 22:38:20 +02:00
Michael Eischer
55cb8d174a Merge pull request #4354 from MichaelEischer/associated-data
Reduce prune memory usage
2024-05-24 22:26:22 +02:00
Michael Eischer
436afbff23 add changelog for memory efficient prune 2024-05-24 22:18:14 +02:00
Michael Eischer
3c7b7efdc9 repository: remove prune plan parts once they are no longer necessary 2024-05-24 22:18:14 +02:00
Michael Eischer
462b82a060 index: reduce size of compressed indexes
use the same index size for compressed and uncompressed indexes.
Otherwise, decoding the index of a compressed repository requires
significantly more memory.
2024-05-24 22:18:14 +02:00
Michael Eischer
77873f5a9d repository: let prune control data structure of usedBlobs set 2024-05-24 22:18:14 +02:00
Michael Eischer
2033c02b09 index: replace CountedBlobSet with AssociatedSet 2024-05-24 22:18:14 +02:00
Michael Eischer
93098e9265 prune: hide implementation details of counted blob set 2024-05-24 21:42:56 +02:00
Michael Eischer
ff4775a15f Merge pull request #4812 from MichaelEischer/streaming-index-rewrite
Resumable prune & memory-efficient index rewrite
2024-05-24 21:41:30 +02:00
Michael Eischer
860b595a8b backend: increase watchdog test timeout for deflaking 2024-05-24 21:33:17 +02:00
Michael Eischer
f680a2331d add changelog for streaming index rewrite 2024-05-24 21:33:17 +02:00
Michael Eischer
027cc64737 repository: fix prune heuristic to allow resuming interrupted runs
Pack files created by interrupted prune runs, appear to consist only of
duplicate blobs on the next run. This caused the previous heuristic to
ignore those pack files. Now, a duplicate blob in a specific pack file
is also selected if that pack file only contains duplicate blobs. This
allows prune to select the already rewritten pack files.
2024-05-24 21:33:17 +02:00
Michael Eischer
e52033a8bd index: slightly reduce Rewrite concurrency
The index operations are likely CPU-bounded. Thus, reduce the
concurrency accordingly.
2024-05-24 21:33:17 +02:00
Michael Eischer
57d69aa640 index: cleanup SaveIndex method 2024-05-24 21:33:17 +02:00
Michael Eischer
2ca1c37721 index: additional tests for new index save methods 2024-05-24 21:33:17 +02:00
Michael Eischer
5f7b48e65f index: replace Save() method with Rewrite and SaveFallback
Rewrite implements a streaming rewrite of the index that excludes the
given packs. For this it loads all index files from the repository and
only modifies those that require changes. This will reduce the index
churn when running prune. Rewrite does not require the in-memory index
and thus can drop it to significantly reduce the memory usage.

However, `prune --unsafe-recovery` cannot use this strategy and requires
a separate method to save the whole in-memory index. This is now handled
using SaveFallback.
2024-05-24 21:33:17 +02:00
Michael Eischer
ad98fbf7dd restic: add IDSet.Clone() method 2024-05-24 21:33:17 +02:00
Michael Eischer
72482ce5bd index: misc cleanups 2024-05-24 21:33:17 +02:00
Michael Eischer
68fa0e0305 prune: no longer disable automatic index updates
this allows prune to resume an interrupted prune run.
2024-05-24 21:33:17 +02:00
Michael Eischer
9aa0c90fb2 index: remove supersedes field
Using the field with its current semantics is nearly impossible to get
right. Remove it as it will be replaced anyways in repository format 3.
2024-05-24 21:33:17 +02:00
Michael Eischer
76e6719f2e repository: make CreateIndexFromPacks method private 2024-05-24 21:33:17 +02:00
Michael Eischer
04ad9f0c0c repository: remove Packer and SavePacker from public interface 2024-05-24 21:33:17 +02:00
Michael Eischer
550d1eeac3 repository: remove SaveIndex from interface
The method is now only indirectly accessible via Prune or RepairIndex.
2024-05-24 21:33:17 +02:00
Michael Eischer
fb59e00614 index: rewrite MasterIndex load/save test to be independent of repository 2024-05-24 21:33:17 +02:00
Michael Eischer
447b486c20 index: deduplicate index loading of check and repository 2024-05-24 21:33:17 +02:00
Michael Eischer
6ca12c1b4a archiver: replace most uses of restic.Repository 2024-05-24 21:33:17 +02:00
Michael Eischer
864995271e repository: unwrap BlobHandle parameters of LookupBlob
The method now uses the same parameters as LookupBlobSize.
2024-05-24 21:33:17 +02:00
Michael Eischer
1266a4932f repository: fix parameter order of LookupBlobSize
All methods should use blobType followed by ID.
2024-05-24 21:33:17 +02:00
Michael Eischer
e848ad651a restic: name parameters in restic interface 2024-05-24 21:33:17 +02:00
Michael Eischer
0bb0720348 test cleanups 2024-05-24 21:33:17 +02:00
Michael Eischer
c01bcb1001 archiver: remove unused masterIndex from test 2024-05-24 21:33:17 +02:00
Michael Eischer
0aa5c53842 repository: replace HasBlob with LookupBlobSize 2024-05-24 21:33:17 +02:00
Michael Eischer
8f1e70cd9b repository: remove clearIndex and packSize from public interface 2024-05-24 21:33:17 +02:00
Michael Eischer
4df887406f repository: inline MasterIndex interface into Repository interface 2024-05-24 21:33:17 +02:00
Michael Eischer
3eeb6723cd Merge pull request #4764 from MichaelEischer/safe-keep-tag
Prevent unsafe uses of `forget --keep-tag`
2024-05-24 20:51:20 +02:00
Michael Eischer
3cc6827f09 forget: add feature flag for safe --keep-tags behavior 2024-05-24 20:45:33 +02:00
Michael Eischer
7948912b31 document --unsafe-allow-remove-all option 2024-05-24 20:45:33 +02:00
Michael Eischer
3f46808898 add forget safety net changelogs 2024-05-24 20:45:33 +02:00
Michael Eischer
82ca0030b7 forget: test --unsafe-allow-remove-all and --keep-tags safety check 2024-05-24 20:45:33 +02:00
Michael Eischer
57f9739573 forget: Add --unsafe-allow-remove-all option
To prevent accidentally wiping all snapshots from a repository, that
option can only be used if either a snapshot filter or a keep policy is
specified.

Essentially, the option allows `forget --tag something
--unsafe-allow-remove-all` calls to remove all snapshots with a specific
tag.
2024-05-24 20:45:33 +02:00
Michael Eischer
5b7952e426 forget: return error if no policy was specified 2024-05-24 20:45:33 +02:00
Michael Eischer
c0e1f36830 forget: refuse deleting the last snapshot in a snapshot group
`--keep-tag invalid-tag` was previously able to wipe all snapshots in a
repository. As a user specified a `--keep-*` option this is likely
unintentional. This forbid deleting all snapshot if a `--keep-*` option
was specified to prevent data loss. (Not specifying such an option
currently also causes the command to abort)
2024-05-24 20:45:33 +02:00
Michael Eischer
d106ad6921 restic: regenerate snapshot keep policy golden test files 2024-05-24 20:45:33 +02:00
Michael Eischer
16ef4d515b Merge pull request #4784 from MichaelEischer/rework-backend-retries
Rework backend retries
2024-05-24 20:29:54 +02:00
Michael Eischer
e4a48085ae backend/retry: feature flag new retry behavior 2024-05-24 20:24:02 +02:00
Michael Eischer
723247c8e5 add changelog for longer retries 2024-05-24 20:24:02 +02:00
Michael Eischer
b1266867d2 repository: wait max 1 minutes for lock removal if context is canceled
The toplevel context in restic only canceled if the user interrupts a
restic operation. If the network connection has failed this can require
waiting the full retry duration of 15 minutes which is a bad user
experience for interactive usage. Thus limit the delay to one minute in
this case.
2024-05-24 20:24:02 +02:00
Michael Eischer
98709a4372 retry: reduce total number of retries
Retries in restic try to solve two main problems:
- retry a temporarily failed operation
- tolerate temporary network interruptions

The first problem only requires a few retries, whereas the last one benefits
primarily from spreading the requests over a longer duration.

Increasing the default multiplier and the initial interval works for
both cases. The first few retries only take a few seconds, while later
retries quickly reach the maximum interval of one minute. This ensures
that the total number of retries issued by restic will remain at around
21 retries for a 15 minute period. As the concurrency in restic is
bounded, retries drastically reduce the number of requests sent to a
backend. This helps to prevent overloading the backend.
2024-05-24 20:24:02 +02:00
Michael Eischer
512cd6ef07 retry: ensure that there's always at least one retry
Previously, if an operation failed after 15 minutes, then it would never
be retried. This means that large backend requests are more unreliable
than smaller ones.
2024-05-24 20:24:02 +02:00
Michael Eischer
a60ee9b764 retry: limit retries based on elapsed time not count
Depending on how long an operation takes to fail, the total retry
duration can currently vary between 1.5 and 15 minutes. In particular
for temporarily interrupted network connections, the former timeout is
too short. Thus always use a limit of 15 minutes.
2024-05-24 20:24:02 +02:00
Michael Eischer
a3633cad9e retry: explicitly log failed requests
This simplifies finding the request in the log output that cause an
operation to fail.
2024-05-24 20:24:02 +02:00
Michael Eischer
b9cbf623fa Merge pull request #4814 from MichaelEischer/fix-tmp-docs
doc: fix tmpdir documentation for windows
2024-05-24 18:17:59 +02:00
greatroar
66d03c797e ui, termstatus: Move WrapStdio
Saves some imports. ui still needs to import ui/termstatus from
message.go.
2024-05-21 11:25:16 +02:00
greatroar
0b56214473 ui: Simplify stdio wrapper
The StdioWrapper type is really just a pair of io.WriteClosers, so
remove it in favor of a function that returns two of those. Test
coverage increases because the removed code was not tested.
2024-05-21 11:23:32 +02:00
greatroar
7f439a9c34 ui: Inline lineWriter into StdioWrapper 2024-05-21 11:00:09 +02:00
Michael Eischer
4021e67d97 doc: fix tmpdir documentation for windows 2024-05-20 20:48:29 +02:00
Michael Eischer
8898f61717 Merge pull request #4809 from MichaelEischer/update-changelog
add retries for corrupted blobs to changelog
2024-05-18 23:04:13 +02:00
Michael Eischer
5f23baabcc add retries for corrupted blobs to changelog 2024-05-18 23:03:24 +02:00
Michael Eischer
9c5bac6f25 Merge pull request #4799 from letmaik/letmaik/azure-force-cli-credential
Azure: add option to force use of CLI credential
2024-05-18 20:22:15 +00:00
Michael Eischer
c56ecec9aa azure: deduplicate cli and default credentials case 2024-05-18 22:15:54 +02:00
Maik Riechert
355f520936 Azure: add option to force use of CLI credential 2024-05-18 22:15:54 +02:00
Michael Eischer
1dfe1b8732 Merge pull request #4802 from MichaelEischer/backend-cleanups
Repository: Remove Backend() method
2024-05-18 22:02:45 +02:00
Michael Eischer
223aa22cb0 replace some uses of restic.Repository with finegrained interfaces 2024-05-18 21:42:51 +02:00
Michael Eischer
291c9677de restic/repository: remove Backend() method 2024-05-18 21:42:51 +02:00
Michael Eischer
673496b091 repository: clean cache between CheckPack retries
The cache cleanup pattern is also used in ListPack etc.
2024-05-18 21:42:51 +02:00
Michael Eischer
3d2410ed50 Replace some repo.RemoveUnpacked usages
These will eventually be blocked as they do not delete Snapshots.
2024-05-18 21:42:51 +02:00
Michael Eischer
d2c26e33f3 repository: remove further usages of repo.Backend() 2024-05-18 21:42:51 +02:00
Michael Eischer
8a425c2f0a remove usages of repo.Backend() from tests 2024-05-18 21:42:51 +02:00
Michael Eischer
aa4647f773 repository: unexport PackBlobIterator 2024-05-18 21:42:51 +02:00
Michael Eischer
94e863885c check: move verification of individual pack file to repository 2024-05-18 21:42:50 +02:00
Michael Eischer
e40943a75d restic: remove backend usage from lock test 2024-05-18 21:38:31 +02:00
Michael Eischer
67e2ba0d40 repository: Lock requires *repository.Repository
This allows the Lock function to access the backend, even once the
Backend method is removed from the interface.
2024-05-18 21:38:31 +02:00
Michael Eischer
d8b184b3d3 repository: convert test helper to return *repository.Repository 2024-05-18 21:38:31 +02:00
Michael Eischer
a1ca5e15c4 migrations: add temporary hack for s3_layout
The migration will be removed after the next restic release anyways.
Thus, there's no need for a clean implementation.
2024-05-18 21:38:31 +02:00
Michael Eischer
34d90aecf9 migrations: move logic of upgrade_repo_v2 to repository package
The migration modifies repository internals and thus should live within
the repository package.
2024-05-18 21:38:31 +02:00
Michael Eischer
ab9077bc13 replace usages of backend.Remove() with repository.RemoveUnpacked()
RemoveUnpacked will eventually block removal of all filetypes other than
snapshots. However, getting there requires a major refactor to provide
some components with privileged access.
2024-05-18 21:38:31 +02:00
Michael Eischer
8274f5b101 prune: remove Backend.IsNotExist()
Only handling one specific error is not particularly useful.
2024-05-18 21:38:31 +02:00
Michael Eischer
9795198189 debug: remove Backend.Stat() usage 2024-05-18 21:38:31 +02:00
Michael Eischer
0c1ba6d95d backend: remove unused Location method 2024-05-18 21:38:31 +02:00
Michael Eischer
eb6c653f89 Merge pull request #4800 from MichaelEischer/cleanup-load
Retry loading of corrupted data from backend / cache
2024-05-18 21:34:54 +02:00
Michael Eischer
74d90653e0 check: use ReadFull to load pack header in checkPack
This ensures that the pack header is actually read completely.
Previously, for a truncated file it was possible to only read a part of
the header, as backend.Load(...) is not guaranteed to return as many
bytes as requested by the length parameter.
2024-05-18 21:28:54 +02:00
Michael Eischer
8f8d872a68 fix compatibility with go 1.19 2024-05-18 21:28:54 +02:00
Michael Eischer
ff0744b3af check: test checkPack retries 2024-05-18 21:28:54 +02:00
Michael Eischer
987c3b250c repository: test retries of ListPack 2024-05-18 21:28:54 +02:00
Michael Eischer
bf16096771 repository: test LoadBlob retries 2024-05-18 21:28:54 +02:00
Michael Eischer
4f45668b7c repository: rework and extend LoadRaw tests 2024-05-18 21:28:54 +02:00
Michael Eischer
ac805d6838 cache: cleanup debug logs 2024-05-18 21:28:54 +02:00
Michael Eischer
5214af88e2 cache: test forget behavior 2024-05-18 21:28:54 +02:00
Michael Eischer
3ff063e913 check: verify pack a second time if broken 2024-05-18 21:28:54 +02:00
Michael Eischer
385cee09dc repository: fix caching of tree packs in LoadBlobsFromPack 2024-05-18 21:28:54 +02:00
Michael Eischer
e734746f75 cache: forget cached file at most once
This is inspired by the circuit breaker pattern used for distributed
systems. If too many requests fails, then it is better to immediately
fail new requests for a limited time to give the backend time to
recover.

By only forgetting a file in the cache at most once, we can ensure that
a broken file is only retrieved once again from the backend. If the file
stored there is broken, previously it would be cached and deleted
continuously. Now, it is retrieved only once again, all later requests
just use the cached copy and either succeed or fail immediately.
2024-05-18 21:28:54 +02:00
Michael Eischer
97a307df1a cache: Always use cached file if it exists
A file is always cached whole. Thus, any out of bounds access will also
fail when directed at the backend. To handle case in which the cached
file is broken, then caller must call Cache.Forget(h) for the file in
question.
2024-05-18 21:28:54 +02:00
Michael Eischer
8cce06d915 repair packs: drop experimental warning
This warning should already have been removed once the feature flag was
dropped.
2024-05-18 21:28:54 +02:00
Michael Eischer
433a6aad29 repository: remove redundant blob loading fallback from RepairPacks
LoadBlobsFromPack already implements the same fallback behavior.
2024-05-18 21:28:54 +02:00
Michael Eischer
e401af07b2 check: fix error message formatting 2024-05-18 21:28:54 +02:00
Michael Eischer
7017adb7e9 repository: retry failed ListPack once 2024-05-18 21:28:54 +02:00
Michael Eischer
e33ce7f408 repository: retry failed LoadBlob once 2024-05-18 21:28:54 +02:00
Michael Eischer
2ace242f36 repository: make reloading broken files explicit 2024-05-18 21:28:54 +02:00
Michael Eischer
e9390352a7 cache: code cleanups 2024-05-18 21:26:00 +02:00
Michael Eischer
503c8140b1 repository: unify blob decoding code 2024-05-18 21:26:00 +02:00
Michael Eischer
6563f1d2ca repository: remove redundant debug log 2024-05-18 21:26:00 +02:00
Michael Eischer
021fb49559 repository: Implement repository.LoadUnpacked using LoadRaw
Both functions were using a similar implementation.
2024-05-18 21:26:00 +02:00
Michael Eischer
779c8d3527 debug/repair packs/upgrade repo v2: use repository.LoadRaw
This replaces calling the low-level backend.Load() method.
2024-05-18 21:26:00 +02:00
Michael Eischer
1d6d3656b0 repository: move backend.LoadAll to repository.LoadRaw
LoadRaw also includes improved context cancellation handling similar to the
implementation in repository.LoadUnpacked.

The removed cache backend test will be added again later on.
2024-05-18 21:26:00 +02:00
Michael Eischer
47232bf8b0 backend: move LimitReadCloser to util package
The helper is only intended for usage by backend implementations.
2024-05-18 21:26:00 +02:00
Michael Eischer
dcd151147c Merge pull request #4803 from restic/permanent-retry-failure
Do not retry permanent backend failures
2024-05-18 20:07:06 +02:00
Michael Eischer
53d15bcd1b retry: add circuit breaker to load method
If a file exhausts its retry attempts, then it is likely not accessible
the next time. Thus, immediately fail all load calls for that file to
avoid useless retries.
2024-05-18 19:59:26 +02:00
Michael Eischer
394c8ca3ed rest/rclone/s3/sftp/swift: move short file detection behind feature gate
These backends tend to use a large variety of server implementations.
Some of those implementations might prove problematic with the new
checks.
2024-05-18 19:59:26 +02:00
Michael Eischer
6328b7e1f5 replace "too small" with "too short" in error messages 2024-05-18 19:59:26 +02:00
Michael Eischer
53561474d9 update changelog with persistent backend error handling 2024-05-18 19:59:26 +02:00
Michael Eischer
aeb7eb245c retry: do not retry permanent errors
This is currently gated behind a feature flag as some unexpected
interactions might show up in the wild.
2024-05-18 19:59:26 +02:00
Michael Eischer
bf8cc59889 Use generic backend-error-redesign feature flag instead of http-timeouts
An individual flag for each change of the backend error handling would
be too finegrained. Thus, add a generic flag.
2024-05-18 19:54:52 +02:00
Michael Eischer
4740528a0b backend: add tests for IsPermanentError 2024-05-18 19:54:52 +02:00
Michael Eischer
6a85df7297 backend: add IsPermanentError() method to interface 2024-05-18 19:54:52 +02:00
Michael Eischer
cfc420664a mem: stricter handling of out of bounds requests 2024-05-18 19:54:52 +02:00
Michael Eischer
d40f23e716 azure/b2/gs/s3/swift: adapt cloud backend 2024-05-18 19:54:51 +02:00
Michael Eischer
e793c002ec local: stricter handling of short files 2024-05-18 19:54:21 +02:00
Michael Eischer
b4895ebd76 rest: rework error reporting and report too short files 2024-05-18 19:54:21 +02:00
Michael Eischer
eaa3f81d6b sftp: check for truncated files without an extra backend request 2024-05-18 19:54:21 +02:00
Michael Eischer
c6d74458ee sftp: improve handling of too short files 2024-05-18 19:54:21 +02:00
aneesh-n
43bc304e42 Add unreleased changelog 2024-05-17 14:54:25 -06:00
aneesh-n
9d1b23964d Update comment for licensing 2024-05-17 14:19:05 -06:00
aneesh-n
5cff6e084e Add support for Windows EA in node
Refactor Extended Attribute related functions in node files as windows apis get and set EA in bulk
2024-05-17 14:18:20 -06:00
aneesh-n
d6708505b9 Add helpers for Windows Extended Attributes 2024-05-17 14:15:03 -06:00
Michael Eischer
7ed560a201 Merge pull request #4796 from MichaelEischer/parallel-dump-load
dump: Parallelize loading large files
2024-05-14 22:35:44 +02:00
Michael Eischer
92221c2a6d Merge pull request #4708 from zmanda/windows-securitydesc
Back up and restore SecurityDescriptors on Windows
2024-05-12 14:14:39 +00:00
Michael Eischer
b5fdb1d637 Merge pull request #4782 from MichaelEischer/fix-sftp-performance
Fix sftp upload performance
2024-05-12 15:28:33 +02:00
Michael Eischer
e4f9bce384 Merge pull request #4792 from restic/request-watchdog
backend: enforce that backend HTTP requests make progress
2024-05-09 23:55:30 +02:00
Michael Eischer
3740700ddc add http timeouts to changelog 2024-05-09 23:46:17 +02:00
Michael Eischer
ebd01a4675 backend: add tests for watchdogRoundTripper 2024-05-09 23:46:17 +02:00
Michael Eischer
8778670232 backend: cancel stuck http requests
requests that make no upload or download progress within a timeout are
canceled.
2024-05-09 23:46:17 +02:00
Michael Eischer
0987c731ec backend: configure protocol-level connection health checks
This should detect a connection that is stuck for more than 2 minutes.
2024-05-09 23:46:17 +02:00
aneesh-n
a4fd1b91e5 Fix review comments
Change lowerPrivileges from bool to atomic.Bool.
Add missing cleanup from upstream go-winio.
Add handling for ERROR_NOT_ALL_ASSIGNED warning.
2024-05-06 16:54:08 -06:00
Michael Eischer
e184538ddf dump: add changelog 2024-05-05 12:12:21 +02:00
Michael Eischer
4d55a62ada bloblru: add test for GetOrCompute 2024-05-05 12:00:25 +02:00
Michael Eischer
7cce667f92 fuse: switch to use bloblru.GetOrCompute 2024-05-05 11:38:42 +02:00
Michael Eischer
bd03af2feb dump: add GetOrCompute to bloblru cache 2024-05-05 11:38:42 +02:00
Michael Eischer
45509eafc8 dump: load blobs of a file from repository in parallel 2024-05-05 11:38:42 +02:00
Michael Eischer
24c1822220 Merge pull request #4794 from flow-c/master
Update 060_forget.rst
2024-05-04 08:25:06 +00:00
flow-c
d4477a5a99 Update 060_forget.rst
Replace deprecated `-1` with `unlimited` in calendar-related `--keep-*` options
2024-05-04 09:32:25 +02:00
Michael Eischer
ffe5439149 Merge pull request #4605 from MichaelEischer/better-restorer-error-handling
Rework repository.StreamPacks & better restorer error handling
2024-05-01 16:37:41 +02:00
Michael Eischer
676f0dc60d add changelog 2024-05-01 16:28:57 +02:00
Michael Eischer
1e57057953 Merge pull request #4789 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/storage/azblob-1.3.2
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from 1.3.1 to 1.3.2
2024-05-01 10:45:47 +00:00
Michael Eischer
1ba0af6993 Merge pull request #4787 from restic/dependabot/go_modules/github.com/klauspost/compress-1.17.8
build(deps): bump github.com/klauspost/compress from 1.17.7 to 1.17.8
2024-05-01 10:44:33 +00:00
Michael Eischer
ffc41ae62a Merge pull request #4786 from restic/dependabot/go_modules/golang.org/x/net-0.24.0
build(deps): bump golang.org/x/net from 0.23.0 to 0.24.0
2024-05-01 10:41:26 +00:00
Michael Eischer
4832c2fbfa Merge pull request #4790 from restic/dependabot/github_actions/golangci/golangci-lint-action-5
build(deps): bump golangci/golangci-lint-action from 4 to 5
2024-05-01 10:37:37 +00:00
dependabot[bot]
30609ae6b2 build(deps): bump golangci/golangci-lint-action from 4 to 5
Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 4 to 5.
- [Release notes](https://github.com/golangci/golangci-lint-action/releases)
- [Commits](https://github.com/golangci/golangci-lint-action/compare/v4...v5)

---
updated-dependencies:
- dependency-name: golangci/golangci-lint-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-05-01 01:45:43 +00:00
dependabot[bot]
502e5867a5 build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
Bumps [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) from 1.3.1 to 1.3.2.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.3.1...sdk/storage/azblob/v1.3.2)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-05-01 01:02:39 +00:00
dependabot[bot]
18a6d6b408 build(deps): bump github.com/klauspost/compress from 1.17.7 to 1.17.8
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.17.7 to 1.17.8.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml)
- [Commits](https://github.com/klauspost/compress/compare/v1.17.7...v1.17.8)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-05-01 01:02:22 +00:00
dependabot[bot]
3bb88e8307 build(deps): bump golang.org/x/net from 0.23.0 to 0.24.0
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.23.0 to 0.24.0.
- [Commits](https://github.com/golang/net/compare/v0.23.0...v0.24.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-05-01 01:02:11 +00:00
aneesh-n
672f6cd776 Fix review comments for privileges and security flags 2024-04-29 17:29:51 -06:00
aneesh-n
08c6945d61 Fix review comments 2024-04-29 16:21:38 -06:00
Aneesh N
3f76b902e5 Merge branch 'master' into windows-securitydesc 2024-04-29 14:40:34 -06:00
Michael Eischer
ccac7c7fb3 Merge pull request #3067 from DRON-666/vss-options
Add options to fine tune VSS snapshots
2024-04-29 18:09:47 +00:00
DRON-666
ccd35565ee s/sec./seconds 2024-04-29 01:48:22 +03:00
DRON-666
125dba23c5 Rearange code 2024-04-29 01:27:34 +03:00
DRON-666
7ee889bb0d Use S_FALSE and MaxInt 2024-04-29 01:25:25 +03:00
DRON-666
90b168eb6c isMountPointExcluded to isMountPointIncluded 2024-04-29 01:23:50 +03:00
DRON-666
24330c19a8 Use kebab case in option names 2024-04-29 01:21:33 +03:00
DRON-666
5703e5a652 Fix texts and comments 2024-04-29 01:18:46 +03:00
DRON-666
0a8f9c5d9c vss: Add tests for "provider" option 2024-04-28 22:45:21 +03:00
DRON-666
739d3243d9 vss: Update docs and changelog 2024-04-28 22:45:21 +03:00
DRON-666
bb0f93ef3d vss: Add "provider" option 2024-04-28 22:45:21 +03:00
DRON-666
3bac1f0135 vss: Fix issues reported by linters 2024-04-28 22:45:21 +03:00
DRON-666
88c509e3e9 vss: Change ErrorHandler signature
We don't need `error` here: the only existing implementation
of `ErrorHandler` always call `Backup.Error` and all
implementations of `Backup.Error` always return nil.
2024-04-28 22:44:16 +03:00
DRON-666
9d3d915e2c vss: Add some tests 2024-04-28 22:44:16 +03:00
DRON-666
9182e6bab5 vss: Update docs and changelog 2024-04-28 22:44:16 +03:00
DRON-666
c4f67c0064 vss: Add volume filtering
Add options to exclude all mountpoints and arbitrary volumes from snapshotting.
2024-04-28 22:44:15 +03:00
DRON-666
7470e5356e vss: Add "timeout" option
Changing multiple "callAsyncFunctionAndWait" with fixed timeout
to calculated timeout based on deadline.
2024-04-28 22:44:15 +03:00
DRON-666
78dbc5ec58 vss: Add initial support for extended options 2024-04-28 22:44:15 +03:00
Michael Eischer
a1d682ce0e add changelog for sftp performance fix 2024-04-28 11:58:08 +02:00
Michael Eischer
935327d480 sftp: slightly increase write concurrency
This should increase upload throughput for high latency links a bit.
2024-04-28 11:50:09 +02:00
Michael Eischer
669a669603 sftp: Fix upload performance issue
Since pkg/sftp 1.13.0 files were uploaded sequentially using 32kb chunks
instead of sending 64 chunks in parallel.
2024-04-28 11:48:26 +02:00
Michael Eischer
faffd15d13 Merge pull request #4734 from maouw/enhancement/envvar-for-host
Add support for specifying --host via environment variable
2024-04-24 20:00:15 +00:00
Michael Eischer
347e9d0765 complete RESITC_HOST environment handling & test 2024-04-24 21:52:39 +02:00
Altan Orhon
871ea1eaf3 Add support for specifying --host via environment variable
This commit adds support for specifying the `--host` option via the `RESTIC_HOST` environment variable. This is done by extending option processing in `cmd_backup.go` and for `restic.SnapshotFilter` in `find.go`.
2024-04-24 21:49:42 +02:00
Michael Eischer
a7b5e09902 Merge pull request #4753 from MichaelEischer/remove-cleanup-handlers
Replace cleanup handlers with context based command cancelation
2024-04-24 21:34:19 +02:00
Michael Eischer
3f9d50865d Merge pull request #4776 from MichaelEischer/cleanup-backend-open
unify backend open and create
2024-04-24 21:24:27 +02:00
Michael Eischer
5f263752d7 init: also apply limiter for non-HTTP backend 2024-04-24 20:42:30 +02:00
Michael Eischer
484dbb1cf4 get rid of a few global variables 2024-04-22 22:39:33 +02:00
Michael Eischer
940a3159b5 let index.Each() and pack.Size() return error on canceled context
This forces a caller to actually check that the function did complete.
2024-04-22 22:39:32 +02:00
Michael Eischer
31624aeffd Improve command shutdown on context cancellation 2024-04-22 22:31:38 +02:00
Michael Eischer
910927670f mount: fix exit code on cancellation 2024-04-22 22:27:19 +02:00
Michael Eischer
6f2a4dea21 remove global shutdown hook 2024-04-22 22:27:19 +02:00
Michael Eischer
699ef5e9de debug: replace cleanup handler usage in profiling setup 2024-04-22 22:27:19 +02:00
Michael Eischer
eb710a28e8 use standalone shutdown hook for readPasswordTerminal
move terminal restoration into readPasswordTerminal
2024-04-22 22:27:19 +02:00
Michael Eischer
86c7909f41 mount: use standalone shutdown hook via goroutine 2024-04-22 22:27:19 +02:00
Michael Eischer
93135dc705 lock: drop cleanup handler 2024-04-22 22:27:19 +02:00
Michael Eischer
21a7cb405c check: replace cleanup handler 2024-04-22 22:27:19 +02:00
Michael Eischer
b15d867414 Merge pull request #4763 from MichaelEischer/refactor-prune
Refactor repair index / prune into the repository package
2024-04-22 22:24:53 +02:00
Michael Eischer
2e6c43c695 Merge pull request #4761 from MichaelEischer/fix-cache-race
cache: ignore ErrNotExist during cleanup of old files
2024-04-22 21:46:06 +02:00
Michael Eischer
f7632de3d6 Merge pull request #4772 from MichaelEischer/better-error-on-too-large-blob
repository: Better error message if blob is larger than 4GB
2024-04-22 21:45:06 +02:00
Michael Eischer
20d8eed400 repository: streamPack: separate requests for gap larger than 1MB
With most cloud providers, traffic is much more expensive than API
calls. Thus slightly bias streamPack towards a bit more API calls in
exchange for slightly less traffic.
2024-04-22 21:21:23 +02:00
Michael Eischer
cf700d8794 repository: streamPack: reuse zstd decoder 2024-04-22 21:21:23 +02:00
Michael Eischer
666a0b0bdb repository: streamPack: replace streaming with chunked download
Due to the interface of streamPack, we cannot guarantee that operations
progress fast enough that the underlying connections remains open. This
introduces partial failures which massively complicate the error
handling.

Switch to a simpler approach that retrieves the pack in chunks of 32MB.
If a blob is larger than this limit, then it is downloaded separately.

To avoid multiple copies in memory, an auxiliary interface
`discardReader` is introduced that allows directly accessing the
downloaded byte slices, while still supporting the streaming used by the
`check` command.
2024-04-22 21:21:23 +02:00
Michael Eischer
621012dac0 repository: Add blob loading fallback to LoadBlobsFromPack
Try to retrieve individual blobs via LoadBlob if streaming did not work.
2024-04-21 21:35:55 +02:00
Michael Eischer
6c6dceade3 global: unify backend open and create 2024-04-19 22:26:14 +02:00
Michael Eischer
10355c3fb6 repository: Better error message if blob is larger than 4GB 2024-04-19 22:00:35 +02:00
Michael Eischer
228b35f074 Merge pull request #4769 from will-ca/patch-1
Tiny wording clarification in `restic-stats.1`.
2024-04-18 19:00:39 +00:00
will-ca
6aced61c72 Tiny docs wording clarification. 2024-04-18 07:29:55 +00:00
Michael Eischer
4d22412e0c Merge pull request #4766 from coderwander/master
Fix struct names
2024-04-18 06:18:19 +00:00
coderwander
a82ed71de7 Fix struct names
Signed-off-by: coderwander <770732124@qq.com>
2024-04-18 10:02:09 +08:00
Michael Eischer
2173c69280 Merge pull request #4770 from testwill/close_files
fix: close files
2024-04-17 16:50:20 +00:00
Michael Eischer
001bb71676 repair packs: Properly close backup files 2024-04-17 18:32:30 +02:00
Michael Eischer
c9191ea72c forget: cleanup verbose output on snapshot deletion error 2024-04-14 14:17:40 +02:00
Michael Eischer
09587e6c08 repository: duplicate a few blobs in prune tests 2024-04-14 13:57:19 +02:00
Michael Eischer
defd7ae729 prune/repair index: reset in-memory index after command
The current in-memory index becomes stale after prune or repair index
have run. Thus, just drop the in-memory index altogether once these
commands have finished.
2024-04-14 13:46:24 +02:00
Michael Eischer
038586dc9d repository: add minimal test for prune 2024-04-14 13:45:17 +02:00
Michael Eischer
d8622c86eb prune: clean up internal interface 2024-04-14 13:45:15 +02:00
Michael Eischer
8d507c1372 repository: add basic test for RepairIndex 2024-04-14 13:45:15 +02:00
Michael Eischer
310db03c0e repair index: improve log output if index cannot be deleted
The operation will always fail with an error if an index cannot be
deleted. Thus, this change is purely cosmetic.
2024-04-14 13:45:13 +02:00
Michael Eischer
7d1b9cde34 repository: use normal Init method in tests 2024-04-14 13:45:11 +02:00
Michael Eischer
b25fc2c89d repository: remove redundant flushes from tests 2024-04-14 13:45:10 +02:00
Michael Eischer
c65459cd8a repository: speed up tests 2024-04-14 13:45:10 +02:00
Michael Eischer
eda9f7beb4 ui/progress: add helper to print messages during tests 2024-04-14 13:45:08 +02:00
Michael Eischer
35277b7797 backend/mem: cleanup not found error message 2024-04-14 13:45:06 +02:00
Michael Eischer
7ba5e95a82 check: allow tests to only verify pack&index integrity 2024-04-14 13:45:04 +02:00
Michael Eischer
4c9a10ca37 repair packs: deduplicate index rebuild 2024-04-14 13:45:02 +02:00
Michael Eischer
85e4021619 prune: move additional option checks to repository 2024-04-14 13:44:58 +02:00
Michael Eischer
55d56db31b Merge pull request #4743 from MichaelEischer/deprecate-s3legacy-layout
Deprecate s3legacy layout
2024-04-11 22:09:34 +02:00
Michael Eischer
fc3b548625 prune: move logic into repository package 2024-04-10 21:30:52 +02:00
Michael Eischer
df9d4b455d prune: prepare for moving code to repository package 2024-04-10 21:30:52 +02:00
Michael Eischer
866ddf5698 repair index: refactor code into repository package 2024-04-10 21:30:52 +02:00
Michael Eischer
32a234b67e prune/forget/repair index: convert output to use progress.Printer 2024-04-10 21:30:52 +02:00
Michael Eischer
739d11c2eb forget: replace usage of DeleteFilesChecked
This simplifies refactoring prune into the repository package.
2024-04-10 21:30:52 +02:00
Michael Eischer
591b421c4a Deprecate s3legacy layout 2024-04-10 21:27:56 +02:00
Michael Eischer
8efc3a8b7d Merge pull request #4668 from MichaelEischer/backup-xattr-parent-enoperm
backup: Ignore xattr.list permission error for parent directories
2024-04-10 21:25:28 +02:00
Michael Eischer
bf054c09d2 backup: Ignore xattr.list permission error for parent directories
On FreeBSD, limited users may not be able to even list xattrs for the
parent directories above the snapshot source paths. As this can cause
the backup to fail, just ignore those errors.
2024-04-10 20:46:15 +02:00
Michael Eischer
0747cf5319 cache: ignore ErrNotExist during cleanup of old files
Two restic processes running concurrently can try to remove the same
files from the cache. This could cause one process to fail with an error
if the other one has already remove a file that the current process also
tries to delete.
2024-04-10 19:25:51 +02:00
Michael Eischer
6091029fd6 Merge pull request #4756 from mgeisler/patch-1
doc: fix typo in 047_tuning_backup_parameters.rst
2024-04-07 19:24:09 +00:00
Martin Geisler
09d2183351 doc: fix typo in 047_tuning_backup_parameters.rst 2024-04-07 18:05:53 +02:00
Michael Eischer
a4b7ebecfc Merge pull request #4750 from restic/dependabot/go_modules/cloud.google.com/go/storage-1.40.0
build(deps): bump cloud.google.com/go/storage from 1.39.0 to 1.40.0
2024-04-03 20:10:41 +00:00
dependabot[bot]
ba136b31b8 build(deps): bump cloud.google.com/go/storage from 1.39.0 to 1.40.0
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.39.0 to 1.40.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/spanner/v1.39.0...spanner/v1.40.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-04-03 20:03:22 +00:00
Michael Eischer
f328111a6e Merge pull request #4751 from restic/dependabot/go_modules/golang.org/x/net-0.23.0
build(deps): bump golang.org/x/net from 0.21.0 to 0.23.0
2024-04-03 19:52:29 +00:00
Michael Eischer
9fb017e67a Merge pull request #4745 from MichaelEischer/full-id-key-list
key list: include full key id in JSON output
2024-04-03 21:50:02 +02:00
Michael Eischer
49f98f25fc Merge pull request #4742 from MichaelEischer/consistent-rtest-import
Use consistent alias for interal/test package
2024-04-03 21:47:32 +02:00
dependabot[bot]
96c602a6de build(deps): bump golang.org/x/net from 0.21.0 to 0.23.0
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.21.0 to 0.23.0.
- [Commits](https://github.com/golang/net/compare/v0.21.0...v0.23.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-04-03 19:41:38 +00:00
Michael Eischer
1d0a20dd58 Merge pull request #4748 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/azcore-1.10.0
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.9.2 to 1.10.0
2024-04-03 19:34:47 +00:00
Michael Eischer
6cca1d5705 Merge pull request #4655 from ae-govau/unixsocket
Enhancement: option to send HTTP over unix socket
2024-04-03 19:29:21 +00:00
dependabot[bot]
f8a72ac2a3 build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore
Bumps [github.com/Azure/azure-sdk-for-go/sdk/azcore](https://github.com/Azure/azure-sdk-for-go) from 1.9.2 to 1.10.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.9.2...sdk/azcore/v1.10.0)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azcore
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-04-01 02:32:19 +00:00
Michael Eischer
5145c8f9c0 key list: include full key id in JSON output 2024-03-31 12:25:20 +02:00
Michael Eischer
831fc4413d Merge pull request #4737 from stephan0307/3117
json output forget command: added id's in snapshots within reasons object
2024-03-29 13:11:24 +00:00
Stephan Paul
df07814ec2 forget json output: added id's in snapshots within reasons object
In order to evaluate the keep reasons for snapshots, there should be also the id's
to compare it with snapshots within the keep object. (See also Issue #3117)

In order to avoid output parameters also changed function addJSONSnapshots to asJSONSnapshots
2024-03-29 11:30:00 +01:00
Michael Eischer
ec2b79834a use consistent alias for interal/test package 2024-03-29 00:24:03 +01:00
Michael Eischer
510f6f06b0 Merge pull request #4709 from MichaelEischer/refactor-locking
Refactor locking into repository package
2024-03-28 23:53:09 +01:00
Michael Eischer
07eb6c315b add changelog for locking refactor 2024-03-28 23:46:58 +01:00
Michael Eischer
5e98f1e2eb repository: fix test setup race conditions 2024-03-28 23:17:02 +01:00
Michael Eischer
8155dbe711 correctly lock repository in integration tests 2024-03-28 23:17:02 +01:00
Michael Eischer
d18726cd70 ls: add missing read lock
As `ls` reads data from the repository, it must acquire a read lock
unless `--no-lock` was specified. The old behavior is equivalent to `ls
--no-lock`.
2024-03-28 23:17:02 +01:00
Michael Eischer
dc441c57a7 repository: unify repository initialization in tests
Tests should use a helper from internal/repository/testing.go to
construct a Repository object.
2024-03-28 23:17:02 +01:00
Michael Eischer
3ba1fa3cee repository: remove a few global variables 2024-03-28 23:17:02 +01:00
Michael Eischer
044e8bf821 repository: parallelize lock tests 2024-03-28 23:17:02 +01:00
Michael Eischer
e8df50fa3c repository: remove global list of locks 2024-03-28 22:46:33 +01:00
Michael Eischer
cbb5f89252 lock: move code to repository package 2024-03-28 22:46:33 +01:00
Michael Eischer
118a69a84b lock: replace lockRepo(Exclusive) with openWith(Read/Write/Exclusive)Lock
The new functions much better convey the intent behind the lock
request. This allows cleanly integrating noLock (for read) and dryRun
(write/exclusive) handling.

There are only minor changes to existing behavior with two exceptions:
- `tag` no longer accepts the `--no-lock` flag. As it replaces files in
  the repository, this always requires an exclusive lock.
- `debug examine` now returns an error if both `--extract-pack` and
  `--no-lock` are given.
2024-03-28 22:46:33 +01:00
Michael Eischer
7f9ad1c3db Merge pull request #4705 from MichaelEischer/snapshot-statistics
Store snapshot statistics & print snapshot size
2024-03-28 22:41:45 +01:00
Michael Eischer
71b6284155 Merge pull request #4006 from MichaelEischer/deviceID-only-for-hardlinks
archiver: only store deviceID for hardlinks
2024-03-28 22:33:28 +01:00
Michael Eischer
cf81f8ced6 stats: only check for hardlinks for files with more than one link 2024-03-28 21:29:27 +01:00
Michael Eischer
21cf38fe96 add changelog for deviceID only for hardlinks 2024-03-28 19:32:50 +01:00
Michael Eischer
d705741571 backup: test that deviceID is only stored for hardlinks 2024-03-28 19:32:50 +01:00
Michael Eischer
a9b3d86c4f features: remove example feature 2024-03-28 19:12:07 +01:00
Michael Eischer
a26d6ffa72 archiver: move deviceID handling behind feature flag 2024-03-28 19:12:07 +01:00
Michael Eischer
2ba21fe72b archiver: only store deviceID for hardlinks
The deviceID can change e.g. when backing up from filesystem snapshot.
It is only used for hardlink detection. Thus there it is not necessary
to store it for everything else.
2024-03-28 19:12:07 +01:00
Michael Eischer
870904d3ae Merge pull request #4731 from facutuesca/powershell-completion-doc
doc: Add instructions to configure PowerShell completions
2024-03-28 17:46:55 +00:00
Facundo Tuesca
15555c9898 doc: Use consistent case for PROFILE env variable in PowerShell 2024-03-28 18:35:18 +01:00
Michael Eischer
63a2350c9e Merge pull request #4741 from MichaelEischer/ci-upgrade-golangci-lint
CI: Update golangci-lint to version 1.57.1
2024-03-28 18:27:05 +01:00
Michael Eischer
8876e3025b Merge pull request #4724 from MichaelEischer/disable-old-index
Deprecated legacy index format
2024-03-28 18:24:47 +01:00
Michael Eischer
4f4979f4e8 Merge pull request #4740 from MichaelEischer/ci-upgrade-docker-actions
CI: update docker actions
2024-03-28 18:16:38 +01:00
Michael Eischer
1497525e15 CI: Update golangci-lint to version 1.57.1 2024-03-28 18:15:50 +01:00
Michael Eischer
a8face3a25 Merge pull request #4739 from MichaelEischer/ci-allow-annotations
CI: Allow golangci-lint to annotate PRs
2024-03-28 18:02:16 +01:00
Michael Eischer
aee6d311f1 CI: update docker actions 2024-03-28 18:01:48 +01:00
Michael Eischer
6ac7519188 add changelog for rest unix socket support 2024-03-28 17:41:41 +01:00
Michael Eischer
add37fcd9f CI: uses rest-server from master branch until unix sockets are released 2024-03-28 17:41:41 +01:00
Adam Eijdenberg
6e775d3787 Enhancement: option to send HTTP over unix socket
add tests for unix socket connection

switch HTTP rest-server test to use any free port

allow rest-server test graceful shutdown opportunity
2024-03-28 17:41:41 +01:00
Michael Eischer
5c4a4b4a30 CI: Allow golangci-lint to annotate PRs 2024-03-28 17:09:59 +01:00
Michael Eischer
d1d773cfcd Merge pull request #4727 from avoidalone/master
fix some typos
2024-03-17 18:52:40 +00:00
Facundo Tuesca
521713fc94 doc: Add instructions to configure PowerShell completions 2024-03-16 18:54:27 +01:00
avoidalone
ac948fccda fix some typos
Signed-off-by: avoidalone <wuguangdong@outlook.com>
2024-03-11 14:35:12 +08:00
rawtaz
9284f7413a Merge pull request #4725 from leoheitmannruiz/master
Minor README.md cleanups
2024-03-10 12:24:07 +00:00
rawtaz
1287b977b4 Merge pull request #4726 from leoheitmannruiz/patch-1
Capitalize Homebrew and minor edit in wording
2024-03-10 12:23:48 +00:00
Leo Heitmann Ruiz
00f762373f Capitalize Homebrew 2024-03-10 00:20:26 +01:00
Leo Heitmann Ruiz
9f3e1462c0 Minor README.md cleanups 2024-03-09 23:56:16 +01:00
Michael Eischer
69ca12d2eb check: treat legacy index format as errors 2024-03-09 18:36:33 +01:00
Michael Eischer
98a6817d01 add changelog for legacy index deprecation 2024-03-09 18:35:00 +01:00
Michael Eischer
f8852f0eb6 repair index: fix deletion of legacy indexes 2024-03-09 18:21:22 +01:00
Michael Eischer
1a8bf358f1 index: deprecate legacy index format 2024-03-09 18:21:14 +01:00
Michael Eischer
396a61a992 Merge pull request #4666 from MichaelEischer/feature-flags
Implement feature flags
2024-03-09 17:36:29 +01:00
Michael Eischer
a9b64cd7ad features: print warning for stable/depreacted feature flags 2024-03-09 17:29:52 +01:00
Michael Eischer
fe68d2cafb add feature flag documentation 2024-03-09 17:29:52 +01:00
Michael Eischer
70839155f2 features: add tests 2024-03-09 17:29:52 +01:00
Michael Eischer
1c77c51a03 features: initialize based on RESTIC_FEATURES environment variable 2024-03-09 17:29:52 +01:00
Michael Eischer
5974a79497 features: add basic feature flag implementation 2024-03-09 17:29:52 +01:00
Michael Eischer
0589da60b3 Merge pull request #4717 from restic/dependabot/go_modules/cloud.google.com/go/storage-1.39.0
build(deps): bump cloud.google.com/go/storage from 1.37.0 to 1.39.0
2024-03-07 14:10:24 +00:00
dependabot[bot]
608116817b build(deps): bump cloud.google.com/go/storage from 1.37.0 to 1.39.0
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.37.0 to 1.39.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/spanner/v1.37.0...spanner/v1.39.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-03-07 13:59:23 +00:00
Michael Eischer
f742da8b2f Merge pull request #4722 from konidev20/upgrade-docker-golang-base-image-to-1-22
docker: update the base image to golang:1.22-alpine
2024-03-07 13:44:48 +00:00
Michael Eischer
af1684743f Merge pull request #4716 from restic/dependabot/go_modules/github.com/klauspost/compress-1.17.7
build(deps): bump github.com/klauspost/compress from 1.17.6 to 1.17.7
2024-03-07 13:41:46 +00:00
Michael Eischer
5b9de4d8b7 Merge pull request #4715 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/storage/azblob-1.3.1
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from 1.2.1 to 1.3.1
2024-03-07 13:40:44 +00:00
Michael Eischer
fceef67abe Merge pull request #4718 from restic/dependabot/go_modules/github.com/spf13/cobra-1.8.0
build(deps): bump github.com/spf13/cobra from 1.7.0 to 1.8.0
2024-03-07 13:37:17 +00:00
Michael Eischer
8506cae710 Merge pull request #4714 from restic/dependabot/github_actions/golangci/golangci-lint-action-4
build(deps): bump golangci/golangci-lint-action from 3 to 4
2024-03-07 13:34:21 +00:00
Michael Eischer
87d47ef189 Merge pull request #4713 from restic/dependabot/github_actions/docker/login-action-5139682d94efc37792e6b54386b5b470a68a4737
build(deps): bump docker/login-action from 3d58c274f17dffee475a5520cbe67f0a882c4dbb to 5139682d94efc37792e6b54386b5b470a68a4737
2024-03-07 13:33:49 +00:00
Michael Eischer
55abf25ea8 Merge pull request #4719 from restic/dependabot/go_modules/golang.org/x/oauth2-0.17.0
build(deps): bump golang.org/x/oauth2 from 0.16.0 to 0.17.0
2024-03-07 13:32:36 +00:00
Srigovind Nayak
b48b1fa2c9 docker: update the base image to golang:1.22-alpine 2024-03-03 16:18:13 +05:30
dependabot[bot]
8e7f29ae28 build(deps): bump golang.org/x/oauth2 from 0.16.0 to 0.17.0
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.16.0 to 0.17.0.
- [Commits](https://github.com/golang/oauth2/compare/v0.16.0...v0.17.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-03-01 01:25:47 +00:00
dependabot[bot]
79e8ddac3f build(deps): bump github.com/spf13/cobra from 1.7.0 to 1.8.0
Bumps [github.com/spf13/cobra](https://github.com/spf13/cobra) from 1.7.0 to 1.8.0.
- [Release notes](https://github.com/spf13/cobra/releases)
- [Commits](https://github.com/spf13/cobra/compare/v1.7.0...v1.8.0)

---
updated-dependencies:
- dependency-name: github.com/spf13/cobra
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-03-01 01:25:41 +00:00
dependabot[bot]
b5a9b5d0bc build(deps): bump github.com/klauspost/compress from 1.17.6 to 1.17.7
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.17.6 to 1.17.7.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml)
- [Commits](https://github.com/klauspost/compress/compare/v1.17.6...v1.17.7)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-03-01 01:25:00 +00:00
dependabot[bot]
f185c80cf0 build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
Bumps [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) from 1.2.1 to 1.3.1.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azidentity/v1.2.1...sdk/azcore/v1.3.1)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-03-01 01:24:46 +00:00
dependabot[bot]
70c8aaa303 build(deps): bump golangci/golangci-lint-action from 3 to 4
Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 3 to 4.
- [Release notes](https://github.com/golangci/golangci-lint-action/releases)
- [Commits](https://github.com/golangci/golangci-lint-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: golangci/golangci-lint-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-03-01 01:17:27 +00:00
dependabot[bot]
e1a588b75c build(deps): bump docker/login-action
Bumps [docker/login-action](https://github.com/docker/login-action) from 3d58c274f17dffee475a5520cbe67f0a882c4dbb to 5139682d94efc37792e6b54386b5b470a68a4737.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](3d58c274f1...5139682d94)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-03-01 01:17:20 +00:00
Michael Eischer
e71660cd1e backup: rename data_added_in_repo statistic to data_added_packed 2024-02-25 20:40:52 +01:00
Aneesh Nireshwalia
062d408987 Clean up SecurityDescriptor helper 2024-02-24 14:23:04 -07:00
Aneesh Nireshwalia
5764300022 Add changelog and fix lint error 2024-02-24 13:47:49 -07:00
Aneesh Nireshwalia
c0a1b9ada5 Update docs for security descriptors 2024-02-24 13:28:18 -07:00
Aneesh Nireshwalia
90916f53de Add test cases for security descriptors 2024-02-24 13:27:01 -07:00
Aneesh Nireshwalia
70cf8e3788 Add support for backup/restore of security descriptors 2024-02-24 13:25:28 -07:00
Aneesh Nireshwalia
e3e59fef24 Fix CombineErrors and fillExtendedAttr error handling 2024-02-24 13:22:34 -07:00
Aneesh Nireshwalia
09ce1b4e58 Create helper for SecurityDescriptor related functions 2024-02-24 13:16:25 -07:00
Michael Eischer
6a13e451b1 document snapshot statistics 2024-02-23 22:32:04 +01:00
Michael Eischer
a8f5684f68 archiver: test statistics in snapshot 2024-02-23 22:05:15 +01:00
Michael Eischer
681395955e archiver: test backup summary calculation 2024-02-23 21:46:39 +01:00
Michael Eischer
b6520038fd snapshots: Print snapshot size stored in snapshots 2024-02-23 20:27:13 +01:00
Michael Eischer
38f91d3b5e backup: store statistics in snapshot 2024-02-23 20:27:13 +01:00
Michael Eischer
86897314d5 backup: expose data_added_in_repo in JSON output
The value describes how much data was added after compression.
Previously, it was only available in the text output.
2024-02-23 20:27:13 +01:00
Michael Eischer
a59f654fa6 archiver: refactor summary collection from ui into the archiver 2024-02-23 20:27:13 +01:00
Michael Eischer
8b1a85711f archiver: unexport save/saveDir/saveTree methods 2024-02-23 20:24:21 +01:00
Michael Eischer
b953dc8f58 Merge pull request #4611 from zmanda/windows-metadata-support
Back up and restore windows metadata like created ts, file attribs like hidden, readonly, encrypted with a common extensible mechanism
2024-02-23 18:16:09 +00:00
Aneesh Nireshwalia
e8211cb64a Add changelog and update docs for windows attr 2024-02-22 17:59:56 -07:00
Aneesh Nireshwalia
4bbd25a37f Add tests for generic attribute changes 2024-02-22 17:55:50 -07:00
Aneesh Nireshwalia
d4be734c73 Handle readonly empty files in windows 2024-02-22 17:54:43 -07:00
Aneesh Nireshwalia
eeb1aa5388 Add ability to report warnings to terminal
Report warnings to terminal when unrecognized generic attributes are found in the repository.
2024-02-22 17:52:26 -07:00
Aneesh Nireshwalia
0962917974 Support windows metadata using generic attribs
Add new generic_attributes attribute in Node.
Use the generic attributes to add support for creation time and file attributes like hidden, readonly, encrypted in windows. Handle permission errors for readonly files in windows.
Handle backup and restore of encrypted attributes using windows system calls.
2024-02-22 17:31:20 -07:00
Aneesh Nireshwalia
62a8a599f1 Add optional messages for Equals helper 2024-02-22 16:58:12 -07:00
Aneesh Nireshwalia
94de87d4b7 Add CombineErrors helper function 2024-02-22 16:57:00 -07:00
Michael Eischer
c6311c1e32 Merge pull request #4703 from ferringb/master
Catch SIGTERM, run cleanup
2024-02-22 21:06:29 +00:00
Michael Eischer
0a65a0f94f update comment 2024-02-22 22:00:42 +01:00
Brian Harring
b41107dcaf Add changelog for SIGTERM bugfix.
Signed-off-by: Brian Harring <ferringb@gmail.com>
2024-02-19 11:31:48 +01:00
Brian Harring
30e979d252 Catch SIGTERM, run cleanup
The previous code only ran cleanup (lock release for example) on SIGINT.  For
anyone running restic in a container, the signal is going to be SIGTERM which
means containerized execution would leave locks behind.

While this could be addressed via interposing dumb-init to translate the signal,
a `kill` invocation is going to default to SIGTERM, so the same problem exists
for non container users.

Signed-off-by: Brian Harring <ferringb@gmail.com>
2024-02-19 11:12:15 +01:00
Michael Eischer
cfbeb2cde5 Merge pull request #4701 from MichaelEischer/better-streampack-errors
repository: Improve StreamPack error messages
2024-02-18 16:57:56 +01:00
Michael Eischer
80754dbf0c Merge pull request #4664 from MichaelEischer/ls-unified-json-output
ls: include standard `message_type` field in output
2024-02-18 15:47:41 +00:00
Michael Eischer
4c3218ef9f repository: include packID in StreamPack for decrypt/decompress errors 2024-02-17 19:38:01 +01:00
Michael Eischer
18b0bbbf42 repository: use fmt.Errorf in StreamPacks 2024-02-17 19:37:32 +01:00
Michael Eischer
6fbb470835 Merge pull request #4665 from MichaelEischer/check-repair-packs
check: Suggest usage of `repair packs` if pack files are damaged
2024-02-17 15:54:07 +00:00
Michael Eischer
0a36d193d8 add changelog for enhanced repair packs 2024-02-12 21:43:35 +01:00
Michael Eischer
69304cd74f check: clarify repair pack usage 2024-02-12 21:43:35 +01:00
Michael Eischer
c3b0e6d004 Merge pull request #4700 from MichaelEischer/remove-duplicate-changelog-entries
remove changelogs that are already included in restic 0.16.4
2024-02-12 21:40:04 +01:00
Michael Eischer
9e3703ded5 remove changelogs that are already included in restic 0.16.4 2024-02-12 20:39:31 +01:00
Michael Eischer
527a3ff2b2 check: link to troubleshooting guide 2024-02-12 20:25:15 +01:00
Michael Eischer
ed4a4f8748 check: exclude inaccessible files from the repair pack suggestion 2024-02-12 20:25:15 +01:00
Michael Eischer
4073299a7c check: fix missing error if blob is invalid 2024-02-12 20:20:13 +01:00
Michael Eischer
6397615fbb check: document that check will show repair pack instructions 2024-02-12 20:20:13 +01:00
Michael Eischer
544fe38786 check: suggest repair pack for all damaged packs 2024-02-12 20:20:13 +01:00
Michael Eischer
772e3416d1 repair pack: drop feature flag 2024-02-12 20:20:12 +01:00
Michael Eischer
22a3cea1b3 checker: wrap all pack errors in ErrPackData 2024-02-12 20:19:32 +01:00
Michael Eischer
19bf2cf52d Merge pull request #4697 from MichaelEischer/report-blob-errors
backup: report files whose chunks failed to upload
2024-02-12 20:18:51 +01:00
Michael Eischer
5b5d506472 backup: report files whose chunks failed to upload 2024-02-11 22:43:26 +01:00
Michael Eischer
dde556e8e8 Merge pull request #4696 from MichaelEischer/fix-exclude-load-error-msg
backup: improve error message if exclude file cannot be loaded
2024-02-11 21:35:31 +00:00
Michael Eischer
ee1ff3c1d0 backup: improve error message if exclude file cannot be loaded 2024-02-11 22:26:13 +01:00
Michael Eischer
667a2f5369 Merge pull request #4694 from restic/update-go-versions
Update go versions
2024-02-11 17:10:12 +00:00
Michael Eischer
2ab18a92e6 CI: keep tests for Go 1.19 2024-02-10 23:42:34 +01:00
Alexander Neumann
c0514dd8ba Fix linter errors (except for tests) 2024-02-10 22:58:10 +01:00
Alexander Neumann
a8cda0119c Upgrade golangci-lint 2024-02-10 22:08:43 +01:00
Alexander Neumann
9720935c56 Update Go version for tests to 1.22 2024-02-10 21:56:01 +01:00
Michael Eischer
68cc327b15 Merge pull request #4692 from 27149chen/dump-to-existing-file
feat: dump flag --target should be allowed to write existing file
2024-02-10 17:44:54 +00:00
Michael Eischer
15d6fa1f83 dump: update docs for --target option 2024-02-10 18:39:06 +01:00
lou
80db02fc35 dump flag --target should be allowed to write existing file
Signed-off-by: lou <alex1988@outlook.com>
2024-02-10 18:39:06 +01:00
Michael Eischer
6a2b10e2a8 Merge pull request #4685 from konidev20/fix-gh-4676-sub-commands-for-key-management
Move key add, list, remove and passwd as separate sub-commands and improve key sub-command documentation
2024-02-08 19:59:16 +00:00
Michael Eischer
e46b21ab80 key: fix integration test for invalid arguments 2024-02-08 20:52:30 +01:00
Michael Eischer
eb389a2d25 Merge pull request #4687 from MichaelEischer/upgrade-zstd-library
Upgrade zstd library to latest version
2024-02-08 20:29:13 +01:00
Srigovind Nayak
795d33b3ee key: move add, list, remove, passwd to sub-commands
docs: improve the sub-command docs

changelog: add the unreleased changelog for the key command updates

key: update integration tests
2024-02-06 01:47:43 +05:30
Michael Eischer
0cffdb7493 Merge pull request #4682 from konidev20/feat-add-target-for-dump-command
feat: set --target for dump command
2024-02-05 19:16:42 +00:00
Michael Eischer
f5ffa40652 dump: minor cleanups 2024-02-05 20:10:52 +01:00
Srigovind Nayak
175c14b5c9 dump: add --target option 2024-02-05 20:10:52 +01:00
Michael Eischer
bca099ac7f Upgrade zstd library to latest version
The data corruption bug is fixed, thus remove the override.
2024-02-05 19:53:02 +01:00
Michael Eischer
0f09a8870c Merge pull request #4684 from konidev20/fix-gh-4658-update-azure-documentation
Update azure storage account authentication documentation
2024-02-04 20:13:18 +00:00
Srigovind Nayak
5771c4ecfb docs: update environment variables for az login to azure backend 2024-02-05 01:32:43 +05:30
Michael Eischer
b63bfd2257 Merge branch 'patch-release' 2024-02-04 20:21:42 +01:00
Alexander Neumann
0f9fa44de5 Set development version for 0.16.4 2024-02-04 19:50:56 +01:00
Alexander Neumann
3786536dc1 Add version for 0.16.4 2024-02-04 19:50:52 +01:00
Alexander Neumann
811be5984d Update manpages and auto-completion 2024-02-04 19:50:51 +01:00
Alexander Neumann
b0ead75de5 Generate CHANGELOG.md for 0.16.4 2024-02-04 19:50:34 +01:00
Alexander Neumann
6cd2804bff Prepare changelog for 0.16.4 2024-02-04 19:50:34 +01:00
Michael Eischer
a72c2b74f3 Apply changelog entry / documentation improvements from review 2024-02-04 19:10:06 +01:00
Michael Eischer
261b1455c7 add documentation for --no-extra-verify option 2024-02-04 19:10:06 +01:00
Michael Eischer
2a0bd2b637 rename --no-verify-pack to --no-extra-verify 2024-02-04 19:10:05 +01:00
Michael Eischer
4589da7eb9 add data verification changelog entry 2024-02-04 19:09:49 +01:00
Michael Eischer
75e72d826c pack: verify integrity of pack file header 2024-02-04 19:09:49 +01:00
Michael Eischer
d8916bc3d9 repository: ask users to report corrupted data while saving blobs 2024-02-04 19:09:49 +01:00
Michael Eischer
dc11d012bb Make --no-verify-pack globally available
Verifying all blobs before upload comes with a notable performance
impact. Allow users to skip it if necessary.
2024-02-04 19:09:49 +01:00
Michael Eischer
8ef5425351 repository: test verification of blobs/unpacked data 2024-02-04 19:09:46 +01:00
Michael Eischer
885431ec2b repository: Allow skipping verification for tests
Some tests have to explicitly create pack files with blobs that don't
match their ID. For those blobs the builtin verification of the
repository must be disabled.
2024-02-04 19:08:30 +01:00
Michael Eischer
cb85fb46dd backup: verify unpacked files before upload 2024-02-04 19:07:48 +01:00
Michael Eischer
2f30c940b2 backup: verify blobs before upload
This only covers the blobs themselves, the pack header is not verified
so far. Unpacked files are also not covered by the integrity check.
2024-02-04 19:07:48 +01:00
Michael Eischer
0ea62b5ac6 repository: make repo.Options configurable for test repos 2024-02-04 19:07:46 +01:00
Michael Eischer
29e1caf825 add changelog draft for data corruption on max compression 2024-02-04 19:05:51 +01:00
Michael Eischer
0164f5310d Downgrade klauspost/compress to fix data corruption at max. compression 2024-02-04 19:05:50 +01:00
Michael Eischer
d5e662315a Merge pull request #4681 from MichaelEischer/verify-integrity-on-upload
backup: verify blobs before upload
2024-02-04 18:04:27 +00:00
Michael Eischer
effe76aaf5 Merge pull request #4679 from MichaelEischer/workaround-compression-bug
Downgrade klauspost/compress to fix data corruption at max. compression
2024-02-04 18:03:34 +00:00
Michael Eischer
5957417b1f Apply changelog entry / documentation improvements from review 2024-02-04 18:55:41 +01:00
Michael Eischer
219d8e3c18 add changelog draft for data corruption on max compression 2024-02-04 18:11:48 +01:00
Michael Eischer
a737fe1e47 add documentation for --no-extra-verify option 2024-02-04 17:11:49 +01:00
Michael Eischer
86b38a0b17 rename --no-verify-pack to --no-extra-verify 2024-02-04 17:01:05 +01:00
Michael Eischer
7d31180fe6 add data verification changelog entry 2024-02-04 15:48:11 +01:00
Michael Eischer
c32e5e2abb pack: verify integrity of pack file header 2024-02-04 15:31:42 +01:00
Michael Eischer
c97a271e89 repository: ask users to report corrupted data while saving blobs 2024-02-04 15:31:42 +01:00
Michael Eischer
66e8971659 Make --no-verify-pack globally available
Verifying all blobs before upload comes with a notable performance
impact. Allow users to skip it if necessary.
2024-02-04 15:31:42 +01:00
Michael Eischer
193140525c repository: test verification of blobs/unpacked data 2024-02-04 15:31:42 +01:00
Michael Eischer
96518d7c4a Merge pull request #4674 from restic/dependabot/go_modules/cloud.google.com/go/storage-1.37.0
build(deps): bump cloud.google.com/go/storage from 1.34.0 to 1.37.0
2024-02-03 17:23:49 +00:00
Michael Eischer
2dbb18128c repository: Allow skipping verification for tests
Some tests have to explicitly create pack files with blobs that don't
match their ID. For those blobs the builtin verification of the
repository must be disabled.
2024-02-03 18:22:47 +01:00
Michael Eischer
30a84e9003 backup: verify unpacked files before upload 2024-02-03 18:22:47 +01:00
Michael Eischer
c01a0c6da7 backup: verify blobs before upload
This only covers the blobs themselves, the pack header is not verified
so far. Unpacked files are also not covered by the integrity check.
2024-02-03 18:22:47 +01:00
Michael Eischer
16e3f79e8b repository: make repo.Options configurable for test repos 2024-02-03 18:22:47 +01:00
Michael Eischer
bb92b487f7 repository: fix repack test 2024-02-03 18:22:47 +01:00
dependabot[bot]
cf7cad11de build(deps): bump cloud.google.com/go/storage from 1.34.0 to 1.37.0
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.34.0 to 1.37.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.34.0...spanner/v1.37.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-02-02 20:40:57 +00:00
Michael Eischer
370d9c31f4 Merge pull request #4671 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/storage/azblob-1.2.1
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from 1.2.0 to 1.2.1
2024-02-02 20:31:16 +00:00
Michael Eischer
6581133e85 Merge pull request #4675 from restic/dependabot/go_modules/golang.org/x/oauth2-0.16.0
build(deps): bump golang.org/x/oauth2 from 0.15.0 to 0.16.0
2024-02-02 20:29:39 +00:00
Michael Eischer
207a4a5e8e Downgrade klauspost/compress to fix data corruption at max. compression 2024-02-02 20:10:29 +01:00
Michael Eischer
cbf9cd4a7f Merge pull request #4670 from joram-berger/patch-1
Link to Go Match syntax directly in 040_backup.rst
2024-02-02 18:40:13 +00:00
dependabot[bot]
552f01662b build(deps): bump golang.org/x/oauth2 from 0.15.0 to 0.16.0
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.15.0 to 0.16.0.
- [Commits](https://github.com/golang/oauth2/compare/v0.15.0...v0.16.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-02-01 01:53:12 +00:00
dependabot[bot]
7f5ea511bc build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
Bumps [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) from 1.2.0 to 1.2.1.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/v1.2...sdk/azidentity/v1.2.1)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-02-01 01:51:32 +00:00
Joram Berger
b07afa9b02 Link to Go Match syntax directly in 040_backup.rst
The docs linked to filepath.Glob (which is used) but the syntax is described in the documentation of filepath.Match. So it makes sense to link that directly.
2024-01-31 23:40:21 +01:00
Michael Eischer
8b08b522c9 Merge pull request #4669 from MichaelEischer/fix-rewrite-typo
rewrite: fix typo in dry-run output
2024-01-31 22:08:12 +01:00
Michael Eischer
eaf9659efc Merge pull request #4657 from numerigraphe/fix-key-add-id
Properly report the ID of newly added keys
2024-01-31 20:59:41 +00:00
Michael Eischer
ba136ff60c rewrite: fix typo in dry-run output 2024-01-31 21:48:37 +01:00
Lionel Sausin
8fbe328371 Properly report the ID of newly added keys
Other commands like key list and key remove show the key's ID.

Showing the ID here lets users easily reuse the ID as a key hint for subsequent
commands.
In particular, a key hint is needed when the repository has many keys - otherwise
opening the repository may fail with "Fatal: maximum number of keys reached" even
when a proper password is provided.

Fixes #4656
2024-01-29 10:12:49 +01:00
Michael Eischer
4273e06a43 Merge pull request #4662 from MichaelEischer/clarify-backup-ignore-inode
backup: clarify that --ignore-inode also ignores ctime
2024-01-27 18:25:11 +01:00
Michael Eischer
248c144f72 Merge pull request #4663 from MichaelEischer/key-subcommand-bugfix
key: return an error if subcommand is unknown
2024-01-27 18:24:50 +01:00
Michael Eischer
5dca8a70d5 ls: include standard message_type field in output 2024-01-27 15:48:24 +01:00
Michael Eischer
765729d009 key: return an error if subcommand is unknown 2024-01-27 15:33:49 +01:00
Michael Eischer
a09d51d96c backup: clarify that --ignore-inode also ignores ctime 2024-01-27 13:42:29 +01:00
Michael Eischer
e44e4b00a6 Merge pull request #4550 from ndecker/ls-ncdu
Ls ncdu
2024-01-27 12:27:35 +00:00
Michael Eischer
10e71af759 describe ls command in docs 2024-01-27 13:22:00 +01:00
Michael Eischer
c90f24a06c Merge pull request #4641 from MichaelEischer/reduce-restic-repository-usage
Misc cleanups
2024-01-27 13:18:20 +01:00
Michael Eischer
d4ed7c8858 walker: add tests for leaveDir 2024-01-27 13:17:33 +01:00
Michael Eischer
2c80cfa4a5 walker: fix missing leaveDir if directory is partially skipped 2024-01-27 13:17:33 +01:00
Michael Eischer
261737abc8 ls: only allow either --json or --ncdu 2024-01-27 13:17:33 +01:00
Michael Eischer
a2f2f8fb4c fix linter warning 2024-01-27 13:17:33 +01:00
Michael Eischer
4bae54d040 ls: test ncdu output format 2024-01-27 13:17:33 +01:00
Michael Eischer
509b339d54 ls: correctly handle setuid/setgit/sticky bit in ncdu output 2024-01-27 13:17:33 +01:00
Michael Eischer
a2fe337610 ls: unify printer implementations 2024-01-27 13:17:33 +01:00
Michael Eischer
1b008c92d3 ls: rework ncdu output to use walker.LeaveDir 2024-01-27 13:17:33 +01:00
Michael Eischer
9ecbda059c walker: add callback to inform about leaving a directory 2024-01-27 13:17:32 +01:00
Nils Decker
b2703a4089 add changelog for ls --ncdu 2024-01-27 13:17:15 +01:00
Nils Decker
a9310948cf command ls: add option for ncdu output
NCDU (NCurses Disk Usage) is a tool to analyse disk usage of directories.
It has an option to save a directory tree and analyse it later.
This patch adds an output option to the ls command.

A snapshot can be seen with

`restic ls latest --ncdu | ncdu -f -`

- https://dev.yorhel.nl/ncdu
2024-01-27 13:06:52 +01:00
Michael Eischer
246559e654 check: cleanup s3 legacy detection 2024-01-27 13:02:04 +01:00
Michael Eischer
1dfd854769 lock: switch to repository.List 2024-01-27 13:02:04 +01:00
Michael Eischer
bfb56b78e1 replace some usages of restic.Repository with more specific interface
This should eventually make it easier to test the code.
2024-01-27 13:02:02 +01:00
Michael Eischer
3424088274 Merge pull request #4644 from MichaelEischer/refactor-repair-packs
Refactor and test `repair packs`
2024-01-27 13:00:51 +01:00
Michael Eischer
724ec179e3 Merge pull request #4648 from MichaelEischer/repository-removekey
repository: Introduce RemoveKey function
2024-01-27 13:00:42 +01:00
Michael Eischer
f0e1ad2285 fix linter warning 2024-01-27 12:51:45 +01:00
Michael Eischer
fd579421dd repository: deduplicate test 2024-01-27 12:51:45 +01:00
Michael Eischer
42c9318b9c repair pack: add tests 2024-01-27 12:51:45 +01:00
Michael Eischer
764b0bacd6 repair pack: add support for truncated files 2024-01-27 12:51:45 +01:00
Michael Eischer
7c351bc53c repair pack: reenable auto index updates
The method is not available on the restic.Repository interface that is
used for testing. Drop the call as a small amount of additional index
writes is not a problem.
2024-01-27 12:51:45 +01:00
Michael Eischer
feeab84204 repair pack: extract the repair logic into the repository package
Currently, the cmd/restic package contains a significant amount of code
that modifies repository internals. This code should in the mid-term
move into the repository package.
2024-01-27 12:51:45 +01:00
Michael Eischer
d7a50fe739 properly show termstatus progress bar if visible less than one frame
If a progress bar using termstatus was only visible for less than one
frame, then its output could be lost.
2024-01-27 12:51:40 +01:00
Michael Eischer
6b65a495b1 backup/restore: fix termstatus initialization
The termstatus must only be canceled once the command has returned.
Otherwise output may be lost when the context gets canceled.
2024-01-27 12:51:08 +01:00
Michael Eischer
d26d2d41f8 backup/restore: extract termstatus initialization 2024-01-27 12:51:08 +01:00
Michael Eischer
cb50832d50 index: let MasterIndex.Save also delete obsolete indexes 2024-01-27 12:51:08 +01:00
Michael Eischer
bedff1ed6d split deleteFiles into UI and logic parts 2024-01-27 12:51:08 +01:00
Michael Eischer
c13bf0b607 repository: Introduce RemoveKey function
This replaces directly removing keys via the backend.
2024-01-27 12:42:58 +01:00
Michael Eischer
25ac1549e7 Merge pull request #4661 from MichaelEischer/clarify-contributing
CONTRIBUTING.md: Clarify handling of small bugfixes
2024-01-26 23:04:58 +00:00
Michael Eischer
ae9683336d CONTRIBUTING.md: Clarify handling of small bugfixes
Opening an issue for a small bugfix is usually not useful. It primarily
adds overhead.
2024-01-26 23:51:54 +01:00
Michael Eischer
446167ae80 Merge pull request #4643 from MichaelEischer/remove-redundant-poly1305-mask
Remove redundant poly1305 key masking
2024-01-23 19:46:40 +01:00
Michael Eischer
5b36c4eb5f Merge pull request #4647 from MichaelEischer/reduce-globals
Remove all usages of the global command-specific options
2024-01-23 19:46:15 +01:00
Michael Eischer
1419baf67a Merge pull request #4645 from MichaelEischer/improve-lock-checking
lock: checkForOtherLocks processes each lock at most once
2024-01-23 19:46:00 +01:00
Michael Eischer
66103aea3d Remove all usages of the global command-specific options
Now, every command uses an options struct, which is passed to the run*
function by the command.RunE method.
2024-01-23 19:21:39 +01:00
Michael Eischer
79f2939eb9 Merge pull request #4654 from adrian5/docfix
docs: fix formatting
2024-01-23 18:08:59 +00:00
Michael Eischer
0e2ee06803 Merge pull request #4650 from MichaelEischer/improve-stdin-from-command-description
backup: Improve help text for `--stdin-from-command`
2024-01-23 19:22:15 +01:00
Michael Eischer
2927982256 Merge pull request #4649 from MichaelEischer/simplify-termstatus-shutdown
ui/termstatus: simplify cleaning up on termination
2024-01-23 19:16:21 +01:00
Michael Eischer
6cc2bec5dd apply suggestion from review 2024-01-23 19:09:04 +01:00
Michael Eischer
18806944f6 doc: remove blockquotes from unordered lists 2024-01-23 19:03:54 +01:00
adrian5
609f84e095 docs: fix formatting 2024-01-22 21:12:12 +01:00
Michael Eischer
767c2539a0 backup: Improve help text for --stdin-from-command 2024-01-21 22:06:54 +01:00
Michael Eischer
6bdca13603 ui/termstatus: simplify cleaning up on termination
`writeStatus` also cleans no longer used status lines.
The old code actually cleaned one line too much. However, as that line
was never used it makes no difference.
2024-01-21 21:27:27 +01:00
Michael Eischer
f1f34eb3e5 lock: checkForOtherLocks processes each lock at most once
If a lock could not be loaded, then restic would check all lock files
again. These repeated checks are not useful as the status of a lock file
cannot change unless its ID changes too. Thus, skip already check lock
files on retries.
2024-01-20 22:40:12 +01:00
Michael Eischer
fee83e1c09 Remove redundant poly1305 key masking
The implementation in crypto/poly1305 already performs the exact same
masking.
2024-01-20 12:36:59 +01:00
Michael Eischer
6696195f38 Merge pull request #4584 from elkemper/fix-stop-archiving-metadata
S3: Don't archive metadata files on S3 Glacier
2024-01-20 10:30:37 +00:00
Michael Eischer
a763a5c67d s3: minor cleanups for archive storage class handling 2024-01-20 11:25:28 +01:00
Vladislav Belous
8ca58b487c S3: do not set storage class for metadata when using archive storage 2024-01-20 11:04:15 +01:00
Michael Eischer
62111f4379 Merge pull request #4625 from MichaelEischer/refactor-streampacks
Refactor repository.StreamPacks
2024-01-19 21:48:37 +01:00
Michael Eischer
2c310a526e repository: Replace StreamPack function with LoadBlobsFromPack method
LoadBlobsFromPack is now part of the repository struct. This ensures
that users of that method don't have to deal will internals of the
repository implementation.

The filerestorer tests now also contain far fewer pack file
implementation details.
2024-01-19 21:40:43 +01:00
Michael Eischer
6b7b5c89e9 repository: prepare StreamPack refactor 2024-01-19 21:40:43 +01:00
Michael Eischer
22d0c3f8dc check: Use PackBlobIterator instead of StreamPack
To only stream the content of a pack file once, check used StreamPack
with a custom pack load function. This combination was always brittle
and complicates using StreamPack everywhere else. Now that StreamPack
internally uses PackBlobIterator use that primitive instead, which is a
much better fit for what the check command requires.
2024-01-19 21:40:36 +01:00
Michael Eischer
fb422497af repository: split StreamPack implementation
Move the actual decoding of the pack data into a separate iterator.
2024-01-19 21:39:55 +01:00
Michael Eischer
54c5c72e5a Merge pull request #4616 from MichaelEischer/fix-rest-connection-close
rest: fix and cleanup closing of http response body
2024-01-19 21:31:35 +01:00
Michael Eischer
5f49eec655 Merge pull request #4615 from MichaelEischer/fix-find-empty-dirs
walker: Remove ignoreTrees functionality
2024-01-19 21:25:40 +01:00
Michael Eischer
ec13105093 Merge pull request #4623 from MichaelEischer/docs-verify-release-binaries
Add documentation for the verify-release-binaries.sh script
2024-01-19 21:17:38 +01:00
Michael Eischer
bd883caae1 CI: enable bodyclose linter 2024-01-19 21:17:18 +01:00
Michael Eischer
b1a8fd1d03 rest: fix and cleanup closing of http response body
If client.Do returns an error, then there's no body that has to be
closed. For requests for which we are not interested in the response
body, immediately drain and close the body to make sure it isn't
forgotten later on.

This change in particular adds the missing `Close()` call for the
`List()` command.
2024-01-19 21:17:17 +01:00
Michael Eischer
fdcbb53017 walker: test skipping for root node 2024-01-19 21:16:06 +01:00
Michael Eischer
0b39940fdb walker: Remove ignoreTrees functionality
It was only used in two places:
- stats: apparently as a minor performance optimization, which is
  unlikely to be important
- find: filtered directories would be ignored. However, this
  optimization missed that it is possible that two directories have the
  exact same content. Such directories would be incorrectly ignored too.
  Example:
```
mkdir test test/a test/b
restic backup test
restic find latest test/b
-> incorrectly does not return anything
```

Thus, remove the functionality as it's apparently too complex to use
correctly.
2024-01-19 21:16:06 +01:00
Michael Eischer
147b0e54cb Merge pull request #4639 from northben/patch-1
Update Backblaze documentation
2024-01-19 20:08:48 +00:00
Ben Northway
5413877d33 Update Backblaze documentation
clarify documentation regarding B2 bucket lifecycle settings. The default lifecycle setting is probably fine for most users now; a custom policy is not necessary.
2024-01-18 16:41:01 -06:00
Michael Eischer
03e06d0797 Merge branch 'patch-release' 2024-01-14 21:38:17 +01:00
Alexander Neumann
0ec9383ba2 Set development version for 0.16.3 2024-01-14 20:21:45 +01:00
Alexander Neumann
abca112404 Add version for 0.16.3 2024-01-14 20:21:45 +01:00
Alexander Neumann
b70b94507a Generate CHANGELOG.md for 0.16.3 2024-01-14 20:21:19 +01:00
Alexander Neumann
d987582594 Prepare changelog for 0.16.3 2024-01-14 20:21:19 +01:00
Leo R. Lundgren
ef2e473b99 doc: Polish changelogs 2024-01-10 00:19:07 +01:00
Michael Eischer
7b2de84763 Merge pull request #4618 from MichaelEischer/workaround-rclone-list-errors
rclone: Workaround for incorrect "not found" errors while listing files
2024-01-09 18:28:31 +01:00
Michael Eischer
e4bbde7036 rclone: Workaround for incorrect "not found" errors while listing files
rclone returns a "not found" error if an internal error occurs while
listing a folder. Ignoring this error lets restic erroneously think that
there are no files, which can cause `prune` to wipe the whole
repository.
2024-01-09 18:28:17 +01:00
Michael Eischer
ec0fb46f6c add changelog for reliable restores 2024-01-09 18:27:48 +01:00
Michael Eischer
103beb96bc restore: separately restore blobs that are frequently referenced
Writing these blobs to their files can take a long time and consequently
cause the backend connection to time out. Avoid that by retrieving these
blobs separately.
2024-01-09 18:27:48 +01:00
Michael Eischer
f0f89d7f27 restore: split error reporting from downloadPack 2024-01-09 18:27:48 +01:00
Michael Eischer
cf352ccafb restore: cleanup downloadPack 2024-01-09 18:27:48 +01:00
Michael Eischer
b856e9489a restore: split downloadPack into smaller methods 2024-01-09 18:27:48 +01:00
Michael Eischer
c31e9418ba Merge pull request #4626 from MichaelEischer/reliable-large-restores
Improve reliability of large restores
2024-01-09 18:23:09 +01:00
Michael Eischer
2e8de9edfd rclone: Workaround for incorrect "not found" errors while listing files
rclone returns a "not found" error if an internal error occurs while
listing a folder. Ignoring this error lets restic erroneously think that
there are no files, which can cause `prune` to wipe the whole
repository.
2024-01-09 18:20:16 +01:00
Michael Eischer
ce7db90e08 sync CI and go dependencies with master branch 2024-01-08 21:33:05 +01:00
Michael Eischer
620518aec6 add changelog for better restore error reporting 2024-01-08 21:33:05 +01:00
Michael Eischer
f2fafbffaa restore: only report errors for blobs that actually failed to load
Previously, errors would be reported for all blobs of a packfile that
failed to stream. Now, only the not yet processed blobs are reported.
2024-01-08 21:33:05 +01:00
Michael Eischer
7a3a884874 repository: test that StreamPack only delivers blobs once 2024-01-08 21:33:05 +01:00
Michael Eischer
772a907533 repository: StreamPack delivers blobs at most once
If an error occurred while streaming a pack file, this could result in
passing some of the blobs multiple times to the callback function. This
significantly complicates using StreamPack correctly and is unnecessary.
Retries do not change the content of a blob and thus only deliver the
same result over and over again.
2024-01-08 21:33:05 +01:00
Michael Eischer
a9446c1184 add changelog for irregular files on windows 2024-01-08 21:33:05 +01:00
Michael Eischer
1bab29c336 archiver: Add filepath to error message if it is not included yet 2024-01-08 21:33:05 +01:00
Michael Eischer
e886c3f6b2 archiver: improve error message for irregular files
Since Go 1.21, most reparse points are considered as irregular files.
Depending on the underlying driver these can exhibit nearly arbitrary
behavior. When encountering such a file, restic returned an
indecipherable error message: `error: invalid node type ""`.

Add the filepath to the error message and state that the file type is
not supported.
2024-01-08 21:33:05 +01:00
Michael Eischer
c95de54726 restic: cleanup node type determination
os.ModeCharDevice is already included in os.ModeType
2024-01-08 21:33:05 +01:00
Michael Eischer
d4b8abd3e2 fix deduplicated files on windows 2024-01-08 21:33:05 +01:00
Joram Berger
948ab3ccaf Add a note that the oldest snapshot may be kept additionally
Documentation enhancement.
2024-01-08 21:33:05 +01:00
Markus Zoppelt
bb0c923298 docs: add pkgx install option
PR in pkgx pantry: https://github.com/pkgxdev/pantry/pull/4098

restic pkg:
https://pkgx.dev/pkgs/restic.net/restic/
2024-01-08 21:33:05 +01:00
Michael Eischer
ff0c975443 regenerate changelog 2024-01-08 21:33:05 +01:00
Michael Eischer
7e61e117d6 cleanup changelog whitespace 2024-01-08 21:33:05 +01:00
mmattel
220a28582e Add a table of contents (TOC) to the changelog template 2024-01-08 21:33:05 +01:00
Giuseppe D'Andrea
f44fd73230 docs: fix typo in working with repos
When using the `copy` command, `--from-password-file` and `--from-password-command` flags are used to specify the password of the source repository, not of the destination repository.
2024-01-08 21:33:05 +01:00
Joda Stößer
76bd975e03 docs(scripting): correct stats output comment to be about the correct command
not about the snapshots command
2024-01-08 21:33:05 +01:00
Quang-Linh LE
64b7aed362 docs: Mention progress for restore command. This is available after https://github.com/restic/restic/pull/3991 2024-01-08 21:33:05 +01:00
Michael Kuhn
3fa6b2de4a Fix repository not being printed when using repository file
When using `RESTIC_REPOSITORY_FILE` in combination with `restic init`,
the repository is missing in the output:
```
$ restic init
created restic repository 3c872be20f at
[...]
```
This is due to the code using `gopts.Repo`, which is empty in this case.
2024-01-08 21:33:05 +01:00
Michael Eischer
5cd000f4b0 CI: update golangci-lint
Necessary to properly support Go 1.21.
2024-01-08 21:33:05 +01:00
Michael Eischer
4ea3796455 add changelog for reliable restores 2024-01-08 21:03:10 +01:00
Michael Eischer
e78be75d1e restore: separately restore blobs that are frequently referenced
Writing these blobs to their files can take a long time and consequently
cause the backend connection to time out. Avoid that by retrieving these
blobs separately.
2024-01-08 21:00:13 +01:00
Michael Eischer
2267910418 restore: split error reporting from downloadPack 2024-01-08 20:57:00 +01:00
Michael Eischer
00d18b7a88 restore: cleanup downloadPack 2024-01-08 20:53:08 +01:00
Michael Eischer
9328f34d43 restore: split downloadPack into smaller methods 2024-01-08 20:52:36 +01:00
Michael Eischer
77434c6e2b Merge pull request #4474 from ekristen/aws-assume-role
Allow AWS Assume Role
2024-01-08 19:07:17 +00:00
Michael Eischer
4248c6c3ca s3: update documentation 2024-01-07 19:30:11 +01:00
Michael Eischer
e4a7eb09ef Merge pull request #4624 from MichaelEischer/better-restorer-error-reporting
Improver restorer error reporting
2024-01-07 11:20:29 +01:00
Michael Eischer
f8b4e932ef Merge pull request #4620 from MichaelEischer/improve-irregular-file-handling
Improve irregular file handling
2024-01-07 11:12:07 +01:00
Michael Eischer
100872308f add changelog for better restore error reporting 2024-01-07 11:06:42 +01:00
Michael Eischer
dac3508170 restore: only report errors for blobs that actually failed to load
Previously, errors would be reported for all blobs of a packfile that
failed to stream. Now, only the not yet processed blobs are reported.
2024-01-07 10:54:56 +01:00
Michael Eischer
77b1c52673 repository: test that StreamPack only delivers blobs once 2024-01-07 10:54:53 +01:00
Michael Eischer
fe5c337ca2 repository: StreamPack delivers blobs at most once
If an error occurred while streaming a pack file, this could result in
passing some of the blobs multiple times to the callback function. This
significantly complicates using StreamPack correctly and is unnecessary.
Retries do not change the content of a blob and thus only deliver the
same result over and over again.
2024-01-07 10:54:49 +01:00
Michael Eischer
3e29f8dddf add changelog for irregular files on windows 2024-01-07 10:52:12 +01:00
Michael Eischer
76f507c775 Merge pull request #4621 from MichaelEischer/fix-windows-dedup-files
Fix backup of deduplicated files on windows
2024-01-07 10:44:50 +01:00
Michael Eischer
6ef23b401b fix deduplicated files on windows 2024-01-07 10:23:31 +01:00
Michael Eischer
62f99a3b2f Add documentation for the verify-release-binaries.sh script 2024-01-07 10:16:30 +01:00
Michael Eischer
0360e540af Merge pull request #4622 from MichaelEischer/fix-outdated-windows-import
termstatus: update import path of golang.org/x/term
2024-01-06 23:24:16 +01:00
Michael Eischer
e6dfefba13 termstatus: update import path of golang.org/x/term 2024-01-06 21:59:26 +01:00
Michael Eischer
02bc73f5eb s3: minor code cleanups 2024-01-06 21:44:53 +01:00
Michael Eischer
20cf4777cb s3: check for EnvAWS credentials before Static credentials
EnvAWS considers more environment variables, including AWS_SESSION_TOKEN
and thus should be checked first.
2024-01-06 21:43:47 +01:00
Erik Kristensen
5ffb536aae feat: support AWS assume role 2024-01-06 21:19:58 +01:00
Michael Eischer
1604922360 Merge pull request #4527 from adamantike/cmd/copy/prefix-hostname-to-snapshot-paths
cmd: Add hostname to snapshot display output
2024-01-06 19:32:14 +00:00
Michael Eischer
c7844530d8 update docs 2024-01-06 20:25:24 +01:00
Michael Eischer
33b7c84a7a deduplicate string formatting of snapshot metadata
This removes the spurious ")" bracket at the end and normalizes the
metadata format used by the `ls` command.
2024-01-06 20:20:51 +01:00
Michael Manganiello
045aa64558 cmd/copy: Prefix hostname to snapshot display output
This change better resembles the output generated by `Snapshot.String()`,
which includes both username and hostname.

Closes #4506

Before:

```
$ restic copy --from-repo /srv/restic-repo
repository 3666882b opened (version 2, compression level auto)
repository 0085c387 opened (version 2, compression level auto)
created new cache in /home/mike/.cache/restic
[0:00] 100.00%  1 / 1 index files loaded
[0:00]          0 index files loaded

snapshot 32b39a20 of [/home/mike/data] at 2023-10-21 16:01:13.979948154 -0300 -03)
  copy started, this may take a while...
[0:00] 100.00%  1 / 1 packs copied
snapshot 10331fdd saved
```

After:

```
$ restic copy --from-repo /srv/restic-repo
repository 3666882b opened (version 2, compression level auto)
repository 0085c387 opened (version 2, compression level auto)
[0:00] 100.00%  1 / 1 index files loaded
[0:00]          0 index files loaded

snapshot 32b39a20 of [/home/mike/data] at 2023-10-21 16:01:13.979948154 -0300 -03 by mike@desktop)
  copy started, this may take a while...
[0:00] 100.00%  1 / 1 packs copied
snapshot a67bd1ee saved
```
2024-01-06 20:20:46 +01:00
Michael Eischer
b2b7669ca0 Merge pull request #4526 from dnnr/detect-bitrot-in-diff
Add bitrot detection to "diff" command
2024-01-06 19:18:34 +00:00
Michael Eischer
4f6b1bb6f6 diff: document limitations regarding metadata 2024-01-06 20:12:47 +01:00
Michael Eischer
3549635243 diff: copy nodes before modifying them for bitrot detection 2024-01-06 20:12:47 +01:00
Daniel Danner
a7dc18e697 Add bitrot detection to "diff" command
This introduces a new modifier to the output of the diff command. It
appears whenever two files being compared only differ in their content
but not in their metadata. As far as we know, under normal
circumstances, this should only ever happen if some kind of bitrot has
happened in the source file. The prerequisite for this detection to work
is that the right-side snapshot of the comparison has been created with
"backup --force".
2024-01-06 20:12:47 +01:00
Michael Eischer
51419c51d3 archiver: Add filepath to error message if it is not included yet 2024-01-06 19:08:24 +01:00
Michael Eischer
6b79834cc8 archiver: improve error message for irregular files
Since Go 1.21, most reparse points are considered as irregular files.
Depending on the underlying driver these can exhibit nearly arbitrary
behavior. When encountering such a file, restic returned an
indecipherable error message: `error: invalid node type ""`.

Add the filepath to the error message and state that the file type is
not supported.
2024-01-06 19:03:11 +01:00
Michael Eischer
0018bb7854 restic: cleanup node type determination
os.ModeCharDevice is already included in os.ModeType
2024-01-06 18:43:16 +01:00
Michael Eischer
634e2a46d9 Merge pull request #4608 from restic/dependabot/go_modules/github.com/minio/minio-go/v7-7.0.66
build(deps): bump github.com/minio/minio-go/v7 from 7.0.63 to 7.0.66
2024-01-06 11:40:02 +00:00
Michael Eischer
dfcab92db2 Merge pull request #4609 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/azcore-1.9.1
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.8.0 to 1.9.1
2024-01-06 11:06:07 +00:00
dependabot[bot]
3666eef76c build(deps): bump github.com/minio/minio-go/v7 from 7.0.63 to 7.0.66
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.63 to 7.0.66.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.63...v7.0.66)

---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-01-06 11:03:52 +00:00
Michael Eischer
3a61622dfe Merge pull request #4610 from restic/dependabot/github_actions/actions/setup-go-5
build(deps): bump actions/setup-go from 4 to 5
2024-01-06 11:01:07 +00:00
Michael Eischer
5c4fca76df Merge pull request #4607 from restic/dependabot/go_modules/golang.org/x/crypto-0.17.0
build(deps): bump golang.org/x/crypto from 0.16.0 to 0.17.0
2024-01-06 10:56:40 +00:00
Michael Eischer
98da0bdd12 Merge pull request #4606 from restic/dependabot/go_modules/github.com/klauspost/compress-1.17.4
build(deps): bump github.com/klauspost/compress from 1.17.2 to 1.17.4
2024-01-06 10:55:00 +00:00
dependabot[bot]
2c60dd97ae build(deps): bump actions/setup-go from 4 to 5
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 4 to 5.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-01-01 01:33:20 +00:00
dependabot[bot]
40905403f4 build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore
Bumps [github.com/Azure/azure-sdk-for-go/sdk/azcore](https://github.com/Azure/azure-sdk-for-go) from 1.8.0 to 1.9.1.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.8.0...sdk/azcore/v1.9.1)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azcore
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-01-01 01:18:36 +00:00
dependabot[bot]
7e7cbe8e19 build(deps): bump golang.org/x/crypto from 0.16.0 to 0.17.0
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.16.0 to 0.17.0.
- [Commits](https://github.com/golang/crypto/compare/v0.16.0...v0.17.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-01-01 01:18:08 +00:00
dependabot[bot]
44646c20be build(deps): bump github.com/klauspost/compress from 1.17.2 to 1.17.4
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.17.2 to 1.17.4.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml)
- [Commits](https://github.com/klauspost/compress/compare/v1.17.2...v1.17.4)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-01-01 01:17:57 +00:00
Michael Eischer
8f9a35779e Merge pull request #4600 from MichaelEischer/doc-forget-note-1
Add a note that the oldest snapshot may be kept additionally
2023-12-29 17:56:54 +01:00
Michael Eischer
23e1b4bbb1 Merge pull request #4573 from gab50000/rewrite_time
Rewrite metadata
2023-12-24 14:42:00 +00:00
Michael Eischer
01b33734ab rewrite: update command output in docs 2023-12-24 15:36:22 +01:00
Michael Eischer
649a6409ee rewrite: cleanup tests 2023-12-24 15:36:22 +01:00
Michael Eischer
c31f5f986c rewrite: Minor cleanups 2023-12-24 15:36:22 +01:00
Michael Eischer
2730d05fce rewrite: Don't walk snapshot content if only metadata is modified 2023-12-24 15:36:22 +01:00
Michael Eischer
893d0d6325 rewrite: cleanup new metadata options and fix no parameters check 2023-12-24 15:36:22 +01:00
Gabriel Kabbe
7de97d7480 rewrite: Add documentation 2023-12-24 15:36:22 +01:00
Gabriel Kabbe
004520a238 rewrite: Add changelog 2023-12-24 15:36:22 +01:00
Gabriel Kabbe
a02d8d75c2 rewrite: Implement rewriting metadata 2023-12-24 15:36:22 +01:00
Gabriel Kabbe
7bf38b6c50 rewrite: Add test TestRewriteMetadata 2023-12-24 15:36:22 +01:00
Gabriel Kabbe
da1704b2d5 rewrite: Add tests
Pass nil instead of metadata to existing tests
2023-12-24 15:36:19 +01:00
Gabriel Kabbe
3026baea07 rewrite: Add structs for tracking metadata changes
Adds

  * snapshotMetadataArgs, which holds the new metadata as strings parsed from
    the command line

  * snapshotMetadata, which holds the new metadata converted to the
    correct types
2023-12-24 14:43:07 +01:00
Michael Eischer
1196c72819 Merge pull request #4570 from MarkusZoppelt/docs/pkgx
docs: add pkgx install option
2023-12-24 11:04:59 +00:00
Michael Eischer
433dd92959 Merge pull request #4590 from renard/optimize-mount-failure
mount: detect mountpoint does not exist before opening the repository
2023-12-24 10:59:56 +00:00
Markus Zoppelt
c14740c50f docs: add pkgx install option
PR in pkgx pantry: https://github.com/pkgxdev/pantry/pull/4098

restic pkg:
https://pkgx.dev/pkgs/restic.net/restic/
2023-12-24 11:59:12 +01:00
Michael Eischer
5537460664 Merge pull request #4598 from MichaelEischer/add_table_of_contents
Add table of contents
2023-12-24 11:56:05 +01:00
Sébastien Gross
f7587be28f mount: detect mountpoint does not exist before opening the repository
Bug #1681 suggests that restic should not be nice to user and should
refrain from creating a mountpoint if it does not exist. Nevertheless,
it currently opens the repository before checking for the mountpoint's
existence. In the case of large or remote repositories, this process
can be time-consuming, delaying the inevitable outcome.

    /restic mount --repo=REMOTE --verbose /tmp/backup
    repository 33f14e42 opened (version 2, compression level max)
    [0:38] 100.00%  162 / 162 index files loaded
    Mountpoint /tmp/backup doesn't exist
    stat /tmp/backup: no such file or directory

    real	0m39.534s
    user	1m53.961s
    sys	0m3.044s

In this scenario, 40 seconds could have been saved if the nonexistence
of the path had been verified beforehand.

This patch relocates the mountpoint check to the beginning of the
runMount function, preceding the opening of the repository.

    /restic mount --repo=REMOTE --verbose /tmp/backup
    Mountpoint /tmp/backup doesn't exist
    stat /tmp/backup: no such file or directory

    real	0m0.136s
    user	0m0.018s
    sys	0m0.027s

Signed-off-by: Sébastien Gross <seb•ɑƬ•chezwam•ɖɵʈ•org>
2023-12-24 11:54:18 +01:00
Michael Eischer
91fb703756 regenerate changelog 2023-12-24 11:47:31 +01:00
Michael Eischer
d7ff862b8d cleanup changelog whitespace 2023-12-24 11:47:31 +01:00
mmattel
db1d920c80 Add a table of contents (TOC) to the changelog template 2023-12-24 11:47:31 +01:00
Michael Eischer
c6299f8dbd Merge pull request #4582 from giuseppedandrea/docs/fix-typo
docs: fix typo in working with repos
2023-12-24 00:09:59 +00:00
Giuseppe D'Andrea
a128976014 docs: fix typo in working with repos
When using the `copy` command, `--from-password-file` and `--from-password-command` flags are used to specify the password of the source repository, not of the destination repository.
2023-12-24 01:04:36 +01:00
Michael Eischer
e2f6109a52 Merge pull request #4580 from restic/dependabot/go_modules/golang.org/x/oauth2-0.15.0
build(deps): bump golang.org/x/oauth2 from 0.13.0 to 0.15.0
2023-12-24 00:02:30 +00:00
Michael Eischer
30e6ed038c Merge pull request #4586 from Gelma/typos
Fix typos
2023-12-23 13:21:50 +00:00
Michael Eischer
e96d1ee33e Merge pull request #4593 from SimJoSt/patch-1
docs(scripting): correct stats output comment to be about the correct command
2023-12-23 13:09:20 +00:00
dependabot[bot]
0054db394f build(deps): bump golang.org/x/oauth2 from 0.13.0 to 0.15.0
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.13.0 to 0.15.0.
- [Commits](https://github.com/golang/oauth2/compare/v0.13.0...v0.15.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-23 13:08:29 +00:00
Michael Eischer
53ebe91a50 Move changelog typo fixes to changelog files 2023-12-23 14:05:47 +01:00
Michael Eischer
9ceaea34dd Merge pull request #4572 from linktohack/patch-1
docs: Mention progress for restore command. This is available after https://github.com/restic/restic/pull/3991
2023-12-23 13:00:32 +00:00
Michael Eischer
356b7aac16 Merge pull request #4571 from michaelkuhn/init-repo-file
Fix repository not being printed when using repository file
2023-12-23 12:59:06 +00:00
Joda Stößer
eef7c65655 docs(scripting): correct stats output comment to be about the correct command
not about the snapshots command
2023-12-23 13:56:40 +01:00
Michael Eischer
97b8629336 Merge pull request #4579 from restic/dependabot/go_modules/golang.org/x/sync-0.5.0
build(deps): bump golang.org/x/sync from 0.4.0 to 0.5.0
2023-12-23 12:51:52 +00:00
Michael Eischer
d2ecd6bef2 Merge pull request #4577 from restic/dependabot/go_modules/golang.org/x/time-0.5.0
build(deps): bump golang.org/x/time from 0.3.0 to 0.5.0
2023-12-23 12:50:59 +00:00
Michael Eischer
634750a732 Merge pull request #4576 from restic/dependabot/github_actions/docker/login-action-3d58c274f17dffee475a5520cbe67f0a882c4dbb
build(deps): bump docker/login-action from 1f401f745bf57e30b3a2800ad308a87d2ebdf14b to 3d58c274f17dffee475a5520cbe67f0a882c4dbb
2023-12-23 12:50:52 +00:00
Quang-Linh LE
c554825e2d docs: Mention progress for restore command. This is available after https://github.com/restic/restic/pull/3991 2023-12-23 13:50:20 +01:00
Michael Kuhn
fd2fb233aa Fix repository not being printed when using repository file
When using `RESTIC_REPOSITORY_FILE` in combination with `restic init`,
the repository is missing in the output:
```
$ restic init
created restic repository 3c872be20f at
[...]
```
This is due to the code using `gopts.Repo`, which is empty in this case.
2023-12-23 13:49:22 +01:00
Michael Eischer
da4e3edbbc Merge pull request #4596 from MichaelEischer/update-golangcilint
CI: update golangci-lint
2023-12-23 13:48:33 +01:00
Michael Eischer
dbbd31bc3a CI: update golangci-lint
Necessary to properly support Go 1.21.
2023-12-23 13:41:30 +01:00
Joram Berger
12af20e606 Add a note that the oldest snapshot may be kept additionally
Documentation enhancement.
2023-12-18 18:24:57 +01:00
Andrea Gelmini
241916d55b Fix typos 2023-12-06 13:11:55 +01:00
dependabot[bot]
427b90cf82 build(deps): bump golang.org/x/sync from 0.4.0 to 0.5.0
Bumps [golang.org/x/sync](https://github.com/golang/sync) from 0.4.0 to 0.5.0.
- [Commits](https://github.com/golang/sync/compare/v0.4.0...v0.5.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sync
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-01 01:15:09 +00:00
dependabot[bot]
eec6e014f4 build(deps): bump golang.org/x/time from 0.3.0 to 0.5.0
Bumps [golang.org/x/time](https://github.com/golang/time) from 0.3.0 to 0.5.0.
- [Commits](https://github.com/golang/time/compare/v0.3.0...v0.5.0)

---
updated-dependencies:
- dependency-name: golang.org/x/time
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-01 01:14:53 +00:00
dependabot[bot]
fa46a47e22 build(deps): bump docker/login-action
Bumps [docker/login-action](https://github.com/docker/login-action) from 1f401f745bf57e30b3a2800ad308a87d2ebdf14b to 3d58c274f17dffee475a5520cbe67f0a882c4dbb.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](1f401f745b...3d58c274f1)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-01 01:02:16 +00:00
Michael Eischer
b72de5a883 Merge branch 'patch-release' 2023-11-12 11:52:21 +01:00
Michael Eischer
6086ae4ca7 Merge pull request #4563 from smlx/mac-signature
fix: drop reference to signature and define MAC
2023-11-08 21:59:13 +00:00
Scott Leggett
aeaf527be1 fix: drop reference to signature and define MAC
Poly1305-AES is not a signature, so don't mention that.

In addition, the term MAC was used without being defined, so add a
definition.

Signed-off-by: Scott Leggett <scott@sl.id.au>
2023-11-06 20:12:42 +08:00
Michael Eischer
19068aa82f Merge pull request #4559 from DRON-666/html-zip
Restore generation of `HTMLZip` files on `readthedocs.com`.
2023-11-02 20:53:12 +00:00
Michael Eischer
81ca9d28f2 Merge pull request #4553 from CommanderRoot/add-version-json
Add --json option to version command
2023-11-01 21:58:38 +00:00
Tobias Speicher
ce53ea32c6 Split go_target into go_os and go_arch 2023-11-01 22:43:38 +01:00
Tobias Speicher
10cbc169c1 Use different function to be more consistent with other code 2023-11-01 22:18:37 +01:00
DRON-666
03f8f494e9 doc: add HTMLZip format to .readthedocs.yaml 2023-11-02 00:16:47 +03:00
Tobias Speicher
ab23d033b6 Add version command output to JSON format documentation 2023-11-01 22:13:57 +01:00
Tobias Speicher
6f1efcb28b Update wording on changelog entry 2023-11-01 22:12:19 +01:00
Michael Eischer
9c399e55e3 Merge pull request #4554 from restic/dependabot/github_actions/docker/login-action-1f401f745bf57e30b3a2800ad308a87d2ebdf14b
build(deps): bump docker/login-action from b4bedf8053341df3b5a9f9e0f2cf4e79e27360c6 to 1f401f745bf57e30b3a2800ad308a87d2ebdf14b
2023-11-01 20:42:11 +00:00
Michael Eischer
e550bc0713 Merge pull request #4555 from restic/dependabot/go_modules/google.golang.org/api-0.149.0
build(deps): bump google.golang.org/api from 0.148.0 to 0.149.0
2023-11-01 20:40:45 +00:00
Michael Eischer
28aa9826af Merge pull request #4557 from restic/dependabot/go_modules/cloud.google.com/go/storage-1.34.0
build(deps): bump cloud.google.com/go/storage from 1.33.0 to 1.34.0
2023-11-01 20:40:40 +00:00
dependabot[bot]
6dde019ac8 build(deps): bump cloud.google.com/go/storage from 1.33.0 to 1.34.0
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.33.0 to 1.34.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.33.0...spanner/v1.34.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-11-01 01:45:02 +00:00
dependabot[bot]
c2f9e21d3c build(deps): bump google.golang.org/api from 0.148.0 to 0.149.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.148.0 to 0.149.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.148.0...v0.149.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-11-01 01:44:28 +00:00
dependabot[bot]
67e6b9104a build(deps): bump docker/login-action
Bumps [docker/login-action](https://github.com/docker/login-action) from b4bedf8053341df3b5a9f9e0f2cf4e79e27360c6 to 1f401f745bf57e30b3a2800ad308a87d2ebdf14b.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](b4bedf8053...1f401f745b)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-11-01 01:25:22 +00:00
Tobias Speicher
6ca07ee004 add changelog/unreleases for issue-4547 2023-10-31 19:39:52 +01:00
Tobias Speicher
d45cc52468 command version: add json output option 2023-10-31 19:31:07 +01:00
Alexander Neumann
59fe24cb2b Set development version for 0.16.2 2023-10-29 19:50:12 +01:00
Alexander Neumann
1a5efcf680 Add version for 0.16.2 2023-10-29 19:50:07 +01:00
Alexander Neumann
d33fe6dd3c Generate CHANGELOG.md for 0.16.2 2023-10-29 19:50:04 +01:00
Alexander Neumann
c8dd95f104 Prepare changelog for 0.16.2 2023-10-29 19:50:04 +01:00
Leo R. Lundgren
7d980b469d doc: Polish changelogs 2023-10-29 00:44:53 +02:00
Michael Eischer
d863234e3e add changelog for missing documentation 2023-10-28 20:25:24 +02:00
Michael Eischer
42ab3ea2b9 Merge pull request #4410 from Enrico204/restic-stdin-command
add --stdin-from-command flag to backup command
2023-10-27 22:09:06 +00:00
Michael Eischer
be28a02626 doc: tweak description for --stdin-from-command 2023-10-27 23:58:52 +02:00
Michael Eischer
5d152c7720 extend changelog for --stdin-from-command 2023-10-27 23:58:52 +02:00
Michael Eischer
ee305e6041 backup: rework error reporting for subcommand 2023-10-27 23:58:52 +02:00
Michael Eischer
8bceb8e359 fs: add tests for CommandReader 2023-10-27 23:58:52 +02:00
Michael Eischer
317144c1d6 fs: merge command startup into CommandReader 2023-10-27 23:58:51 +02:00
Michael Eischer
7d879705ad fs: cleanup CommandReader implementation 2023-10-27 23:58:51 +02:00
Enrico204
37a312e505 restic-from-command: use standard behavior when no output and exit code 0 from command
The behavior of the new option should reflect the behavior of normal backups: when the command exit code is zero and there is no output in the stdout, emit a warning but create the snapshot. This commit fixes the integration tests and the ReadCloserCommand struct.
2023-10-27 23:58:51 +02:00
Enrico204
c0ca54dc8a restic-from-command: add tests 2023-10-27 23:58:51 +02:00
Enrico204
81f8d473df restic-from-command: abort snapshot on non-zero exit codes 2023-10-27 23:58:51 +02:00
Enrico204
6990b0122e Add issue-4251 (stdin-from-command) in the changelog 2023-10-27 23:58:51 +02:00
Enrico204
072b227544 stdin-from-command: add documentation in backup sub-command 2023-10-27 23:58:51 +02:00
Enrico204
4e5caab114 stdin-from-command: implemented suggestions in #4254
The code has been refactored so that the archiver is back to the original code, and the stderr is handled using a go routine to avoid deadlock.
2023-10-27 23:58:51 +02:00
Sebastian Hoß
c133065a9f Check command result before snapshotting
Return with an error containing the stderr of the given command in case it fails. No new snapshot will be created and future prune operations on the repository will remove the unreferenced data.

Signed-off-by: Sebastian Hoß <seb@xn--ho-hia.de>
2023-10-27 23:58:51 +02:00
Sebastian Hoß
25350a9c55 Extend SnapshotOptions w/ command data
In order to determine whether to save a snapshot, we need to capture the exit code returned by a command. In order to provide a nice error message, we supply stderr as well.

Signed-off-by: Sebastian Hoß <seb@xn--ho-hia.de>
2023-10-27 23:58:51 +02:00
Sebastian Hoß
a2b76ff34f Start command from --stdin-from-command
It acts similar to --stdin but reads its data from the stdout of the given command instead of os.Stdin.

Signed-off-by: Sebastian Hoß <seb@xn--ho-hia.de>
2023-10-27 23:58:51 +02:00
Sebastian Hoß
333fe1c3cf Align Stdin and StdinCommand in conditionals
In order to run with --stdin-from-command we need to short-circuit some functions similar to how it is handled for the --stdin flag. The only difference here is that --stdin-from-command actually expects that len(args) should be greater 0 whereas --stdin does not expect any args at all.

Signed-off-by: Sebastian Hoß <seb@xn--ho-hia.de>
2023-10-27 23:58:51 +02:00
Sebastian Hoß
a8657bde68 Add --stdin-from-command option
This new flag is added to the backup subcommand in order to allow restic to control the execution of a command and determine whether to save a snapshot if the given command succeeds.

Signed-off-by: Sebastian Hoß <seb@xn--ho-hia.de>
2023-10-27 23:58:51 +02:00
Michael Eischer
104107886a Merge pull request #4503 from MichaelEischer/fix-stats-with-hardlinks
stats: Fix hardlink tracking across multiple filesystems
2023-10-27 23:53:03 +02:00
Michael Eischer
731b3a4357 stats: fix hardlink tracking in a snapshot
inodes are only unique within a device. Use the HardlinkIndex from the
restorer instead of the custom (broken) hashmap to correctly account for
both inode and deviceID.
2023-10-27 23:40:42 +02:00
Michael Eischer
a8fdcf79b7 restorer: Make hardlink index generic
This will allow reusing it for the stats command without regressing the
memory usage due to storing an unnecessary file path.
2023-10-27 23:40:42 +02:00
Michael Eischer
45962c2847 Merge pull request #4499 from MichaelEischer/modular-backend-code
Split backend code from restic package
2023-10-27 20:19:20 +02:00
Michael Eischer
4be45de1c2 Restore support for ARMv5 platforms 2023-10-27 19:52:14 +02:00
Leo R. Lundgren
8c1125fe13 doc: Correct two typos 2023-10-27 19:48:00 +02:00
Martin Michlmayr
0b6ccea461 Fix typos 2023-10-27 19:48:00 +02:00
Martin Michlmayr
de6135351e Format option correctly 2023-10-27 19:48:00 +02:00
Michael Eischer
d47581b25e verify-release-binaries.sh: don't show warning if binaries are correct 2023-10-27 19:48:00 +02:00
Michael Eischer
69dec02a14 Remove readthedocs special case from docs configuration
Apparently it's now required to bring your own theme.
2023-10-27 19:47:19 +02:00
Michael Eischer
826d880614 Fix doc path typo in readthedocs configuration 2023-10-27 19:47:15 +02:00
Michael Eischer
dbf7ef72b9 Add read the docs config file version 2
The config file is by now necessary to build documentation:
https://blog.readthedocs.com/migrate-configuration-v2/
2023-10-27 19:47:11 +02:00
Michael Eischer
50ef01131a Merge pull request #4542 from MichaelEischer/support-armv6
Only support ARMv6 on ARM platforms
2023-10-27 19:42:49 +02:00
rawtaz
6be3a8fe51 Merge pull request #4539 from tbm/docs
Fix typos in docs
2023-10-27 17:30:08 +00:00
Michael Eischer
5166bde386 Only support ARMv6 on ARM platforms
Go 1.21 has switched the default from GOARM=5 to GOARM=7. So far there
have been complaints from Raspberry Pi 1 users, as the first raspberry
pi version only supports ARMv6. Exclude older ARM versions as these are
likely not relevant (rest-server also only supports ARMv6/7) and enforce
the use of software floating point emulation.
2023-10-27 19:12:12 +02:00
Leo R. Lundgren
aafb806a8c doc: Correct two typos 2023-10-27 18:56:32 +02:00
Martin Michlmayr
41e6a02bcc Fix typos 2023-10-27 18:56:32 +02:00
Martin Michlmayr
b51fe2fb69 Format option correctly 2023-10-27 18:56:32 +02:00
Michael Eischer
56537fb48e Merge pull request #4545 from restic/fix-rtd3
Try to fix documentation build
2023-10-26 22:00:20 +02:00
Michael Eischer
feea567868 Remove readthedocs special case from docs configuration
Apparently it's now required to bring your own theme.
2023-10-26 21:56:36 +02:00
Michael Eischer
2968b52f84 Merge pull request #4543 from MichaelEischer/fix-rtd2
Fix doc path typo in readthedocs configuration
2023-10-26 20:40:45 +02:00
Michael Eischer
619e80d7cc Fix doc path typo in readthedocs configuration 2023-10-26 20:39:43 +02:00
Michael Eischer
3804e50d64 Merge pull request #4541 from MichaelEischer/fix-binary-check-script
verify-release-binaries.sh: don't show warning if binaries are correct
2023-10-26 20:37:56 +02:00
Michael Eischer
c19e39968f verify-release-binaries.sh: don't show warning if binaries are correct 2023-10-26 19:59:27 +02:00
Michael Eischer
550be5c1e9 Merge pull request #4538 from MichaelEischer/fix-rtd
Add read the docs config file version 2
2023-10-26 19:42:36 +02:00
Michael Eischer
249605843b prune: get backend connection count via repository 2023-10-25 23:01:54 +02:00
Michael Eischer
c7b770eb1f convert MemorizeList to be repository based
Ideally, code that uses a repository shouldn't directly interact with
the underlying backend. Thus, move MemorizeList one layer up.
2023-10-25 23:01:35 +02:00
Michael Eischer
1b8a67fe76 move Backend interface to backend package 2023-10-25 23:00:18 +02:00
Michael Eischer
ceb0774af1 backend: make LoadAll independent of restic package 2023-10-25 22:58:39 +02:00
Michael Eischer
b6d79bdf6f restic: decouple restic.Handle 2023-10-25 22:54:07 +02:00
Michael Eischer
7881309d63 backend: move backend implementation helpers to util package
This removes code that is only used within a backend implementation from
the backend package. The latter now only contains code that also has
external users.
2023-10-25 22:54:07 +02:00
Michael Eischer
8e6fdf5edf Merge pull request #4520 from awannabeengineer/load-retry-nonexistent
retry: Do not retry Load() if file does not exist
2023-10-25 20:42:05 +00:00
Michael Eischer
c2e3e8d6ea Add read the docs config file version 2
The config file is by now necessary to build documentation:
https://blog.readthedocs.com/migrate-configuration-v2/
2023-10-25 22:00:42 +02:00
Alexander Neumann
27ec320eae Set development version for 0.16.1 2023-10-24 20:02:58 +02:00
Alexander Neumann
baca3f6303 Add version for 0.16.1 2023-10-24 20:02:55 +02:00
Alexander Neumann
524c2721b4 Update manpages and auto-completion 2023-10-24 20:02:55 +02:00
Alexander Neumann
be1b978ac8 Generate CHANGELOG.md for 0.16.1 2023-10-24 20:02:29 +02:00
Alexander Neumann
d4d9c1b8f1 Prepare changelog for 0.16.1 2023-10-24 20:02:29 +02:00
Michael Eischer
ead8dd0173 Merge pull request #4524 from MichaelEischer/fix-autocompletion
Don't load password when autocompleting CLI commands
2023-10-23 20:06:53 +00:00
Michael Eischer
cd09ef4614 don't load password when autocompleting CLI commands 2023-10-23 21:44:22 +02:00
Michael Eischer
d399e32590 Merge pull request #4535 from MichaelEischer/fix-cat-command
Only show index loading progress bar if stdout is a terminal
2023-10-23 21:28:13 +02:00
Michael Eischer
54a4034ec0 Only show index progress bar if stdout is a terminal
This ensures that there is no interference when using restic for
scripting use cases.
2023-10-23 21:16:52 +02:00
rawtaz
138b7b3328 Merge pull request #4534 from MichaelEischer/polish-changelogs
Polish changelogs
2023-10-23 19:00:01 +00:00
Leo R. Lundgren
6d19e0260d doc: Polish changelogs 2023-10-23 20:45:15 +02:00
Michael Eischer
85abceb99c doc: update minimum required Go version 2023-10-23 19:02:20 +02:00
Michael Eischer
85c15e6fa3 polish changelogs 2023-10-23 19:01:45 +02:00
Michael Eischer
d6917c7e00 Merge pull request #4533 from MichaelEischer/update-dependencies
update dependencies
2023-10-23 19:01:03 +02:00
Michael Eischer
8c20301172 update dependencies 2023-10-23 18:48:13 +02:00
Michael Eischer
4b4f916bdc Merge pull request #4532 from MichaelEischer/update-zstd
Update klauspost/compress to fix data corruption at maximum compression
2023-10-23 18:45:35 +02:00
Michael Eischer
9707956375 Merge pull request #4530 from MichaelEischer/repair-packs
Add `repair packs` command to simplify recovery from damaged pack
2023-10-23 18:44:42 +02:00
Michael Eischer
d1d4510974 repair packs: Add stub feature flag implementation 2023-10-23 18:36:28 +02:00
Michael Eischer
a28940ea29 check: Suggest usage of restic repair packs for corrupted blobs
For now, the guide is only shown if the blob content does not match its
hash. The main intended usage is to handle data corruption errors when
using maximum compression in restic 0.16.0
2023-10-23 18:36:28 +02:00
Michael Eischer
db26dc75e1 repair packs: add experimental command
This allows recovering a repository from several cases of damaged blobs.
2023-10-23 18:36:28 +02:00
Michael Eischer
efef38d0e8 drop support for Go 1.18 2023-10-23 18:30:38 +02:00
Michael Eischer
d00e72fed4 Update klauspost/compress to fix data corruption at maximum compression 2023-10-23 18:30:38 +02:00
Arash Farr
d15ffd9c92 retry: Do not retry Load() if file does not exist 2023-10-22 13:25:32 -05:00
Michael Eischer
62af0d769a Merge pull request #4528 from adamantike/misc/run-tests-with-go-1.21
Run tests with Go 1.21
2023-10-22 16:00:27 +00:00
Michael Eischer
ae83a9002a Merge pull request #4531 from MichaelEischer/silence-automaxprocs
Disable log output from automaxprocs library
2023-10-22 17:43:36 +02:00
Michael Eischer
ceff4af1ac Disable log output from automaxprocs library 2023-10-22 17:09:46 +02:00
Michael Manganiello
b15ba553a4 Run tests with Go 1.21 2023-10-22 10:47:42 -03:00
Michael Eischer
46cb1df1bc Merge pull request #4505 from MichaelEischer/verify-release-binaries
Add script to verify release binaries
2023-10-21 20:00:25 +02:00
Michael Eischer
bd3816fa14 CI: Ensure that github containers match the official binaries
The binaries accidentally included VCS information whereas binaries
built from the release tarball do not. For consistency remove the .git
directory before building the container on Github.
2023-10-21 19:44:59 +02:00
Michael Eischer
b2b0856908 Add helper script to verify release binaries
The script checks that the released binaries and the container binaries
can be reproduced.
2023-10-21 19:44:59 +02:00
Michael Eischer
7f05af02b9 Merge pull request #4480 from AgathaSorceress/add-rest-auth-env
Support reading basic auth credentials for REST server from environment variables
2023-10-21 17:41:08 +00:00
Michael Eischer
eabc177a42 Merge pull request #4442 from Refutable4890/copy-doc
Add a description of the resumption of `copy`
2023-10-21 17:38:15 +00:00
Michael Eischer
ab6defbace Merge pull request #4519 from adamantike/backend/sftp/add-args-option
backend/sftp: Add sftp.args option
2023-10-21 17:37:46 +00:00
Michael Eischer
fe1f61570b Merge pull request #4518 from adamantike/backend/b2/switch-to-official-library
backend/b2: Switch to official library `Backblaze/blazer`
2023-10-21 17:29:19 +00:00
Michael Eischer
baf9b54891 sftp: simplify documentation for sftp.args 2023-10-21 19:28:27 +02:00
Michael Eischer
6a4d6d5da4 sftp: test that Args and Command options cannot be set at the same time 2023-10-21 19:26:39 +02:00
Michael Manganiello
41f70f1f4f backend/sftp: Add sftp.args option
Allow setting custom arguments for the `sftp` backend, by using the
`sftp.args` option. This is similar to the approach already implemented
in the `rclone` backend, to support new arguments without requiring
future code changes for each different SSH argument.

Closes #4241
2023-10-21 19:25:44 +02:00
Michael Eischer
6c7560e537 backend/b2: Switch to official library Backblaze/blazer
According to https://github.com/kurin/blazer Readme, the repository is
no longer maintained, and the official one at
https://github.com/Backblaze/blazer took over.

Closes #4148
2023-10-21 19:18:13 +02:00
Agatha V. Lovelace
0f97356b21 docs: add new REST server env variables 2023-10-20 13:04:58 +02:00
Agatha V. Lovelace
2089c54310 changelog: add unreleased entry 2023-10-20 13:04:58 +02:00
Agatha V. Lovelace
f1877e721e feat: support reading REST credentials from env 2023-10-20 13:04:58 +02:00
Michael Eischer
17f2301cc2 Merge pull request #4514 from grembo/key-list-no-lock
Make `key list` command honor `--no-lock`
2023-10-19 20:50:56 +00:00
Michael Eischer
79deb99605 Merge pull request #4500 from MichaelEischer/simplify-issue-template
Simplify issue template
2023-10-19 20:48:36 +00:00
Michael Eischer
643180b415 simplify issue template 2023-10-19 22:16:38 +02:00
Michael Eischer
d27cfd10a9 Merge pull request #4487 from lbausch/tweak-restore-summary
Tweak restore summary
2023-10-19 20:14:14 +00:00
Michael Gmelin
34f3b13b7c Make key list command honor --no-lock
Fixes #4513
2023-10-19 22:09:35 +02:00
Michael Eischer
495982232c Merge pull request #4511 from dnnr/inode-in-json
Include inode in JSON output for find and ls
2023-10-17 21:06:34 +00:00
Michael Eischer
d173d1c532 Merge pull request #4486 from MichaelEischer/separate-large-blobs
repository: store oversized blobs in separate pack files
2023-10-17 23:03:52 +02:00
Michael Eischer
f955222750 Merge pull request #4502 from MichaelEischer/replace-ratelimit
backend/limiter: replace juju/ratelimit with x/time/rate
2023-10-17 23:03:17 +02:00
Michael Eischer
cb9cbe55d9 repository: store oversized blobs in separate pack files
Store oversized blobs in separate pack files as the blobs is large
enough to warrant its own pack file. This simplifies the garbage
collection of such blobs and keeps the cache smaller, as oversize (tree)
blobs only have to be downloaded if they are actually used.
2023-10-17 22:52:16 +02:00
Michael Eischer
f750aa8dfb backend/limiter: replace juju/ratelimit with x/time/rate 2023-10-17 22:49:25 +02:00
Michael Eischer
c635e30e3f Merge pull request #4485 from MichaelEischer/uber-automaxprocs
Automatically adjust GOMAXPROCS when run in a cpu-constrained container
2023-10-17 22:47:10 +02:00
Michael Eischer
f5d5e8fd0a automatically adjust GOMAXPROCS when run in a cpu-constrained container 2023-10-17 20:31:57 +02:00
Daniel Danner
a858ab254b Include inode in JSON output for find and ls
This introduces the inode attribute to the JSON output emitted for nodes
in `ls` and matches in `find`. There doesn't seem to be any discernible
reason to omit the inode and it can be useful in scripting scenarios.
2023-10-11 21:10:01 +02:00
Michael Eischer
4087c3aff7 Merge pull request #4498 from PeterDaveHello/ImproveDockerfile
Remove additional `--update` for apk in Dockerfile
2023-10-09 20:53:49 +00:00
Refutable4890
aa86c76aab Add a description of the resumption of copy 2023-10-08 12:07:40 +08:00
Michael Eischer
76ef94d15c Merge pull request #4419 from arjunajesh/issue#299
Progress Bar when loading index files
2023-10-01 18:06:35 +00:00
Michael Eischer
91aef00df3 check: add index loading progress bar 2023-10-01 19:55:29 +02:00
Michael Eischer
b0da0f152f deduplicate index progress bar setup 2023-10-01 19:53:26 +02:00
Michael Eischer
75f6bd89ed hide index loading progress for JSON output 2023-10-01 19:53:26 +02:00
Michael Eischer
3fd0ad7448 repository: list index files only once 2023-10-01 19:53:26 +02:00
arjunajesh
b6593ad7df changelog 2023-10-01 19:53:26 +02:00
arjunajesh
ed65a7dbca implement progress bar for index loading 2023-10-01 19:52:59 +02:00
Michael Eischer
eac1c4a8d0 Merge pull request #4471 from jalseth/master
docs(examples): Use restic binary from system path
2023-10-01 12:31:28 +00:00
Michael Eischer
f519454f33 Merge pull request #4493 from restic/dependabot/go_modules/cloud.google.com/go/storage-1.33.0
build(deps): bump cloud.google.com/go/storage from 1.31.0 to 1.33.0
2023-10-01 12:09:43 +00:00
Michael Eischer
54ae8a0c40 Merge pull request #4492 from restic/dependabot/github_actions/docker/login-action-b4bedf8053341df3b5a9f9e0f2cf4e79e27360c6
build(deps): bump docker/login-action from cf8514a65188af1d4f94f8c28a7a4153af1088ce to b4bedf8053341df3b5a9f9e0f2cf4e79e27360c6
2023-10-01 12:08:10 +00:00
Michael Eischer
a36b5b6391 Merge pull request #4491 from restic/dependabot/github_actions/docker/setup-buildx-action-3
build(deps): bump docker/setup-buildx-action from 2 to 3
2023-10-01 12:03:48 +00:00
Michael Eischer
5e36e4da96 Merge pull request #4489 from restic/dependabot/github_actions/docker/setup-qemu-action-3
build(deps): bump docker/setup-qemu-action from 2 to 3
2023-10-01 12:03:19 +00:00
Michael Eischer
8ee08e5d09 Merge pull request #4490 from restic/dependabot/github_actions/actions/checkout-4
build(deps): bump actions/checkout from 3 to 4
2023-10-01 12:03:09 +00:00
Michael Eischer
9f9f736ec2 Merge pull request #4495 from restic/dependabot/go_modules/golang.org/x/oauth2-0.12.0
build(deps): bump golang.org/x/oauth2 from 0.11.0 to 0.12.0
2023-10-01 12:01:12 +00:00
Michael Eischer
af98c3ccbe Merge pull request #4496 from restic/dependabot/go_modules/github.com/hashicorp/golang-lru/v2-2.0.7
build(deps): bump github.com/hashicorp/golang-lru/v2 from 2.0.6 to 2.0.7
2023-10-01 11:59:38 +00:00
Peter Dave Hello
6edfc73879 Remove additional --update for apk in Dockerfile 2023-10-01 18:00:15 +08:00
dependabot[bot]
d985ed27d1 build(deps): bump github.com/hashicorp/golang-lru/v2 from 2.0.6 to 2.0.7
Bumps [github.com/hashicorp/golang-lru/v2](https://github.com/hashicorp/golang-lru) from 2.0.6 to 2.0.7.
- [Release notes](https://github.com/hashicorp/golang-lru/releases)
- [Commits](https://github.com/hashicorp/golang-lru/compare/v2.0.6...v2.0.7)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/golang-lru/v2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-01 01:45:19 +00:00
dependabot[bot]
4278ec6553 build(deps): bump golang.org/x/oauth2 from 0.11.0 to 0.12.0
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.11.0 to 0.12.0.
- [Commits](https://github.com/golang/oauth2/compare/v0.11.0...v0.12.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-01 01:45:07 +00:00
dependabot[bot]
15cb498c47 build(deps): bump cloud.google.com/go/storage from 1.31.0 to 1.33.0
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.31.0 to 1.33.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.31.0...pubsub/v1.33.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-01 01:44:54 +00:00
dependabot[bot]
59e217b003 build(deps): bump docker/login-action
Bumps [docker/login-action](https://github.com/docker/login-action) from cf8514a65188af1d4f94f8c28a7a4153af1088ce to b4bedf8053341df3b5a9f9e0f2cf4e79e27360c6.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](cf8514a651...b4bedf8053)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-01 01:24:15 +00:00
dependabot[bot]
3457f50c8c build(deps): bump docker/setup-buildx-action from 2 to 3
Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 2 to 3.
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](https://github.com/docker/setup-buildx-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: docker/setup-buildx-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-01 01:24:10 +00:00
dependabot[bot]
bbe2f1ecf2 build(deps): bump actions/checkout from 3 to 4
Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-01 01:24:05 +00:00
dependabot[bot]
362917afb9 build(deps): bump docker/setup-qemu-action from 2 to 3
Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 2 to 3.
- [Release notes](https://github.com/docker/setup-qemu-action/releases)
- [Commits](https://github.com/docker/setup-qemu-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: docker/setup-qemu-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-01 01:24:01 +00:00
Michael Eischer
b92ab458b0 Merge pull request #4459 from restic/dependabot/go_modules/github.com/go-ole/go-ole-1.3.0
build(deps): bump github.com/go-ole/go-ole from 1.2.6 to 1.3.0
2023-09-28 20:42:14 +00:00
Lorenz Bausch
2657217574 Tweak restore summary 2023-09-20 10:59:55 +02:00
James Alseth
02ab511c2f docs(examples): Use restic binary from system path
Signed-off-by: James Alseth <james@jalseth.me>
2023-09-17 19:39:45 -07:00
Michael Eischer
6e586b64e4 Merge pull request #4450 from MichaelEischer/cleanup-master-index-test
master_index: fix inconsistent length blob length in test
2023-09-06 22:55:48 +02:00
Michael Eischer
fb4d458cce Merge pull request #4452 from lbausch/docs/add-current-key-hint
docs: explain meaning of asterisk when managing repository keys
2023-09-06 20:28:33 +00:00
Michael Eischer
c7b5ddc012 Merge pull request #4451 from greatroar/bh-string
internal/restic: Return summary from BlobSet.String
2023-09-06 20:27:26 +00:00
Michael Eischer
3eb825e47c Merge pull request #4464 from restic/dependabot/go_modules/golang.org/x/term-0.12.0
build(deps): bump golang.org/x/term from 0.11.0 to 0.12.0
2023-09-06 20:18:34 +00:00
dependabot[bot]
4d60011030 build(deps): bump golang.org/x/term from 0.11.0 to 0.12.0
Bumps [golang.org/x/term](https://github.com/golang/term) from 0.11.0 to 0.12.0.
- [Commits](https://github.com/golang/term/compare/v0.11.0...v0.12.0)

---
updated-dependencies:
- dependency-name: golang.org/x/term
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-09-05 21:43:28 +00:00
Michael Eischer
507ed32469 Merge pull request #4457 from restic/dependabot/go_modules/google.golang.org/api-0.138.0
build(deps): bump google.golang.org/api from 0.134.0 to 0.138.0
2023-09-05 21:32:34 +00:00
dependabot[bot]
22fdfe1ffe build(deps): bump github.com/go-ole/go-ole from 1.2.6 to 1.3.0
Bumps [github.com/go-ole/go-ole](https://github.com/go-ole/go-ole) from 1.2.6 to 1.3.0.
- [Release notes](https://github.com/go-ole/go-ole/releases)
- [Changelog](https://github.com/go-ole/go-ole/blob/master/ChangeLog.md)
- [Commits](https://github.com/go-ole/go-ole/compare/v1.2.6...v1.3.0)

---
updated-dependencies:
- dependency-name: github.com/go-ole/go-ole
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-09-05 21:17:03 +00:00
Michael Eischer
e05cd9abca Merge pull request #4462 from restic/dependabot/github_actions/docker/login-action-cf8514a65188af1d4f94f8c28a7a4153af1088ce
build(deps): bump docker/login-action from a9794064588be971151ec5e7144cb535bcb56e36 to cf8514a65188af1d4f94f8c28a7a4153af1088ce
2023-09-05 21:07:57 +00:00
Michael Eischer
ea55ca5303 Merge pull request #4458 from restic/dependabot/go_modules/github.com/hashicorp/golang-lru/v2-2.0.6
build(deps): bump github.com/hashicorp/golang-lru/v2 from 2.0.4 to 2.0.6
2023-09-05 21:05:45 +00:00
Michael Eischer
df53f4782b Merge pull request #4453 from lbausch/docs/fix-table
docs: fix malformed table
2023-09-05 20:37:30 +00:00
Michael Eischer
35055adbc4 Merge pull request #4454 from lbausch/tweak-readme
README: add Mastodon link
2023-09-05 20:36:32 +00:00
dependabot[bot]
bd9eb528c0 build(deps): bump docker/login-action
Bumps [docker/login-action](https://github.com/docker/login-action) from a9794064588be971151ec5e7144cb535bcb56e36 to cf8514a65188af1d4f94f8c28a7a4153af1088ce.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](a979406458...cf8514a651)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-09-01 01:45:03 +00:00
dependabot[bot]
02032f3109 build(deps): bump github.com/hashicorp/golang-lru/v2 from 2.0.4 to 2.0.6
Bumps [github.com/hashicorp/golang-lru/v2](https://github.com/hashicorp/golang-lru) from 2.0.4 to 2.0.6.
- [Release notes](https://github.com/hashicorp/golang-lru/releases)
- [Commits](https://github.com/hashicorp/golang-lru/compare/v2.0.4...v2.0.6)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/golang-lru/v2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-09-01 01:41:13 +00:00
dependabot[bot]
cfff1367c1 build(deps): bump google.golang.org/api from 0.134.0 to 0.138.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.134.0 to 0.138.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.134.0...v0.138.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-09-01 01:40:56 +00:00
Lorenz Bausch
353265a329 README: add Mastodon link 2023-08-27 10:52:12 +02:00
Lorenz Bausch
84a1170dee docs: remove trailing whitespace 2023-08-27 10:49:34 +02:00
Lorenz Bausch
0b4d9c9a51 docs: add missing whitespace to avoid malformed table warning 2023-08-27 10:46:55 +02:00
Lorenz Bausch
5422a7daa5 docs: explain meaning of asterisk when managing repository keys 2023-08-27 10:40:55 +02:00
greatroar
691c01963b internal/restic: Return summary from BlobSet.String
Fixes #4449: error messages from prune were too long to fit in scroll
buffers.
2023-08-25 21:41:30 +02:00
Michael Eischer
2bec99dc6f master_index: fix inconsistent length blob length in test
Two blobs with the same hash must always have the same content length.
2023-08-19 20:04:25 +02:00
Michael Eischer
e60c5b2d7f Merge pull request #4436 from MichaelEischer/add-snapshot-subfolder-syntax-to-help-text
Add snapshot subfolder syntax in help output of diff/dump/restore
2023-08-13 19:36:38 +02:00
Michael Eischer
a04964bb86 Merge pull request #4446 from 2000yeshu/feat-cat
Check for arguments before opening remote connection in cat command
2023-08-13 17:25:37 +00:00
Michael Eischer
fe54912a46 cat: extract parameter validation and add a test 2023-08-13 19:17:46 +02:00
Michael Eischer
feb6abb7bb Merge pull request #4394 from greatroar/acl
dump: Rewrite Linux ACL handling
2023-08-13 17:01:31 +00:00
greatroar
aaf5254e26 dump: Rewrite Linux ACL handling
The old version was taken from an MPL-licensed library. This is a
cleanroom implementation. The code is shorter and it's now explicit that
only Linux ACLs are supported.
2023-08-13 13:37:11 +02:00
Yakul Garg
705556f134 Check for arguments before opening remote connection in cat command
Signed-off-by: Yakul Garg <2000yeshu@gmail.com>
2023-08-12 23:54:40 +05:30
Michael Eischer
c23eebc225 Merge pull request #4428 from MichaelEischer/fix-restore-help
restore: Fix help text for `--iinclude` and `--iexclude`
2023-08-12 20:21:22 +02:00
Refutable4890
e09f6f540f Add a description of the resumption of copy 2023-08-08 12:51:40 +08:00
Michael Eischer
0fca028491 restore: Fix help text for --iinclude and --iexclude 2023-08-05 18:21:03 +02:00
Michael Eischer
57a08291f5 Merge pull request #4429 from restic/dependabot/github_actions/docker/login-action-a9794064588be971151ec5e7144cb535bcb56e36
build(deps): bump docker/login-action from 65b78e6e13532edd9afa3aa52ac7964289d1a9c1 to a9794064588be971151ec5e7144cb535bcb56e36
2023-08-05 13:41:08 +00:00
Michael Eischer
2117ce4cfb Merge pull request #4434 from lbausch/docs-fix-wording
docs: fix `percent_done` description
2023-08-05 13:39:02 +00:00
Michael Eischer
3a478bc522 unify snapshotID spelling in command parameters 2023-08-05 12:35:45 +02:00
Michael Eischer
9a7704fa2b diff/dump/restore: describe snapshot:subfolder syntax in help text 2023-08-05 12:35:08 +02:00
Lorenz Bausch
9a69f44de2 docs: fix percent_done description 2023-08-04 16:15:18 +02:00
dependabot[bot]
7a6339180b build(deps): bump docker/login-action
Bumps [docker/login-action](https://github.com/docker/login-action) from 65b78e6e13532edd9afa3aa52ac7964289d1a9c1 to a9794064588be971151ec5e7144cb535bcb56e36.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](65b78e6e13...a979406458)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-08-01 01:19:51 +00:00
Alexander Neumann
82e6e28781 Set development version for 0.16.0 2023-07-31 20:24:29 +02:00
Alexander Neumann
eb33e564c9 Add version for 0.16.0 2023-07-31 20:23:24 +02:00
Alexander Neumann
62680af734 Update manpages and auto-completion 2023-07-31 20:23:24 +02:00
Alexander Neumann
68460fd3d1 Generate CHANGELOG.md for 0.16.0 2023-07-31 20:22:50 +02:00
Alexander Neumann
9459328d8d Prepare changelog for 0.16.0 2023-07-31 20:22:50 +02:00
rawtaz
9d71990c26 Merge pull request #4424 from MichaelEischer/tweak-changelog
Tweak changelog entries
2023-07-29 11:29:30 +00:00
rawtaz
9219b6a0ef Merge pull request #4427 from MichaelEischer/changelog-template
Extend style guide in changelog template
2023-07-29 11:15:17 +00:00
Michael Eischer
9a0ec05145 Extend style guide in changelog template 2023-07-29 12:55:56 +02:00
Michael Eischer
2a5b6af2e9 extend gc tweak changelog 2023-07-29 12:51:05 +02:00
Leo R. Lundgren
51e4e5ef82 doc: Polish changelog 2023-07-29 12:51:04 +02:00
Michael Eischer
e998314088 tweak changelog entries 2023-07-29 12:50:01 +02:00
Michael Eischer
be03c1a457 Merge pull request #4422 from MichaelEischer/document-non-utf-symlinks
Document `linktarget_raw` field for non utf symlinks
2023-07-28 23:14:08 +02:00
Michael Eischer
95050117eb Merge pull request #4426 from MichaelEischer/consistent-forget-policy
forget: replace `--keep-* -1` with `--keep-* unlimited`
2023-07-28 19:06:00 +00:00
Michael Eischer
21a3a41b69 Merge pull request #4423 from MichaelEischer/update-dependencies
Update dependencies
2023-07-28 19:25:02 +02:00
Michael Eischer
3f919f2371 forget: simplify test 2023-07-28 19:21:40 +02:00
Michael Eischer
50b43fbac0 forget: replace --keep-* -1 with --keep-* unlimited
This ensures consistency with the `prune --max-unused unlimited` option.
2023-07-28 19:21:40 +02:00
Michael Eischer
f689e2638d reenable aix builds 2023-07-27 21:27:31 +02:00
Michael Eischer
f9ef2b1e44 update all dependencies 2023-07-27 21:26:44 +02:00
Michael Eischer
b87a37f318 doc: Fix typos 2023-07-25 20:14:56 +02:00
Michael Eischer
647ebf352a doc: Describe linktarget_raw field 2023-07-25 20:14:47 +02:00
Michael Eischer
62d3ef4a93 Merge pull request #4417 from MichaelEischer/fix-flaky-list-cancel-test
backend: Fix unreliable TestListCancel/Timeout test
2023-07-23 18:04:11 +02:00
Michael Eischer
6c5fc32967 Merge pull request #3225 from wojas/restic-cmd-more-help
Extend help output with short overview
2023-07-23 15:43:03 +00:00
Michael Eischer
ae13cf15c1 doc: move quickstart to docs 2023-07-23 17:32:37 +02:00
Michael Eischer
1fe1ec40a2 Merge pull request #4409 from arjunajesh/issue#1495
snapshots sorted by timestamp in restic find
2023-07-23 11:00:41 +00:00
arjunajesh
32f5ee6f4e snapshots sorted by timestamp 2023-07-23 12:49:23 +02:00
Konrad Wojas
5bd8a6d7eb Extend help output with short overview
The list of subcommands can be confusing for first time users. This
adds a short overview of the most important commands to help get people
started.
2023-07-23 12:01:40 +02:00
Michael Eischer
4a33370072 backend: Fix unreliable TestListCancel/Timeout test
The test uses `WithTimeout` to create a context that cancels the List
operation after a given delay. Several backends internally use a derived
child context created using WithCancel.

The cancellation of a context first closes the done channel of the
context (here: the `WithTimeout` context) and _afterwards_ propagates
the cancellation to child contexts (here: the `WithCancel` context).
Therefor if the List implementation uses a child context, then it may
take a moment until that context is also cancelled. Thus give the
context cancellation a moment to propagate.
2023-07-23 11:47:56 +02:00
Michael Eischer
98fb56baa6 Merge pull request #3802 from MichaelEischer/support-non-utf-symlinks
Support non utf symlink targets
2023-07-23 00:15:01 +02:00
Michael Eischer
f12bbd9229 restic: check that Node.LinkTarget can handle non-utf8 characters 2023-07-23 00:05:14 +02:00
Michael Eischer
6adb629608 Add support for non-utf8 symlink targets 2023-07-23 00:05:14 +02:00
Michael Eischer
25ff9fa893 Merge pull request #4334 from MichaelEischer/snapshot-subtree-syntax
Add support for snapshot:path syntax
2023-07-22 23:59:26 +02:00
Michael Eischer
bdaec8fdb8 Merge pull request #4374 from MichaelEischer/try-refresh-stale-locks
Try to refresh stale locks
2023-07-22 23:50:52 +02:00
Michael Eischer
55b440b520 tweak snapshot:subfolder docs 2023-07-22 23:50:17 +02:00
Michael Eischer
3f63b53090 lock: Shrink critical section in RefreshStaleLock
A cleaning up after a failed lock refresh attempt does not require a
mutex.
2023-07-22 23:29:14 +02:00
Michael Eischer
76253b2a20 Hopefully fix flaky TestLockWaitCancel test 2023-07-22 23:23:17 +02:00
Michael Eischer
20e82d1fcf lock: add changelog for stale lock refresh 2023-07-22 23:23:17 +02:00
Michael Eischer
96eada3d5f backend/sema: add test for freeze functionality 2023-07-22 23:23:17 +02:00
Michael Eischer
b8f4267a36 lock: add more debug information to flaky test 2023-07-22 23:23:17 +02:00
Michael Eischer
1f6883a05c lock: test stale lock refresh 2023-07-22 23:23:17 +02:00
Michael Eischer
8154f6a77a Ensure consistent naming for <snapshot>:<subfolder> syntax 2023-07-22 19:55:57 +02:00
Michael Eischer
184b7616ba document snapshot:subfolder syntax 2023-07-22 19:55:57 +02:00
Michael Eischer
67f237b4f3 add changelog for snapshot:path syntax 2023-07-22 19:55:57 +02:00
Michael Eischer
090f9d6237 restic: Cleanup and simplify TestCreateSnapshot 2023-07-22 19:55:57 +02:00
Michael Eischer
321cc35cde restic: add test for FindTreeDirectory 2023-07-22 19:55:57 +02:00
Michael Eischer
a97915642c restic: Ensure snapshots created by TestCreateSnapshot are valid
All nodes in a tree must be sorted by node name. Otherwise functionality
like searching for a specific tree node will break.
2023-07-22 19:55:57 +02:00
Michael Eischer
c64d81063e test snapshot filtering with snapshot:path syntax 2023-07-22 19:55:57 +02:00
Michael Eischer
233b841ad9 Add cat tree snapshot:subpath command
This command prints the tree for the given path.
2023-07-22 19:55:57 +02:00
Michael Eischer
85860e6e97 Add support for snapshot:subpath syntax
This snapshot specification syntax is supported by the cat, diff, dump,
ls and restore command.
2023-07-22 19:55:57 +02:00
Michael Eischer
2dd6769429 lock: Fix possible deadlock during refresh of stale lock
A delayed lock refresh could send a signal on the `refreshed` channel
while the `monitorLockRefresh` goroutine waits for a reply to its
`refreshLockRequest`. As the channels are unbuffered, this resulted in a
deadlock.
2023-07-22 19:55:44 +02:00
Michael Eischer
5d9b0d894e lock: add unit test for RefreshStaleLock 2023-07-22 19:55:44 +02:00
Michael Eischer
d4bf7a3cb1 restic: cleanup lock test 2023-07-22 19:55:44 +02:00
Michael Eischer
24c8a33da9 restic: reduce sleeps in lock tests 2023-07-22 19:55:44 +02:00
Michael Eischer
399f8e84a1 lock: use mem backend for locking tests
This hopefully fixes the timing related test failures which appear to be
caused by very slow fsync calls.
2023-07-22 19:55:44 +02:00
Michael Eischer
6a436d731d lock: rename test helper function 2023-07-22 19:55:44 +02:00
Michael Eischer
b2fcbc21cb lock: rework stale lock refresh to avoid data race 2023-07-22 19:55:44 +02:00
Michael Eischer
05e5e29a8c backup: don't pretend to lock repo for dry run
The dry run did not actually create lock files, but still ran all other
parts of the lock processing.
2023-07-22 19:55:44 +02:00
Michael Eischer
f490288738 lock: freeze backend operations while refreshing stale lock
Freeze new backend operations while trying to refresh a stale lock.
2023-07-22 19:55:44 +02:00
Michael Eischer
51718ec561 restic: generic implemention of AsBackend 2023-07-22 19:55:44 +02:00
Michael Eischer
11eb88a2ea lock: cleanup 2023-07-22 19:55:44 +02:00
Michael Eischer
bee3231ed4 lock: try refreshing of stale locks
A stale lock may be refreshed if it continues to exist until after a
replacement lock has been created. This ensures that a repository was
not unlocked in the meantime.
2023-07-22 19:55:44 +02:00
Michael Eischer
60d8066568 Merge pull request #4407 from MichaelEischer/simpler-required-ci
CI: Add check that verifies the results of other checks
2023-07-22 19:55:06 +02:00
Michael Eischer
08dea911bd CI: Add check that verifies the results of other checks
To properly use merge queues, it is necessary to mark certain checks as
"required" in the branch protection rules. However, this doesn't scale
with matrix jobs and would require manual tweaks of the CI jobs change.
The new analyze job simplifies this by allowing the branch protection
rule just check for that job. All further dependencies are then
configured within the CI config.
2023-07-16 14:13:41 +02:00
Michael Eischer
47206a6579 Merge pull request #4351 from kenny-y-dev/human-readable-ls
Add --human-readable to ls and find output
2023-07-16 11:44:41 +00:00
Michael Eischer
594689db32 Merge pull request #4400 from MichaelEischer/rest-handle-missing-dirs
rest: Ignore missing directories
2023-07-14 23:29:37 +02:00
Michael Eischer
5705326bb8 add changelog 2023-07-14 22:39:38 +02:00
Michael Eischer
978ebaac49 rest: use http status code constants 2023-07-14 22:39:38 +02:00
Michael Eischer
307aeb6849 rest: Don't return error if listing non-existent directory
When transferring a repository from S3 to, for example, a local disk
then all empty folders will be missing.

When saving files, the missing intermediate folders are created
automatically. Therefore, missing directories can be ignored by the
`List()` operation.
2023-07-14 22:39:38 +02:00
Michael Eischer
9cd85d5956 Merge pull request #4029 from ekarlso/add-azure-workload-identity-support
add azure workload identity support
2023-07-09 19:06:59 +00:00
Michael Eischer
c9f506925c azure: Document additional auth options 2023-07-09 20:56:21 +02:00
Michael Eischer
e990d3d483 azure: Support authentication using managed / workload identity
See https://github.com/Azure/azure-sdk-for-go/tree/sdk/azidentity/v1.3.0/sdk/azidentity
2023-07-09 14:47:03 +02:00
Michael Eischer
7042190807 Merge pull request #4403 from MichaelEischer/fix-certificate-env-handling
Fix certificate env handling
2023-07-09 14:46:38 +02:00
Michael Eischer
c158741e2e CI: add minimal CLI test
Just create a repository and run a minimal backup.
2023-07-09 14:29:43 +02:00
Michael Eischer
89fbd39e59 Don't print stacktrace on invalid cacert option 2023-07-09 14:23:54 +02:00
Michael Eischer
1ce599d2ae Fix handling of empty cacert environment variable
This resulted in a "empty filename for root certificate supplied" error.
2023-07-09 14:23:54 +02:00
Michael Eischer
789fec3da7 Merge pull request #4402 from MichaelEischer/merge-queue-setup
CI: Enable missing CI tests for Github merge queue
2023-07-09 13:57:10 +02:00
Michael Eischer
8a120c8800 CI: Enable missing CI tests for Github merge queue 2023-07-08 23:51:39 +02:00
Michael Eischer
41395e83c5 Merge pull request #2740 from metalsp0rk/document-json
Add JSON documentation
2023-07-08 21:34:02 +00:00
Michael Eischer
2a193195b0 Merge pull request #4364 from MichaelEischer/multiplatform-docker
Setup multiplatform docker builds
2023-07-08 23:31:58 +02:00
Michael Eischer
229c7b24a4 doc: add program_version field of snapshot JSON output 2023-07-08 23:31:20 +02:00
Michael Eischer
b34ce57dd4 doc: describe JSON output of restore command 2023-07-08 23:31:20 +02:00
Michael Eischer
1ce839228e doc: Expand JSON documentation 2023-07-08 23:31:20 +02:00
Kyle Brennan
fb1170c1d6 doc tweaks 2023-07-08 23:21:41 +02:00
Kyle Brennan
e457fe22bc Documentation improvement 2023-07-08 23:21:41 +02:00
Kyle Brennan
39299e36ef add json documentation 2023-07-08 23:21:41 +02:00
Michael Eischer
0512f292e8 Merge pull request #4361 from MichaelEischer/stats-repo-debug-info
stats: Add `debug` mode to collect repository statistics
2023-07-08 23:20:23 +02:00
Michael Eischer
3a93e28605 CI: Remove .dockerignore to ensure reproducible builds
Since go 1.18, built binaries also include VCS information such as the
built commit. This information is also included in the official
binaries. To ensure that the Docker container recreates the same
binaries, the .git folder must also be transferred into the container.
Thus, remove the .dockerignore file.

The copied files must also be owned by the current user within the
container, as git refuses to work otherwise.
2023-07-08 23:16:14 +02:00
Michael Eischer
6b82cce1bd add changelog for multiplatform containers 2023-07-08 23:16:14 +02:00
Michael Eischer
31e07cecbb docker: update to Go 1.20 for custom container builds 2023-07-08 23:05:39 +02:00
Michael Eischer
c181b51360 doc: Update docker build process 2023-07-08 23:05:38 +02:00
Michael Eischer
ccd19b7e88 CI: run cloud backend tests only on restic/restic 2023-07-08 23:05:02 +02:00
Michael Eischer
b0987ff570 CI: only build containers on restic/restic 2023-07-08 23:05:02 +02:00
Michael Eischer
eff3124f15 CI: Setup automatic container builds for ghcr.io
Containers are built for new tags and pushes to the master branch.
2023-07-08 23:05:02 +02:00
Michael Eischer
43fa051546 Directly build restic binary in release Docker container 2023-07-08 23:05:02 +02:00
Michael Eischer
2293835242 Release multi-platform docker containers 2023-07-08 23:05:02 +02:00
Michael Eischer
0fcb6c7f94 ask for debug statistics in resource usage issues 2023-07-08 23:04:47 +02:00
Michael Eischer
325fa916b5 stats: Add debug mode to collect repository statistics 2023-07-08 23:04:47 +02:00
Michael Eischer
170e495334 Merge pull request #4401 from MichaelEischer/fix-windows-build
fuse: fix windows build
2023-07-08 23:01:22 +02:00
Michael Eischer
9d44682e3e fuse: fix windows build 2023-07-08 22:40:22 +02:00
Michael Eischer
4d43509423 Merge pull request #4379 from chenxiaolong/symlink_xattrs
Add support for extended attributes on symlinks
2023-07-08 16:56:51 +00:00
Michael Eischer
ea9ad77e05 restic: refactor node test 2023-07-08 18:54:56 +02:00
Michael Eischer
cc84884d2e restic: basic xattr test for files/dirs 2023-07-08 18:49:21 +02:00
Michael Eischer
4a5ae2ba84 restic: test NodeFromFileInfo for symlinks 2023-07-08 18:18:13 +02:00
Michael Eischer
1f1e50f49e fuse: add test for symlink xattr 2023-07-08 18:02:17 +02:00
Michael Eischer
f3c3b0f377 fuse: deduplicate xattr code 2023-07-08 17:41:45 +02:00
Michael Eischer
7e2be9e081 Merge pull request #4384 from arjunajesh/issue#1926
certificates can be passed through env vars
2023-07-08 07:46:31 +00:00
Michael Eischer
cc3c218baf small cleanups for certificate environment variables 2023-07-08 09:44:59 +02:00
arjunajesh
c1578a2035 certificates can be passed through env vars 2023-07-08 09:44:59 +02:00
Michael Eischer
678b983300 Merge pull request #4360 from Refutable4890/master
Add an description of `check` temporary cache directory location
2023-07-07 21:51:52 +00:00
Michael Eischer
1b3870dc43 Merge pull request #4378 from MichaelEischer/add-version-to-snapshot
Add program version to snapshot
2023-07-07 23:50:44 +02:00
Michael Eischer
ef40aee2bd Merge pull request #4373 from MichaelEischer/performance-issue-help
document how to analyze performance / memory usage issues
2023-07-07 23:50:27 +02:00
Michael Eischer
3a32c4e59f document how to analyze performance / memory usage issues 2023-07-07 23:38:53 +02:00
Michael Eischer
e703e89e9b add changelog for program version 2023-07-07 23:36:57 +02:00
Michael Eischer
389f6ee74c backup: add minimal test for program versioni 2023-07-07 23:33:23 +02:00
Michael Eischer
bbac74b172 add program version to snapshot 2023-07-07 23:27:10 +02:00
Michael Eischer
a280b7364e Merge pull request #4395 from greatroar/parsebytes
cmd, ui: Move size parsing code and make it more robust
2023-07-07 21:24:47 +00:00
Michael Eischer
825651a135 Merge pull request #4387 from arjunajesh/issue#2468
azure domain can be overridden with env variable
2023-07-07 21:17:01 +00:00
Michael Eischer
e36d17a6f8 azure: tweak documentation for endpoint suffix 2023-07-07 23:10:26 +02:00
arjunajesh
068b115abc added azure domain parameter 2023-07-07 23:10:26 +02:00
greatroar
41a5bf357f cmd, ui: Move size parsing code and make it more robust 2023-07-02 21:44:50 +02:00
Michael Eischer
f96896a9c0 Merge pull request #4383 from LouisMT/add-docker-hostname
Add hostname flag to Docker docs
2023-07-01 18:58:43 +02:00
dependabot[bot]
2ab9a3b9c3 Merge pull request #4389 from restic/dependabot/go_modules/google.golang.org/api-0.129.0 2023-07-01 16:57:18 +00:00
Louis Matthijssen
dbe2eef80c Add hostname flag to Docker docs
Fixes #4380
2023-07-01 18:46:31 +02:00
dependabot[bot]
6e7c6674ad build(deps): bump google.golang.org/api from 0.116.0 to 0.129.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.116.0 to 0.129.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.116.0...v0.129.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-07-01 16:11:44 +00:00
Michael Eischer
7fe830ee1e Merge pull request #4392 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/azcore-1.6.1
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.6.0 to 1.6.1
2023-07-01 18:10:47 +02:00
Michael Eischer
a07c7166ba Merge pull request #4390 from restic/dependabot/go_modules/golang.org/x/oauth2-0.9.0
build(deps): bump golang.org/x/oauth2 from 0.8.0 to 0.9.0
2023-07-01 18:10:38 +02:00
dependabot[bot]
d2f8f9de23 build(deps): bump golang.org/x/oauth2 from 0.8.0 to 0.9.0
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.8.0 to 0.9.0.
- [Commits](https://github.com/golang/oauth2/compare/v0.8.0...v0.9.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-07-01 15:32:55 +00:00
dependabot[bot]
d5fe5107c8 Merge pull request #4391 from restic/dependabot/go_modules/golang.org/x/sys-0.9.0 2023-07-01 15:31:42 +00:00
dependabot[bot]
f08ba1a005 build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore
Bumps [github.com/Azure/azure-sdk-for-go/sdk/azcore](https://github.com/Azure/azure-sdk-for-go) from 1.6.0 to 1.6.1.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.6.0...sdk/azcore/v1.6.1)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azcore
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-07-01 01:23:19 +00:00
dependabot[bot]
70fb554854 build(deps): bump golang.org/x/sys from 0.8.0 to 0.9.0
Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.8.0 to 0.9.0.
- [Commits](https://github.com/golang/sys/compare/v0.8.0...v0.9.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sys
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-07-01 01:23:05 +00:00
Andrew Gunnerson
8c02ebb029 Add support for extended attributes on symlinks
Linux allows the use of non-`user.` extended attributes on symlinks. One
of the main users of this functionality is SELinux's `security.selinux`
xattr for storing a path's label. By storing symlink xattrs, restic is
now suitable for backing up the root filesystem on Linux distributions
that use SELinux.

This commit adds support for symlink xattrs when backing up data,
restoring data, and mounting snapshots via a fuse mount. All calls to
the xattr library have been updated to the use `L` variants of the
various functions, which always operate on the path given, without
following symlinks.

Fixes: #4375

Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
2023-06-19 14:37:31 -04:00
Michael Eischer
8dcb0c4a9d doc: improve description of caching behavior of the check command 2023-06-17 22:18:50 +02:00
Michael Eischer
74ca82a6f8 Merge pull request #4362 from restic/unified-backend-factory
Unified backend initialization
2023-06-17 15:52:37 +02:00
Michael Eischer
9cbc2502c6 Merge pull request #4366 from gautammenghani/master
Issue #3624: Preserve oldest snapshot when keep-* values are not sati…
2023-06-17 15:41:17 +02:00
Michael Eischer
93038ed8f4 s3: restore retries for minio tests 2023-06-17 15:25:08 +02:00
Gautam Menghani
8da5a6649b Preserve oldest snapshot when keep-within* does not collect enough 2023-06-17 15:17:20 +02:00
Michael Eischer
3888c21a27 reword changelog 2023-06-17 15:16:58 +02:00
Michael Eischer
1257c2c075 forget: Add comments to snapshot policy 2023-06-17 15:16:58 +02:00
Gautam Menghani
182b9796e4 Issue #3624: Preserve oldest snapshot when keep-* values are not satisfied 2023-06-17 15:16:58 +02:00
Michael Eischer
cbf87fbdb3 init: don't include password in debug log 2023-06-17 15:15:58 +02:00
Michael Eischer
b5511e8e4c Fix linter warnings 2023-06-17 15:15:58 +02:00
Michael Eischer
50e0d5e6b5 backend: Hardcode backend scheme in Factory
Our ParseConfig implementations always expect a specific scheme, thus no
other scheme would work.
2023-06-17 15:15:58 +02:00
Michael Eischer
705ad51bcc backend: check that StripPassword can be called 2023-06-17 15:15:58 +02:00
Michael Eischer
13a8b5822f backend: Adjust tests to use the Factory to instantiate the backend
This drastically reduces the amount of duplicated test code.
2023-06-17 15:15:58 +02:00
Michael Eischer
3d3bb88745 b2: remove duplicate check for config file during repository creation
No other backend implements that check. The check that a repository is
not yet initialized is handled by the Repository later on.
2023-06-17 15:15:58 +02:00
Michael Eischer
3a3cf608f5 b2/s3: Move config validation from ApplyEnvironment to Open/Create
Conceptually the backend configuration should be validated when creating
or opening the backend, but not when filling in information from
environment variables into the configuration.
2023-06-17 15:15:58 +02:00
Michael Eischer
19ac12d95b location: make genericBackendFactory private 2023-06-17 15:15:58 +02:00
Michael Eischer
3325a7c862 location: extract backend specific part of StripPassword
The tests for the rest backend now reside there.
2023-06-17 15:15:58 +02:00
Michael Eischer
555be49a79 location: Make ParseConfig-test backend agnostic
The backend specific parts of the test are now directly handled by the
respective backend. Duplicate tests were removed.
2023-06-17 15:15:58 +02:00
Michael Eischer
9aa9e0d1ec local/sftp: move limiter setup into backend 2023-06-17 15:15:58 +02:00
Michael Eischer
7d12c29286 backend: Unify backend construction using factory and registry
This unified construction removes most backend-specific code from
global.go. The backend registry will also enable integration tests to
use custom backends if necessary.
2023-06-17 15:15:57 +02:00
Michael Eischer
56836364a4 backend: pass context into every backend constructor 2023-06-17 15:15:57 +02:00
Michael Eischer
4df77e9f26 Merge pull request #4314 from MichaelEischer/restore-json
Add JSON support for the restore progress bar
2023-06-16 23:53:17 +02:00
Michael Eischer
2545c84321 Merge pull request #4365 from MichaelEischer/ci-rework-build-tests
CI: rework build tests
2023-06-16 23:29:39 +02:00
Michael Eischer
191c47d30e Merge pull request #4353 from MichaelEischer/tune-gc
Tune Go garbage collector
2023-06-16 23:24:39 +02:00
Michael Eischer
dd1ef13c1c Merge pull request #4352 from MichaelEischer/pointerless-index
index: optimize go GC performance
2023-06-16 23:23:14 +02:00
Michael Eischer
4d5ee987a7 add changelog about missing AIX builds 2023-06-16 23:19:36 +02:00
Michael Eischer
b2ed42cec4 index: add basic hat test 2023-06-16 23:12:30 +02:00
Michael Eischer
61042a77a4 building on aix is currently not possible 2023-06-09 13:35:35 +02:00
Michael Eischer
4e9e2c3229 CI: Use build-release-binaries to run the cross-compilation tests
gox silently ignored linux/mips and aix/ppc64. This change also removes
the duplicate platform list.
2023-06-09 13:27:57 +02:00
Michael Eischer
faec0ff816 build-release-binaries: support building a subset of all platforms 2023-06-09 13:27:33 +02:00
Michael Eischer
07d1f8047e restore: More descriptive field names for the JSON output 2023-06-09 12:07:55 +02:00
Michael Eischer
b2b0760eb0 restore: add test for json output 2023-06-09 12:03:33 +02:00
Michael Eischer
cf16239058 restore: print output via termStatus 2023-06-09 12:03:33 +02:00
Michael Eischer
1531eab746 mention restore json support in changelog 2023-06-09 12:03:33 +02:00
Michael Eischer
d54176ce5d restore: add basic json progress 2023-06-09 12:03:33 +02:00
Michael Eischer
a9aff885d6 restore: reorganize progress bar code
The structure is now much more similar to that of the backup command.
2023-06-09 12:03:33 +02:00
Michael Eischer
bb20078641 restore: pass termStatus to restore in tests 2023-06-09 12:03:33 +02:00
Michael Eischer
237f32c651 Merge pull request #3261 from DRON-666/prealloc-packs
Prevent local backend file fragmentation by file preallocation.
2023-06-09 11:50:46 +02:00
DRON-666
74e4656850 Update changelog 2023-06-09 11:49:06 +03:00
DRON-666
c37d587f81 Use PreallocateFile in local backend 2023-06-09 11:49:06 +03:00
DRON-666
ffc6b3d887 Move PreallocateFile to fs package 2023-06-09 11:49:06 +03:00
Michael Eischer
88c63a029c Merge pull request #3563 from matta/improve-eta
Improve the ETA displayed during backup
2023-06-08 20:49:19 +02:00
Michael Eischer
0b908bb1fb Address review comments 2023-06-08 20:24:21 +02:00
Matt Armstrong
0372c7ef04 Improve the ETA displayed during backup
The ETA restic displays was based on a rate computed across the entire
backup operation. Often restic can progress at uneven rates. In the worst
case, restic progresses over most of the backup at a very high rate and
then finds new data to back up. The displayed ETA is then unrealistic and
never adapts.

Restic now estimates the transfer rate based on a sliding window, with the
goal of adapting to observed changes in rate. To avoid wild changes in the
estimate, several heuristics are used to keep the sliding window wide
enough to be relatively stable.
2023-06-08 20:05:56 +02:00
Michael Eischer
9464c63550 Make formatNode test timezone independent
formatNode formats the timestamp according to the current time zone. Pin
the local timezone to UTC to ensure the test works everywhere.
2023-06-08 19:18:30 +02:00
Michael Eischer
6ebf2dd235 Reduce duplicate code in test for fomatNode 2023-06-08 19:16:16 +02:00
Michael Eischer
5f153109ba Refactor formatNode 2023-06-08 19:12:49 +02:00
Michael Eischer
2beaa74892 tweak changelog 2023-06-08 19:12:26 +02:00
Michael Eischer
55c21846b1 Revert "index: remove redundant storage of indexmap size"
This reverts commit f1c388c623.

For an uninitialized indexmap the returned size was `-1` which is
unexpected and could cause problems.
2023-06-08 18:08:46 +02:00
Michael Eischer
0f80b6a137 add changelog for gc tuning 2023-06-08 18:02:46 +02:00
Michael Eischer
e14ccb1142 Merge pull request #4298 from restic/backend-parseconfig-cleanup
Unified and slightly type-safer backend config parsing
2023-06-08 12:02:27 +02:00
Michael Eischer
609367195a backend: Fix test compilation with Go 1.18 and 1.19 2023-06-07 22:31:15 +02:00
Michael Eischer
18eb1d3ab0 backend: test cleanup 2023-06-07 22:31:15 +02:00
Michael Eischer
32a6b66267 backend: add standardized Config.ApplyEnvironment
This removes the backend specific special cases while parsing the
configuration in `global.go`.
2023-06-07 22:31:15 +02:00
Michael Eischer
f903db492c backend: let ParseConfig return a Config pointer
In order to change the backend initialization in `global.go` to be able
to generically call cfg.ApplyEnvironment() for supported backends, the
`interface{}` returned by `ParseConfig` must contain a pointer to the
configuration.

An alternative would be to use reflection to convert the type from
`interface{}(Config)` to `interface{}(*Config)` (from value to pointer
type). However, this would just complicate the type mess further.
2023-06-07 22:31:15 +02:00
Michael Eischer
25a0be7f26 backend: fix linter warnings 2023-06-07 22:31:15 +02:00
Michael Eischer
a27b7f1370 backend: use generic instead of any type for test suite 2023-06-07 22:31:14 +02:00
Michael Eischer
fa361dbfbd backend: use generic implementation for ParseConfig tests 2023-06-07 22:30:35 +02:00
Michael Eischer
5260d38980 backend: let ParseConfig return concrete type 2023-06-07 22:30:35 +02:00
Michael Eischer
2f7b4ceae1 backend: Move environment based configuration into backend 2023-06-07 22:30:35 +02:00
Refutable4890
aea7538936 Add an description of check temporary cache directory location 2023-06-07 22:35:33 +08:00
Michael Eischer
49a6a4f5bf Merge pull request #4356 from MichaelEischer/fix-unlock-hint
lock: fix missing hint to unlock command if repository is locked
2023-06-06 21:03:06 +02:00
Michael Eischer
237d00000e lock: fix missing hint to unlock command if repository is locked 2023-06-02 23:16:49 +02:00
dependabot[bot]
0c727f6ad1 Merge pull request #4355 from restic/dependabot/go_modules/github.com/minio/minio-go/v7-7.0.56 2023-06-02 20:51:26 +00:00
Michael Eischer
4e7d3efad4 Merge pull request #4345 from DanielGibson/patch-1
restic restore docs: Mention that restic restore is fastest
2023-06-02 22:44:38 +02:00
Michael Eischer
17446da5fd Merge pull request #4343 from greatroar/cache
cache: Restructure New to remove redundant operations
2023-06-02 22:41:17 +02:00
dependabot[bot]
a3cee840d2 build(deps): bump github.com/minio/minio-go/v7 from 7.0.52 to 7.0.56
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.52 to 7.0.56.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.52...v7.0.56)

---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-06-02 20:35:42 +00:00
dependabot[bot]
c76f3a1e27 Merge pull request #4348 from restic/dependabot/go_modules/github.com/spf13/cobra-1.7.0 2023-06-02 20:34:38 +00:00
dependabot[bot]
1e9714088d Merge pull request #4347 from restic/dependabot/go_modules/golang.org/x/sync-0.2.0 2023-06-02 20:32:27 +00:00
Michael Eischer
58e3f5955c Merge pull request #4350 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/azcore-1.6.0
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.4.0 to 1.6.0
2023-06-02 22:26:55 +02:00
dependabot[bot]
dd8d8b1ae0 build(deps): bump github.com/spf13/cobra from 1.6.1 to 1.7.0
Bumps [github.com/spf13/cobra](https://github.com/spf13/cobra) from 1.6.1 to 1.7.0.
- [Release notes](https://github.com/spf13/cobra/releases)
- [Commits](https://github.com/spf13/cobra/compare/v1.6.1...v1.7.0)

---
updated-dependencies:
- dependency-name: github.com/spf13/cobra
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-06-02 20:22:39 +00:00
dependabot[bot]
e1ac0f0e0c build(deps): bump golang.org/x/sync from 0.1.0 to 0.2.0
Bumps [golang.org/x/sync](https://github.com/golang/sync) from 0.1.0 to 0.2.0.
- [Commits](https://github.com/golang/sync/compare/v0.1.0...v0.2.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sync
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-06-02 20:22:36 +00:00
Michael Eischer
a88d90b8e5 Merge pull request #4346 from restic/dependabot/go_modules/golang.org/x/oauth2-0.8.0
build(deps): bump golang.org/x/oauth2 from 0.7.0 to 0.8.0
2023-06-02 22:21:22 +02:00
Michael Eischer
2fcb3947df prune: trigger GC after prune planning 2023-06-02 21:57:40 +02:00
Michael Eischer
eef0ee7a85 repository: trigger GC after loading the index
Loading the index requires some scratch space, thus make sure that this
memory does not factor into the targeted gc memory usage limit.
2023-06-02 21:56:14 +02:00
Michael Eischer
f1b73c9301 Reduce GOGC to 50
The index used by restic consumes a major part of the total memory
usage. This leads to an unnecessarily large amount of memory that
contains ephemeral objects that are only used for a short time.
2023-06-02 21:51:50 +02:00
Michael Eischer
ac1dfc99bb index: fix blocklist size 2023-06-02 19:39:12 +02:00
Kenny Y
098de3554c Add pull request link 2023-06-01 21:30:51 -04:00
Kenny Y
8812dcd56a Add --human-readable to ls and find output
Modifies format module to add options for human readable storage size formatting, using size parsing already in ui/format.
Cmd flag --human-readable added to ls and find commands.
Additional option added to formatNode to support printing size in regular or new human readable format
2023-06-01 21:18:18 -04:00
dependabot[bot]
379282299a build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore
Bumps [github.com/Azure/azure-sdk-for-go/sdk/azcore](https://github.com/Azure/azure-sdk-for-go) from 1.4.0 to 1.6.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.4.0...sdk/azcore/v1.6.0)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azcore
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-06-01 01:59:16 +00:00
dependabot[bot]
0c796dbd9b build(deps): bump golang.org/x/oauth2 from 0.7.0 to 0.8.0
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.7.0 to 0.8.0.
- [Commits](https://github.com/golang/oauth2/compare/v0.7.0...v0.8.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-06-01 01:58:29 +00:00
Daniel Gibson
f9dded83b3 doc/050_restore.rst: Mention that restic restore is fastest
Especially with lots of files (like when backing up git repos),
restic mount + cp -r can be really slow.
2023-05-31 22:44:36 +02:00
Michael Eischer
88a10a368f Merge pull request #4342 from greatroar/errors
cmd: Don't check for errno == 0
2023-05-31 22:17:07 +02:00
Michael Eischer
9a7056a479 index: implement indexmap.grow() without random access 2023-05-30 20:13:33 +02:00
Michael Eischer
fc05e35a08 index: let indexmap.Each iterate in allocation order
Iterating through the indexmap according to the bucket order has the
problem that all indexEntries are accessed in random order which is
rather cache inefficient.

As we already keep a list of all allocated blocks, just iterate through
it. This allows iterating through a batch of indexEntries without random
memory accesses. In addition, the packID will likely remain similar
across multiple blobs as all blobs of a pack file are added as a single
batch.
2023-05-30 20:12:36 +02:00
Michael Eischer
f1c388c623 index: remove redundant storage of indexmap size 2023-05-30 20:11:53 +02:00
Michael Eischer
12141afbad index: Allow inlining of HAT 2023-05-30 20:11:14 +02:00
Michael Eischer
fed33295c3 index: store indexEntries in hashed array tree
This data structure reduces the wasted memory to O(sqrt(n)). The
top-layer of the hashed array tree (HAT) also has a size of O(sqrt(n)),
which makes it cache efficient. The top-layer should be small enough to
easily fit into the CPU cache and thus only adds little overhead
compared to directly accessing an index entry via a pointer.
2023-05-29 00:24:15 +02:00
Michael Eischer
b217f38ee7 index: Remove pointers from within indexentrys
The indexEntry objects are now allocated in a separate array. References
to an indexEntry are now stored as array indices. This has the benefit
of allowing the garbage collector to ignore the indexEntry objects as
these do not contain pointers and are part of a single large allocation.
2023-05-29 00:24:15 +02:00
Michael Eischer
0c1240360d index: add garbage collection benchmark
Allocates an index and repeatedly triggers the GC.
2023-05-29 00:23:04 +02:00
Michael Eischer
ffca602315 repository: Fix panic in benchmarkLoadIndex 2023-05-28 23:55:47 +02:00
greatroar
da419be43c cache: Restructure New to remove redundant operations
New and its helpers used to create the cache directories several times
over. They now only do so once. The added test ensures that the cache is
produced in a consistent state when parts are deleted.
2023-05-27 10:32:08 +02:00
greatroar
8f1ca8fabe cmd: Don't check for errno == 0
Maybe term.Restore's predecessor could return this, but the current
implementation never does.
2023-05-27 08:52:37 +02:00
Michael Eischer
f74dad2afb Merge pull request #4340 from restic/least-privilege-ci
CI: least privilege for workflows
2023-05-26 20:29:05 +02:00
Michael Eischer
3e287afdbf Merge pull request #4226 from Sohalt/gcs-region
gs: support other regions than us
2023-05-26 20:28:52 +02:00
Michael Eischer
06894484a1 shorten changelog title 2023-05-26 20:05:49 +02:00
Michael Eischer
6e5b42d5c4 CI: least privilege for workflows 2023-05-26 19:58:02 +02:00
sohalt
ed5b2c2c9b gs: support other regions than us 2023-05-26 19:54:42 +02:00
Michael Eischer
13c32b0fbe Merge pull request #4328 from MichaelEischer/cleanup-integration-tests
Cleanup integration tests
2023-05-26 19:49:16 +02:00
Michael Eischer
9747cef338 fix linter warnings 2023-05-26 19:38:44 +02:00
Michael Eischer
8e913e6d3a repair index: always read Quiet flags from GlobalOptions passed as parameter 2023-05-26 19:38:44 +02:00
Michael Eischer
b93459cbb0 repair snapshots: use local copy of globalOptions to open repository 2023-05-26 19:38:44 +02:00
Michael Eischer
692f81ede8 cleanup prune integration test 2023-05-26 19:38:44 +02:00
Michael Eischer
7a268e4aba always access stdout/stderr via globalOptions 2023-05-26 19:38:44 +02:00
Michael Eischer
4b3a0b4104 read JSON/Quiet flag from the passed in globalOptions 2023-05-26 19:38:44 +02:00
Michael Eischer
cebce52c16 test: add helper to capture stdout for integration tests 2023-05-26 19:38:44 +02:00
Michael Eischer
675a49a95b Restructure integration tests
The tests are now split into individual files for each command. The
separation isn't perfect as many tests make use of multiple commands. In
particular `init`, `backup`, `check` and `list` are used by a larger
number of test cases.

Most tests now reside in files name cmd_<name>_integration_test.go. This
provides a certain indication which commands have significant test
coverage.
2023-05-26 19:38:43 +02:00
Michael Eischer
e2dba9f5c7 test: cleanup a some check calls 2023-05-26 19:34:57 +02:00
Michael Eischer
06fd6b54d7 test: print log output if testRunCheck fails 2023-05-26 19:34:57 +02:00
Michael Eischer
419e6f26b1 tests: Simplify checks that a specific number of snapshots exists 2023-05-26 19:34:57 +02:00
Michael Eischer
c3212ab6a6 test: use standard logging methods from testing for the test helpers
Use the logging methods from testing.TB to make use of tb.Helper(). This
allows the tests to log the filename and line number in which the test
helper was called. Previously the test helper was logged which is rarely
useful.
2023-05-26 19:34:57 +02:00
Michael Eischer
658aa4c0f7 Merge pull request #4339 from greatroar/ioctl
backend, termstatus: fix ioctl calls for 64-bit big-endian platforms
2023-05-26 19:34:14 +02:00
Michael Eischer
998cf5a7f8 Merge pull request #4333 from MichaelEischer/staticcheck
Update golangci-lint and resolve all errors
2023-05-26 19:34:00 +02:00
Michael Eischer
7eec91f841 add changelog 2023-05-26 19:23:51 +02:00
greatroar
51dc80be5b backend: Use IoctlSetPointerInt for tcsetpgrp
This function casts its argument to int32 before passing it to the
system call, so that big-endian CPUs read the lower rather than the
upper 32 bits of the pid.

This also gets rid of the last import of "unsafe" in the Unix build.
I changed syscall to x/sys/unix while I was at it, to remove one more
import line. The constants and types there are aliases for their syscall
counterparts.
2023-05-25 17:31:51 +02:00
greatroar
ddbc0c1b37 termstatus: Fix IsProcessBackground for 64-bit big-endian Linux
Fixes #4223. Verified with QEMU on linux/amd64.
2023-05-25 17:20:42 +02:00
Michael Eischer
ecbf8e055c CI: upgrade golangci-lint and check full codebase 2023-05-18 21:17:53 +02:00
Michael Eischer
16ba237d8b azure: Fix missing rate limiting 2023-05-18 21:17:53 +02:00
Michael Eischer
a466e945d9 stats: pass StatsOptions via parameter 2023-05-18 21:17:53 +02:00
Michael Eischer
03b9764bce init/generate: check that no parameters are passed 2023-05-18 21:17:53 +02:00
Michael Eischer
22c9276719 rest: remove unused parameter 2023-05-18 21:17:53 +02:00
Michael Eischer
1e33b285c1 lock: remove unused parameter 2023-05-18 21:17:53 +02:00
Michael Eischer
c05f96e6b9 repair index: remove no longer used paramter 2023-05-18 21:17:53 +02:00
Michael Eischer
94752b7ee2 backup: Remove unused parameters from rejectFunc collection 2023-05-18 21:17:53 +02:00
Michael Eischer
0058745881 test: use parameter instead of hardcoded constant 2023-05-18 21:17:53 +02:00
Michael Eischer
a719d10e22 termstatus: remove unused parameter from helpers 2023-05-18 21:17:53 +02:00
Michael Eischer
b0a01ae68a ui/backup: remove unused parameter from ProgressPrinter interface 2023-05-18 21:17:53 +02:00
Michael Eischer
472bf5184f Replace lots of unused parameters with _
The parameters are required by the implemented function signature or interface.
2023-05-18 21:17:53 +02:00
Michael Eischer
d1a5ec7839 Rename unused testing parameter to _
The parameter is an additional marker that the test helper must only be
used for tests.
2023-05-18 21:17:53 +02:00
Michael Eischer
1514593f22 Remove unused context or testing parameters 2023-05-18 21:17:53 +02:00
Michael Eischer
5e4e268bdc Use _ as parameter name for unused Context
The context is required by the implemented interface.
2023-05-18 21:15:45 +02:00
Michael Eischer
3252f60df5 Ignore deprecated imports for poly1305 and openpgp 2023-05-18 21:15:45 +02:00
Michael Eischer
2fa8b96843 Don't shadow builtins 2023-05-18 21:15:45 +02:00
Michael Eischer
7a01bd3b67 Remove unnecessary else block 2023-05-18 21:15:45 +02:00
Michael Eischer
319087c056 Remove redundant if ...; err != nil; return pattern 2023-05-18 21:15:45 +02:00
Michael Eischer
6ed73ed408 CI: remove deprecated linters 2023-05-18 21:15:45 +02:00
Michael Eischer
c832a492ac fuse: use syscall errnos directly to fix deprecations 2023-05-18 21:15:45 +02:00
Michael Eischer
e01baeabba Use either test or rtest to refer to internal test helpers
A single test file should not use both names.
2023-05-18 21:15:45 +02:00
Michael Eischer
bfc9c6c971 Merge pull request #4331 from MichaelEischer/fix-mount-failures
Hopefully fix `TestMount` failures
2023-05-18 20:22:57 +02:00
Michael Eischer
5773b86d02 repository: Push all usage of errors.Fatal out of the package
As the `Fatal` error type only includes a string, it becomes impossible
to inspect the contained error. This is for a example a problem for the
fuse implementation, which must be able to detect context.Canceled
errors.

Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2023-05-18 17:27:41 +02:00
Michael Eischer
a013014c24 backup: return normal error if --group-by cannot be parsed
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2023-05-18 17:23:31 +02:00
Michael Eischer
f9850b79b5 rest/sftp: Remove redundant fatal from error message
This caused restic to exit with error messages like
`Fatal: parsing repository location failed: Fatal: sftp path [...]`
`Fatal: create repository at rest:http://localhost:12345/ failed: Fatal: config file already exists`
2023-05-18 17:22:26 +02:00
Michael Eischer
2f518b7241 Merge pull request #4304 from MichaelEischer/unlimited-lock-refresh
lock: Do not limit backend concurrency for lock files
2023-05-13 21:40:29 +02:00
Michael Eischer
49be202cb0 Merge pull request #4315 from MichaelEischer/restore-hardlink-progress
restore: Correctly account for hardlinks in progress bar
2023-05-13 21:37:40 +02:00
Michael Eischer
19ebc1b786 restore: Add basic test for progress bar accounting of hardlinks 2023-05-08 20:51:49 +02:00
Michael Eischer
23a122a901 restore: count files in the same way as the stats command 2023-05-08 20:51:49 +02:00
Michael Eischer
e77002f841 restore: correctly count hardlinks in progress bar
For hardlinked files, only the first instance of that file increases the
amount of bytes to restore. All later instances only increase the file
count but not the restore size.
2023-05-08 20:51:49 +02:00
Michael Eischer
d05f6211d1 lock: Do not limit backend concurrency for lock files
restic must be able to refresh lock files in time. However, large
uploads over slow connections can cause the lock refresh to be stuck
behind the large uploads and thus time out.
2023-05-08 19:04:46 +02:00
Michael Eischer
ee3c55ea3d Merge pull request #2876 from aawsome/new-repair-command
Add repair command
2023-05-05 23:22:24 +02:00
Michael Eischer
db046c0acc Merge pull request #4318 from MichaelEischer/status-output-truncation
Fix status output truncation
2023-05-05 23:09:59 +02:00
Michael Eischer
3e6a26e2e9 Merge pull request #4317 from MichaelEischer/debug-test-mount-failures
mount: enable debug logging for the flaky TestMount test
2023-05-05 23:08:03 +02:00
greatroar
65c5e511a1 ui/termstatus: Add test for Terminal.SetStatus 2023-05-05 22:51:41 +02:00
Michael Eischer
6d10c655a0 termstatus: test status line sanitization 2023-05-05 22:51:41 +02:00
Michael Eischer
bb40e49e75 ui/termstatus: Fix truncation of status output
The last line was not truncated as expected
2023-05-05 22:45:52 +02:00
Michael Eischer
fefe15d7a1 Merge pull request #4316 from MichaelEischer/fix-check-modified-data
Fix flaky TestCheckerModifiedData test
2023-05-05 22:38:28 +02:00
Michael Eischer
78e5aa6d30 repair snapshots: add basic tests 2023-05-04 23:00:46 +02:00
Michael Eischer
1cb11ad8ad mount: enable debug logging for the flaky TestMount test
The test case fails from time to time with an Input/Output error while
trying to access the snapshots directory.
2023-05-01 18:03:17 +02:00
Michael Eischer
90a663c94f Merge pull request #4308 from greatroar/tar-header
dump: Report filename with tar.ErrFieldTooLong
2023-05-01 17:30:31 +02:00
Michael Eischer
88a7231217 report snapshot id if loading failed 2023-05-01 17:24:13 +02:00
Michael Eischer
c0627dc80d check: Fix flaky TestCheckerModifiedData
The test had a 4% chance of not modified the data read from the
repository, in which case the test would fail. Change the data
manipulation to just modified each read operation.
2023-05-01 17:18:19 +02:00
Michael Eischer
e71367e6b9 repair snapshots: update changelog 2023-05-01 16:06:17 +02:00
Michael Eischer
5aa37acdaa repair snapshots: cleanup command help 2023-05-01 16:06:17 +02:00
Michael Eischer
9c64a95df8 doc: rewrite troubleshooting section 2023-05-01 16:06:17 +02:00
Michael Eischer
7c8dd61e8c repair snapshots: cleanup warnings 2023-05-01 15:22:30 +02:00
Michael Eischer
f6cc10578d repair snapshots: Always sanitize file nodes
If the node for a file is intact, this is a no-op.
2023-05-01 15:22:18 +02:00
Michael Eischer
4ce87a7f64 repair snapshots: port to filterAndReplaceSnapshot
The previous approach of rewriting all snapshots first, then flushing
the repository data and finally removing old snapshots has the downside
that an interrupted command execution leaves behind broken snapshots as
not all new data is already flushed.
2023-05-01 15:22:03 +02:00
Michael Eischer
e17ee40a31 repair snapshots: Port to use walker.TreeRewriter 2023-05-01 15:20:36 +02:00
Michael Eischer
1bd1f3008d walker: extend TreeRewriter to support snapshot repairing
This adds support for caching already rewritten trees, handling of load
errors and disabling the check that the serialization doesn't lead to
data loss.
2023-05-01 15:20:24 +02:00
Michael Eischer
38dac78180 walker: restructure FilterTree into TreeRewriter
The more generic RewriteNode callback replaces the SelectByName and
PrintExclude functions. The main part of this change is a preparation to
allow using the TreeRewriter for the `repair snapshots` command.
2023-05-01 15:20:12 +02:00
Michael Eischer
bc2399fbd9 walker: recurse into directory based on node type
A broken directory might also not have a subtree.
2023-05-01 15:20:00 +02:00
Michael Eischer
1a9705fc95 walker: Simplify change detection in FilterTree
Now the rewritten tree is always serialized which makes sure that we
don't accidentally miss any relevant changes.
2023-05-01 15:19:48 +02:00
Michael Eischer
8c4caf09a8 repair snapshots: Do not rename repaired files
The files in a tree must be sorted in lexical order. However, this
cannot be guaranteed when appending a filename suffix. For two files

file, file.rep

where "file" is broken, this would result in

file.repaired, file.rep

which is no longer sorted.

In addition, adding a filename suffix is also prone to filename
collisions which would require a rather complex search for a
collision-free name in order to work reliably.
2023-05-01 15:19:36 +02:00
Michael Eischer
375189488c rewrite: prepare for code sharing with rewrite snapshots 2023-05-01 15:19:24 +02:00
Michael Eischer
903651c719 repair snapshots: partially synchronize code with rewrite command
Simplify CLI options:
* Rename "DeleteSnapshots" to "Forget"
* Replace "AddTag" and "Append" with hardcoded values

Change output and snapshot modifications to be more in line with the
"rewrite" command.
2023-05-01 15:19:11 +02:00
Michael Eischer
118d599d0a Rename 'rebuild-index' to 'repair index'
The old name still works, but is deprecated.
2023-05-01 15:16:44 +02:00
Michael Eischer
db459eda21 move to subcommand 2023-05-01 15:15:48 +02:00
Michael Eischer
a14a63cd29 modernize code 2023-05-01 15:12:09 +02:00
Alexander Weiss
947f0c345e correct typos 2023-05-01 14:56:42 +02:00
Alexander Weiss
d23a2e1925 better error handling and correct nil tree behavior 2023-05-01 14:56:42 +02:00
Alexander Weiss
08ae708b3b make linter happy 2023-05-01 14:56:42 +02:00
Alexander Weiss
99a05d5ab2 Update troubleshooting documentation 2023-05-01 14:56:42 +02:00
Alexander Weiss
6557f36f61 Add changelog and docu for #2876 2023-05-01 14:56:42 +02:00
Alexander Weiss
5f58797ba7 Add repair command 2023-05-01 14:56:42 +02:00
Alexander Weiss
9cef6b4c69 Add troubleshooting documentation 2023-05-01 14:56:42 +02:00
Michael Eischer
8a78a042db Merge pull request #4310 from restic/dependabot/go_modules/github.com/klauspost/compress-1.16.5
build(deps): bump github.com/klauspost/compress from 1.16.0 to 1.16.5
2023-05-01 14:35:57 +02:00
Michael Eischer
b491af2b57 Merge pull request #4311 from restic/dependabot/go_modules/golang.org/x/crypto-0.8.0
build(deps): bump golang.org/x/crypto from 0.7.0 to 0.8.0
2023-05-01 14:35:37 +02:00
dependabot[bot]
d747a9c401 build(deps): bump golang.org/x/crypto from 0.7.0 to 0.8.0
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.7.0 to 0.8.0.
- [Release notes](https://github.com/golang/crypto/releases)
- [Commits](https://github.com/golang/crypto/compare/v0.7.0...v0.8.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-05-01 11:15:38 +00:00
dependabot[bot]
888c1ae63a build(deps): bump github.com/klauspost/compress from 1.16.0 to 1.16.5
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.16.0 to 1.16.5.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml)
- [Commits](https://github.com/klauspost/compress/compare/v1.16.0...v1.16.5)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-05-01 11:07:29 +00:00
Michael Eischer
5eeb257c95 Merge pull request #4312 from restic/dependabot/go_modules/github.com/minio/minio-go/v7-7.0.52
build(deps): bump github.com/minio/minio-go/v7 from 7.0.50 to 7.0.52
2023-05-01 13:00:48 +02:00
Michael Eischer
ae6729cf89 Merge pull request #4309 from restic/dependabot/go_modules/golang.org/x/oauth2-0.7.0
build(deps): bump golang.org/x/oauth2 from 0.6.0 to 0.7.0
2023-05-01 12:58:55 +02:00
dependabot[bot]
6abd494915 build(deps): bump github.com/minio/minio-go/v7 from 7.0.50 to 7.0.52
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.50 to 7.0.52.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.50...v7.0.52)

---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-05-01 01:59:50 +00:00
dependabot[bot]
7147a54ceb build(deps): bump golang.org/x/oauth2 from 0.6.0 to 0.7.0
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.6.0 to 0.7.0.
- [Release notes](https://github.com/golang/oauth2/releases)
- [Commits](https://github.com/golang/oauth2/compare/v0.6.0...v0.7.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-05-01 01:59:10 +00:00
Michael Eischer
8d971172c4 Merge pull request #4306 from MichaelEischer/document-restic-compression-env
Document that the compression mode can be set via $RESTIC_COMPRESSION
2023-04-30 18:26:10 +02:00
Michael Eischer
37d0e323eb doc: update help output in manual_rest.rst 2023-04-30 16:12:29 +02:00
Michael Eischer
face5bd7f7 Document that the compression mode can be set via $RESTIC_COMPRESSION 2023-04-30 16:11:53 +02:00
Michael Eischer
1daf928a77 Merge pull request #4305 from MichaelEischer/stracktrace-for-windows
Print stacktrace in SIGINT handler if RESTIC_DEBUG_STACKTRACE_SIGINT set
2023-04-30 16:08:58 +02:00
Michael Eischer
37bab08181 Merge pull request #4302 from MichaelEischer/logging-cleanup
Tweak debug logs
2023-04-30 16:08:08 +02:00
Michael Eischer
6dc2324d2e Merge pull request #4301 from MichaelEischer/less-restore-memory
restore: slightly reduce memory usage while restoring files
2023-04-30 16:07:43 +02:00
Michael Eischer
850cd9aace Merge pull request #4299 from MichaelEischer/update-scripting-docs
doc: recommend `cat config` to check if a repository exists
2023-04-30 16:04:59 +02:00
greatroar
b50ff04cf3 dump: Report filename with tar.ErrFieldTooLong
Updates #4307.
2023-04-27 11:45:41 +02:00
Michael Eischer
c8641f4479 Merge branch 'patch-release' 2023-04-24 20:44:38 +02:00
Alexander Neumann
ac7ac0cb97 Set development version for 0.15.2 2023-04-24 20:28:37 +02:00
Alexander Neumann
be8be3397c Add version for 0.15.2 2023-04-24 20:28:37 +02:00
Alexander Neumann
db6b4f8912 Update manpages and auto-completion 2023-04-24 20:28:37 +02:00
Alexander Neumann
1f3f042f32 Generate CHANGELOG.md for 0.15.2 2023-04-24 20:28:24 +02:00
Alexander Neumann
0aaa4e6cbe Prepare changelog for 0.15.2 2023-04-24 20:28:24 +02:00
Leo R. Lundgren
0bac935dac doc: Polish changelogs 2023-04-23 22:10:04 +02:00
Michael Eischer
306a29980a Print stacktrace in SIGINT handler if RESTIC_DEBUG_STACKTRACE_SIGINT set
The builtin mechanism to capture a stacktrace in Go is to send a SIGQUIT
to the running process. However, this mechanism is not avaiable on
Windows. Thus, tweak the SIGINT handler to dump a stacktrace if the
environment variable `RESTIC_DEBUG_STACKTRACE_SIGINT` is set.
2023-04-23 15:50:40 +02:00
Michael Eischer
1e6e9f9bd0 tweak changelogs 2023-04-23 12:46:55 +02:00
greatroar
f342db7666 ui/termstatus: Quote funny filenames
Fixes #2260, #4191.
2023-04-23 12:45:40 +02:00
Michael Eischer
41cc320145 Merge pull request #4166 from MichaelEischer/cache-mkdirall
cache: Create missing intermediate directories when caching a file
2023-04-23 11:44:18 +02:00
Michael Eischer
cdb0fb9c06 tweak debug logs 2023-04-23 11:38:06 +02:00
Michael Eischer
94cbc6392d restore: slightly reduce memory usage while restoring files
The information which target files are contained in a pack file is no
longer necessary after processing a pack.
2023-04-23 11:33:21 +02:00
Michael Eischer
78a1757e5a Cancel current command if cache becomes unusable
If the cache suddenly disappears, the current command will now fail.
2023-04-23 11:31:15 +02:00
Michael Eischer
22562d2132 Merge pull request #4300 from MichaelEischer/less-flaky-tests
Increase timeouts for lock refresh tests
2023-04-23 11:24:59 +02:00
Michael Eischer
51d823348d Merge pull request #4286 from MichaelEischer/backend-cleanup-logging
Normalize backend logging and connection limiting
2023-04-23 11:24:26 +02:00
Michael Eischer
831f593b87 backend/sema: Add tests 2023-04-22 13:20:20 +02:00
Michael Eischer
179e11c2ae Increase timeouts for lock refresh tests
When saving files to the local backend, in some cases the used fsync
calls are slow enough to cause the tests to time out. Thus, increase the
test timeouts as a stopgap measure until we can use the mem backend for
these tests.
2023-04-22 12:45:59 +02:00
Michael Eischer
ebba233a3a backend/sema: rename constructor to NewBackend 2023-04-22 12:32:57 +02:00
rawtaz
8479390d7c Merge pull request #3939 from Fabien-jrt/resticprofile
doc: Add resticprofile to scheduling section
2023-04-22 01:24:34 +02:00
Fabien-Jrt
756f43d5f9 doc: Add resticprofile to scheduling section 2023-04-22 01:04:48 +02:00
Michael Eischer
affd04c125 doc: recommend cat config to check if a repository exists
This will be much faster if a large number of snapshots exists.
2023-04-21 22:56:31 +02:00
Michael Eischer
f27750e270 backend/sema: rename type to connectionLimitedBackend 2023-04-21 22:53:09 +02:00
Michael Eischer
1dd873b706 Merge pull request #4293 from MichaelEischer/fix-groups-by
Fix parent snapshot selection for relative paths
2023-04-21 22:47:03 +02:00
Michael Eischer
7a60d9e54f Merge pull request #4288 from MichaelEischer/log-warnings-to-debug-log
Add warnings via Warnf to the debug log
2023-04-21 22:39:00 +02:00
Michael Eischer
3001dd8c2b Add test to verify that the backup parent is correctly selected 2023-04-21 22:35:02 +02:00
Michael Eischer
4503aea0df Merge pull request #4296 from MichaelEischer/rewrite-reorder-output
rewrite: log snapshot saved before removal of the old snapshot
2023-04-19 22:00:32 +02:00
Michael Eischer
09cddb8927 rewrite: log snapshot saved before removal of the old snapshot
The snapshot was already saved before removing the old snapshot. Only
the log messages were printed in the wrong order.
2023-04-17 21:00:45 +02:00
Michael Eischer
913eab3361 Merge pull request #4234 from thndrbrrr/forget-opts-neg1-means-forever-issue-2565
restic forget --keep-* options will interpret -1 as "forever"
2023-04-14 23:18:47 +02:00
Michael Eischer
a9c7c12276 Merge pull request #4265 from HeikoSchlittermann/chg/progress-to-stderr
generate: do not write progress to STDOUT
2023-04-14 23:18:23 +02:00
Michael Eischer
85eef232e6 Merge pull request #4232 from thndrbrrr/copy-chunker-params-verbose-msg
init: Add --copy-chunker-params verbose msg
2023-04-14 23:17:56 +02:00
Michael Eischer
07a44a88f2 Fix snapshot filtering for relative paths in the backup command
The snapshot filtering internally converts relative paths to absolute
ones to ensure that the parent snapshots selection works for backups of
relative paths.
2023-04-14 22:58:34 +02:00
Michael Eischer
48e065d971 Sync dependency upgrades from master
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob still uses v0.5.1
as upgrading it would increase the minimum Go version on Solaris to
1.20.
2023-04-14 22:58:34 +02:00
Michael Eischer
a06d927dce Merge pull request #4291 from greatroar/widechars
ui/termstatus: Optimize Truncate
2023-04-14 22:48:34 +02:00
Michael Eischer
fd3ed9e2f4 Merge pull request #4192 from greatroar/quote
ui/termstatus: Quote funny filenames
2023-04-14 22:39:09 +02:00
Michael Eischer
6042df075f migrations: Fix S3 backend detection 2023-04-14 22:32:16 +02:00
Michael Eischer
c934c99d41 gs: replace usage of context.Background() 2023-04-14 22:32:15 +02:00
Michael Eischer
616926d2c1 gs: use IsNotExist to check error 2023-04-14 22:32:15 +02:00
Michael Eischer
05abc6d6f5 backend: deduplicate implementation of Delete() method 2023-04-14 22:32:15 +02:00
Michael Eischer
45244fdf68 backend: remove parameter validation tests
These parameter validations have been factored out into
SemaphoreBackend.
2023-04-14 22:32:15 +02:00
Michael Eischer
803640ba4b backend: remove a few unnecessary debug logs 2023-04-14 22:32:15 +02:00
Michael Eischer
8e1e3844aa backend: factor out connection limiting and parameter validation
The SemaphoreBackend now uniformly enforces the limit of concurrent
backend operations. In addition, it unifies the parameter validation.

The List() methods no longer uses a semaphore. Restic already never runs
multiple list operations in parallel.

By managing the semaphore in a wrapper backend, the sections that hold a
semaphore token grow slightly. However, the main bottleneck is IO, so
this shouldn't make much of a difference.

The key insight that enables the SemaphoreBackend is that all of the
complex semaphore handling in `openReader()` still happens within the
original call to `Load()`. Thus, getting and releasing the semaphore
tokens can be refactored to happen directly in `Load()`. This eliminates
the need for wrapping the reader in `openReader()` to release the token.
2023-04-14 22:32:15 +02:00
Michael Eischer
8b5ab5b59f dryrun: fix outdated comments 2023-04-14 22:32:15 +02:00
Michael Eischer
4703473ec5 backend: extract most debug logs into logger backend 2023-04-14 22:32:15 +02:00
Michael Eischer
8bfc2519d7 backend: Deduplicate sanity checks for parameters of Load() method
The check is now handled by backend.DefaultLoad. This also guarantees
consistent behavior across all backends.
2023-04-14 22:32:15 +02:00
Michael Eischer
ba16904eed backup: Add test to verify parent snapshot selection for relative paths 2023-04-14 22:21:43 +02:00
Michael Eischer
2841a87cc6 Fix snapshot filtering for relative paths in the backup command
The snapshot filtering internally converts relative paths to absolute
ones to ensure that the parent snapshots selection works for backups of
relative paths.
2023-04-14 21:53:55 +02:00
Michael Eischer
fab4a8a4d2 Properly initialize the --group-by option for backup tests 2023-04-14 21:53:01 +02:00
greatroar
3b24c15c3d fuse: Mix inode hashes in a non-symmetric way
Since 0.15 (#4020), inodes are generated as hashes of names, xor'd with
the parent inode. That means that the inode of a/b/b is

	h(a/b/b) = h(a) ^ h(b) ^ h(b) = h(a).

I.e., the grandchild has the same inode as the grandparent. GNU find
trips over this because it thinks it has encountered a loop in the
filesystem, and fails to search a/b/b. This happens more generally when
the same name occurs an even number of times.

Fix this by multiplying the parent by a large prime, so the combining
operation is not longer symmetric in its arguments. This is what the FNV
hash does, which we used prior to 0.15. The hash is now

	h(a/b/b) = h(b) ^ p*(h(b) ^ p*h(a))

Note that we already ensure that h(x) is never zero.

Collisions can still occur, but they should be much less likely to occur
within a single path.

Fixes #4253.
2023-04-14 20:50:39 +02:00
greatroar
4304e01ca2 fuse: Report fuse.Attr.Blocks correctly
Fixes #4239.
2023-04-14 20:50:08 +02:00
Torben Giesselmann
6d6c04abef forget: Simplify usage text 2023-04-14 10:05:23 -07:00
greatroar
49e32f3f8a ui/termstatus: Optimize Truncate
x/text/width.LookupRune has to re-encode its argument as UTF-8,
while LookupString operates on the UTF-8 directly.
The uint casts get rid of a bounds check.

Benchmark results, with b.ResetTimer introduced first:

name               old time/op  new time/op  delta
TruncateASCII-8    69.7ns ± 1%  55.2ns ± 1%  -20.90%  (p=0.000 n=20+18)
TruncateUnicode-8   350ns ± 1%   171ns ± 1%  -51.05%  (p=0.000 n=20+19)
2023-04-14 11:13:39 +02:00
greatroar
9412f37e50 ui/termstatus: Quote funny filenames
Fixes #2260, #4191.
2023-04-14 09:36:02 +02:00
Ian Muge
593eb710b4 added changelog 2023-04-13 22:54:36 +02:00
greatroar
97274ecabd cmd, restic: Refactor and fix snapshot filtering
This turns snapshotFilterOptions from cmd into a restic.SnapshotFilter
type and makes restic.FindFilteredSnapshot and FindFilteredSnapshots
methods on that type. This fixes #4211 by ensuring that hosts and paths
are named struct fields instead of unnamed function arguments in long
lists of such.

Timestamp limits are also included in the new type. To avoid too much
pointer handling, the convention is that time zero means no limit.
That's January 1st, year 1, 00:00 UTC, which is so unlikely a date that
we can sacrifice it for simpler code.
2023-04-13 22:51:45 +02:00
Michael Eischer
74f7dd0b38 Make help for --verbose less confusing
The output is now
```
-v, --verbose                    be verbose (specify multiple times or a level using --verbose=n, max level/times is 2)
```
instead of
```
-v, --verbose n                  be verbose (specify multiple times or a level using --verbose=n, max level/times is 2)
```
2023-04-13 22:46:37 +02:00
Michael Eischer
21ad357c10 add linux/riscv64 builds 2023-04-13 22:45:35 +02:00
Heiko Schlittermann (HS12-RIPE)
087cf7e114 generate: write progress to STDOUT if this is a terminal
This allows usage as:

  eval "$(restic generated --bash-completion /dev/stdout)"
2023-04-12 22:39:59 +02:00
Michael Eischer
e604939e72 Debug log status code if execution is interrupted
Currently, there is no clear indication in the debug log whether restic
exited or not.
2023-04-12 21:58:35 +02:00
Michael Eischer
37aca6bec0 Add warnings via Warnf to the debug log 2023-04-12 21:37:37 +02:00
Michael Eischer
bdcafbc11c Merge pull request #4282 from TorstenC/patch-1
040_backup.rst line 228 - #4280 --exclude-caches
2023-04-11 22:34:16 +02:00
Torsten C
ba33e41068 040_backup.rst line 228 - #4280 --exclude-caches
Proposal for issue #4280
2023-04-11 22:33:10 +02:00
Michael Eischer
4661f45a8c Merge pull request #4220 from Skaronator/master
Add jq to container image
2023-04-11 22:26:02 +02:00
Michael Eischer
2091fc0dde Merge pull request #4107 from jooola/feature-wait-for-unlock
Add a global option --retry-lock
2023-04-11 22:21:01 +02:00
Michael Eischer
9238dcc81a Merge pull request #4201 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/storage/azblob-1.0.0
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from 0.5.1 to 1.0.0
2023-04-11 22:20:38 +02:00
Torben Giesselmann
71537da4b0 init: Add --copy-chunker-params verbose msg 2023-04-09 14:59:07 -07:00
Torben Giesselmann
ce51d2f3c0 forget: Update usage text 2023-04-09 12:59:15 -07:00
Torben Giesselmann
403b7ca2be forget: Update documentation 2023-04-09 12:57:37 -07:00
Torben Giesselmann
309cf0586a Add changelog entry 2023-04-09 12:05:15 -07:00
Torben Giesselmann
9f9e91eb0d Fix comment to comply with linter 2023-04-09 11:47:10 -07:00
Michael Eischer
1f43003cc1 Merge branch 'master' into feature-wait-for-unlock 2023-04-07 20:32:21 +02:00
Michael Eischer
8ce5f29758 lock: increase test timeout tolerances to avoid test failures 2023-04-07 20:07:05 +02:00
Michael Eischer
8afc117aa3 Merge pull request #4285 from MichaelEischer/debug-lock-refresh-failures
lock: Improve debug logging in the test
2023-04-07 20:06:15 +02:00
Michael Eischer
cf1cc1fb72 lock: Print stacktrace if TestLockSuccessfulRefresh fails 2023-04-07 19:51:31 +02:00
Michael Eischer
64233ca0a7 lock: Improve debug logging in the test 2023-04-07 19:43:00 +02:00
jo
ea59896bd6 Add a global option --retry-lock
Fixes restic#719

If the option is passed, restic will wait the specified duration of time
and retry locking the repo every 10 seconds (or more often if the total
timeout is relatively small).

- Play nice with json output
- Reduce wait time in lock tests
- Rework timeout last attempt
- Reduce test wait time to 0.1s
- Use exponential back off for the retry lock
- Don't pass gopts to lockRepo functions
- Use global variable for retry sleep setup
- Exit retry lock on cancel
- Better wording for flag help
- Reorder debug statement
- Refactor tests
- Lower max sleep time to 1m
- Test that we cancel/timeout in time
- Use non blocking sleep function
- Refactor into minDuration func

Co-authored-by: Julian Brost <julian@0x4a42.net>
2023-04-07 16:24:14 +02:00
Michael Eischer
806a0cdce3 bump minimum go version on Solaris to 1.20 2023-04-07 15:40:57 +02:00
Michael Eischer
faa83db9e4 azure: Adapt code to API change 2023-04-07 15:05:55 +02:00
dependabot[bot]
9358a5fb37 build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
Bumps [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) from 0.5.1 to 1.0.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/armcore/v0.5.1...v1.0)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-04-07 11:03:25 +00:00
Michael Eischer
71c9516b26 Merge pull request #4247 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/azcore-1.4.0
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.3.1 to 1.4.0
2023-04-07 13:01:04 +02:00
Michael Eischer
17ac91fba3 Merge pull request #4283 from restic/dependabot/go_modules/google.golang.org/api-0.116.0
build(deps): bump google.golang.org/api from 0.114.0 to 0.116.0
2023-04-07 12:58:54 +02:00
Michael Eischer
26a3c47c5c Merge pull request #4255 from greatroar/fuse-hash
fuse: Mix inode hashes in a non-symmetric way
2023-04-07 12:56:57 +02:00
Michael Eischer
7896e50301 Merge pull request #4271 from restic/dependabot/go_modules/github.com/minio/minio-go/v7-7.0.50
build(deps): bump github.com/minio/minio-go/v7 from 7.0.49 to 7.0.50
2023-04-07 12:47:30 +02:00
dependabot[bot]
9584cbda90 build(deps): bump google.golang.org/api from 0.114.0 to 0.116.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.114.0 to 0.116.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.114.0...v0.116.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-04-07 10:27:38 +00:00
dependabot[bot]
cacc48fc09 build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore
Bumps [github.com/Azure/azure-sdk-for-go/sdk/azcore](https://github.com/Azure/azure-sdk-for-go) from 1.3.1 to 1.4.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.3.1...sdk/azcore/v1.4.0)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azcore
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-04-07 10:27:30 +00:00
Michael Eischer
7fbaca577b Merge pull request #4272 from restic/dependabot/go_modules/cloud.google.com/go/storage-1.30.1
build(deps): bump cloud.google.com/go/storage from 1.29.0 to 1.30.1
2023-04-07 12:26:20 +02:00
Michael Eischer
1f9dd84d1e Merge pull request #3991 from mark-herrmann/feature/restore-progress
restore: show progress
2023-04-07 12:24:55 +02:00
Michael Eischer
c8ff5592b8 Merge pull request #4273 from ducalex/gitignore-restic-exe
Added restic.exe to the .gitignore
2023-04-07 12:10:09 +02:00
Mark Herrmann
f875a8843d restore: Add progress bar
Co-authored-by: Mark Herrmann <mark.herrmann@mailbox.org>
2023-04-07 12:08:23 +02:00
Alex Duchesne
0ed5c20c57 Added restic.exe to the .gitignore 2023-03-28 23:07:25 -04:00
dependabot[bot]
21edbdc3ac build(deps): bump cloud.google.com/go/storage from 1.29.0 to 1.30.1
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.29.0 to 1.30.1.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.29.0...spanner/v1.30.1)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-03-26 20:14:09 +00:00
dependabot[bot]
220d937975 build(deps): bump github.com/minio/minio-go/v7 from 7.0.49 to 7.0.50
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.49 to 7.0.50.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.49...v7.0.50)

---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-03-26 20:13:34 +00:00
rawtaz
5f13bbc118 Merge pull request #4270 from MichaelEischer/monthly-dependabot
Switch back to monthly dependabot updates
2023-03-26 22:09:21 +02:00
Michael Eischer
9672670756 Merge pull request #4240 from greatroar/fuse-blocks
fuse: Report fuse.Attr.Blocks correctly
2023-03-26 21:57:53 +02:00
Michael Eischer
5ac24a9744 Switch back to monthly dependabot updates 2023-03-26 21:55:27 +02:00
Michael Eischer
99e247caa6 Merge pull request #4259 from restic/dependabot/github_actions/actions/setup-go-4
build(deps): bump actions/setup-go from 3 to 4
2023-03-26 21:53:47 +02:00
Michael Eischer
0c705e07db Merge pull request #4249 from restic/dependabot/go_modules/golang.org/x/oauth2-0.6.0
build(deps): bump golang.org/x/oauth2 from 0.5.0 to 0.6.0
2023-03-26 10:38:53 +02:00
Michael Eischer
024d01d85b Merge pull request #4266 from restic/linkfix
doc: Correct broken links
2023-03-22 22:59:54 +01:00
Leo R. Lundgren
0666fa11b8 doc: Correct broken links 2023-03-22 17:37:58 +01:00
rawtaz
caa0e89114 Merge pull request #4246 from greatroar/doc
doc: Fix links, focus less on implementation details
2023-03-22 14:47:31 +01:00
rawtaz
46a7072f3f Merge pull request #4264 from GuitarBilly/patch-1
Update 110_talks.rst - Add changelog podcast.
2023-03-22 14:13:40 +01:00
GuitarBilly
1f12915b0c Update 110_talks.rst
Update 110_talks.rst

align date to 2nd of April as agreed.
( match podcast publication date )
2023-03-22 13:54:27 +01:00
greatroar
a0885d5d69 fuse: Mix inode hashes in a non-symmetric way
Since 0.15 (#4020), inodes are generated as hashes of names, xor'd with
the parent inode. That means that the inode of a/b/b is

	h(a/b/b) = h(a) ^ h(b) ^ h(b) = h(a).

I.e., the grandchild has the same inode as the grandparent. GNU find
trips over this because it thinks it has encountered a loop in the
filesystem, and fails to search a/b/b. This happens more generally when
the same name occurs an even number of times.

Fix this by multiplying the parent by a large prime, so the combining
operation is not longer symmetric in its arguments. This is what the FNV
hash does, which we used prior to 0.15. The hash is now

	h(a/b/b) = h(b) ^ p*(h(b) ^ p*h(a))

Note that we already ensure that h(x) is never zero.

Collisions can still occur, but they should be much less likely to occur
within a single path.

Fixes #4253.
2023-03-21 17:33:18 +01:00
dependabot[bot]
5dccab701a build(deps): bump actions/setup-go from 3 to 4
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 3 to 4.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-03-20 02:18:58 +00:00
Torben Giesselmann
5069c9edd9 Represent -1 as "all" in ExpirePolicy's Stringer 2023-03-15 15:07:51 -07:00
Torben Giesselmann
1a584cb16e Refactor policy sum calculation & duration parsing
- Convert policy sum calculation to function and move to tests.
- Remove parseDuration(...) and use ParseDurationOrPanic(...) instead.
2023-03-14 19:29:08 -07:00
Torben Giesselmann
84ede6ad7a forget: Prevent neg. values in --keep-within* opts 2023-03-14 19:20:03 -07:00
Torben Giesselmann
b7f03d01b8 Add helper function for Duration parsing
Tests in cmd_forget_test.go need the same convenience function
that was implemented in snapshot_policy_test.go.

Function parseDuration(...) was moved to testing.go  and renamed to
ParseDurationOrPanic(...).
2023-03-14 19:16:24 -07:00
dependabot[bot]
eaceaca113 build(deps): bump golang.org/x/oauth2 from 0.5.0 to 0.6.0
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.5.0 to 0.6.0.
- [Release notes](https://github.com/golang/oauth2/releases)
- [Commits](https://github.com/golang/oauth2/compare/v0.5.0...v0.6.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-03-13 02:07:39 +00:00
greatroar
a7ac9a4769 doc: Fix some links, focus less on implementation details
The godoc for filepath.Match has the syntax, which is what is important
for writing patterns. Use pkg.go.dev instead of golang.org/pkg.

For #4245. Not all links fixed yet.
2023-03-13 00:23:15 +01:00
Michael Eischer
23ed03a267 Merge pull request #4243 from restic/update-go
Run tests with Go 1.20
2023-03-12 18:15:54 +01:00
Alexander Neumann
aac8c5a7ca Merge pull request #4244 from MichaelEischer/fix-doc-typos
Fix documentation typos and build with Sphinx 6
2023-03-12 15:04:26 +01:00
Michael Eischer
7c8a401d97 Merge pull request #4176 from LXGaming/fix/impostor
Fix scan_finished JSON MessageType
2023-03-12 12:14:40 +01:00
Alexander Neumann
d83332315c Update golangci-lint 2023-03-12 12:05:59 +01:00
Alexander Neumann
c2703e5024 Increase timeout for golangci-lint 2023-03-12 12:02:22 +01:00
Michael Eischer
eb7dbc88b5 Merge pull request #4210 from aawsome/patch-1
docs: Corrections/extra information in design.rst
2023-03-12 12:00:25 +01:00
Michael Eischer
e02aadf1d2 doc: use extlink for links to issues 2023-03-12 11:55:36 +01:00
Michael Eischer
c4e6b198ae doc: fix building with Sphinx 6 2023-03-12 11:45:26 +01:00
Michael Eischer
f47c8eebb7 doc: fix broken link in rewrite command docs 2023-03-12 11:44:56 +01:00
Alexander Neumann
996e2ac7c5 Run tests with Go 1.20 2023-03-12 11:41:22 +01:00
Michael Eischer
a67d3781a3 doc: apply review comments and improve link formatting 2023-03-12 11:37:37 +01:00
Michael Eischer
153a73ebba Merge pull request #4242 from MichaelEischer/fix-mixed-index-in-design-docs
doc: Remove mixed pack files from index example
2023-03-12 11:24:21 +01:00
Michael Eischer
f9d6e3a035 Merge pull request #4236 from restic/dependabot/go_modules/golang.org/x/crypto-0.7.0
build(deps): bump golang.org/x/crypto from 0.5.0 to 0.7.0
2023-03-11 20:42:38 +01:00
dependabot[bot]
51656e8764 build(deps): bump golang.org/x/crypto from 0.5.0 to 0.7.0
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.5.0 to 0.7.0.
- [Release notes](https://github.com/golang/crypto/releases)
- [Commits](https://github.com/golang/crypto/compare/v0.5.0...v0.7.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-03-11 19:17:59 +00:00
Michael Eischer
9e23200bff Merge pull request #4219 from ianmuge/feat_upgrade_minio_for_new_region
(feat) Upgrade Minio to 7.0.49
2023-03-11 20:17:03 +01:00
Michael Eischer
b884643b40 Merge pull request #4224 from restic/dependabot/go_modules/github.com/klauspost/compress-1.16.0
build(deps): bump github.com/klauspost/compress from 1.15.15 to 1.16.0
2023-03-11 20:14:55 +01:00
Ian Muge
5be4845710 added changelog 2023-03-11 20:04:12 +01:00
Ian Muge
cfa3c6abc5 upgrade Minio to 7.0.49 2023-03-11 20:04:12 +01:00
Michael Eischer
f499e66032 Merge pull request #4235 from restic/dependabot/go_modules/google.golang.org/api-0.111.0
build(deps): bump google.golang.org/api from 0.109.0 to 0.111.0
2023-03-11 20:02:03 +01:00
Michael Eischer
00575ecffe doc: Remove mixed pack files from index example
The example given for the format of an index shows a mixed pack file.
Mixing tree and data blobs has been deprecated for a long time. Thus,
change the pack to only contain "data" blobs.
2023-03-11 19:10:31 +01:00
greatroar
cb5694d136 fuse: Report fuse.Attr.Blocks correctly
Fixes #4239.
2023-03-07 22:14:07 +01:00
dependabot[bot]
100b06d806 build(deps): bump google.golang.org/api from 0.109.0 to 0.111.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.109.0 to 0.111.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.109.0...v0.111.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-03-06 02:11:38 +00:00
Torben Giesselmann
667536cea4 forget: Allow neg. values in "--keep-within*" opts 2023-03-05 14:18:08 -08:00
Torben Giesselmann
ba183c44c3 forget: Add test with regular and "forever" opts 2023-03-05 13:56:16 -08:00
Torben Giesselmann
32e6a438be forget: Add test for "keep all hourly snapshots" 2023-03-04 22:13:34 -08:00
Torben Giesselmann
b77b0749fa forget: Treat -1 as forever for all "last n" opts 2023-03-04 19:01:37 -08:00
Torben Giesselmann
6aca7dac21 forget: Verify forget opts 2023-03-03 19:10:11 -08:00
Torben Giesselmann
8161605f1b snapshot_group: Fix typo 2023-03-03 19:07:57 -08:00
dependabot[bot]
b78607c9d8 build(deps): bump github.com/klauspost/compress from 1.15.15 to 1.16.0
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.15.15 to 1.16.0.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml)
- [Commits](https://github.com/klauspost/compress/compare/v1.15.15...v1.16.0)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-27 02:52:10 +00:00
Niklas Wagner
c1101ede19 Add jq to container image 2023-02-21 14:01:43 +01:00
Michael Eischer
f646406822 Merge pull request #4182 from BenWiederhake/dev-with-cache-description
check: Document behavior of --with-cache a bit better
2023-02-19 19:41:38 +01:00
Ben Wiederhake
9888443f5c check: Document behavior of --with-cache a bit better
Also see https://forum.restic.net/t/how-to-speed-up-tiny-incremental-checks/5905
2023-02-19 17:45:47 +01:00
Michael Eischer
cf6dfd6d36 Merge pull request #4212 from greatroar/snapshotfilter
cmd, restic: Refactor and fix snapshot filtering
2023-02-19 15:20:18 +01:00
greatroar
a7786c67f1 cmd, restic: Refactor and fix snapshot filtering
This turns snapshotFilterOptions from cmd into a restic.SnapshotFilter
type and makes restic.FindFilteredSnapshot and FindFilteredSnapshots
methods on that type. This fixes #4211 by ensuring that hosts and paths
are named struct fields instead of unnamed function arguments in long
lists of such.

Timestamp limits are also included in the new type. To avoid too much
pointer handling, the convention is that time zero means no limit.
That's January 1st, year 1, 00:00 UTC, which is so unlikely a date that
we can sacrifice it for simpler code.
2023-02-19 15:04:25 +01:00
Michael Eischer
15b7d9c80b Merge pull request #4213 from restic/dependabot/go_modules/golang.org/x/net-0.7.0
build(deps): bump golang.org/x/net from 0.6.0 to 0.7.0
2023-02-19 12:42:01 +01:00
dependabot[bot]
ee4128281e build(deps): bump golang.org/x/net from 0.6.0 to 0.7.0
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.6.0 to 0.7.0.
- [Release notes](https://github.com/golang/net/releases)
- [Commits](https://github.com/golang/net/compare/v0.6.0...v0.7.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-19 11:32:55 +00:00
Michael Eischer
3c8aefa0cb Merge pull request #4205 from restic/dependabot/go_modules/golang.org/x/oauth2-0.5.0
build(deps): bump golang.org/x/oauth2 from 0.4.0 to 0.5.0
2023-02-19 12:31:43 +01:00
aawsome
e2df73b0ac Update doc/design.rst
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2023-02-17 10:07:53 +01:00
aawsome
f4329a20f6 docs: Corrections/extra information in design.rst 2023-02-16 22:41:30 +01:00
dependabot[bot]
11ebc0c5db build(deps): bump golang.org/x/oauth2 from 0.4.0 to 0.5.0
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.4.0 to 0.5.0.
- [Release notes](https://github.com/golang/oauth2/releases)
- [Commits](https://github.com/golang/oauth2/compare/v0.4.0...v0.5.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-13 02:54:09 +00:00
Michael Eischer
f137be42fe Merge pull request #4081 from MichaelEischer/backup-group-by
Add support for `--group-by` to the backup command
2023-02-11 11:38:13 +01:00
Michael Eischer
0ce182f044 document backup --group-by 2023-02-11 00:55:11 +01:00
Alex Thomson
1b50faf03e Add changelog 2023-02-10 23:45:20 +01:00
Alex Thomson
4cbbf5d952 Fix scan_finished JSON MessageType
Corrected the scan_finished JSON MessageType from status to verbose_status
2023-02-10 23:45:20 +01:00
Michael Eischer
2885db7902 backup: add group-by option 2023-02-10 23:18:14 +01:00
Michael Eischer
acb40d2b94 Refactor group-by to parse options into a struct 2023-02-10 23:18:14 +01:00
Michael Eischer
fa73b50b45 Merge pull request #4198 from MichaelEischer/better-verbose-description
Make help for --verbose less confusing
2023-02-09 22:28:18 +01:00
Michael Eischer
2d700c3887 Make help for --verbose less confusing
The output is now
```
-v, --verbose                    be verbose (specify multiple times or a level using --verbose=n, max level/times is 2)
```
instead of
```
-v, --verbose n                  be verbose (specify multiple times or a level using --verbose=n, max level/times is 2)
```
2023-02-06 22:11:21 +01:00
Michael Eischer
91251f2d57 Merge pull request #4196 from restic/dependabot/github_actions/docker/build-push-action-4
build(deps): bump docker/build-push-action from 3 to 4
2023-02-06 21:54:06 +01:00
Michael Eischer
3df4ec7c61 Merge pull request #4195 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/azcore-1.3.1
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.3.0 to 1.3.1
2023-02-06 21:20:33 +01:00
Michael Eischer
b1d3a1a5e3 Merge pull request #4194 from restic/dependabot/go_modules/google.golang.org/api-0.109.0
build(deps): bump google.golang.org/api from 0.108.0 to 0.109.0
2023-02-06 21:17:57 +01:00
dependabot[bot]
4f31c2699d build(deps): bump docker/build-push-action from 3 to 4
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 3 to 4.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-06 02:13:17 +00:00
dependabot[bot]
96b1ff5e38 build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore
Bumps [github.com/Azure/azure-sdk-for-go/sdk/azcore](https://github.com/Azure/azure-sdk-for-go) from 1.3.0 to 1.3.1.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.3.0...sdk/azcore/v1.3.1)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azcore
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-06 02:07:34 +00:00
dependabot[bot]
6b5ffce9dc build(deps): bump google.golang.org/api from 0.108.0 to 0.109.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.108.0 to 0.109.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.108.0...v0.109.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-06 02:06:57 +00:00
Michael Eischer
590eb9efd7 Merge pull request #4180 from MichaelEischer/add-riscv
Add linux/riscv64 builds
2023-01-30 22:57:39 +01:00
Michael Eischer
55c4ca66f7 add linux/riscv64 builds 2023-01-30 22:24:12 +01:00
Michael Eischer
56ad761b19 Merge pull request #4177 from greatroar/loadunpacked
repository: Reuse buffers in Repository.LoadUnpacked
2023-01-30 22:12:13 +01:00
greatroar
d129baba7a repository: Reuse buffers in Repository.LoadUnpacked
This method had a buffer argument, but that was nil at all call sites.
That's removed, and instead LoadUnpacked now reuses whatever it
allocates inside its retry loop.
2023-01-30 22:01:01 +01:00
Alexander Neumann
febb32b5b4 Set development version for 0.15.1 2023-01-30 20:43:52 +01:00
Alexander Neumann
7d4b7ad9cb Add version for 0.15.1 2023-01-30 20:43:47 +01:00
Alexander Neumann
a883bb6596 Generate CHANGELOG.md for 0.15.1 2023-01-30 20:43:15 +01:00
Alexander Neumann
91acef90b2 Prepare changelog for 0.15.1 2023-01-30 20:43:15 +01:00
Michael Eischer
b2b7727b31 Merge pull request #4178 from rawtaz/changelogs
Polish changelogs
2023-01-27 22:46:46 +01:00
Leo R. Lundgren
0e4c9a5421 Polish changelogs 2023-01-27 22:27:57 +01:00
Michael Eischer
49fa8fe6dd Merge pull request #4175 from fergus-dall/deadlock-fix
Fix deadlock in Lock.Stale
2023-01-26 22:45:13 +01:00
Michael Eischer
12f167ee79 Merge pull request #4167 from aneesh-n/progress-percent-eta-fix
ui/backup: Fix percent and eta in backup progress
2023-01-26 22:42:10 +01:00
Michael Eischer
bb018fbc3e Merge pull request #4163 from MichaelEischer/fix-windows-self-upgrade
self-upgrade: Fix handling of `--output` on windows
2023-01-26 22:40:36 +01:00
Michael Eischer
3b24e0ac55 Merge pull request #4154 from MichaelEischer/cleanup-prune-tests
deduplicate prune tests a bit
2023-01-26 22:39:15 +01:00
Fergus Dall
04da31af2b Fix deadlock in Lock.Stale
With debug logging enabled this method would take a lock and then
format the lock as a string. Since PR #4022 landed the string
formatting method has also taken the lock, so this deadlocks.

Instead just record the lock ID, as is done elsewhere.
2023-01-27 03:21:37 +11:00
Alexander Neumann
65923e9c26 Merge pull request #4152 from MichaelEischer/invalid-locks
Improve handling of invalid locks
2023-01-25 08:19:53 +01:00
Michael Eischer
b903081804 Merge pull request #4169 from restic/dependabot/go_modules/github.com/klauspost/compress-1.15.15
build(deps): bump github.com/klauspost/compress from 1.15.14 to 1.15.15
2023-01-23 22:35:52 +01:00
Michael Eischer
beb1e872cc Merge pull request #4170 from restic/dependabot/go_modules/cloud.google.com/go/storage-1.29.0
build(deps): bump cloud.google.com/go/storage from 1.28.1 to 1.29.0
2023-01-23 22:25:09 +01:00
Michael Eischer
db350c0430 Merge pull request #4171 from restic/dependabot/go_modules/google.golang.org/api-0.108.0
build(deps): bump google.golang.org/api from 0.107.0 to 0.108.0
2023-01-23 22:23:51 +01:00
dependabot[bot]
716a5dd20d build(deps): bump google.golang.org/api from 0.107.0 to 0.108.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.107.0 to 0.108.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.107.0...v0.108.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-23 02:23:51 +00:00
dependabot[bot]
dbd07ade98 build(deps): bump cloud.google.com/go/storage from 1.28.1 to 1.29.0
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.28.1 to 1.29.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/storage/v1.28.1...spanner/v1.29.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-23 02:23:33 +00:00
dependabot[bot]
7adf1e5d37 build(deps): bump github.com/klauspost/compress from 1.15.14 to 1.15.15
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.15.14 to 1.15.15.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml)
- [Commits](https://github.com/klauspost/compress/compare/v1.15.14...v1.15.15)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-23 02:23:15 +00:00
Michael Eischer
8f94eb5420 add changelog for less strict lock handling 2023-01-22 15:54:07 +01:00
Michael Eischer
8aaba83719 add changelog for self-upgrade bug on windows 2023-01-22 15:39:42 +01:00
Michael Eischer
e16a6d4c50 self-update: add basic test for extractToFile 2023-01-22 15:39:42 +01:00
Michael Eischer
34e67e3510 self-update: Fix handling of --output on windows
The code always assumed that the upgrade happens in place. Thus writing
the upgrade to a separate file fails, when trying to remove the file
stored at that location.
2023-01-22 15:39:42 +01:00
Michael Eischer
c527c05590 add changelog for missing backup ETA 2023-01-22 15:32:55 +01:00
Aneesh Nireshwalia
ed23edeb62 ui/backup: Fix percent and eta in backup progress
Added missing call to scanFinished=true.
This was causing the percent and eta to never get
printed for backup progress even after the scan was finished.
2023-01-21 13:25:08 -07:00
Michael Eischer
0f398b82e3 Merge pull request #4145 from greatroar/index-encode
index: Optimize generatePackList
2023-01-21 01:01:12 +01:00
greatroar
99755c634b index: Optimize generatePackList
name                 old time/op    new time/op    delta
EncodeIndex/100-8      1.56ms ± 2%    1.48ms ± 3%   -5.37%  (p=0.000 n=10+10)
EncodeIndex/1000-8     14.5ms ± 2%    13.1ms ± 2%   -9.49%  (p=0.000 n=9+10)
EncodeIndex/10000-8     120ms ± 2%     116ms ± 2%   -3.58%  (p=0.000 n=10+10)

name                 old alloc/op   new alloc/op   delta
EncodeIndex/100-8       306kB ± 1%     275kB ± 1%  -10.28%  (p=0.000 n=10+10)
EncodeIndex/1000-8     3.69MB ±11%    2.88MB ± 5%  -22.07%  (p=0.000 n=10+9)
EncodeIndex/10000-8    35.9MB ±11%    31.9MB ±10%  -11.13%  (p=0.005 n=10+10)

name                 old allocs/op  new allocs/op  delta
EncodeIndex/100-8       3.39k ± 0%     2.39k ± 0%  -29.61%  (p=0.000 n=10+10)
EncodeIndex/1000-8      32.6k ± 0%     22.9k ± 0%  -29.63%  (p=0.000 n=10+9)
EncodeIndex/10000-8      326k ± 0%      229k ± 0%  -29.71%  (p=0.000 n=10+10)

The bulk of the allocation rate improvement comes from just removing the
debug.Log calls: every one of those copied a restic.ID to the heap.
2023-01-14 20:41:07 +01:00
Michael Eischer
f5f13f6648 Merge pull request #4153 from MichaelEischer/fix-quiet-prune
prune: Don't show "packs processed" for quiet runs
2023-01-14 20:11:34 +01:00
Michael Eischer
00216d54a1 deduplicate prune tests a bit 2023-01-14 19:05:45 +01:00
Michael Eischer
1f3f68b2c0 prune: Don't show "packs processed" for quiet runs 2023-01-14 18:48:32 +01:00
Michael Eischer
57acc769b4 lock: Ignore empty lock files
These may be left behind by backends which do not guarantee atomic
uploads.
2023-01-14 18:15:46 +01:00
Michael Eischer
20ad14e362 lock: add help message how to recover from invalid locks 2023-01-14 18:04:22 +01:00
Michael Eischer
c995b5be52 lock: cleanup error message
The error message is now `Fatal: unable to create lock in backend:
[...]` instead of `unable to create lock in backend: Fatal: [...]`.
2023-01-14 17:57:02 +01:00
Michael Eischer
1adf28a2b5 repository: properly return invalid data error in LoadUnpacked
The retry backend does not return the original error, if its execution
is interrupted by canceling the context. Thus, we have to manually
ensure that the invalid data error gets returned.

Additionally, use the retry backend for some of the repository tests, as
this is the configuration which will be used by restic.
2023-01-14 17:57:02 +01:00
Michael Eischer
6d9675c323 repository: cleanup error message on invalid data
The retry printed the filename twice:
```
Load(<lock/04804cba82>, 0, 0) returned error, retrying after 720.254544ms: load(<lock/04804cba82>): invalid data returned
```
now the warning has changed to
```
Load(<lock/04804cba82>, 0, 0) returned error, retrying after 720.254544ms: invalid data returned
```
2023-01-14 17:57:02 +01:00
Michael Eischer
551b31ce3c Merge pull request #4146 from MichaelEischer/update-blazer
Update blazer to fix `b2_download_file_by_name: 404`
2023-01-14 15:23:32 +01:00
Michael Eischer
ec99507e4c update blazer
This removes a stray warning, that was printed when checking the status
of non-existent files.
2023-01-14 14:56:25 +01:00
Michael Eischer
5f97f534b1 Merge pull request #4150 from restic/dependabot/go_modules/google.golang.org/api-0.107.0
build(deps): bump google.golang.org/api from 0.106.0 to 0.107.0
2023-01-14 14:55:15 +01:00
Michael Eischer
ed11bbd0e2 Merge pull request #4151 from restic/dependabot/go_modules/github.com/minio/minio-go/v7-7.0.47
build(deps): bump github.com/minio/minio-go/v7 from 7.0.46 to 7.0.47
2023-01-14 14:50:01 +01:00
dependabot[bot]
5bb9cb056d build(deps): bump github.com/minio/minio-go/v7 from 7.0.46 to 7.0.47
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.46 to 7.0.47.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.46...v7.0.47)

---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-14 13:38:58 +00:00
dependabot[bot]
cd9bd22563 build(deps): bump google.golang.org/api from 0.106.0 to 0.107.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.106.0 to 0.107.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.106.0...v0.107.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-14 13:38:42 +00:00
Michael Eischer
ecc826ef7d Merge pull request #4149 from MichaelEischer/more-frequent-dependabot
CI: let dependabot check for updates weekly
2023-01-14 14:37:30 +01:00
Michael Eischer
fb43cbab49 Merge pull request #4129 from greatroar/cleanup
cache: Replace readCloser+LimitedReader by backend.LimitedReadCloser
2023-01-14 12:17:25 +01:00
Michael Eischer
41d31b1e27 Merge pull request #4116 from MichaelEischer/extract-progress-updater
ui/progress: Extract progress updater
2023-01-14 12:07:26 +01:00
Michael Eischer
f6ea5c5865 CI: let dependabot check for updates weekly 2023-01-14 12:02:42 +01:00
Michael Eischer
4a7a6b06af ui/backup: Use progress.Updater for progress updates 2023-01-14 01:20:43 +01:00
Michael Eischer
e499bbe3ae progress: extract progress updating into Updater struct
This allows reusing the code to create periodic progress updates.
2023-01-14 01:13:08 +01:00
Michael Eischer
52682b1c7b Merge pull request #4111 from MichaelEischer/extract-stdio-wrapper
backup: extract StdioWrapper from ProgressPrinters
2023-01-14 01:12:23 +01:00
Michael Eischer
c15b4bceae backup: extract StdioWrapper from ProgressPrinters
The StdioWrapper is not used at all by the ProgressPrinters. It is
called a bit earlier than previously. However, as the password prompt
directly accessed stdin/stdout this doesn't cause problems.
2023-01-14 00:58:13 +01:00
Alexander Neumann
74348be3fa Set development version for 0.15.0 2023-01-12 20:51:23 +01:00
Alexander Neumann
37d0e1fe58 Add version for 0.15.0 2023-01-12 20:51:19 +01:00
Alexander Neumann
da196aa43e Update manpages and auto-completion 2023-01-12 20:51:19 +01:00
Alexander Neumann
099774c2aa Generate CHANGELOG.md for 0.15.0 2023-01-12 20:50:45 +01:00
Alexander Neumann
cd2f53e3f9 Prepare changelog for 0.15.0 2023-01-12 20:50:44 +01:00
Alexander Neumann
0c5a55d1bd Merge pull request #4142 from restic/update-deps
Upgrade dependencies
2023-01-12 20:23:25 +01:00
Michael Eischer
9ddca65f6d update dependencies 2023-01-11 23:22:10 +01:00
Michael Eischer
06fee601bc Merge pull request #4141 from restic/doc-no-scan-correction
doc: Correct heading level for --no-scan
2023-01-11 22:01:05 +01:00
Leo R. Lundgren
1cb920cc57 doc: Correct heading level for --no-scan 2023-01-11 21:47:38 +01:00
Michael Eischer
8f53ffb921 Merge pull request #4140 from restic/doc-manual
doc: Update manual page with --no-scan
2023-01-11 21:42:42 +01:00
Michael Eischer
351cbb4f94 Merge pull request #4139 from restic/doc-no-scan
doc: Move and update documentation for --no-scan
2023-01-11 21:41:20 +01:00
Leo R. Lundgren
b8b5508d15 doc: Update manual page with --no-scan 2023-01-11 02:27:43 +01:00
Leo R. Lundgren
c5542ddcd2 doc: Move and update documentation for --no-scan 2023-01-11 00:09:24 +01:00
rawtaz
dffb8e0c14 Merge pull request #4138 from MichaelEischer/doc-sparse
doc: add description for restore --sparse
2023-01-10 23:39:51 +01:00
Michael Eischer
b151fa498a doc: add description for restore --sparse 2023-01-10 23:27:42 +01:00
Michael Eischer
c354b55e62 Merge pull request #4136 from restic/doc-small-files
doc: Clarify text about tuning backups for small files
2023-01-08 22:00:09 +01:00
Michael Eischer
375953a001 Merge pull request #4088 from MichaelEischer/doc-cifs-backup-source
doc: reading from CIFS can be a problem on linux
2023-01-08 21:56:48 +01:00
Leo R. Lundgren
6306797238 doc: Clarify text about tuning backups for small files 2023-01-08 21:49:55 +01:00
Michael Eischer
ef9164fcbb doc: reading from CIFS can be a problem on linux 2023-01-08 21:47:34 +01:00
rawtaz
e2bcfd68dd Merge pull request #4135 from restic/changelogs
Polish changelogs
2023-01-08 20:04:24 +01:00
Leo R. Lundgren
33fb351386 Polish changelogs 2023-01-08 19:48:51 +01:00
Michael Eischer
c9840da4f8 Merge pull request #4134 from MichaelEischer/fix-verbose-help-text
Correct maximum verbosity level in help message
2023-01-07 22:24:04 +01:00
Michael Eischer
732184a849 Correct maximum verbosity level in help message
The maximum for `--verbose=n` is n=2. Internally it is translated into a
scale from 0 to 3. However, the default (without verbose) is 1, thus the
verbosity level can only be increased two times.
2023-01-07 22:02:13 +01:00
Michael Eischer
24178c97e9 Merge pull request #4117 from MichaelEischer/prune-dry-run-help
prune: make it clearer when prune is used in dry-run mode
2023-01-04 23:18:53 +01:00
Michael Eischer
7a36306901 forget: Clarify log message for --dry-run --prune 2023-01-04 00:44:46 +01:00
Michael Eischer
b404ad4eaa prune: make it clearer when prune is used in dry-run mode 2023-01-04 00:44:46 +01:00
Michael Eischer
e02a10c58a Merge pull request #4109 from MichaelEischer/fix-prune-uncompressed-accounting
prune: Fix calculation of remaining uncompressed data
2023-01-03 23:28:10 +01:00
Michael Eischer
81dc8c8d13 prune: Fix calculation of remaining uncompressed data
Only the repacking of *un*compressed packs reduces the amount of
uncompressed data. Previously the counter even overflowed for fully
compressed repositories.
2023-01-03 22:34:36 +01:00
greatroar
72922a79ed cache: Replace readCloser+LimitedReader by backend.LimitedReadCloser 2023-01-03 19:03:36 +01:00
Michael Eischer
89a8006578 Merge pull request #4104 from philaris/fix_max_uint32_uid_gid_to_zero
in tar dump, convert uid, gid of value -1 to zero
2023-01-02 22:28:28 +01:00
Panagiotis Cheilaris
3b516d4b70 convert uid/gid -1 to 0 only in 32-bit tar dump
Only for a 32-bit build of restic, convert a uid or gid value of -1 to 0.
2022-12-30 18:12:12 +01:00
Michael Eischer
0de3b24756 Merge pull request #4110 from MichaelEischer/remove-exitf
Remove Exitf function
2022-12-29 12:07:51 +01:00
Michael Eischer
0fbff39ae8 Merge pull request #4108 from MichaelEischer/cleanup-check-output
Cleanup check output
2022-12-29 11:59:18 +01:00
Michael Eischer
68b1f30733 Remove Exitf function
Commands should use the normal shutdown path. In addition, the Exitf
function was only used by `dump` and `restore` but not any other command
which introduces the risk of inconsistent behavior.
2022-12-28 21:42:38 +01:00
Panagiotis Cheilaris
a86a56cf3b fix lint issue with function name 'tarId'
See https://github.com/golang/lint/issues/89 and
https://github.com/golang/lint/issues/124
2022-12-28 18:46:58 +01:00
Panagiotis Cheilaris
050ed616ae be more explicit with uid or gid of value -1 2022-12-28 18:44:36 +01:00
Michael Eischer
8430399fce check: Partially fix garbled output
When reporting an error for a tree, the output message can overlap with
the progress bar output, e.g. `error for tree e91ef6fb:napshots`.

The fix only applies for this specific message and does not work on
Windows.
2022-12-28 17:47:27 +01:00
Michael Eischer
aea96b7d86 check: Slightly improve help message
If a repository has both pack/index related warnings and errors, then
the help message is quite misleading. Reword it slightly to be more
clear.
2022-12-28 17:46:06 +01:00
Michael Eischer
bcae28afb4 Merge pull request #4100 from klemensn/tag-self-update
Reinstate selfupdate tag to make builds without self-update work
2022-12-28 15:49:11 +01:00
Panagiotis Cheilaris
10fa5cde0a in tar dump, convert uid, gid of value -1 to zero 2022-12-27 16:36:04 +01:00
Klemens Nanni
61e7386384 Bugfix: Make distribution package builds without self-update work 2022-12-26 21:52:24 +04:00
Klemens Nanni
94f6e7d4a6 Reinstate selfupdate tag to make builds without self-update work
Revert what seems to be a typo introduced as part of the fix for #2041
in 2018 7d0f2eaf24.

`xbuild` does not look like a go build/tag keyword to me, I failed to
find documentation for it and using `go install -tags '!selfupdate' ...`
has no effect, i.e. self-update code is still compiled.

`+build` however works;  updating the OpenBSD port/binary package
security/restic to apply this PR works as expected:

```
	$ restic help | grep self
	$ restic self-update
	unknown command "self-update" for "restic"
```

(Using `go:build` now as per restic's style and gofmt.)

Previously, using `restic-0.14.0p1` on OpenBSD/amd64 7.2-current would
check for a newer version and probably attempt replacing the system wide
root-owned executable (on a read-only filesystem) as unprivileged user:

```
	$ restic version
	restic 0.14.0 compiled with go1.19.2 on openbsd/amd64
	$ restic help | grep self
	  self-update   Update the restic binary
	$ restic self-update
	writing restic to /usr/local/bin/restic
	find latest release of restic at GitHub
	restic is up to date
```

(It never tried to actually write besaid path;  doing so would fail, so
the current message can be considered misleading.)
2022-12-26 21:46:22 +04:00
Michael Eischer
90fb6f70b4 Merge pull request #4089 from greatroar/errors
Clean up error handling further
2022-12-24 10:41:56 +01:00
Michael Eischer
29b8500254 Merge pull request #4090 from restic/upgrade-dependencies
Upgrade dependencies
2022-12-23 22:34:39 +01:00
Michael Eischer
705cabb304 Merge pull request #3981 from MichaelEischer/prune-uncompressed-stats
prune: report how much data must be repacked to compressed the repo
2022-12-23 22:34:04 +01:00
Michael Eischer
a6f3ae5790 Merge pull request #4094 from googol42/master
remove duplicated init
2022-12-23 22:33:01 +01:00
Andreas Dominik Preikschat
ea37240597 remove duplicated init
the documentation contained the `init` command twice
2022-12-20 17:24:56 +01:00
Michael Eischer
bd2f6aaac3 azure: downgrade azblob dependency due to build breakages on Solaris 2022-12-17 23:35:07 +01:00
Michael Eischer
583372956b Upgrade dependencies
Nothing special has changed.
2022-12-17 15:23:11 +01:00
greatroar
1678392a6d checker: Make ErrLegacyLayout a value, not a type 2022-12-17 09:41:07 +01:00
greatroar
d9002f050e backend: Don't Wrap errors from url.Parse
The messages from url.Error.Error already start with the word "parse".
2022-12-17 09:41:07 +01:00
greatroar
b150dd0235 all: Replace some errors.Wrap calls by errors.WithStack
Mostly changed the ones that repeat the name of a system call, which is
already contained in os.PathError.Op. internal/fs.Reader had to be
changed to actually return such errors.
2022-12-17 09:41:07 +01:00
Michael Eischer
cccc17e4e9 Merge pull request #4086 from blackpiglet/modify_access_denied_code
Fix: change error code in function isAccessDenied to AccessDenied
2022-12-16 21:55:52 +01:00
Michael Eischer
2723159ed4 Merge pull request #3931 from kjetilho/feature/optional_scanner
add --no-scan to backup command
2022-12-16 21:42:22 +01:00
Xun Jiang/Bruce Jiang
ecc62c8be2 Update changelog/unreleased/issue-4085
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
Signed-off-by: Xun Jiang <blackpiglet@gmail.com>
2022-12-16 21:41:16 +01:00
Xun Jiang
cc5325d22b Fix: change error code in function isAccessDenied to AccessDenied
Signed-off-by: Xun Jiang <blackpiglet@gmail.com>
2022-12-16 21:41:16 +01:00
Michael Eischer
da0e45cf40 Merge pull request #4083 from greatroar/cleanup
repository: Remove empty cleanup functions in tests
2022-12-16 21:39:30 +01:00
Kjetil Torgrim Homme
14aa6f2a00 add --disable-scanner to backup command
The scanner process has only cosmetic effect for the progress printer,
and can be disabled without impacting functionality when the user does
not need an estimate of completion.

In many cases the scanner process can provide beneficial priming of
the file system cache, so as general advice it should not be disabled.
However, tests have shown that backup of NFS and fuse based filesystems,
where stat(2) is relatively expensive, can be significantly faster
without the scanner.
2022-12-16 21:29:59 +01:00
Michael Eischer
7bdb985dde Merge pull request #4079 from MichaelEischer/rewrite-set-original
rewrite: Always set the Original field in a rewritten snapshot
2022-12-13 22:56:20 +01:00
Michael Eischer
1bfe98bdc0 Merge pull request #2398 from DanielG/b2-hide-file
b2: Fallback to b2_hide_file when delete returns unauthorized
2022-12-13 22:52:23 +01:00
Michael Eischer
1c071a462e Merge pull request #4084 from ekarlso/azure-stat-fix
fix: Make create not error out when ContainerNotFound
2022-12-13 22:49:05 +01:00
Michael Eischer
25d22d5241 Merge pull request #4082 from MichaelEischer/unbuffered-logger-for-testing
Don't buffer the golang `log` package output when running tests
2022-12-13 22:45:50 +01:00
Endre Karlson
7dd33c0ecc azure: Make create not error out when ContainerNotFound 2022-12-11 22:57:23 +01:00
greatroar
c0b5ec55ab repository: Remove empty cleanup functions in tests
TestRepository and its variants always returned no-op cleanup functions.
If they ever do need to do cleanup, using testing.T.Cleanup is easier
than passing these functions around.
2022-12-11 11:06:25 +01:00
Michael Eischer
2e3d4640be Don't buffer the golang log output when running tests 2022-12-10 16:08:27 +01:00
Michael Eischer
38b2e9b42c rewrite: Always set the Original field in a rewritten snapshot
The Original field is meant to remember the original snapshot id if e.g.
changing its tags. It was only set by the `rewrite` command if it was
not set previously. However, a rewritten snapshot is potentially rather
different from the original snapshot. Thus just always set the Original
field. This also makes it easier to later on detect and potentially
remove the original snapshots.
2022-12-10 12:47:00 +01:00
Michael Eischer
049a105ba5 Merge pull request #4077 from greatroar/cleanup
test: Use testing.T.Cleanup to remove tempdirs
2022-12-09 22:17:46 +01:00
Michael Eischer
4b98b5562d Merge pull request #4075 from greatroar/sftp-enospc
sftp: Fix ENOSPC check
2022-12-09 22:00:13 +01:00
greatroar
f90bf84ba7 test: Use testing.T.Cleanup to remove tempdirs 2022-12-09 14:23:55 +01:00
greatroar
83d23b3ae8 Changelog for ENOSPC handling bug 2022-12-09 08:50:30 +01:00
Michael Eischer
eae7366563 Merge pull request #4028 from ekarlso/use-az-blob-sdk
Switch to azblob sdk
2022-12-07 21:58:03 +01:00
Endre Karlson
25648e2501 azure: Switch to azblob sdk 2022-12-07 21:46:07 +01:00
greatroar
62520bb7b4 sftp: Fix ENOSPC check
We now check for space that is not reserved for the root user on the
remote, and the check is no longer in a defer block because it wouldn't
fire. Some change in the surrounding code may have led the deferred
function to capture the wrong err variable.

Fixes #3336.
2022-12-07 21:06:46 +01:00
rawtaz
4ba31df08f Merge pull request #4074 from greatroar/lobaro-docker
doc: Remove ref to Lobaro's Docker image
2022-12-04 18:00:08 +01:00
greatroar
5efcbe143c doc: Remove ref to Lobaro's Docker image
It hasn't been updated for a while and has restic 0.12.0. Fixes #4002.
2022-12-04 16:20:42 +01:00
Michael Eischer
0df585dd99 Merge pull request #4066 from sedlund/fix#4033
fix#4033 cmd: copy no longer lists skipped existing snapshots by default
2022-12-03 19:22:11 +01:00
Michael Eischer
223da7344e Merge pull request #4070 from restic/fix-cloud-tests
Fix cloud tests
2022-12-03 19:21:25 +01:00
Michael Eischer
2b67862420 backend/test: check that IsNotExist actually works 2022-12-03 18:56:55 +01:00
Michael Eischer
2f934f5803 gs: check against the correct error in IsNotExist 2022-12-03 18:49:54 +01:00
Michael Eischer
04d101fa94 gs/s3: remove useless os.IsNotExist check 2022-12-03 18:49:54 +01:00
Michael Eischer
579cd6dc64 azure: fix totally broken IsNotExist 2022-12-03 18:49:54 +01:00
Michael Eischer
3ebdadc58f Merge pull request #4069 from greatroar/cleanup
cache, prune, restic: Cleanup
2022-12-03 17:50:52 +01:00
Michael Eischer
bc8b2455b9 Merge pull request #4064 from MichaelEischer/flaky-abort-early-on-error
archiver: Fix flaky TestArchiverAbortEarlyOnError
2022-12-03 17:43:44 +01:00
Michael Eischer
60c6a09324 Merge pull request #4065 from MichaelEischer/flaky-rclone-failed-start
rclone: treat "file already closed" as command startup error
2022-12-03 17:42:56 +01:00
Michael Eischer
8bf6b2b80d Merge pull request #4067 from MichaelEischer/remove-backend-test-method
Remove `Test()` method from Backend
2022-12-03 17:40:55 +01:00
Michael Eischer
78ea69082a Merge pull request #4068 from MichaelEischer/debug-lock-refresh-test
Add more debug logging to `TestLockSuccessfulRefresh`
2022-12-03 17:38:21 +01:00
Scott Edlund
cbe73ace3f Update changelog/unreleased/issue-4033
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2022-12-03 20:07:37 +08:00
greatroar
63bed34608 restic: Clean up restic.IDs type
IDs.Less can be rewritten as

	string(list[i][:]) < string(list[j][:])

Note that this does not copy the ID's.

The Uniq method was no longer used.

The String method has been reimplemented without first copying into a
separate slice of a custom type.
2022-12-03 12:38:20 +01:00
greatroar
0c749dd358 prune: Pass fewer options around 2022-12-03 12:14:04 +01:00
greatroar
d45a2475e1 cache: Rewrite unnecessary if-else 2022-12-03 12:13:54 +01:00
Michael Eischer
6b5d6b9f2c Add more debug logging to TestLockSuccessfulRefresh
The test fails from time to time. Add some more logging to hopefully get
an idea where things go wrong.
2022-12-03 12:05:38 +01:00
Michael Eischer
648edeca40 retry: Do not retry Stat() if file does not exist
In non test/debug code, Stat() is used exclusively to check whether a
file exists. Thus, do not retry if a file is reported as not existing.
2022-12-03 11:42:48 +01:00
Michael Eischer
40ac678252 backend: remove Test method
The Test method was only used in exactly one place, namely when trying
to create a new repository it was used to check whether a config file
already exists.

Use a combination of Stat() and IsNotExist() instead.
2022-12-03 11:28:10 +01:00
sedlund
06ee0339aa fix#4033 cmd: copy no longer lists skipped existing snapshots by default 2022-12-03 09:55:39 +08:00
Michael Eischer
57d8eedb88 Merge pull request #4020 from greatroar/fuse-inode
fuse: Better inode generation
2022-12-02 22:28:15 +01:00
Michael Eischer
ca1803cacb Merge pull request #4063 from MichaelEischer/replace-ioutil-usage
Replace ioutil usage
2022-12-02 21:49:40 +01:00
Michael Eischer
0af89a5738 Merge pull request #3132 from metalsp0rk/init-json
Init command JSON output
2022-12-02 21:49:22 +01:00
Michael Eischer
364a396fd6 init: use standard name message_type to distinguish JSON messages 2022-12-02 21:33:03 +01:00
Michael Eischer
9a9f559806 init: cleanup json print code 2022-12-02 21:33:03 +01:00
Kyle Brennan
933c9af328 create changelog entry for issue-3124 and pull-3132 2022-12-02 21:32:30 +01:00
Kyle Brennan
a6ae79b39e support json output for init command 2022-12-02 21:32:30 +01:00
Michael Eischer
f3d964a8c1 rclone: treat "file already closed" as command startup error
Since #3940 the rclone backend returns the commands exit code if it
fails to start. The list of expected errors was missing the "file
already closed"-error which can occur if the http test request first
learns about the closed pipe to rclone before noticing the canceled
context.

Go internally makes sure that a file descriptor is unusable once it was
closed, thus this cannot have unintended side effects (like accidentally
reading from the wrong file due to a reused file descriptor).
2022-12-02 20:46:02 +01:00
Michael Eischer
a9972dbe7d archiver: Fix flaky TestArchiverAbortEarlyOnError
Saving the blobs of a file by now happens asynchronously to the
processing in the FileSaver. Thus we have to account for the blobs
queued for saving.
2022-12-02 20:07:34 +01:00
Michael Eischer
f755233210 Replace usages of ioutil.ReadDir
This changes the return type to []fs.DirEntry. However, as we only use
the filenames anyways, this doesn't make a difference.
2022-12-02 19:54:27 +01:00
Michael Eischer
fa20a78bb6 Merge pull request #4056 from greatroar/cleanup
backend, fs, options: Minor cleanup
2022-12-02 19:44:54 +01:00
Michael Eischer
ff7ef5007e Replace most usages of ioutil with the underlying function
The ioutil functions are deprecated since Go 1.17 and only wrap another
library function. Thus directly call the underlying function.

This commit only mechanically replaces the function calls.
2022-12-02 19:36:43 +01:00
greatroar
65612d797c backend, options: Prefer strings.Cut to SplitN
Also realigned the various "split into host:bucket:prefix"
implementations.
2022-12-02 19:19:14 +01:00
Michael Eischer
2d5e28e777 Merge pull request #4059 from restic/dependabot/go_modules/github.com/minio/minio-go/v7-7.0.45
build(deps): bump github.com/minio/minio-go/v7 from 7.0.44 to 7.0.45
2022-12-01 21:20:57 +01:00
dependabot[bot]
4fefa2ade2 build(deps): bump github.com/minio/minio-go/v7 from 7.0.44 to 7.0.45
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.44 to 7.0.45.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.44...v7.0.45)

---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-12-01 01:03:42 +00:00
Michael Eischer
3c5d1eabe9 Merge pull request #4051 from restic/dependabot/go_modules/github.com/klauspost/compress-1.15.12
build(deps): bump github.com/klauspost/compress from 1.15.9 to 1.15.12
2022-11-28 21:47:55 +01:00
Michael Eischer
bec391ee26 Merge pull request #4053 from greatroar/xattr
Upgrade pkg/xattr to version with Solaris FIFO fix
2022-11-28 21:06:19 +01:00
greatroar
daafcaf380 Upgrade pkg/xattr to version with Solaris FIFO fix
This version doesn't have a release tag yet, but it's 0.4.9 + one patch.

Fixes #4003.
2022-11-28 20:43:51 +01:00
Alexander Neumann
1d7e7fcd6b Merge pull request #4049 from MichaelEischer/fix-rewrite-docs
rewrite: fix link anchors in documentation
2022-11-28 19:39:38 +01:00
dependabot[bot]
57d59c71e3 build(deps): bump github.com/klauspost/compress from 1.15.9 to 1.15.12
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.15.9 to 1.15.12.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml)
- [Commits](https://github.com/klauspost/compress/compare/v1.15.9...v1.15.12)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-28 18:39:35 +00:00
Alexander Neumann
bb83c78ee5 Merge pull request #4047 from MichaelEischer/clean-ci-configuration
Cleanup CI configuration
2022-11-28 19:38:33 +01:00
greatroar
60aa87bbab fs: Remove explicit type check in extendedStat
Without comma-ok, the runtime inserts the same check with a similar
enough panic message:

    interface conversion: interface {} is nil, not *syscall.Stat_t
2022-11-27 19:58:06 +01:00
Michael Eischer
34609bca0e Merge pull request #4050 from greatroar/lruv2
bloblru: Upgrade to hashicorp/golang-lru/v2
2022-11-27 17:37:14 +01:00
greatroar
e5d597fd22 bloblru: Upgrade to hashicorp/golang-lru/v2
The new genericized LRU cache no longer needs to have the IDs separately
allocated:

name   old time/op    new time/op    delta
Add-8     494ns ± 2%     388ns ± 2%  -21.46%  (p=0.000 n=10+9)

name   old alloc/op   new alloc/op   delta
Add-8      176B ± 0%      152B ± 0%  -13.64%  (p=0.000 n=10+10)

name   old allocs/op  new allocs/op  delta
Add-8      5.00 ± 0%      3.00 ± 0%  -40.00%  (p=0.000 n=10+10)
2022-11-27 17:18:13 +01:00
Michael Eischer
0eddc89e98 doc: design.rst: Fix highlighting for index snippet
JSON does not support comments. As JSON is a subset of Javascript, use
the latter instead.
2022-11-27 17:01:27 +01:00
Michael Eischer
41b0f1d43a doc: fix link to amazon s3 section 2022-11-27 17:01:22 +01:00
Michael Eischer
6a793db9ca rewrite: fix link anchors in documentation 2022-11-27 16:38:10 +01:00
Michael Eischer
05cebc1c4b Merge pull request #4044 from restic/dependabot/go_modules/cloud.google.com/go/storage-1.28.0
build(deps): bump cloud.google.com/go/storage from 1.25.0 to 1.28.0
2022-11-27 15:23:06 +01:00
Michael Eischer
ce39727846 Merge pull request #4036 from restic/dependabot/go_modules/github.com/pkg/profile-1.7.0
build(deps): bump github.com/pkg/profile from 1.6.0 to 1.7.0
2022-11-27 15:22:16 +01:00
Michael Eischer
9aa06ce959 CI: remove option to configure command used to install go tools
With the minimum required go version of 1.18, we always use `go
install`.
2022-11-27 15:07:29 +01:00
Michael Eischer
5968971313 CI: remove dependabot ignore for bazil.org/fuse
We've switched to a fork of the original library, thus the ignore is no
longer necessary.
2022-11-27 15:06:30 +01:00
dependabot[bot]
95374767de build(deps): bump github.com/pkg/profile from 1.6.0 to 1.7.0
Bumps [github.com/pkg/profile](https://github.com/pkg/profile) from 1.6.0 to 1.7.0.
- [Release notes](https://github.com/pkg/profile/releases)
- [Commits](https://github.com/pkg/profile/compare/v1.6.0...v1.7.0)

---
updated-dependencies:
- dependency-name: github.com/pkg/profile
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-27 14:02:32 +00:00
dependabot[bot]
c100a62ebf build(deps): bump cloud.google.com/go/storage from 1.25.0 to 1.28.0
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.25.0 to 1.28.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.25.0...spanner/v1.28.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-27 14:01:36 +00:00
Michael Eischer
c41a1b66e1 Merge pull request #4037 from restic/dependabot/go_modules/google.golang.org/api-0.103.0
build(deps): bump google.golang.org/api from 0.93.0 to 0.103.0
2022-11-27 15:00:16 +01:00
dependabot[bot]
705aed0ecb build(deps): bump google.golang.org/api from 0.93.0 to 0.103.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.93.0 to 0.103.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.93.0...v0.103.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-27 13:45:08 +00:00
Michael Eischer
28d6de648c Merge pull request #4040 from restic/dependabot/go_modules/github.com/spf13/cobra-1.6.1
build(deps): bump github.com/spf13/cobra from 1.5.0 to 1.6.1
2022-11-27 14:44:25 +01:00
Michael Eischer
bb40b55d1c Merge pull request #4038 from restic/dependabot/go_modules/github.com/cenkalti/backoff/v4-4.2.0
build(deps): bump github.com/cenkalti/backoff/v4 from 4.1.3 to 4.2.0
2022-11-27 14:13:39 +01:00
dependabot[bot]
a24c1e99a6 build(deps): bump github.com/cenkalti/backoff/v4 from 4.1.3 to 4.2.0
Bumps [github.com/cenkalti/backoff/v4](https://github.com/cenkalti/backoff) from 4.1.3 to 4.2.0.
- [Release notes](https://github.com/cenkalti/backoff/releases)
- [Commits](https://github.com/cenkalti/backoff/compare/v4.1.3...v4.2.0)

---
updated-dependencies:
- dependency-name: github.com/cenkalti/backoff/v4
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-27 12:57:41 +00:00
dependabot[bot]
fd56ead4a8 build(deps): bump github.com/spf13/cobra from 1.5.0 to 1.6.1
Bumps [github.com/spf13/cobra](https://github.com/spf13/cobra) from 1.5.0 to 1.6.1.
- [Release notes](https://github.com/spf13/cobra/releases)
- [Commits](https://github.com/spf13/cobra/compare/v1.5.0...v1.6.1)

---
updated-dependencies:
- dependency-name: github.com/spf13/cobra
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-27 12:57:23 +00:00
Michael Eischer
cc679c6494 Merge pull request #4041 from MichaelEischer/require-go-1.18
Require go 1.18
2022-11-27 13:56:43 +01:00
greatroar
c9c7671c58 fuse: Clean up inode generation 2022-11-27 13:53:42 +01:00
Michael Eischer
530f129a39 rest: remove workaround for content-length handling bug 2022-11-27 13:18:44 +01:00
Michael Eischer
8ad231bcad bump version numbers in instructions to reproduce binaries 2022-11-27 13:18:44 +01:00
Michael Eischer
a1eb923876 remove no longer necessary conditional compiles 2022-11-27 13:18:44 +01:00
Michael Eischer
bcdfc2a8ea CI: allow dependabot update of oauth2
Our minimum go version is new enough to allow updating the library.
2022-11-27 13:18:44 +01:00
Michael Eischer
686b0b2a3e update the minimum required go version to 1.18 2022-11-27 13:18:43 +01:00
Michael Eischer
69a2e81bd3 Merge pull request #4039 from restic/dependabot/go_modules/github.com/google/go-cmp-0.5.9
build(deps): bump github.com/google/go-cmp from 0.5.8 to 0.5.9
2022-11-26 17:39:27 +01:00
dependabot[bot]
278e93f738 build(deps): bump github.com/google/go-cmp from 0.5.8 to 0.5.9
Bumps [github.com/google/go-cmp](https://github.com/google/go-cmp) from 0.5.8 to 0.5.9.
- [Release notes](https://github.com/google/go-cmp/releases)
- [Commits](https://github.com/google/go-cmp/compare/v0.5.8...v0.5.9)

---
updated-dependencies:
- dependency-name: github.com/google/go-cmp
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-26 11:35:05 +00:00
Michael Eischer
747d2ecd7b Merge pull request #4042 from restic/skip-cloud-test-for-dependabot
CI: skip cloud tests for dependabot pull requests
2022-11-26 12:34:16 +01:00
Michael Eischer
98c6ca9d8f CI: skip cloud tests for dependabot pull requests 2022-11-26 12:23:55 +01:00
Michael Eischer
9113b2620f Merge pull request #4024 from MichaelEischer/macos-fuse
mount: switch to anacrolix fork of bazil/fuse
2022-11-26 12:15:14 +01:00
Michael Eischer
f115d64634 Merge pull request #4022 from MichaelEischer/race-checker
CI: Run the golang race checker
2022-11-26 12:13:50 +01:00
Michael Eischer
923c06cea0 Merge pull request #4025 from MichaelEischer/update-minio
Update minio library to add `credential_process` support
2022-11-25 23:21:57 +01:00
Michael Eischer
f4d3ed77c4 update minio library 2022-11-25 22:36:21 +01:00
greatroar
189e0fe5a9 fuse: Better inode generation
Hard links to the same file now get the same inode within the FUSE
mount. Also, inode generation is faster and, more importantly, no longer
allocates.

Benchmarked on Linux/amd64. Old means the benchmark with

        sink = fs.GenerateDynamicInode(1, sub.node.Name)

instead of calling inodeFromNode. Results:

name                   old time/op    new time/op    delta
Inode/no_hard_links-8     137ns ± 4%      34ns ± 1%   -75.20%  (p=0.000 n=10+10)
Inode/hard_link-8        33.6ns ± 1%     9.5ns ± 0%   -71.82%  (p=0.000 n=9+8)

name                   old alloc/op   new alloc/op   delta
Inode/no_hard_links-8     48.0B ± 0%      0.0B       -100.00%  (p=0.000 n=10+10)
Inode/hard_link-8         0.00B          0.00B           ~     (all equal)

name                   old allocs/op  new allocs/op  delta
Inode/no_hard_links-8      1.00 ± 0%      0.00       -100.00%  (p=0.000 n=10+10)
Inode/hard_link-8          0.00           0.00           ~     (all equal)
2022-11-16 08:35:01 +01:00
Michael Eischer
32ffcd86a2 Merge pull request #3993 from MichaelEischer/backup-json-full-snapshot-id
backup: print full snapshot id in JSON summary
2022-11-12 20:42:35 +01:00
Michael Eischer
f032a9d0ad prune: report how much data must be repacked to compressed the repo
prune now reports the remaining size of pack files containing
uncompressed blobs. The displayed value is suitable for use with `--max-repack-size`.
2022-11-12 20:20:23 +01:00
Michael Eischer
66818a8f98 Merge pull request #3980 from MichaelEischer/prune-compression-stats
prune: Correctly count used/duplicate blobs for partially compressed repos
2022-11-12 20:06:56 +01:00
Michael Eischer
4b5234924b Merge pull request #2875 from fgma/issue2699
issue2699: restore symlinks on windows when run as admin user
2022-11-12 20:06:45 +01:00
Michael Eischer
726a1969cd Merge pull request #2731 from dionorgua/rewrite-snapshot
Implement 'rewrite' command to exclude files from existing snapshots
2022-11-12 20:06:35 +01:00
Michael Eischer
bb0fa76c06 Cleanup exclude pattern collection 2022-11-12 19:55:22 +01:00
Michael Eischer
537cfe2e4c rewrite: Fix check that an exclude pattern was passed
The old check did not consider files containing case insensitive
excludes. The check is now implemented as a function of the
excludePatternOptions struct to improve cohesion.
2022-11-12 19:55:22 +01:00
Leo R. Lundgren
f175da2756 rewrite: Polish documentation 2022-11-12 19:55:22 +01:00
Leo R. Lundgren
f86ef4d3dd rewrite: Polish code and add missing messages 2022-11-12 19:55:22 +01:00
Leo R. Lundgren
c15bedccc0 rewrite: Revert unrelated documentation change 2022-11-12 19:55:22 +01:00
Michael Eischer
f88acd4503 rewrite: Fail if a tree contains an unknown field
In principle, the JSON format of Tree objects is extensible without
requiring a format change. In order to not loose information just play
it safe and reject rewriting trees for which we could loose data.
2022-11-12 19:55:22 +01:00
Michael Eischer
11b8c3a158 rewrite: add documentation 2022-11-12 19:55:22 +01:00
Michael Eischer
ec0c91e233 rewrite: Add tests for further ways to use the command 2022-11-12 19:55:22 +01:00
Michael Eischer
0224e276ec walker: Add tests for FilterTree 2022-11-12 19:55:22 +01:00
Michael Eischer
73f54cc5ea rewrite: rename --inplace to --forget 2022-11-12 19:55:22 +01:00
Michael Eischer
a47d9a1c40 rewrite: use unified snapshot filter options 2022-11-12 19:55:22 +01:00
Michael Eischer
b044649118 rewrite: add minimal test 2022-11-12 19:55:22 +01:00
Michael Eischer
375a3db64d rewrite: non-exclusive lock if snapshots are only added 2022-11-12 19:55:22 +01:00
Michael Eischer
327f418a9c rewrite: cleanup err handling and output 2022-11-12 19:55:22 +01:00
Michael Eischer
ad14d6e4ac rewrite: use SelectByName like in the backup command 2022-11-12 19:55:22 +01:00
Michael Eischer
7ebaf6e899 rewrite: start repository uploader goroutines 2022-11-12 19:55:22 +01:00
Michael Eischer
559acea0d8 unify exclude pattern options 2022-11-12 19:55:22 +01:00
Michael Eischer
4cace1ffe9 unify exclude patterns with backup command 2022-11-12 19:55:22 +01:00
Michael Eischer
2b69a1c53b rewrite: filter all snapshots if none are specified 2022-11-12 19:55:22 +01:00
Michael Eischer
f6339b88af rewrite: extract tree filtering 2022-11-12 19:55:22 +01:00
Michael Eischer
c0f7ba2388 rewrite: simplify dryrun 2022-11-12 19:55:22 +01:00
Michael Eischer
4d6ab83019 rewrite: use treejsonbuilder 2022-11-12 19:55:22 +01:00
Michael Eischer
82592b88b5 rewrite: address most review comments 2022-11-12 19:55:22 +01:00
Michael Eischer
b922774343 rewrite: fix compilation 2022-11-12 19:55:22 +01:00
Dmitry Nezhevenko
dc29709742 Implement 'rewrite' command to exclude files from existing snapshots 2022-11-12 19:55:22 +01:00
Michael Eischer
220eaee76b mount: switch to anacrolix fork of bazil/fuse
The anacrolix fork contains the latest changes from bazil/fuse and
additionally provides support for recent versions of macFUSE.
2022-11-12 19:22:31 +01:00
Michael Eischer
6fa45d0d39 Merge pull request #4011 from greatroar/backup-stdin-password
cmd: Don't read password from stdin for backup --stdin
2022-11-12 19:18:56 +01:00
Michael Eischer
bbd180ae21 Merge pull request #4017 from Rajpratik71/Rajpratik71-patch-1
feat: dependabot workflow automation for updating dependency
2022-11-12 15:48:48 +01:00
Pratik Raj
bef1064b8e chore: ignore upgrade for 'bazil/fuse' and 'golang.org/x/oauth2' 2022-11-12 19:39:16 +05:30
Michael Eischer
7b4fe7bad5 Merge pull request #4021 from greatroar/mac-fsync
backend/local: Ignore ENOTTY for fsync on Mac
2022-11-11 23:10:37 +01:00
greatroar
348e966daa backend/local: Ignore ENOTTY for fsync on Mac
Fixes #4016.
2022-11-11 22:51:51 +01:00
Michael Eischer
0e5fe4c6ab CI: run golang race checker 2022-11-11 22:15:22 +01:00
Michael Eischer
13fbc96ed3 lock: Synchronize Refresh() and Stale()
The lock test creates a lock and checks that it is not stale. However,
it is possible that the lock is refreshed concurrently, which updates
the lock timestamp. Checking the timestamp in `Stale()` without
synchronization results in a data race. Thus add a lock to prevent
concurrent accesses.
2022-11-11 21:52:53 +01:00
Michael Eischer
e1ba7ab684 lock: Don't copy the lock when checking for process existence
The lock test creates a lock and checks that it is not stale. This also
tests whether the corresponding process still exists. However, it is
possible that the lock is refreshed concurrently, which updates the lock
timestamp. Calling `processExists()` with a value receiver, however,
creates an unsynchronized copy of this field. Thus call the method using
a pointer receiver.
2022-11-11 21:45:55 +01:00
Michael Eischer
dc060356c2 mount: only start next test after mount command cleanup is complete
The test did not wait for the mount command to fully shutdown all
running goroutines. This caused the go race detector to report a data
race related to lock refreshes.

==================
WARNING: DATA RACE
Write at 0x0000021bdfdb by goroutine 667:
  github.com/restic/restic/internal/backend/retry.TestFastRetries()
      /restic/restic/internal/backend/retry/testing.go:7 +0x18f
  github.com/restic/restic/cmd/restic.withTestEnvironment()
      /restic/restic/cmd/restic/integration_helpers_test.go:175 +0x183
  github.com/restic/restic/cmd/restic.TestMountSameTimestamps()
      /restic/restic/cmd/restic/integration_fuse_test.go:202 +0xac
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      /usr/lib/go/src/testing/testing.go:1493 +0x47

Previous read at 0x0000021bdfdb by goroutine 609:
  github.com/restic/restic/internal/backend/retry.(*Backend).retry()
      /restic/restic/internal/backend/retry/backend_retry.go:72 +0x9e
  github.com/restic/restic/internal/backend/retry.(*Backend).Remove()
      /restic/restic/internal/backend/retry/backend_retry.go:149 +0x17d
  github.com/restic/restic/internal/cache.(*Backend).Remove()
      /restic/restic/internal/cache/backend.go:38 +0x11d
  github.com/restic/restic/internal/restic.(*Lock).Unlock()
      /restic/restic/internal/restic/lock.go:190 +0x249
  github.com/restic/restic/cmd/restic.refreshLocks.func1()
      /restic/restic/cmd/restic/lock.go:86 +0xae
  runtime.deferreturn()
      /usr/lib/go/src/runtime/panic.go:476 +0x32
  github.com/restic/restic/cmd/restic.lockRepository.func2()
      /restic/restic/cmd/restic/lock.go:61 +0x71

[...]

Goroutine 609 (finished) created at:
  github.com/restic/restic/cmd/restic.lockRepository()
      /restic/restic/cmd/restic/lock.go:61 +0x488
  github.com/restic/restic/cmd/restic.lockRepo()
      /restic/restic/cmd/restic/lock.go:25 +0x219
  github.com/restic/restic/cmd/restic.runMount()
      /restic/restic/cmd/restic/cmd_mount.go:126 +0x1f8
  github.com/restic/restic/cmd/restic.testRunMount()
      /restic/restic/cmd/restic/integration_fuse_test.go:61 +0x1ce
  github.com/restic/restic/cmd/restic.checkSnapshots.func1()
      /restic/restic/cmd/restic/integration_fuse_test.go:90 +0x124
==================
2022-11-11 21:43:01 +01:00
Michael Eischer
32c9667990 Merge pull request #4019 from MichaelEischer/fix-file-saver-race
archiver: Fix race condition resulting in files containing null IDs
2022-11-11 20:52:33 +01:00
Michael Eischer
d268552a0a Merge pull request #4014 from MichaelEischer/fix-debug-examine
debug: fix crash in `debug examine --reupload-blobs`
2022-11-10 20:37:32 +01:00
Michael Eischer
5756c96c9f archiver: Fix race condition resulting in files containing null IDs
In some rare cases files could be created which contain null IDs (all
zero) in their content list. This was caused by a race condition between
growing the `Content` slice and inserting the blob IDs into it. In some
cases the blob ID was written to the old slice, which a short time
afterwards was replaced with a larger copy, that did not yet contain the
blob ID.
2022-11-10 20:19:37 +01:00
Pratik Raj
df614fff26 feat: dependabot workflow automation for updating dependency
Signed-off-by: Pratik Raj <Rajpratik71@gmail.com>
2022-11-10 16:02:03 +05:30
Michael Eischer
11a4bb051e debug: fix crash in debug examine --reupload-blobs 2022-11-09 22:13:17 +01:00
Michael Eischer
5f9ac2b165 Merge pull request #4010 from MichaelEischer/file-saver-sanity-check
archiver: Check that saved file does not have null IDs in content
2022-11-08 23:07:32 +01:00
Michael Eischer
b1d1202b1d archiver: Check that saved file does not have null IDs in content
Null IDs in the file content indicate that something went wrong. Thus
fails before saving the affected file.
2022-11-08 22:57:41 +01:00
greatroar
5dceadeb72 cmd: Don't read password from stdin for backup --stdin 2022-11-06 14:55:57 +01:00
Michael Eischer
1ccab95bc4 b2: Support file hiding instead of deleting them permanently
Automatically fall back to hiding files if not authorized to permanently
delete files. This allows using restic with an append-only application
key with B2.  Thus, an attacker cannot directly delete backups with the
API key used by restic.

To use this feature create an application key without the deleteFiles
capability. It is recommended to restrict the key to just one bucket.
For example using the b2 command line tool:

    b2 create-key --bucket <bucketName> <keyName> listBuckets,readFiles,writeFiles,listFiles

Suggested-by: Daniel Gröber <dxld@darkboxed.org>
2022-11-05 20:10:45 +01:00
Michael Eischer
24a2e5cab9 Merge pull request #4008 from MichaelEischer/tweak-lock-refresh-test
lock: Tweak timeouts for lock refresh test
2022-11-05 10:53:13 +01:00
Michael Eischer
403390479c Merge pull request #3997 from greatroar/fuse-hash
fuse: Better check for whether snapshots changed
2022-11-05 10:52:11 +01:00
Michael Eischer
d29abc1a31 Merge pull request #4007 from MichaelEischer/hide-compression-level-for-v1-repo
Only print compression level starting from repository version 2
2022-11-05 10:33:25 +01:00
greatroar
c091e43b33 fuse: Better check for whether snapshots changed
We previously checked whether the set of snapshots might have changed
based only on their number, which fails when as many snapshots are
forgotten as are added. Check for the SHA-256 of their id's instead.
2022-11-05 09:32:45 +01:00
Michael Eischer
aaac63da8d lock: Tweak timeouts for lock refresh test
For some reason the test fails from time to time. Increase the timeouts
to hopefully avoid this issue.
2022-11-04 22:48:18 +01:00
Michael Eischer
fd4d23460f only print compression level starting from repository version 2 2022-11-04 22:40:07 +01:00
Alexander Neumann
8dd95b710e Merge pull request #3992 from MichaelEischer/err-on-invalid-compression
Return error if RESTIC_COMPRESSION env variable is invalid
2022-11-04 19:41:34 +01:00
Alexander Neumann
783b8781a7 Merge pull request #4000 from restic/min-go-version
build: Correct checks for minimum Go version
2022-11-04 10:31:02 +01:00
Alexander Neumann
543649f2f2 Merge pull request #4001 from restic/docker-go-version
docker: Increase Go version to 1.19
2022-11-04 10:30:11 +01:00
Leo R. Lundgren
0a4cddb34d docker: Increase Go version to 1.19 2022-11-03 22:59:59 +01:00
Leo R. Lundgren
333c2c6ed4 build: Correct checks for minimum Go version 2022-11-03 22:50:07 +01:00
rawtaz
92df039e5d Merge pull request #3996 from MichaelEischer/fix-ui-progress
backup: fix stuck status bar
2022-11-02 21:48:16 +01:00
Michael Eischer
9354262b1b backup: fix stuck status bar
The status bar got stuck once the first error was reported, the scanner
completed or some file was backed up. Either case sets a flag that the
scanner has started.

This flag is used to hide the progress bar until the flag is set. Due to
an inverted condition, the opposite happened and the status stopped
refreshing once the flag was set.

In addition, the scannerStarted flag was not set when the scanner just
reported progress information.
2022-11-02 21:31:13 +01:00
Michael Eischer
06141ce1f4 backup: print full snapshot id in JSON summary 2022-10-31 19:03:42 +01:00
Michael Eischer
59a90943bb Merge pull request #3983 from greatroar/formatting
Centralize and fix formatting of bytes, percentages, durations
2022-10-31 18:52:24 +01:00
greatroar
5ab3e6276a ui: Fix FormatBytes at exactly 1024 time a unit
1024 would be displayed as "1024 bytes" instead of "1.000 KiB", etc.
2022-10-31 18:39:28 +01:00
rawtaz
4f1fae9c98 Merge pull request #3982 from MichaelEischer/show-compression-mode
Show selected compression level when opening repository
2022-10-30 21:29:42 +01:00
Michael Eischer
8fe159cc5a enable ysmlink tests for windows 2022-10-30 18:43:04 +01:00
Michael Eischer
3499c6354e Merge pull request #3955 from MichaelEischer/async-futurefile-completion
Improve archiver performance for small files
2022-10-30 18:38:04 +01:00
Michael Eischer
144257f8bd restore symlink timestamps on windows 2022-10-30 11:04:04 +01:00
Michael Eischer
c0f34af9db backup: hide files from status which are read completely but not saved
As the FileSaver is asynchronously waiting for all blobs of a file to be
stored, the number of active files is higher than the number of files
from which restic is reading concurrently. Thus to not confuse users,
only display files in the status from which restic is currently reading.
2022-10-30 10:29:12 +01:00
Michael Eischer
a571fc4aa1 add changelog for faster backups with small files 2022-10-30 10:29:12 +01:00
Michael Eischer
b52a8ff05c ui: Properly clear lines no longer used for status
Previously, the old status text remained until it was overwritten.
2022-10-30 10:29:12 +01:00
Michael Eischer
b4de902596 archiver: Asynchronously complete FutureFile
After reading and chunking all data in a file, the FutureFile still has
to wait until the FutureBlobs are completed. This was done synchronously
which results in blocking the file saver and prevents the next file from
being read.

By replacing the FutureBlob with a callback, it becomes possible to
complete the FutureFile asynchronously.
2022-10-30 10:29:11 +01:00
Michael Eischer
47e05080a9 Merge pull request #3990 from MichaelEischer/lock-refresh-test
lock: add test to check that refreshing works
2022-10-30 10:15:44 +01:00
Michael Eischer
c7ace314f6 Merge pull request #3989 from greatroar/eachbypack
More compact data structure for Index.EachByPack
2022-10-30 00:02:55 +02:00
greatroar
0e8893dae9 index: Compact data structure for Index.EachByPack 2022-10-29 23:09:17 +02:00
greatroar
137f0bc944 repository: Fix benchmarkSaveAndEncrypt 2022-10-29 23:09:17 +02:00
Michael Eischer
01f0db4e56 return error if RESTIC_COMPRESSION env variable is invalid 2022-10-29 22:03:39 +02:00
Michael Eischer
7c87fb941c Merge pull request #3986 from greatroar/counter
ui/progress: Load both values in a single Lock/Unlock
2022-10-29 21:50:55 +02:00
Michael Eischer
3b0bb02a68 Merge pull request #3977 from greatroar/progress
ui/backup: Replace channels with a mutex
2022-10-29 21:33:04 +02:00
Michael Eischer
0d260cfd82 enable symlink test on windows 2022-10-29 21:26:34 +02:00
fgma
8e5eb1090c issue2699: restore symlinks on windows when run as admin user 2022-10-29 21:19:33 +02:00
rawtaz
af3f7c866f Merge pull request #3988 from FelixBurkhard/FelixBurkhard-patch-1
Clearify what Azure account name means
2022-10-29 13:32:44 +02:00
Michael Eischer
24267e9a9d lock: add test to check that refreshing works 2022-10-29 11:26:00 +02:00
Michael Eischer
8e51e1e605 shorten 'repository opened' output 2022-10-29 11:22:00 +02:00
FelixBurkhard
575d26ec87 Clearify what Azure account name means
When reading it first it was not clear to me the 'account name' meant the name of the
Azure Storage Account and not the Azure account itself.
2022-10-29 00:27:43 +02:00
greatroar
2dafda9164 ui/progress: Load both values in a single Lock/Unlock
We always need both values, except in a test, so we don't need to lock
twice and risk scheduling in between.

Also, removed the resetting in Done. This copied a mutex, which isn't
allowed. Static analyzers tend to trip over that.
2022-10-25 07:55:24 +02:00
Michael Eischer
f8910bc4ff Merge pull request #3985 from saltsa/fix_lock_refresh
Fix bug in lock refresh monitoring
2022-10-24 22:59:18 +02:00
Joonas Aunola
b06427c9f6 fix Unix to UnixNano 2022-10-23 23:40:21 +03:00
greatroar
006380199e cmd, ui: Deduplicate formatting utilities 2022-10-23 13:40:07 +02:00
greatroar
04216eb9aa ui/backup: Replace channels with a mutex
The channel-based algorithm had grown quite complicated. This is easier
to reason about and likely to be more performant with very many
CompleteBlob calls.
2022-10-23 13:28:41 +02:00
Michael Eischer
4fea3a413d show selected compression level when opening repository 2022-10-22 20:18:46 +02:00
Michael Eischer
ba58ccbe07 prune: add remark about non-deterministic blob selection 2022-10-22 19:46:10 +02:00
Michael Eischer
05651d6d4f prune: Correctly count used/duplicate blobs for partially compressed repos
Counting the first occurrence of a duplicate blob as used and counting
all other as duplicates, independent of which instance of the blob is
kept, is only accurate if all copies of the blob have the same size. This
is no longer the case for a repository containing both compressed and
uncompressed blobs.

Thus for duplicated blobs first count all instances as duplicates and
then subtract the actually used instance later on.
2022-10-22 19:24:36 +02:00
Michael Eischer
b57d42905c Merge pull request #3899 from MichaelEischer/less-prune-mem
Optimize prune memory usage
2022-10-22 18:56:02 +02:00
Michael Eischer
d966c52707 prune: allow gc of set of repacked blobs before index rebuild 2022-10-22 18:45:12 +02:00
Michael Eischer
1e2794fa55 add prune memory optimization changelog 2022-10-22 18:45:12 +02:00
Michael Eischer
68c9cb9c6a prune: Shrink keepBlobs set if possible
As long as only a small fraction of the data in a repository is
rewritten, the keepBlobs set will be rather small after cleaning it up.
As golang maps do not shrink their memory usage, just copy the contents
over to a new map. However, only copy the map if the cleanup removed at
least half the entries.
2022-10-22 18:45:12 +02:00
Michael Eischer
c4fc5c97f9 prune: Use a single CountedBlobSet to track blobs
The set covers necessary, existing and duplicate blobs. This removes the
duplicate sets used to track whether all necessary blobs also exist.
This reduces the memory usage of prune by about 20-30%.
2022-10-22 18:45:12 +02:00
Michael Eischer
b21241ec1c restic: Add CountedBlobSet type
This allows maintaining a usage counter for each blob.
2022-10-22 18:45:12 +02:00
Michael Eischer
ee6688a9f6 Merge pull request #3915 from plumbeo/compression-stats
restic stats: print uncompressed size in mode raw-data
2022-10-21 22:10:29 +02:00
Michael Eischer
27634a1a68 Merge pull request #3978 from MichaelEischer/fix-negative-pattern-example
Remove misleading wildcard from negative exclude pattern example
2022-10-21 22:04:30 +02:00
Michael Eischer
aa77702e49 Merge pull request #3971 from MichaelEischer/parallel-list
Unify ForAllIndex/Snapshot/Lock functions
2022-10-21 21:58:33 +02:00
Michael Eischer
6877aaa8aa Merge pull request #3967 from MichaelEischer/archiver-extract-exclude-options
backup: extract exclude pattern options
2022-10-21 21:50:00 +02:00
Michael Eischer
2e9ee8577a Merge pull request #3970 from MichaelEischer/split-retry-backend
Split backend package into smaller parts
2022-10-21 21:49:46 +02:00
Michael Eischer
59d46bb3f5 backup: extract exclude pattern options
This is a preparation to make the exclude options usable for the
upcoming `rewrite` command.
2022-10-21 21:40:59 +02:00
Michael Eischer
5c7a9a739a backend: Split RetryBackend into own package
The RetryBackend tests depend on the mock backend. When the Backend
interface is eventually split from the restic package, this will lead to
a dependency cycle between backend and backend/mock. Thus split the
RetryBackend into a separate package to avoid this problem.
2022-10-21 21:38:17 +02:00
Michael Eischer
32603d49c4 backend: remove unused ErrorBackend 2022-10-21 21:36:05 +02:00
Michael Eischer
8c18c65b3b backend: remove unused Paths variable 2022-10-21 21:36:05 +02:00
Michael Eischer
4ccd5e806b backend: split layout code into own subpackage 2022-10-21 21:36:05 +02:00
Michael Eischer
b361284f28 Merge pull request #3979 from MichaelEischer/backup-less-time-now
backup: reduce calls to time.Now
2022-10-21 21:33:34 +02:00
Michael Eischer
738b2a0445 parallelize more List usages 2022-10-21 21:26:45 +02:00
Michael Eischer
ae45f3b04f restic: Unify code to load Index/Lock/Snapshot 2022-10-21 21:25:11 +02:00
Michael Eischer
8e2695be0b Merge pull request #3973 from MichaelEischer/speedup-integration-tests
speed-up integration tests by reducing the RetryBackend timeout
2022-10-21 21:17:35 +02:00
Michael Eischer
35d968bcde Merge pull request #3969 from MichaelEischer/key-by-id
Port restic.Find to return IDs and identify keys by restic.ID
2022-10-21 21:15:40 +02:00
Michael Eischer
4133fee6f9 Merge pull request #3972 from MichaelEischer/fix-flaky-lock-cancel-test
lock: fix flaky TestLockFailedRefresh
2022-10-21 21:12:34 +02:00
Michael Eischer
c8c8391b21 Merge pull request #3974 from greatroar/cleanup
More cleanups and a micro-optimization
2022-10-21 21:11:37 +02:00
Michael Eischer
ee7c28f5e6 backup: reduce calls to time.Now
Archiver.Save queries the current time multiple times. This commit
removes one of these calls as they showed up while profiling a backup of
a nearly unchanged dataset containing 3 million files.
2022-10-21 20:55:01 +02:00
Michael Eischer
3e60d38a23 Remove misleading wildcard from negative exclude pattern example
There is no need to use a special wildcard `**` to demonstrate negative
patterns. Actually, it is both slower than the simpler variant and seems
to confuse users.
2022-10-21 20:48:45 +02:00
greatroar
9adae5521d cache: Call interface method once 2022-10-21 14:32:46 +02:00
greatroar
201e5c7e74 backup: Clean up progress reporting code 2022-10-21 13:48:30 +02:00
plumbeo
a6f83e0011 Add changelog 2022-10-17 15:38:42 +02:00
plumbeo
bc945d0bf0 restic stats: add more compression statistics
Calculate and display compression ratio, space saving and progress
2022-10-17 15:38:38 +02:00
greatroar
b513597546 internal/restic: Make FileType a uint8 instead of a string
The string form was presumably useful before the introduction of
layouts, but right now it just makes call sequences and garbage
collection more expensive (the latter because every string contains
a pointer to be scanned).
2022-10-16 10:59:01 +02:00
greatroar
22147e1e02 all: Minor cleanups
if x { return true } return false => return x

	fmt.Sprintf("%v", x) => fmt.Sprint(x) or x.String()

The fmt.Sprintf idiom is still used in the SecretString tests, where it
serves security hardening.
2022-10-16 10:50:39 +02:00
greatroar
d03460010f internal/restic: Fix ID.UnmarshalJSON, ParseID
ID.UnmarshalJSON accepted non-JSON input with ' as the string delimiter.
Also, the error message for non-hex input was less informative than it
could be and it performed too many checks.

Changed ParseID to keep the error messages consistent.
2022-10-16 10:39:52 +02:00
Michael Eischer
aa39bf3cf6 backend/test: remove duplicate test
The test is identical to the tests for the mem backend.
2022-10-15 23:15:07 +02:00
Michael Eischer
28e1c4574b mem: use cheaper hash for backend 2022-10-15 23:14:33 +02:00
Michael Eischer
c3400d3c55 backend: speedup RetryBackend tests 2022-10-15 23:13:44 +02:00
Michael Eischer
99547518cd lock: fix flaky TestLockFailedRefresh
The comparison of the current time and the last lock refresh were using
seconds represented as integers. As the test only waits for up to one
second, the associated number truncation can cause the test to take
longer than once second and thus to fail.

Switch to nanoseconds to avoid this problem. This also slightly speeds
up the test.
2022-10-15 22:36:32 +02:00
Michael Eischer
e10420553b speed-up integration tests by reducing the RetryBackend timeout
On my machine this decreases the runtime for `./cmd/restic` from 9.5s to
6.5s.
2022-10-15 22:29:58 +02:00
Michael Eischer
367f35db27 Merge pull request #3968 from MichaelEischer/cleanup-complete-blob
backup: Remove unused filename parameter from CompleteBlob callback
2022-10-15 16:11:16 +02:00
Michael Eischer
8d62a7adb4 identify keys by ID and not name 2022-10-15 16:07:43 +02:00
Michael Eischer
02634dce7a restic: change Find to return ids
That way consumers no longer have to manually convert the returned name
to an id.
2022-10-15 16:06:54 +02:00
Michael Eischer
964977677f backup: Remove unused filename parameter from CompleteBlob callback 2022-10-15 15:21:17 +02:00
Michael Eischer
258b487d8f Merge pull request #3951 from MichaelEischer/rework-snapshot-filter
Rework snapshot filtering
2022-10-15 14:47:47 +02:00
Michael Eischer
de9bc031df add changelog for ls handling of missing snapshots 2022-10-15 13:34:50 +02:00
Michael Eischer
246d3032ae restic: Don't list snapshots if FindSnapshot gets full id 2022-10-15 13:34:34 +02:00
Michael Eischer
d8c00b9726 add comment 2022-10-15 13:34:21 +02:00
Michael Eischer
a3113c6097 restic: Change FindSnapshot functions to return the snapshot 2022-10-15 13:34:04 +02:00
Michael Eischer
b50f48594d restic: cleanup arguments of findLatestSnapshot 2022-10-15 13:33:48 +02:00
Michael Eischer
61e827ae4f restic: hide findLatestSnapshot 2022-10-15 13:33:32 +02:00
Michael Eischer
fcad5e6f5d backup: use unified FindFilteredSnapshot 2022-10-15 13:33:29 +02:00
Michael Eischer
0aa73bbd39 ls: proper error handling for non-existent snapshot
Use restic.FindFilteredSnapshot to resolve the snapshot ID. This ensures
consistent behavior for all commands using initSingleSnapshotFilterOptions.
2022-10-15 13:32:00 +02:00
Michael Eischer
a81f0432e9 restic: Add unified method to resolve a single snapshot 2022-10-15 13:31:45 +02:00
Michael Eischer
95a1bb4261 restic: Rework error handling of FindFilteredSnapshots and handle snapshotIDs
FindFilteredSnapshots no longer prints errors during snapshot loading on
stderr, but instead passes the error to the callback to allow the caller
to decide on what to do.

In addition, it moves the logic to handle an explicit snapshot list from
the main package to restic.
2022-10-15 13:31:26 +02:00
Michael Eischer
cff22a5f01 dump: use correct help text for filter options 2022-10-15 13:31:10 +02:00
Michael Eischer
7a6dcb4831 Merge pull request #3966 from MichaelEischer/cleanup-walker-test
walker: Convert tests to use TreeJSONBuilder
2022-10-15 11:25:11 +02:00
Michael Eischer
7cf042118f walker: Convert tests to use TreeJSONBuilder
The old code marshalled the tree blobs different than other places in
restic. The hashed tree blob did not contain a final newline character.
2022-10-15 11:04:13 +02:00
Michael Eischer
cea7191995 Merge pull request #3959 from MichaelEischer/buffered-backup-progress
backup: Use buffered channels to collect backup status
2022-10-15 10:57:19 +02:00
Michael Eischer
ba688aad20 Merge pull request #3961 from greatroar/cleanup
Misc. cleanup
2022-10-14 21:49:35 +02:00
Michael Eischer
9c290a8093 Merge pull request #3960 from greatroar/errors
errors: Drop WithMessage
2022-10-14 21:41:28 +02:00
greatroar
0e155fd9a6 internal/restic: Fix UID/GID parsing
The helper function uidGidInt used strconv.ParseInt instead of
ParseUint, so it silently ignored some invalid user/group IDs.

Also, improve the error message. "Invalid UID" is more informative than
having "ParseInt" twice (*strconv.NumError displays the function name).

Finally, the user.User struct can be passed by pointer to get reduce
code size.
2022-10-14 18:21:00 +02:00
greatroar
e0b743c64d internal/restic: Remove unused ID.EqualString 2022-10-14 18:20:11 +02:00
greatroar
6922360179 ui/backup: Remove unused ProgressReporter type, Progress field 2022-10-14 14:36:19 +02:00
greatroar
d4aadfa389 all: Drop ctxhttp
This package is no longer needed, since we can use the stdlib's
http.NewRequestWithContext.

backend/rclone already did, but it needed a different error check due to
a difference between net/http and ctxhttp.

Also, store the http.Client by value in the REST backend (changed to a
pointer when ctxhttp was introduced) and use errors.WithStack instead
of errors.Wrap where the message was no longer accurate. Errors from
http.NewRequestWithContext will start with "net/http" or "net/url", so
they're easy to identify.
2022-10-14 14:33:49 +02:00
greatroar
16849d5361 internal/archiver: Missing argument to errors.Errorf 2022-10-14 14:18:52 +02:00
greatroar
09c14f33c8 internal/checker: Pass Error.Error pointer receiver 2022-10-14 14:13:32 +02:00
greatroar
feb790f497 internal/restic: Use errors.New when no formatting is needed 2022-10-14 14:07:20 +02:00
greatroar
ba44666704 errors: Drop WithMessage 2022-10-14 14:06:47 +02:00
Michael Eischer
1a6160d152 Merge pull request #3880 from MichaelEischer/archiver-savedir-cleanup
archiver: Improve handling of "file xxx already present" error
2022-10-08 21:48:14 +02:00
Michael Eischer
21b1d7a880 Merge pull request #3948 from MichaelEischer/split-index
repository: split index into a separate package
2022-10-08 21:41:57 +02:00
Michael Eischer
5278ab51c8 archiver: Check that duplicates are only ignored if identical 2022-10-08 21:38:36 +02:00
Michael Eischer
403b01b788 backup: Only return a warning for duplicate directory entries
The backup command failed if a directory contains duplicate entries.
Downgrade the severity of this problem from fatal error to a warning.
This allows users to still create a backup.
2022-10-08 21:38:21 +02:00
Michael Eischer
d7d7b4ab27 archiver: refactor TreeSaverTest 2022-10-08 21:29:32 +02:00
Michael Eischer
8e38c43c27 archiver: let FutureNode.Take return an error if no data is available
This ensures that we cannot accidentally store an invalid node.
2022-10-08 21:28:39 +02:00
Michael Eischer
2b88cd6eab archiver: Restructure SaveTree to work like SaveDir
SaveTree did not use the TreeSaver but rather managed the tree
collection and upload itself. This prevents using the parallelism
offered by the TreeSaver and duplicates all related code. Using the
TreeSaver can provide some speed-ups as all steps within the backup tree
now rely on FutureNodes. This can be especially relevant for backups
with large amounts of explicitly specified files.

The main difference between SaveTree and SaveDir is, that only the
former can save tree blobs in which nodes have a different name than the
actual file on disk. This is the result of resolving name conflicts
between multiple files with the same name. The filename that must be
used within the snapshot is now passed directly to
restic.NodeFromFileInfo. This ensures that a FutureNode already contains
the correct filename.
2022-10-08 21:28:39 +02:00
Michael Eischer
2e3f1c08c5 repository: split index into a separate package 2022-10-08 21:15:34 +02:00
Michael Eischer
5760ba6989 Merge pull request #3949 from MichaelEischer/simplify-mixedpacks
repository: remove IsMixedPack and add replacement for checker
2022-10-08 21:14:14 +02:00
Michael Eischer
5ee25e669a Merge pull request #3940 from MichaelEischer/better-rclone-error
Better error message if connection to rclone fails
2022-10-08 21:14:00 +02:00
Michael Eischer
5600f11696 rclone: Fix stderr handling if command exits unexpectedly
According to the documentation of exec.Cmd Wait() must not be called
before completing all reads from the pipe returned by StdErrPipe(). Thus
return a context that is canceled once rclone has exited and use that as
a precondition to calling Wait(). This should ensure that all errors
printed to stderr have been copied first.
2022-10-08 20:16:06 +02:00
Michael Eischer
b8acad4da0 rclone: return rclone error instead of canceled context
When rclone fails during the connection setup this currently often
results in a context canceled error. Replace this error with the exit
code from rclone.
2022-10-08 20:15:24 +02:00
Michael Eischer
d3ebec8f21 backup: Use buffered channels to collect backup status
When backing up many small files, the unbuffered channels frequently
cause the FileSaver to block when reporting progress information. Thus,
add buffers to these channels to avoid unnecessary scheduling.

As the status information is purely informational, it doesn't matter
that the status reporting shutdown is somewhat racy and could miss a few
final updates.
2022-10-08 18:20:41 +02:00
Michael Eischer
f9d4e0c2af Merge pull request #3958 from greatroar/errors
errors: Drop Cause in favor of Go 1.13 error handling
2022-10-08 18:06:35 +02:00
Michael Eischer
119e6aee01 Merge pull request #3957 from greatroar/typo
cmd: Typo in --read-concurrency description
2022-10-08 14:41:35 +02:00
greatroar
07e5c38361 errors: Drop Cause in favor of Go 1.13 error handling
The only use cases in the code were in errors.IsFatal, backend/b2,
which needs a workaround, and backend.ParseLayout. The last of these
requires all backends to implement error unwrapping in IsNotExist.
All backends except gs already did that.
2022-10-08 13:08:08 +02:00
greatroar
4eae4d3e1a cmd: Typo in --read-concurrency description 2022-10-08 11:27:39 +02:00
Michael Eischer
83cb58b4f3 Merge pull request #3956 from MichaelEischer/fix-lock-refresh
lock: Use the correct duration to check for expired locks
2022-10-07 22:58:10 +02:00
Michael Eischer
7c5d63a794 lock: Use the correct duration to check for expired locks 2022-10-07 22:39:53 +02:00
Michael Eischer
8b7c952f17 Merge pull request #3953 from keachi/typo
Fix typo
2022-10-07 22:18:32 +02:00
Michael Eischer
e43d2d45f7 Merge pull request #3952 from hoelzro/master
Update copy documentation to use --from-repo option
2022-10-07 22:18:06 +02:00
Rob Hoelz
03e9a26018 Update copy documentation to use --from-repo option
Removing the last references to the deprecated --repo2 option
2022-10-07 22:00:12 +02:00
tr
43cc01d63e doc: Fix typo 2022-10-05 21:03:14 +02:00
Michael Eischer
7112a132c3 Merge pull request #3950 from MichaelEischer/misc-cleanups
Cleanups for cmd_debug/repository and remove dead code from restic package
2022-10-03 12:46:32 +02:00
Michael Eischer
4bb5240720 repository: remove unused PrefixLength 2022-10-03 12:15:53 +02:00
Michael Eischer
999fe29976 repository: hide prepareCache 2022-10-03 12:15:53 +02:00
Michael Eischer
9197c63007 debug: use repository.ListPack wrapper 2022-10-03 12:09:08 +02:00
Michael Eischer
ddcf549eba repository: remove IsMixedPack and add replacement for checker
Repositories with mixed packs are probably quite rare by now. When
loading data blobs from a mixed pack file, this will no longer trigger
caching that file. However, usually tree blobs are accessed first such
that this shouldn't make much of a difference.

The checker gets a simpler replacement.
2022-10-03 12:03:59 +02:00
Michael Eischer
a61fbd287a Merge pull request #3569 from MichaelEischer/strict-locking
Strict repository lock handling
2022-10-03 00:44:44 +02:00
Michael Eischer
6d2d297215 pass global context through cobra 2022-10-03 00:19:46 +02:00
Michael Eischer
49126796d0 lock: fix timer expiry monitoring during standby
Monotonic timers are paused during standby. Thus these timers won't fire
after waking up. Fall back to periodic polling to detect too large clock
jumps. See https://github.com/golang/go/issues/35012 for a discussion of
go timers during standby.
2022-10-03 00:19:46 +02:00
Michael Eischer
401e432e9d lock: Do not ignore invalid lock files
While searching for lock file from concurrently running restic
instances, restic ignored unreadable lock files. These can either be
in fact invalid or just be temporarily unreadable. As it is not really
possible to differentiate between both cases, just err on the side of
caution and consider the repository as already locked.

The code retries searching for other locks up to three times to smooth
out temporarily unreadable lock files.
2022-10-03 00:19:46 +02:00
Michael Eischer
aeed420e1a add changelog 2022-10-03 00:19:46 +02:00
Michael Eischer
9959190e39 lock: Add integration test
The tests check that the wrapped context is properly canceled whenever
the repository is unlock or when the lock refresh fails.
2022-10-03 00:19:46 +02:00
Michael Eischer
c3538b063a lock: Use repository interface instead of struct 2022-10-03 00:19:46 +02:00
Michael Eischer
d92957dd78 lock: Implement strict lock expiry monitoring
Restic continued e.g. a backup task even when it failed to renew the
lock or failed to do so in time. For example if a backup client enters
standby during the backup this can allow other operations like `prune`
to run in the meantime (after calling `unlock`). After leaving standby
the backup client will continue its backup and upload indexes which
refer pack files that were removed in the meantime.

This commit introduces a goroutine explicitly monitoring for locks that
are not refreshed in time. To simplify the implementation there's now a
separate goroutine to refresh the lock and monitor for timeouts for each
lock. The monitoring goroutine would now cause the backup to fail as the
client has lost it's lock in the meantime.

The lock refresh goroutines are bound to the context used to lock the
repository initially. The context returned by `lockRepo` is also
cancelled when any of the goroutines exits. This ensures that the
context is cancelled whenever for any reason the lock is no longer
refreshed.
2022-10-03 00:19:46 +02:00
Michael Eischer
928914f821 Prepare for context bound to lock lifetime 2022-10-03 00:19:46 +02:00
Michael Eischer
985722b102 Remove ctx from globalOptions
Previously the global context was either accessed via gopts.ctx,
stored in a local variable and then used within that function or
sometimes both. This makes it very hard to follow which ctx or a wrapped
version of it reaches which method.

Thus just drop the context from the globalOptions struct and pass it
explicitly to every command line handler method.
2022-10-03 00:19:46 +02:00
Michael Eischer
ab819b2344 key: Cleanup method signatures 2022-10-03 00:19:46 +02:00
Michael Eischer
d0668b695d Remove unnecessary context.WithCancel calls
The gopts.ctx is cancelled when the main() method of restic exits.
2022-10-03 00:19:46 +02:00
Michael Eischer
7ce4cb7908 Merge pull request #3947 from MichaelEischer/fix-cache-verify-test
cache: Fix file descriptor leak in TestBackendRemoveBroken
2022-10-03 00:19:26 +02:00
Michael Eischer
430ab32941 cache: Fix file descriptor leak in TestBackendRemoveBroken 2022-10-03 00:06:44 +02:00
Michael Eischer
e99ad39b34 Merge pull request #2750 from metalsp0rk/min-packsize
Add `backup --file-read-concurrency` flag
2022-10-02 23:11:47 +02:00
Michael Eischer
2e606ca70b backup: rework read concurrency 2022-10-02 22:55:14 +02:00
Kyle Brennan
4a501d7118 backup: add option for file read concurrency 2022-10-02 22:51:45 +02:00
Michael Eischer
9ec7eee803 Merge pull request #3521 from MichaelEischer/redownload-broken-files
Redownload files with wrong hash
2022-10-02 22:50:03 +02:00
Michael Eischer
b25d0773b6 Merge pull request #3944 from MichaelEischer/fix-linter-errors
CI: ignore warning about missing package comment
2022-09-27 21:41:55 +02:00
Michael Eischer
5265550ff3 CI: ignore warning about missing package comment 2022-09-27 21:31:37 +02:00
Michael Eischer
e89fc2a29d Merge pull request #3943 from MichaelEischer/find-match-only-valid-ids
ignore filenames which are not IDs when expanding a prefix
2022-09-27 20:56:48 +02:00
Michael Eischer
67e4620cd6 Merge pull request #3938 from restic/errdot
rclone/sftp: Improve handling of ErrDot errors
2022-09-27 20:33:42 +02:00
Michael Eischer
5d3c5b9e50 restic: ignore filenames which are not IDs when expanding a prefix
Some backends generate additional files for each existing file, e.g.

1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef
1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef.sha256

For some commands this leads to an "multiple IDs with prefix" error when
trying to reference a snapshot.
2022-09-27 20:30:40 +02:00
Leo R. Lundgren
ebe9f2c969 rclone/sftp: Improve handling of ErrDot errors
Restic now yields a more informative error message when exec.ErrDot occurs.
2022-09-25 16:19:03 +02:00
Michael Eischer
d114e483c4 Add changelog for corrupt data downloads 2022-09-25 11:55:09 +02:00
Michael Eischer
34c1a83340 cache: Drop cache entry if it cannot be processed
Failing to process data requested from the cache usually indicates a
problem with the returned data. Assume that the cache entry is somehow
damaged and retry downloading it once.
2022-09-25 11:55:09 +02:00
Michael Eischer
aa3b1925b4 cache: Simplify loadFromCacheOrDelegate 2022-09-25 11:35:35 +02:00
Michael Eischer
5c6b6edefe retry index, lock and snapshot loading on hash mismatch 2022-09-25 11:35:35 +02:00
Michael Eischer
822422ef03 retry key loading on hash mismatch 2022-09-25 11:35:35 +02:00
Michael Eischer
d6575f53ca Merge pull request #3942 from MichaelEischer/split-cross-compile-test
Split cross compile test
2022-09-24 22:27:08 +02:00
Michael Eischer
78d2312ee9 Merge pull request #3854 from MichaelEischer/sparsefiles
restore: Add support for sparse files
2022-09-24 22:04:02 +02:00
Michael Eischer
46b30b9826 split cross compilation into three parts
The cross compilation tasks are currently the slowest part of the CI
runs. Splitting it into three parts should reduce its time to roughly
that of the windows CI run.
2022-09-24 22:00:25 +02:00
Michael Eischer
bd191ec60b update golang-ci to version 1.49 2022-09-24 22:00:08 +02:00
Michael Eischer
519059cca4 update ci actions 2022-09-24 21:59:36 +02:00
Michael Eischer
19afad8a09 restore: support sparse restores also on windows 2022-09-24 21:39:39 +02:00
Michael Eischer
0f89f443c7 update sparse restore changelog 2022-09-24 21:39:39 +02:00
Michael Eischer
c147422ba5 repository: special case SaveBlob for all zero chunks
Sparse files contain large regions containing only zero bytes. Checking
that a blob only contains zeros is possible with over 100GB/s for modern
x86 CPUs. Calculating sha256 hashes is only possible with 500MB/s (or
2GB/s using hardware acceleration). Thus we can speed up the hash
calculation for all zero blobs (which always have length
chunker.MinSize) by checking for zero bytes and then using the
precomputed hash.

The all zeros check is only performed for blobs with the minimal chunk
size, and thus should add no overhead most of the time. For chunks which
are not all zero but have the minimal chunks size, the overhead will be
below 2% based on the above performance numbers.

This allows reading sparse sections of files as fast as the kernel can
return data to us. On my system using BTRFS this resulted in about
4GB/s.
2022-09-24 21:39:39 +02:00
Michael Eischer
34fe1362da restorer: move zeroPrefixLen to restic package 2022-09-24 21:39:39 +02:00
Michael Eischer
a5ebd5de4b restorer: Fix race condition in partialFile.WriteAt
The restorer can issue multiple calls to WriteAt in parallel. This can
result in unexpected orderings of the Truncate and WriteAt calls and
sometimes too short restored files.
2022-09-24 21:39:39 +02:00
Michael Eischer
5b6a77058a Enable sparseness only conditionally
We can either preallocate storage for a file or sparsify it. This
detects a pack file as sparse if it contains an all zero block or
consists of only one block. As the file sparsification is just an
approximation, hide it behind a `--sparse` parameter.
2022-09-24 21:20:00 +02:00
greatroar
3047bf611c Changelog entry for sparse file restoring 2022-09-24 21:18:48 +02:00
greatroar
5d4568d393 Write sparse files in restorer
This writes files by using (*os.File).Truncate, which resolves to the
truncate system call on Unix.

Compared to the naive loop,

	for _, b := range p {
		if b != 0 {
			return false
		}
	}

the optimized allZero is about 10× faster:

name       old time/op    new time/op     delta
AllZero-8    1.09ms ± 1%     0.09ms ± 1%    -92.10%  (p=0.000 n=10+10)

name       old speed      new speed       delta
AllZero-8  3.84GB/s ± 1%  48.59GB/s ± 1%  +1166.51%  (p=0.000 n=10+10)
2022-09-24 21:18:48 +02:00
Michael Eischer
eb83402d39 Merge pull request #3935 from miles170/master
Only display the message if there were locks to be removed
2022-09-24 20:53:13 +02:00
Michael Eischer
ef58ddd7b1 Merge pull request #3923 from MichaelEischer/fix-flaky-cache-test
cache: fix flaky TestFileSaveConcurrent on windows
2022-09-24 20:52:55 +02:00
Michael Eischer
7fc178aaf4 internal/cache: extend description of cache sharing test failure 2022-09-24 13:07:01 +02:00
Miles Liu
1acbda18f8 Only display the message if there were locks to be removed
`restic unlock` now only shows `successfully removed locks` if there were locks to be removed.
In addition, it also reports the number of the removed lock files.
2022-09-24 19:02:24 +08:00
Michael Eischer
da1a359c8b Merge pull request #3927 from MichaelEischer/faster-index-each
Speed up MasterIndex.Each
2022-09-24 12:35:23 +02:00
Michael Eischer
041a51512a Merge pull request #3780 from jkmw/fix/2578
Remove existing path before restoring a symlink
2022-09-24 12:34:42 +02:00
Michael Eischer
1ebd57247a repository: optimize MasterIndex.Each
Sending data through a channel at very high frequency is extremely
inefficient. Thus use simple callbacks instead of channels.

> name                old time/op  new time/op  delta
> MasterIndexEach-16   6.68s ±24%   0.96s ± 2%  -85.64%  (p=0.008 n=5+5)
2022-09-24 12:21:59 +02:00
Michael Eischer
825b95e313 repository: add benchmark for MasterIndex.Each 2022-09-24 12:21:59 +02:00
greatroar
1220fe9650 internal/cache: Concurrent use of cache not working on Windows 2022-09-17 19:49:44 +02:00
Jerome Küttner
ef618bdd3f use os.Remove if path already exists on symlink restore 2022-09-14 08:14:31 +02:00
Michael Eischer
b48766d7b8 Merge pull request #3928 from restic/rawtaz-doc-b2-s3
doc: Clarify S3 recommendation for B2 slightly
2022-09-13 20:59:50 +02:00
rawtaz
20f1913ef7 doc: Clarify S3 recommendation for B2 slightly
This gives slightly more background to the recommendation for using restic's S3 backend with Backblaze B2.
2022-09-12 17:48:59 +02:00
rawtaz
d79e61ce5d Merge pull request #3925 from hgraeber/add-powershell-completion
Add powershell completion
2022-09-11 01:04:57 +02:00
Herbert Graeber
988b386e8b Add powershell completion
- Add code for powersehll complition available in cobra
- Add documentation for powershell completion
- Add changelog for pr3925
2022-09-11 00:44:12 +02:00
rawtaz
14d09a6081 Merge pull request #3912 from MichaelEischer/cleanup-snapshot-filter-options
Clean up snapshot filter options
2022-09-11 00:18:42 +02:00
Michael Eischer
381da0443a tweak snapshot filter descriptions 2022-09-10 23:50:20 +02:00
Michael Eischer
8b9778d537 Merge pull request #3900 from MichaelEischer/b2-init-timeout
Add timeout for the initial connection to B2
2022-09-10 23:28:59 +02:00
Michael Eischer
17c27400f8 Merge pull request #3921 from MichaelEischer/filter-cleanup-error-handling
filter: deduplicate error handling for pattern validation
2022-09-10 23:24:50 +02:00
Michael Eischer
f76643bd2e Merge pull request #3894 from MichaelEischer/filter-mount-exit-code
Mount should return exit code 0 after pressing Ctrl-C
2022-09-10 23:22:01 +02:00
Michael Eischer
be9ccc186e Merge pull request #3875 from MichaelEischer/fix-fuse-context-cancel
mount: Fix input/output errors for canceled syscalls
2022-09-10 23:20:29 +02:00
Michael Eischer
2363e5c083 Merge pull request #3913 from MichaelEischer/better-migrate-error-message
migrate: Report why an migration cannot be applied
2022-09-09 23:37:25 +02:00
Michael Eischer
8e0ca80547 filter: deduplicate error handling for pattern validation 2022-09-09 23:12:41 +02:00
plumbeo
d66e755ac7 Change uncompressed size calculation to account for the encryption overhead 2022-09-08 10:15:19 +02:00
plumbeo
837b816358 restic stats: print uncompressed size in mode raw-data 2022-09-05 17:38:32 +02:00
Michael Eischer
d6309961c5 deduplicate the snapshot filter cli option setup 2022-09-04 10:27:33 +02:00
Michael Eischer
8b4dd70013 migrate: Report why an migration cannot be applied
Just returning that `Migration upgrade cannot be applied: check failed`
is not too useful when running `migrate upgrade_repo_v2`.
2022-09-03 11:49:31 +02:00
Michael Eischer
7689d6c679 normalize help text for host, tag and path options 2022-09-03 00:06:38 +02:00
Michael Eischer
6c69f08a7b Merge pull request #3905 from DRON-666/haspaths-linear
Reduce quadratic time complexity of `Snapshot.HasPaths`
2022-08-30 20:35:56 +02:00
Michael Eischer
3e70bac56e Merge pull request #3898 from MichaelEischer/fix-copy-hang
don't hang when `copy` uses a single connection
2022-08-30 20:23:39 +02:00
DRON-666
2a630c51c1 Add changelog 2022-08-30 20:22:07 +02:00
DRON-666
d0f1060df7 Fix quadratic time complexity of Snapshot.HasPaths 2022-08-30 04:38:17 +03:00
Michael Eischer
f481ad64c8 Merge pull request #3904 from lbausch/add-newline
Add newline to keep prompt intact
2022-08-29 21:43:18 +02:00
Lorenz Bausch
7ddd803e46 Add newline to keep prompt intact 2022-08-29 17:37:49 +02:00
Michael Eischer
e5b2c4d571 b2: sniff the error that caused init retry loops 2022-08-28 17:46:03 +02:00
Michael Eischer
dc2db2de5e b2: cancel connection setup after a minute
If the connection to B2 fails, the library enters an endless loop.
2022-08-28 14:56:17 +02:00
Michael Eischer
7682149c9d repository: cleanup copy connection count check 2022-08-28 11:40:56 +02:00
Michael Eischer
b03277ead5 repository: don't hang when copying using a single connection 2022-08-28 11:40:31 +02:00
Michael Eischer
1b233c4e2e Merge pull request #2661 from creativeprojects/issue-1734
"Everything is ok" message after retrying
2022-08-28 11:04:59 +02:00
Fred
4042db5169 Add changelog 2022-08-27 22:36:19 +02:00
Fred
be6baaec12 Add success callback to the backend 2022-08-27 22:27:15 +02:00
Fred
baf58fbaa8 Add unit tests 2022-08-27 22:21:06 +02:00
Fred
d629333efe Add function to notify of success after retrying 2022-08-27 22:21:06 +02:00
Alexander Neumann
c169e37139 Merge pull request #3895 from MichaelEischer/refactor-cat-key
cat: Simplify implementation of 'cat key'
2022-08-27 18:40:46 +02:00
Michael Eischer
1b4af0c6e5 cat: Simplify implementation of 'cat key' 2022-08-26 23:21:51 +02:00
Michael Eischer
3174641ca4 add changelog for mount exit code filtering 2022-08-26 23:17:04 +02:00
Michael Eischer
5478ab22c5 mount: return exit code 0 after receiving a SIGINT 2022-08-26 23:07:07 +02:00
Michael Eischer
d768c1c3e4 Allow cleanup handlers to filter the exit code 2022-08-26 23:04:59 +02:00
Michael Eischer
908f7441fe Merge pull request #3885 from MichaelEischer/delete-fixes
Improve reliability of upload retries and B2 file deletions
2022-08-26 22:30:50 +02:00
Michael Eischer
4c90d91d4d backend: Test that failed uploads are not removed for backends with atomic replace 2022-08-26 21:20:52 +02:00
Michael Eischer
694dfa026a add changelog for reliable B2 deletes 2022-08-26 21:20:46 +02:00
MichaelEischer
582167d671 Merge pull request #3882 from MichaelEischer/sftp-init-single-connection
sftp: Only connect once to server during `init`
2022-08-26 21:13:28 +02:00
MichaelEischer
3822ded0b3 Merge pull request #3877 from MichaelEischer/no-env-in-help
Do not include the actual values of environment variables in help output
2022-08-26 20:59:54 +02:00
Michael Eischer
cf0a8d7758 sftp: Only connect once for repository creation
This is especially useful if ssh asks for a password or if closing the
initial connection could return an error due to a problematic server
implementation.
2022-08-26 20:50:40 +02:00
Michael Eischer
dd7cd5b9b3 fuse: remove unused context parameter 2022-08-26 20:48:48 +02:00
Michael Eischer
a0c1ae9f90 mount: Correctly return context.Canceled for interrupted syscalls
bazil/fuse expects us to return context.Canceled to signal that a
syscall was successfully interrupted. Returning a wrapped version of
that error however causes the fuse library to signal an EIO (input/output
error). Thus unwrap context.Canceled errors before returning them.
2022-08-26 20:48:48 +02:00
Michael Eischer
5d0649faaf Update help output in docs 2022-08-26 20:44:01 +02:00
Michael Eischer
faa4597af1 Set name for option values of cli 2022-08-26 20:42:34 +02:00
Michael Eischer
6ed157aee6 Do not include the actual values of environment variables in help output
This results in printing a `(default: $ENV) (default: value)` suffix for
the corresponding options which looks strange. In addition, some of the
environment variables might contain secrets which should not be
displayed.
2022-08-26 20:39:54 +02:00
MichaelEischer
f7808245aa Merge pull request #3878 from MichaelEischer/cheaper-cache-load
cache: Just try to open cache entry without calling stat first
2022-08-26 20:33:36 +02:00
MichaelEischer
bee15dd555 Merge pull request #3879 from MichaelEischer/mem-optimize
Some random (minor) memory-allocation optimizations
2022-08-26 20:33:02 +02:00
MichaelEischer
0e1d082b12 Merge pull request #3886 from MichaelEischer/recommend-s3-over-b2
doc: recommend usage of B2's S3 API
2022-08-26 20:29:05 +02:00
Alexander Neumann
d464543171 Update repo version table 2022-08-25 21:30:25 +02:00
Alexander Neumann
6b40456db7 Set development version for 0.14.0 2022-08-25 19:55:05 +02:00
Alexander Neumann
1bc87e1718 Add version for 0.14.0 2022-08-25 19:55:00 +02:00
Alexander Neumann
c1a5da56e3 Update manpages and auto-completion 2022-08-25 19:55:00 +02:00
Alexander Neumann
193c62dfc3 Generate CHANGELOG.md for 0.14.0 2022-08-25 19:54:02 +02:00
Alexander Neumann
a825e0d409 Prepare changelog for 0.14.0 2022-08-25 19:54:01 +02:00
rawtaz
b824d8cdcc Merge pull request #3891 from MichaelEischer/fix-secondary-repo-typo
Fix typo in the environment variable name for `--from-password-file`
2022-08-24 22:38:47 +02:00
Michael Eischer
a2e89234fc Fix typo in the environment variable name for --from-password-file 2022-08-24 22:25:18 +02:00
MichaelEischer
b4ae05627f Merge pull request #3876 from MichaelEischer/document-aws-session-token
doc: document aws session token
2022-08-24 22:20:26 +02:00
MichaelEischer
bd7bca2b51 Merge pull request #3889 from restic/prepare-0-14
Polish changelog entries
2022-08-24 21:28:44 +02:00
MichaelEischer
be90a565cc Merge pull request #3887 from MichaelEischer/rclone-permanent-error
rclone: Return a permanent error if rclone already exited
2022-08-24 21:19:00 +02:00
Michael Eischer
7a5d29ce24 Further changelog polishing 2022-08-24 21:13:14 +02:00
MichaelEischer
d198a77d86 Merge pull request #3888 from restic/doc-prepare-repo
doc: Improve/clarify preparing and versions of repositories
2022-08-24 20:40:49 +02:00
Leo R. Lundgren
cb5a61c46e doc: Improve/clarify preparing and versions of repositories 2022-08-24 02:08:46 +02:00
Leo R. Lundgren
ee6b9dc492 Polish changelog entries 2022-08-24 01:23:51 +02:00
Michael Eischer
506d92e87c rclone: Return a permanent error if rclone already exited
rclone can exit early for example when the connection to rclone is
relayed for example via ssh: `-o rclone.program='ssh user@example.org
forced-command'`
2022-08-23 22:05:04 +02:00
MichaelEischer
ad6eabbfa5 Merge pull request #3883 from MichaelEischer/update-dependencies
Update dependencies
2022-08-23 21:44:38 +02:00
Michael Eischer
7681a63fdb restic: Cleanup xattr error handling for Solaris
Since xattr 0.4.8 (https://github.com/pkg/xattr/pull/68) returns ENOTSUP
similar to Linux.
2022-08-23 21:25:15 +02:00
Michael Eischer
99e4ccbd94 remain compatible with go 1.15 2022-08-23 21:25:14 +02:00
Michael Eischer
22f46c18f9 downgrade bazil/fuse again to retain macOS support 2022-08-23 21:24:47 +02:00
Michael Eischer
6db979b3a6 update dependencies 2022-08-23 21:24:46 +02:00
MichaelEischer
9cdc8da10f Merge pull request #3881 from MichaelEischer/cleanup-release-helper
Cleanup release helper
2022-08-23 21:10:57 +02:00
MichaelEischer
b51e73e78f Merge pull request #3884 from MichaelEischer/fix-index-saving-progress
repository: Do not report ignored packs in EachByPack
2022-08-23 21:10:07 +02:00
MichaelEischer
98dcd0a887 Merge pull request #3874 from MichaelEischer/disk-wear-note
Add note that larger packs increase disk wear
2022-08-21 22:18:08 +02:00
Michael Eischer
e0d6bf525c doc: fix typo 2022-08-21 19:12:19 +02:00
Michael Eischer
5d0523e2f1 Add note that larger packs increase disk wear 2022-08-21 19:12:19 +02:00
Michael Eischer
c586a5e20f doc: recommend usage of B2's S3 API 2022-08-21 11:39:03 +02:00
Michael Eischer
623556bab6 b2: Increase list size to maximum
Just request as many files as possible in one call to reduce the number
of network roundtrips.
2022-08-21 11:20:03 +02:00
Michael Eischer
de0162ea76 backend/retry: Overwrite failed uploads instead of deleting them
For backends which are able to atomically replace files, we just can
overwrite the old copy, if it is necessary to retry an upload. This has
the benefit of issuing one operation less and might be beneficial if a
backend storage, due to bugs or similar, could mix up the order of the
upload and delete calls.
2022-08-21 11:14:53 +02:00
Michael Eischer
fc506f8538 b2: Repeat deleting until all file versions are removed
When hard deleting the latest file version on B2, this uncovers earlier
versions. If an upload required retries, multiple version might exist
for a file. Thus to reliably delete a file, we have to remove all
versions of it.
2022-08-21 11:11:00 +02:00
Michael Eischer
cc4728d287 repository: Do not report ignored packs in EachByPack
Ignored packs were reported as an empty pack by EachByPack. The most
immediate effect of this is that the progress bar for rebuilding the
index reports processing more packs than actually exist.
2022-08-21 10:38:40 +02:00
Michael Eischer
c3374b3ea5 helper: download modules as first step
There's no use in running that step in parallel.
2022-08-20 12:11:54 +02:00
Michael Eischer
7f0929e519 helper: Reduce number of parallel builds a bit
The go compiler is already parallelized. The high concurrency caused my
podman container to hit a resource limit.
2022-08-20 12:10:48 +02:00
Michael Eischer
ed94678820 helper: cleanups 2022-08-20 12:10:29 +02:00
Michael Eischer
e530d422a0 helper: don't setup cmd paths twice 2022-08-20 12:09:42 +02:00
Michael Eischer
7a992fc794 repository: Reduce buffer reallocations in ForAllIndexes
Previously the buffer was grown incrementally inside `repo.LoadUnpacked`.
But we can do better as we already know how large the index will be.
Allocate a bit more memory to increase the chance that the buffer can be
reused in the future.
2022-08-19 21:13:40 +02:00
Michael Eischer
77b1980d8e repository: MasterIndex.Packs: reduce allocations 2022-08-19 21:10:43 +02:00
Michael Eischer
6ff9517e45 repository: MasterIndex.ListPacks / Index.EachByPack allow earlier GC
Allow earlier garbage collection of some of the intermediate data
structures.
2022-08-19 21:06:33 +02:00
Michael Eischer
ce902aac67 cache: Just try to open cache entry without calling stat first
Instead of first checking whether a file is in the repository cache and
then opening it, we just can open the file. This saves one stat call. If
the file is in the cache, everything is fine and otherwise the code
follows its normal fallback path.
2022-08-19 20:59:06 +02:00
Michael Eischer
b6c86ababe doc: document aws session token 2022-08-19 20:41:15 +02:00
MichaelEischer
0d9ac78437 Merge pull request #3873 from MichaelEischer/gofmt-comments
gofmt comments
2022-08-19 19:54:30 +02:00
MichaelEischer
7e96a5af62 Merge pull request #3872 from MichaelEischer/fuse-fix
mount: Only remember successful snapshot refreshes
2022-08-19 19:21:29 +02:00
Michael Eischer
f414db987d gofmt all files
Apparently the rules for comment formatting have changed with go 1.19.
2022-08-19 19:12:26 +02:00
Michael Eischer
522406b4f0 mount: Only remember successful snapshot refreshes
If the context provided by the fuse library is canceled before the index
was loaded this could lead to missing snapshots.
2022-08-19 19:07:07 +02:00
MichaelEischer
dbca93da28 Merge pull request #3742 from MichaelEischer/from-repo
copy: replace --repo2 with --from-repo
2022-08-19 19:01:04 +02:00
MichaelEischer
b4dfab002a Merge pull request #3691 from greatroar/tag-filenames
Sanitize tags when used as filenames by restic mount
2022-08-19 18:33:44 +02:00
Michael Eischer
2758d76b77 copy: replace --repo2 with --from-repo
`init` and `copy` use `--repo2` with two different meaning which has
proven to be confusing for users. `--from-repo` now consistently marks a
source repository from which data is read. `--repo` is now always the
target/destination repository.
2022-08-19 18:33:26 +02:00
Michael Eischer
af50fe9ac0 mount: Map slashes in tags to underscores
Suggested-by: greatroar <>
2022-08-19 18:17:57 +02:00
MichaelEischer
4cccffab58 Merge pull request #3862 from restic/3861-forget-invalid-policy
forget: Error when invalid unit is given in duration policy
2022-08-18 20:45:26 +02:00
Michael Eischer
2ea6c82cf6 comment cleanup
gofmt reformatted the comment
2022-08-18 20:15:38 +02:00
Michael Eischer
bb27f7408c forget: Fail test if duration parsing error is missing 2022-08-18 20:14:09 +02:00
MichaelEischer
c4b3a154ba Merge pull request #3865 from restic/rawtaz-doc-comp-typo
doc: Fix typo in compression section
2022-08-18 19:37:44 +02:00
rawtaz
2cb2aa31cd doc: Fix typo in compression section 2022-08-12 22:17:38 +02:00
Leo R. Lundgren
6f517858e8 forget: Error when invalid unit is given in duration policy 2022-08-10 13:37:26 +02:00
rawtaz
f0bb4f8708 Merge pull request #3857 from restic/rawtaz-gcs-account
doc: Update links to GCS documentation
2022-08-08 22:52:30 +02:00
Michael Eischer
40c8755b13 doc: Update more links to GCS documentation 2022-08-08 20:00:44 +02:00
rawtaz
f673068dbb doc: Update link to GCS documentation
Updates the link to Google Cloud Storage documentation about creating a service account key.
2022-08-08 12:51:43 +02:00
rawtaz
f26231c9e6 Merge pull request #3852 from MichaelEischer/snapshots-processed
stats: Add snapshots count to json output
2022-08-08 00:05:15 +02:00
MichaelEischer
80e3efffef Merge pull request #3855 from mattxtaz/master
Fix typo with double percentage in help text
2022-08-07 22:44:36 +02:00
mattxtaz
01ab36336f Fix typo with double percentage in help text 2022-08-07 20:21:05 +01:00
Michael Eischer
6e92d852a8 stats: Add snapshots count to json output 2022-08-07 15:44:09 +02:00
MichaelEischer
9ad3ad5972 Merge pull request #3850 from lbausch/go1.19
Update tests to Go 1.19
2022-08-07 14:56:17 +02:00
MichaelEischer
2930a102de Merge pull request #3731 from metalsp0rk/feature/min-packsize-flag
Feature: min packsize flag
2022-08-07 14:54:45 +02:00
MichaelEischer
8fa64a8f99 Merge pull request #3036 from greatroar/refactor-fuse
Clean up internal/fuse
2022-08-07 14:46:30 +02:00
Michael Eischer
f3fdc66b32 restic: Use stable sorting in snapshot policy
sort.Sort is not guaranteed to be stable. Go 1.19 has changed the
sorting algorithm which resulted in changes of the sort order. When
comparing snapshots with identical timestamp but different paths and
tags lists, there is not meaningful order among them. So just keep their
order stable.
2022-08-07 14:10:40 +02:00
Lorenz Bausch
b82f4824f0 Bump golangci-lint version 2022-08-07 14:10:39 +02:00
Lorenz Bausch
0b9b4c52ad Update tests to Go 1.19 2022-08-07 14:10:39 +02:00
Michael Eischer
0b7291b8b2 mount: Fix parent inode used by snapshots dir 2022-08-07 13:03:32 +02:00
greatroar
cfa80e2c6b mount: remove unused inode field from root node 2022-08-07 13:03:26 +02:00
MichaelEischer
74ae76036f Merge pull request #2913 from aawsome/mount-snapshot-slashes
mount: Make snapshots dir structure customizable
2022-08-07 12:27:59 +02:00
MichaelEischer
09497aec02 Merge pull request #3826 from MichaelEischer/debug-log-for-release
Debug log for release build
2022-08-07 12:15:27 +02:00
Michael Eischer
83b4c50ee3 Mention --snapshot-template and --time-template in changelog 2022-08-07 12:13:06 +02:00
Michael Eischer
caa17988a3 fuse: Redesign snapshot dirstruct
Cleanly separate the directory presentation and the snapshot directory
structure. SnapshotsDir now translates the dirStruct into a format
usable by the fuse library and contains only minimal special case rules.
All decisions have moved into SnapshotsDirStructure which now creates a
fully preassembled tree data structure.
2022-08-07 12:13:06 +02:00
Michael Eischer
a3e48da3a3 Add changelog for DEBUG_LOG available in release builds 2022-08-05 23:49:39 +02:00
Michael Eischer
b3cdee66a9 update documentation to reflect DEBUG_LOG for release builds 2022-08-05 23:49:39 +02:00
Michael Eischer
1ed775e3a8 debug: support roundtripper logging also for release builds
Different from debug builds do not use the eofDetectRoundTripper if
logging is disabled.
2022-08-05 23:49:39 +02:00
Michael Eischer
38becfc436 debug: enable debug support for release builds 2022-08-05 23:49:39 +02:00
Michael Eischer
82c268c917 Remove unused hooks mechanism 2022-08-05 23:49:39 +02:00
Michael Eischer
7266f07c87 repository: StreamPack in parts if there are too large gaps
For large pack sizes we might be only interested in the first and last
blob of a pack file. Thus stream a pack file in multiple parts if the
gaps between requested blobs grow too large.
2022-08-05 23:48:36 +02:00
Michael Eischer
55a11c1396 Reword prune --repack-small description 2022-08-05 23:48:36 +02:00
Michael Eischer
eaf43607f9 Add note that pack-size is not an exact limit 2022-08-05 23:48:36 +02:00
Michael Eischer
7f3b2be1e8 s3: Disable multipart uploads below 200MB 2022-08-05 23:48:36 +02:00
Michael Eischer
176b387d98 Always repack very small pack files 2022-08-05 23:48:36 +02:00
Michael Eischer
324935cb80 Only repack small files if there are multiple of them 2022-08-05 23:48:34 +02:00
Michael Eischer
1b076cda97 rename option to --pack-size 2022-08-05 23:47:43 +02:00
Michael Eischer
d7e2892048 Add changelog for packsize option 2022-08-05 23:47:43 +02:00
Michael Eischer
8a44258b6f update restic help snippets in documentation 2022-08-05 23:47:43 +02:00
Michael Eischer
420ddc03c9 rework pack size parameter documentation 2022-08-05 23:47:43 +02:00
Kyle Brennan
e43be84eb8 document minPackSize 2022-08-05 23:47:41 +02:00
Kyle Brennan
1e3f05c3f1 repository: prevent header overfill 2022-08-05 23:47:12 +02:00
Michael Eischer
6a6d313c9a prune: reduce priority of repacking small packs 2022-08-05 23:47:12 +02:00
Kyle Brennan
0269381b8d prune: add repack-small parameter 2022-08-05 23:47:12 +02:00
Michael Eischer
0a6fa602c8 add option for setting min pack size 2022-08-05 23:47:12 +02:00
Michael Eischer
2db7733ee3 fuse: remove unused MetaDir 2022-08-05 23:46:46 +02:00
Michael Eischer
f678f7cb04 fuse: cleanup test 2022-08-05 23:46:46 +02:00
Alexander Weiss
1751afae26 Make snapshots dirs in mount command customizable 2022-08-05 23:46:46 +02:00
Alexander Weiss
57f4003f2f Generalize fuse snapshot dirs implemetation
+ allow "/" in tags and snapshot template
2022-08-05 23:46:46 +02:00
Alexander Weiss
696c18e031 Add possibility to set snapshot ID (used in test) 2022-08-05 23:46:46 +02:00
MichaelEischer
04a8ee80fb Merge pull request #3829 from MichaelEischer/prune-refactor
Split prune into slightly small functions
2022-08-05 23:29:52 +02:00
MichaelEischer
15679be858 Merge pull request #3841 from MichaelEischer/compression-env
Add environment variable RESTIC_COMPRESSION
2022-08-05 23:20:04 +02:00
MichaelEischer
7d14b1baf1 Merge pull request #3845 from greatroar/solaris-xattr
internal/restic: Handle EINVAL for xattr on Solaris
2022-08-04 22:04:52 +02:00
greatroar
ad6ac680af internal/restic: Handle EINVAL for xattr on Solaris
Also make the errors a bit less verbose by not prepending the operation,
since pkg/xattr already does that. Old errors looked like

    Listxattr: xattr.list /myfiles/.zfs/snapshot: invalid argument
2022-08-01 12:45:17 +02:00
MichaelEischer
846d021db5 Merge pull request #3840 from greatroar/sftp-init
Speed up restic init over slow SFTP links
2022-07-31 19:54:59 +02:00
greatroar
b9fa6e05bd Add changelog for #3837/#3840 2022-07-30 22:53:52 +02:00
Michael Eischer
73053674d9 repository: Test fallback to existing blobs 2022-07-30 17:37:07 +02:00
Michael Eischer
e85a21eda2 prune: move code 2022-07-30 17:37:07 +02:00
Michael Eischer
623770eebb repository: try to recover from invalid blob while repacking
If a blob that should be kept is invalid, Repack will now try to request
the blob using LoadBlob. Only return an error if that fails.
2022-07-30 17:37:07 +02:00
Michael Eischer
d0590b7841 prune: Add internal integrity check
After repacking every blob that should be kept must have been repacked.
We have seen a few cases in which a single blob went missing, which
could have been caused by a bitflip somewhere. This sanity check might
help catch some of these cases.
2022-07-30 17:37:07 +02:00
Michael Eischer
5cbde03eae prune: split into smaller functions 2022-07-30 17:37:07 +02:00
Alexander Weiss
7643237da5 prune: separate collecting/printing/pruning 2022-07-30 17:37:07 +02:00
Michael Eischer
5723c72eb1 Add environment variable RESTIC_COMPRESSION 2022-07-30 16:21:53 +02:00
greatroar
2bdc40e612 Speed up restic init over slow SFTP links
pkg/sftp.Client.MkdirAll(d) does a Stat to determine if d exists and is
a directory, then a recursive call to create the parent, so the calls
for data/?? each take three round trips. Doing a Mkdir first should
eliminate two round trips for 255/256 data directories as well as all
but one of the top-level directories.

Also, we can do all of the calls concurrently. This may reintroduce some
of the Stat calls when multiple goroutines try to create the same
parent, but at the default number of connections, that should not be
much of a problem.
2022-07-30 13:09:08 +02:00
greatroar
23ebec717c Remove stale comments from backend/sftp
The preExec and postExec functions were removed in
0bdb131521 from 2018.
2022-07-30 13:07:25 +02:00
MichaelEischer
4ffd479ba4 Merge pull request #3773 from MichaelEischer/efficient-dir-json
Reduce memory usage for large directories/files
2022-07-23 17:47:32 +02:00
Michael Eischer
2ba14160de Add changelog for the optimized tree serialization 2022-07-23 14:49:08 +02:00
Michael Eischer
4a10ebed15 archiver: reduce memory usage for large files
FutureBlob now uses a Take() method as a more memory-efficient way to
retrieve the futures result. In addition, futures are now collected
while saving the file. As only a limited number of blobs can be queued
for uploading, for a large file nearly all FutureBlobs already have
their result ready, such that the FutureBlob object just consumes
memory.
2022-07-23 14:45:07 +02:00
Michael Eischer
b817681a11 archiver: Incrementally serialize tree nodes
That way it is not necessary to keep both the Nodes forming a Tree and
the serialized JSON version in memory.
2022-07-23 14:45:07 +02:00
Michael Eischer
c206a101a3 archiver: unify FutureTree/File into futureNode
There is no real difference between the FutureTree and FutureFile
structs. However, differentiating both increases the size of the
FutureNode struct.

The FutureNode struct is now only 16 bytes large on 64bit platforms.
That way is has a very low overhead if the corresponding file/directory
was not processed yet.

There is a special case for nodes that were reused from the parent
snapshot, as a go channel seems to have 96 bytes overhead which would
result in a memory usage regression.
2022-07-23 14:45:07 +02:00
Michael Eischer
32f4997733 archiver: remove unused fileInfo from progress callback 2022-07-23 14:16:23 +02:00
Michael Eischer
dcb00fd2d1 archiver: cleanup Saver interface 2022-07-23 14:16:23 +02:00
Michael Eischer
79321a195c archiver: remove dead attribute from FutureNode 2022-07-23 14:16:23 +02:00
MichaelEischer
049f4c4144 Merge pull request #3730 from MichaelEischer/stricter-check
Let `check` warn about legacy variants of the repo format version 1
2022-07-23 14:14:50 +02:00
Michael Eischer
3bf53da672 Add changelog for stricter checks 2022-07-23 11:21:26 +02:00
Michael Eischer
5a6f2f9fa0 Fix S3 legacy layout migration 2022-07-23 11:19:32 +02:00
Michael Eischer
04e49924fb checker: Fix S3 legacy layout detection 2022-07-23 11:19:32 +02:00
Michael Eischer
768c890fcb check: Deprecate --check-unused
Unused blobs are not a problem but rather expected to exist now that
prune by default does not remove every unused blob. However, the option
has caused questions from users whether a repository is damaged or not,
so just remove that option.

Note that the remaining code is left intact as it is still useful for
our test cases.
2022-07-23 11:19:32 +02:00
Michael Eischer
fcb3ddf181 check: Complain about usage of s3 legacy layout 2022-07-23 11:19:32 +02:00
Michael Eischer
8b8bd4e8ac check: complain about mixed pack files 2022-07-23 11:19:32 +02:00
MichaelEischer
443cc49afd Merge pull request #3830 from MichaelEischer/cleanup-repo
Extract Load/SaveTree/JSONUnpacked from repository
2022-07-23 10:46:13 +02:00
MichaelEischer
1f5369e072 Merge pull request #3831 from MichaelEischer/move-code
Move code out of the restic package and consolidate backend specific code
2022-07-23 10:33:05 +02:00
MichaelEischer
827ab02eea Merge pull request #3661 from rgammans/azure_sas_support
add SAS authentication option for Azure repos
2022-07-23 10:32:03 +02:00
MichaelEischer
e9c39442fb Merge pull request #3827 from MichaelEischer/backup-doc
backup: clarify usage string
2022-07-23 10:31:33 +02:00
Michael Eischer
9729e6d7ef backend: extract readerat from restic package 2022-07-17 15:29:09 +02:00
Michael Eischer
c44b21d366 restorer: extract hardlinks index from restic package 2022-07-17 13:45:42 +02:00
Michael Eischer
8c11fc3ec9 crypto: move crypto buffer helpers 2022-07-17 13:42:23 +02:00
Michael Eischer
a0cef9f247 limiter: move to internal/backend 2022-07-17 13:40:15 +02:00
Michael Eischer
163ab9c025 mock: move to internal/backend 2022-07-17 13:40:06 +02:00
Michael Eischer
89d3ce852b repository: extract Load/StoreJSONUnpacked
A Load/Store method for each data type is much clearer. As a result the
repository no longer needs a method to load / store json.
2022-07-17 13:22:00 +02:00
Michael Eischer
fbcbd5318c repository: extract LoadTree/SaveTree
The repository has no real idea what a Tree is. So these methods never
belonged there.
2022-07-17 13:11:28 +02:00
MichaelEischer
d9ea1e9ee2 Merge pull request #3290 from aawsome/prune-handle-duplicates
prune: Handle duplicate blobs more efficiently
2022-07-17 11:51:54 +02:00
Michael Eischer
715d457aad prune: code cleanups 2022-07-17 11:41:56 +02:00
Michael Eischer
9be1bd2acc prune: handle very high duplication of some blobs
Suggested-By: Alexander Weiss <alex@weissfam.de>
2022-07-17 11:39:56 +02:00
Alexander Weiss
7478cbf70e prune: Enhance treatment of duplicates 2022-07-17 00:22:23 +02:00
Michael Eischer
b2043e8198 backup: clarify usage string
Using the `--files-from` options it is possible to run `backup` without
specifying any source paths directly on the command line.
2022-07-17 00:03:22 +02:00
Michael Eischer
5639c41b6a azure: Strip ? prefix from sas token 2022-07-16 23:55:18 +02:00
Roger Gammans
64a7ec5341 azure: add SAS authentication option 2022-07-16 23:55:18 +02:00
MichaelEischer
6cbeb4a9f9 Merge pull request #3825 from restic/rawtaz-doc-gdrive
doc: Add note about using rclone for Google Drive
2022-07-16 19:21:59 +02:00
rawtaz
f5c219f5a2 doc: Add note about using rclone for Google Drive
It wasn't clear that Google Cloud Storage and Google Drive are two different services and that one should use the rclone backend for the latter. This commit adds a note with this information.
2022-07-16 13:22:38 +02:00
MichaelEischer
d71b29221b Merge pull request #3822 from JsBergbau/doc-max-compression
Added hint for --compression max in migration process
2022-07-12 23:02:45 +02:00
Michael Eischer
71ff6b77f0 doc: Rework hint to repack with max compression 2022-07-12 21:24:40 +02:00
MichaelEischer
6970d05d47 Merge pull request #3741 from lbausch/repository-wording
Wording: change repo to repository
2022-07-12 21:15:33 +02:00
MichaelEischer
3934480da4 Merge pull request #3819 from lbausch/restore-validate-patterns
restore: validate include/exclude patterns
2022-07-12 20:58:41 +02:00
MichaelEischer
71a0157c2c Merge pull request #3820 from lbausch/patch-1
Fix wording in changelog template
2022-07-12 20:51:52 +02:00
Michael Eischer
2aad6f24b5 doc: update sample help output 2022-07-12 20:49:01 +02:00
Michael Eischer
ec4dfa3c66 Wording: replace further repo occurrences with repository 2022-07-12 20:48:01 +02:00
JsBergbau
8d3f04aefa Added hint for --compression max in migration process
Added hint for --compression max in migration process. Since this is a onetime process users should be aware of this and consider this step.
2022-07-12 00:08:26 +02:00
Lorenz Bausch
b609523582 Add changelog entry 2022-07-09 22:24:56 +02:00
lbausch
ac96a4138d Fix wording in changelog template 2022-07-09 22:13:54 +02:00
Lorenz Bausch
7e36ec279d Test restore fails when using invalid patterns 2022-07-08 20:09:26 +02:00
Lorenz Bausch
be524f0b78 Add testRunRestoreAssumeFailure function 2022-07-08 20:09:25 +02:00
Lorenz Bausch
9a7db6675c Restore: validate provided patterns 2022-07-08 20:09:25 +02:00
Lorenz Bausch
d6e3c7f28e Wording: change repo to repository 2022-07-08 20:05:35 +02:00
MichaelEischer
98a3125ce4 Merge pull request #3816 from mattxtaz/master
Fix minor typo in docs
2022-07-05 21:30:26 +02:00
mattxtaz
87d899c099 Fix minor typo in docs 2022-07-05 20:06:27 +01:00
Jerome Küttner
6f3883c9d2 add changelog 2022-07-05 08:47:48 +02:00
Alexander Neumann
545220803b Merge pull request #3813 from MichaelEischer/fix-blob-saver-data-race
Fix data race in blob_saver
2022-07-04 08:10:01 +02:00
Michael Eischer
ce89018902 Fix data race in blob_saver
After the `BlobSaver` job is submitted, the buffer can be released and
reused by another `FileSaver` even before `BlobSaver.Save` returns. That
FileSaver will the change `buf.Data` leading to wrong backup statistics.

Found by `go test -race ./...`:

WARNING: DATA RACE
Write at 0x00c0000784a0 by goroutine 41:
  github.com/restic/restic/internal/archiver.(*FileSaver).saveFile()
      /home/michael/Projekte/restic/restic/internal/archiver/file_saver.go:176 +0x789
  github.com/restic/restic/internal/archiver.(*FileSaver).worker()
      /home/michael/Projekte/restic/restic/internal/archiver/file_saver.go:242 +0x2af
  github.com/restic/restic/internal/archiver.NewFileSaver.func2()
      /home/michael/Projekte/restic/restic/internal/archiver/file_saver.go:88 +0x5d
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/michael/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x91

Previous read at 0x00c0000784a0 by goroutine 29:
  github.com/restic/restic/internal/archiver.(*BlobSaver).Save()
      /home/michael/Projekte/restic/restic/internal/archiver/blob_saver.go:57 +0x1dd
  github.com/restic/restic/internal/archiver.(*BlobSaver).Save-fm()
      <autogenerated>:1 +0xac
  github.com/restic/restic/internal/archiver.(*FileSaver).saveFile()
      /home/michael/Projekte/restic/restic/internal/archiver/file_saver.go:191 +0x855
  github.com/restic/restic/internal/archiver.(*FileSaver).worker()
      /home/michael/Projekte/restic/restic/internal/archiver/file_saver.go:242 +0x2af
  github.com/restic/restic/internal/archiver.NewFileSaver.func2()
      /home/michael/Projekte/restic/restic/internal/archiver/file_saver.go:88 +0x5d
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/michael/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x91
2022-07-03 14:47:53 +02:00
MichaelEischer
b6a38d43b3 Merge pull request #3611 from MichaelEischer/auto-concurrency
Adjust worker goroutines to number of backend connections
2022-07-03 12:33:32 +02:00
Michael Eischer
3af9c2cc58 Document automatic CPU/IO-concurrency 2022-07-03 12:19:26 +02:00
Michael Eischer
6f53ecc1ae adapt workers based on whether an operation is CPU or IO-bound
Use runtime.GOMAXPROCS(0) as worker count for CPU-bound tasks,
repo.Connections() for IO-bound task and a combination if a task can be
both. Streaming packs is treated as IO-bound as adding more worker
cannot provide a speedup.

Typical IO-bound tasks are download / uploading / deleting files.
Decoding / Encoding / Verifying are usually CPU-bound. Several tasks are
a combination of both, e.g. for combined download and decode functions.
In the latter case add both limits together. As the backends have their
own concurrency limits restic still won't download more than
repo.Connections() files in parallel, but the additional workers can
decode already downloaded data in parallel.
2022-07-03 12:19:26 +02:00
MichaelEischer
cd50feb66f Merge pull request #3489 from MichaelEischer/async-pack-uploads
Asynchronously upload pack files
2022-07-03 11:56:05 +02:00
Michael Eischer
74df9d5998 Add changelog for async pack uploads 2022-07-03 11:34:01 +02:00
Michael Eischer
dbb5860dc9 Document connections and compression option 2022-07-03 11:19:24 +02:00
Michael Eischer
753e56ee29 repository: Limit to a single pending pack file
Use only a single not completed pack file to keep the number of open and
active pack files low. The main change here is to defer hashing the pack
file to the upload step. This prevents the pack assembly step to become
a bottleneck as the only task is now to write data to the temporary pack
file.

The tests are cleaned up to no longer reimplement packer manager
functions.
2022-07-02 22:42:34 +02:00
Michael Eischer
fa25d6118e archiver: Reduce tree saver concurrency
Large amount of tree savers have no obvious benefit, however they can
increase the amount of (potentially large) trees kept in memory.
2022-07-02 22:42:34 +02:00
Michael Eischer
bba1e81719 archiver: Limit blob saver count to GOMAXPROCS
Now with the asynchronous uploaders there's no more benefit from using
more blob savers than we have CPUs. Thus use just one blob saver for
each CPU we are allowed to use.
2022-07-02 22:42:34 +02:00
Michael Eischer
120ccc8754 repository: Rework blob saving to use an async pack uploader
Previously, SaveAndEncrypt would assemble blobs into packs and either
return immediately if the pack is not yet full or upload the pack file
otherwise. The upload will block the current goroutine until it
finishes.

Now, the upload is done using separate goroutines. This requires changes
to the error handling. As uploads are no longer tied to a SaveAndEncrypt
call, failed uploads are signaled using an errgroup.

To count the uploaded amount of data, the pack header overhead is no
longer returned by `packer.Finalize` but rather by
`packer.HeaderOverhead`. This helper method is necessary to continue
returning the pack header overhead directly to the responsible call to
`repository.SaveBlob`. Without the method this would not be possible,
as packs are finalized asynchronously.
2022-07-02 22:42:34 +02:00
Michael Eischer
bba4c69a2a tag: Remove unnecessary flush call 2022-07-02 22:42:23 +02:00
MichaelEischer
3e1de52e0a Merge pull request #3805 from greatroar/global
cmd/restic, limiter: Move config knowledge to internal packages
2022-07-02 21:56:35 +02:00
MichaelEischer
621023a50b Merge pull request #3772 from MichaelEischer/fix-mixed-index
rebuild-index: correctly rebuild index for mixed packs
2022-07-02 20:10:02 +02:00
MichaelEischer
90e9c5c4cc Merge pull request #3729 from MichaelEischer/full-ids-in-check
Include full IDs in check output
2022-07-02 20:09:39 +02:00
MichaelEischer
7137034517 Merge pull request #3811 from restic/fix-secret-string-crash
Don't crash if SecretString is uninitialized
2022-07-02 19:56:36 +02:00
Michael Eischer
cdaf9b4f26 Don't crash if SecretString is uninitialized 2022-07-02 19:44:28 +02:00
Michael Eischer
5e0f1c3cef check: remove dead code 2022-07-02 19:28:57 +02:00
Michael Eischer
0df022fa6d check: Print full ids
The short ids are not always unique. In addition, recovering from
damages is easier when having the full ids as that makes it easier to
access the corresponding files.
2022-07-02 19:28:57 +02:00
Michael Eischer
04c23fa95d rebuild-index: correctly rebuild index for mixed packs
For mixed packs, data and tree blobs were stored in separate index
entries. This results in warning from the check command and maybe other
problems.
2022-07-02 19:24:02 +02:00
MichaelEischer
bb5f196b09 Merge pull request #3733 from restic/improve-stats
Improve stats
2022-07-02 19:07:31 +02:00
MichaelEischer
c16f989d4a Merge pull request #3470 from MichaelEischer/sanitize-debug-log
Sanitize debug log
2022-07-02 19:00:54 +02:00
Michael Eischer
00d7fcff96 extend compression feature changelog entry 2022-07-02 18:55:59 +02:00
Michael Eischer
a6e9e08034 Account for pack header overhead at each entry
This will miss the pack header crypto overhead and the length field,
which only amount to a few bytes per pack file.
2022-07-02 18:55:58 +02:00
Michael Eischer
856d5e4303 stats: return storage size for raw-data mode
raw-data summed up the size of the blob plaintexts. However, with
compression this makes little sense as the storage size in the
repository is lower due to compression. Thus sum up the actual size each
blob takes in the repository.
2022-07-02 18:55:12 +02:00
Alexander Neumann
6c4ceaf1e7 Print number of bytes added to the repo
This includes optional compression and crypto overhead.
2022-07-02 18:55:12 +02:00
Alexander Neumann
99634c0936 Return real size from SaveBlob 2022-07-02 18:55:12 +02:00
MichaelEischer
fdc53a9d32 Merge pull request #3787 from MichaelEischer/refactor-repository
repository: (Mostly) index-related cleanups
2022-07-02 18:54:04 +02:00
Michael Eischer
6923353c43 redact swift auth token in debug output 2022-07-02 18:47:35 +02:00
Michael Eischer
5a11d14082 redacted keys/token in backend config debug log 2022-07-02 18:47:35 +02:00
Michael Eischer
0936d864a4 redact http authorization header in debug log output 2022-07-02 18:47:35 +02:00
Michael Eischer
ec7c9ce88b drop unused repository.Loader interface 2022-07-02 18:39:59 +02:00
Michael Eischer
2cd7e90ad1 repository: cleanup 2022-07-02 18:39:59 +02:00
Michael Eischer
c1a8fa4290 repository: remove unused packIDToIndex field 2022-07-02 18:39:59 +02:00
Michael Eischer
e68c3a4e62 repository: simplify CreateIndexFromPacks 2022-07-02 18:39:59 +02:00
Michael Eischer
1974ad7ce2 repository: hide MasterIndex.FinalizeFullIndexes / FinalizeNotFinalIndexes 2022-07-02 18:39:59 +02:00
Michael Eischer
ef53ca4a5a repository: remove MasterIndex.All() 2022-07-02 18:39:59 +02:00
Michael Eischer
bf81bf0795 repository: Properly set id for finalized index
As MergeFinalIndex and index uploads can occur concurrently, it is
necessary for MergeFinalIndex to check whether the IDs for an index were
already set before merging it. Otherwise, we'd loose the ID of an index
which is set _after_ uploading it.
2022-07-02 18:39:59 +02:00
Michael Eischer
e0a7852b8b repository: remove unused (Master)Index.Count 2022-07-02 18:39:58 +02:00
Michael Eischer
8ef2968f28 repository: remove unused index.ListPack 2022-07-02 18:39:12 +02:00
Michael Eischer
e4f20dea61 repository: inline index.encode 2022-07-02 18:39:12 +02:00
Michael Eischer
fe5a8e137a repository: remove unused index.Store 2022-07-02 18:39:12 +02:00
Michael Eischer
628ae799ca repository: make flushPacks private 2022-07-02 18:39:12 +02:00
Michael Eischer
ed8aa15376 repository: add Save method to MasterIndex interface 2022-07-02 18:38:56 +02:00
Michael Eischer
a77d5c4d11 repository: index saving belongs into the MasterIndex 2022-07-02 18:38:56 +02:00
MichaelEischer
19641bf828 Merge pull request #3810 from greatroar/revert-3786
Revert "restic prune: Merge three loops over the index"
2022-07-01 23:13:39 +02:00
greatroar
a0fa9c6e9f Revert "restic prune: Merge three loops over the index"
This reverts commit 8bdfcf779f.
Should fix #3809. Also needed to make #3290 apply cleanly.
2022-06-30 15:27:34 +02:00
greatroar
90d2c0502b cmd/restic, limiter: Move config knowledge to internal packages
The GlobalOptions struct now embeds a backend.TransportOptions, so it
doesn't need to construct one in open and create. The upload and
download limits are similarly now a struct in internal/limiter that is
embedded in GlobalOptions.
2022-06-22 18:29:58 +02:00
MichaelEischer
bc96879d41 Merge pull request #3785 from MichaelEischer/replace-tomb-usage
Remove usage of tomb package
2022-06-19 14:42:48 +02:00
MichaelEischer
307f14604f Merge pull request #3795 from greatroar/sema
backend: Move semaphores to a dedicated package
2022-06-18 17:12:01 +02:00
MichaelEischer
19581dbc18 Merge pull request #3786 from greatroar/prune
restic prune: Merge three loops over the index
2022-06-18 16:54:50 +02:00
greatroar
8bdfcf779f restic prune: Merge three loops over the index
There were three loops over the index in restic prune, to find
duplicates, to determine sizes (in pack.Size) and to generate packInfos.
These three are now one loop. This way, prune doesn't need to construct
a set of duplicate blobs, pack.Size doesn't need to contain special
logic for prune's use case (the onlyHdr argument) and pack.Size doesn't
need to construct a map only to have it immediately transformed into a
different map.

Some quick testing on a 160GiB local repo doesn't show running time or
memory use of restic prune --dry-run changing significantly.
2022-06-18 10:40:33 +02:00
greatroar
910d917b71 backend: Move semaphores to a dedicated package
... called backend/sema. I resisted the temptation to call the main
type sema.Phore. Also, semaphores are now passed by value to skip a
level of indirection when using them.
2022-06-18 10:01:58 +02:00
MichaelEischer
2c893fe43c Merge pull request #3798 from greatroar/errors
all: Move away from pkg/errors, easy cases
2022-06-17 19:01:40 +02:00
greatroar
f92ecf13c9 all: Move away from pkg/errors, easy cases
github.com/pkg/errors is no longer getting updates, because Go 1.13
went with the more flexible errors.{As,Is} function. Use those instead:
errors from pkg/errors already support the Unwrap interface used by 1.13
error handling. Also:

* check for io.EOF with a straight ==. That value should not be wrapped,
  and the chunker (whose error is checked in the cases changed) does not
  wrap it.
* Give custom Error methods pointer receivers, so there's no ambiguity
  when type-switching since the value type will no longer implement error.
* Make restic.ErrAlreadyLocked private, and rename it to
  alreadyLockedError to match the stdlib convention that error type
  names end in Error.
* Same with rest.ErrIsNotExist => rest.notExistError.
* Make s3.Backend.IsAccessDenied a private function.
2022-06-14 08:36:38 +02:00
MichaelEischer
0c0e7b6957 Merge pull request #3776 from wjiec/bugfix/maxkeys-in-search
Limit number of key files tested while opening a repository
2022-06-12 15:51:10 +02:00
Michael Eischer
c9ef873192 tweak password test count changelog 2022-06-12 15:39:06 +02:00
Michael Eischer
d08549f0ae fix flaky key test 2022-06-12 14:19:06 +02:00
Jayson Wang
f144920ed5 fix handling of maxKeys in SearchKey 2022-06-12 14:19:06 +02:00
Alexander Neumann
1dd4b9b60e Merge pull request #3788 from greatroar/sftp-posix-rename
backend/sftp: Support atomic rename
2022-06-06 19:39:48 +02:00
Alexander Neumann
07114ccb21 Merge pull request #3789 from greatroar/fix-loadblob
internal/repository: Fix LoadBlob + fuzz test
2022-06-06 19:33:42 +02:00
greatroar
c9557b2822 internal/repository: Fix LoadBlob + fuzz test
When given a buf that is big enough for a compressed blob but not its
decompressed contents, the copy at the end of LoadBlob would skip the
last part of the contents.

Fixes #3783.
2022-06-06 17:02:28 +02:00
greatroar
fa8f02292e backend/sftp: Support atomic rename
... if the server has posix-rename@openssh.com.
OpenSSH introduced this extension in 2008:
7c29661471
2022-06-06 13:40:42 +02:00
Alexander Neumann
7d64aa7f57 Merge pull request #3784 from MichaelEischer/cleanup-migrate-check
migrate: Cleanup option to request repository check
2022-06-05 17:37:45 +02:00
Michael Eischer
853ceb3bec get rid of tomb package 2022-06-05 15:54:57 +02:00
Michael Eischer
e002b09d57 archiver: free workers once finished 2022-06-05 15:48:10 +02:00
Michael Eischer
408ac1a0c2 archiver: remove tomb usage 2022-06-05 15:47:52 +02:00
Michael Eischer
5eba1217e7 migrate: Cleanup option to request repository check 2022-06-04 23:45:00 +02:00
MichaelEischer
0cb6b3d80a Merge pull request #3778 from greatroar/ellipsis
cmd/restic: Remove trailing "..." from progress messages
2022-06-04 19:16:00 +02:00
MichaelEischer
60ca6b1418 Merge pull request #3774 from greatroar/archiver-pool
archiver: Remove cleanup goroutine from BufferPool
2022-06-04 18:50:24 +02:00
Jerome Küttner
9adaa6e240 Delete existing path before restoring a symlink 2022-06-01 17:26:25 +02:00
greatroar
b7c990871f cmd/restic: Remove trailing "..." from progress messages
These were added after message since the last refactor of the progress
printing code. Also skips an allocation in the common case.
2022-05-31 19:06:26 +02:00
greatroar
0db1d11b2e archiver: Remove cleanup goroutine from BufferPool
This isn't doing anything. Channels should get cleaned up by the GC when
the last reference to them disappears, just like all other data
structures. Also inlined BufferPool.Put in Buffer.Release, its only
caller.
2022-05-29 17:09:16 +02:00
Alexander Neumann
74f7fe2b98 Merge pull request #3767 from MichaelEischer/fix-prune-empty-snapshot
prune: Fix crash on snapshot loading error
2022-05-29 16:52:45 +02:00
Alexander Neumann
d2c5843c68 Merge pull request #3704 from MichaelEischer/compression-migrations
Support migration to repository format with compression
2022-05-29 15:52:21 +02:00
Alexander Neumann
78a21bbccf Merge pull request #3752 from MichaelEischer/fix-dir-sync-errors
local: Ignore additional errors for directory syncing
2022-05-29 12:54:51 +02:00
MichaelEischer
2ce8587598 Merge pull request #3771 from greatroar/id-marshaljson
internal/restic: Custom ID.MarshalJSON
2022-05-28 16:24:51 +02:00
Michael Eischer
a73fc31b50 Fix linter check 2022-05-28 16:13:46 +02:00
greatroar
dde8e9e296 internal/restic: Custom ID.MarshalJSON
This skips an allocation. internal/archiver benchmarks, Linux/amd64:

name                     old time/op    new time/op    delta
ArchiverSaveFileSmall-8    3.94ms ± 6%    3.91ms ± 6%    ~     (p=0.947 n=20+20)
ArchiverSaveFileLarge-8     304ms ± 3%     301ms ± 4%    ~     (p=0.265 n=18+18)

name                     old speed      new speed      delta
ArchiverSaveFileSmall-8  1.04MB/s ± 6%  1.05MB/s ± 6%    ~     (p=0.803 n=20+20)
ArchiverSaveFileLarge-8   142MB/s ± 3%   143MB/s ± 4%    ~     (p=0.421 n=18+19)

name                     old alloc/op   new alloc/op   delta
ArchiverSaveFileSmall-8    17.9MB ± 0%    17.9MB ± 0%  -0.01%  (p=0.000 n=19+19)
ArchiverSaveFileLarge-8     382MB ± 2%     382MB ± 1%    ~     (p=0.687 n=20+19)

name                     old allocs/op  new allocs/op  delta
ArchiverSaveFileSmall-8       540 ± 1%       528 ± 0%  -2.19%  (p=0.000 n=19+19)
ArchiverSaveFileLarge-8     1.93k ± 3%     1.79k ± 4%  -7.06%  (p=0.000 n=20+20)
2022-05-27 12:26:37 +02:00
Alexander Neumann
bc27c370e7 Update gopkg.in/yaml
This fixes a panic in invalid input, but I think we aren't affected.
2022-05-26 14:23:49 +02:00
Alexander Neumann
9e30152f3c Merge pull request #3770 from lbausch/update-minio-go
Update github.com/minio/minio-go/v7 to v7.0.27
2022-05-26 14:21:19 +02:00
Lorenz Bausch
cc3f8d3732 Update github.com/minio/minio-go/v7 to v7.0.27
This version adds support for Cloudflare R2, as discussed in #3757
2022-05-26 13:05:13 +02:00
Michael Eischer
c8e1ac4049 prune: Don't print stack trace if snapshot can't be loaded 2022-05-23 22:38:45 +02:00
Michael Eischer
173695104c prune: Fix crash on empty snapshot 2022-05-23 22:32:59 +02:00
MichaelEischer
ded783dd61 Merge pull request #3762 from marigbede/marigbede-gs-documentation-fix
Update 030_preparing_a_new_repo.rst
2022-05-23 22:10:38 +02:00
Arigbede Moses
2aa75d6272 Update 030_preparing_a_new_repo.rst 2022-05-18 22:03:59 +01:00
MichaelEischer
88a8701fb5 Merge pull request #3734 from lbausch/validate-patterns
Validate exclude patterns
2022-05-14 16:20:15 +02:00
MichaelEischer
b2a2e5f727 Merge pull request #3753 from greatroar/indexmap-alloc
repository: Re-tune indexmap allocation strategy
2022-05-14 15:44:08 +02:00
MichaelEischer
b52c631bd3 Merge pull request #3754 from greatroar/simplify-hashing
hashing: Fix up comments
2022-05-14 15:33:51 +02:00
Lorenz Bausch
e7df66cc91 Add changelog entry for validating exclude patterns 2022-05-11 22:41:05 +02:00
Lorenz Bausch
36bd464e8c Add tests for validating exclude patterns 2022-05-11 22:41:00 +02:00
greatroar
39a335e690 hashing: Fix up comments 2022-05-11 21:36:10 +02:00
greatroar
5141228e0c repository: Re-tune indexmap allocation strategy
fd05037e1a changed the allocation batch
size from 256 to 128 under the assumption that an indexEntry is 60 bytes
on amd64, but it's 64: structs are padded out to a multiple of 8 for
alignment reasons. That means we'd waste no space in malloc even without
the batch allocation, at least on 64-bit machines. While that strategy
cuts the overallocation down dramatically for many small indexes, it also
seems to slow allocation down (Go 1.18, Linux, amd64, -benchtime=2s):

    name                   old time/op    new time/op    delta
    DecodeIndex-8             4.67s ± 5%     4.60s ± 1%      ~     (p=0.953 n=10+5)
    DecodeIndexParallel-8     4.67s ± 3%     4.60s ± 1%      ~     (p=0.953 n=10+5)
    IndexHasUnknown-8        37.8ns ± 8%    36.5ns ±14%      ~     (p=0.841 n=5+5)
    IndexHasKnown-8          38.5ns ±12%    37.7ns ±10%      ~     (p=0.968 n=5+5)
    IndexAlloc-8              615ms ±18%     607ms ± 1%      ~     (p=1.000 n=10+5)
    IndexAllocParallel-8      245ms ±11%     285ms ± 6%   +16.40%  (p=0.001 n=10+5)
    MasterIndexAlloc-8        286ms ± 9%     275ms ± 2%      ~     (p=1.000 n=10+5)
    LoadIndex/v1-8           27.0ms ± 4%    26.8ms ± 1%      ~     (p=0.690 n=5+5)
    LoadIndex/v2-8           22.4ms ± 1%    22.8ms ± 2%    +1.48%  (p=0.016 n=5+5)

    name                   old alloc/op   new alloc/op   delta
    IndexAlloc-8              446MB ± 0%     446MB ± 0%    -0.00%  (p=0.000 n=8+4)
    IndexAllocParallel-8      446MB ± 0%     446MB ± 0%    -0.00%  (p=0.008 n=8+5)
    MasterIndexAlloc-8        213MB ± 0%     159MB ± 0%   -25.47%  (p=0.000 n=10+5)

    name                   old allocs/op  new allocs/op  delta
    IndexAlloc-8               913k ± 0%     2632k ± 0%  +188.19%  (p=0.008 n=5+5)
    IndexAllocParallel-8       913k ± 0%     2632k ± 0%  +188.21%  (p=0.008 n=5+5)
    MasterIndexAlloc-8         318k ± 0%     1172k ± 0%  +267.86%  (p=0.008 n=5+5)

Instead, this patch sets a batch size of 4, which means no space is
wasted by malloc on 64-bit and very little on 32-bit. It still gets very
close to the savings from not allocating in batches, without requiring
special code for bits.UintSize==64. Benchmark results, again for
Linux/amd64:

    name                   old time/op    new time/op    delta
    DecodeIndex-8             4.67s ± 5%     4.83s ± 9%     ~     (p=0.315 n=10+10)
    DecodeIndexParallel-8     4.67s ± 3%     4.68s ± 4%     ~     (p=0.315 n=10+10)
    IndexHasUnknown-8        37.8ns ± 8%    44.5ns ±19%     ~     (p=0.095 n=5+5)
    IndexHasKnown-8          38.5ns ±12%    36.9ns ± 8%     ~     (p=0.690 n=5+5)
    IndexAlloc-8              615ms ±18%     628ms ±18%     ~     (p=0.218 n=10+10)
    IndexAllocParallel-8      245ms ±11%     262ms ± 9%   +7.02%  (p=0.043 n=10+10)
    MasterIndexAlloc-8        286ms ± 9%     287ms ±13%     ~     (p=1.000 n=10+10)
    LoadIndex/v1-8           27.0ms ± 4%    26.8ms ± 0%     ~     (p=1.000 n=5+5)
    LoadIndex/v2-8           22.4ms ± 1%    22.5ms ± 0%     ~     (p=0.056 n=5+5)

    name                   old alloc/op   new alloc/op   delta
    IndexAlloc-8              446MB ± 0%     446MB ± 0%     ~     (p=1.000 n=8+10)
    IndexAllocParallel-8      446MB ± 0%     446MB ± 0%   -0.00%  (p=0.000 n=8+8)
    MasterIndexAlloc-8        213MB ± 0%     160MB ± 0%  -25.02%  (p=0.000 n=10+9)

    name                   old allocs/op  new allocs/op  delta
    IndexAlloc-8               913k ± 0%     1333k ± 0%  +45.94%  (p=0.000 n=8+10)
    IndexAllocParallel-8       913k ± 0%     1333k ± 0%  +45.94%  (p=0.000 n=8+8)
    MasterIndexAlloc-8         318k ± 0%      525k ± 0%  +64.99%  (p=0.000 n=10+10)

The allocation method indexmap.newEntry has also been rewritten in a
form that is a few instructions shorter.
2022-05-11 21:22:14 +02:00
Michael Eischer
48a0d83143 local: Ignore additional errors for directory syncing
Apparently SMB/CIFS on Linux/macOS returns somewhat random errnos when
trying to sync a windows share which does not support calling fsync for
a directory.
2022-05-11 20:37:59 +02:00
MichaelEischer
ac36fda155 Merge pull request #3749 from greatroar/simplify-hashing
hashing: Remove io.WriterTo implementation
2022-05-11 20:03:43 +02:00
MichaelEischer
df554e5f69 Merge pull request #3748 from greatroar/runworkers
repository: Remove RunWorkers, report ctx.Err()
2022-05-11 19:38:46 +02:00
greatroar
54b8337813 hashing: Remove io.WriterTo implementation
This functionality has gone unused since
4b3dc415ef changed hashing.Reader's only
client to use ioutil.ReadAll on a bufio.Reader wrapping the hashing
Reader.

Reverts bcb852a8d0.
2022-05-10 23:41:18 +02:00
greatroar
2e0f1f5113 repository: Remove RunWorkers, report ctx.Err()
This removes RunWorkers, which had become mere overhead by successive
refactors. It also ensures that each former user of that function
returns any context error that occurs, so failure to complete an
operation is always reported as an error.
2022-05-10 22:26:00 +02:00
MichaelEischer
47c56dea5c Merge pull request #3746 from greatroar/cache-lstat
cache: Don't Lstat before creating CACHEDIR.TAG
2022-05-10 20:31:15 +02:00
Alexander Neumann
c270ab1e08 Merge pull request #3744 from MichaelEischer/fix-windows-temp-file
Fix error on temp file deletion on windows
2022-05-10 19:53:25 +02:00
greatroar
2da377c582 cache: Don't Lstat before creating the tag file
The tag file is opened with O_CREATE|O_EXCL and ErrExist is handled, so
we don't need to check for existence first.
2022-05-10 18:52:39 +02:00
Michael Eischer
ae7e51382a Fix error on temp file deletion on windows
Apparently it can take a moment between closing a tempfile marked as
DELETE_ON_CLOSE and it actually being deleted. During that time the file
is inaccessible. Thus just skip deleting the temp file on windows.
2022-05-09 22:43:26 +02:00
Michael Eischer
5c6db534d4 Add compression migration support to changelog 2022-05-09 22:39:02 +02:00
Michael Eischer
c1bbbcd0dc migrate: Allow migrations to request a check run
This is currently only used by upgrade_repo_v2.
2022-05-09 22:31:30 +02:00
Michael Eischer
59eb132dcd check: Better differentiate between warnings and errors 2022-05-09 22:31:30 +02:00
Michael Eischer
5815f727ee checker: convert error type to use pointer-receivers 2022-05-09 22:31:30 +02:00
Michael Eischer
4faff0debe doc: Describe repository upgrade process 2022-05-09 22:31:30 +02:00
Michael Eischer
e36a40db10 upgrade_repo_v2: Use atomic replace for supported backends 2022-05-09 22:31:30 +02:00
Michael Eischer
7559d2f105 Document repository version and minimum restic version 2022-05-09 22:31:30 +02:00
Michael Eischer
381bd94c6c prune: Add option to repack uncompressed data 2022-05-09 22:31:30 +02:00
Michael Eischer
5406743102 prune: Automatically repack uncompressed trees for repo v2
Tree packs are cached locally at clients and thus benefit a lot from
being compressed. Ensure this be having prune always repack pack files
containing uncompressed trees.
2022-05-09 22:31:30 +02:00
Alexander Neumann
c8c0d659ec Add migration to compress all data 2022-05-09 22:31:30 +02:00
Alexander Neumann
8c244214bf Add tests for upgrade migration 2022-05-09 22:31:30 +02:00
Alexander Neumann
a5f1d318ac Try to make repo upgrade migration more failsafe 2022-05-09 22:31:30 +02:00
Alexander Neumann
82ed5a3a15 Add repo upgrade migration 2022-05-09 22:31:30 +02:00
Alexander Neumann
3af6c180e4 Improve migrate command 2022-05-09 22:31:30 +02:00
Michael Eischer
92816fa966 init: Enable compression support by default 2022-05-09 22:31:30 +02:00
MichaelEischer
ab49c14621 Merge pull request #3740 from MichaelEischer/fix-restore-size-stats
Fix restore size calculation for multiple snapshots
2022-05-09 21:39:45 +02:00
Michael Eischer
2c07f7fff3 stats: hardlinks only reduce restore within a snapshot
The `stats` command checks inodes to not count hardlinked files multiple
times into the restore size. This check applies across all snapshots and
not only within snapshots. As a result the result size was far too low
when calculating it for multiple snapshots and it would vary depending
on the order in which snapshots were listed.
2022-05-09 21:26:24 +02:00
Lorenz Bausch
9fb81c4246 Validate exclude patterns 2022-05-07 21:12:47 +02:00
Lorenz Bausch
e7fd200237 Keep original pattern for later use 2022-05-07 21:08:09 +02:00
MichaelEischer
cc8a03b1d0 Merge pull request #3735 from HenrikBengtsson/master
DOCS: Incorrectly used SI units when IEC units were meant (fix #3669)
2022-05-07 11:15:27 +02:00
Henrik Bengtsson
9bb532672a DOCS: Incorrectly used SI units when IEC units were meant (fix #3669) 2022-05-02 21:21:39 -07:00
MichaelEischer
26c333325c Merge pull request #3715 from ema/master
doc: specify AWS Region via AWS_DEFAULT_REGION
2022-05-02 21:46:56 +02:00
Emanuele Rocca
38c0531b52 doc: specify AWS Region via AWS_DEFAULT_REGION
If no specific Region is mentioned in RESTIC_REPOSITORY, AWS defaults to
us-east-1. For this reason, users that follow the tutorial and create
their S3 bucket in any other region get the following error:

"Fatal: create repository at [...] client.BucketExists"

Explicitly specifying the AWS region name fixes the issue.
2022-05-02 21:26:58 +02:00
Alexander Neumann
fb5b9370f3 Merge pull request #3728 from MichaelEischer/debug-examine-upload
Add support to reupload blobs in `debug examine`
2022-04-30 20:39:19 +02:00
Alexander Neumann
ffbd48c0c6 Merge pull request #3481 from MichaelEischer/recover-enospace
Recover from no free space errors
2022-04-30 20:22:26 +02:00
Michael Eischer
95bcc9ea31 debug: Support pack ID prefixes in debug examine 2022-04-30 20:20:31 +02:00
Michael Eischer
2d6a943911 debug: Add switch to upload blobs extracted by debug examine
This simplifies salvaging a damaged pack file. Reuploading tree blobs
was previously not possible.
2022-04-30 20:20:31 +02:00
Alexander Neumann
9af499d8a4 Tidy go.sum 2022-04-30 20:16:09 +02:00
Alexander Neumann
2e3d23c1d7 Update github.com/klauspost/compress 2022-04-30 20:03:21 +02:00
Michael Eischer
dbbeac7174 prune: Add unsafe option to recover from no free space
The new option allows prune to operate with nearly no scratch space by only removing
no longer necessary pack files and first deleting the index before
rebuilding it. By first deleting the index it becomes safe to just
delete no longer necessary pack files. However, as a downside there's
now the risk that the repository becomes inaccessible if prune fails.

To recover from that problem a user might have to manually delete the
repository index and then run (a full) `rebuild-index` again.
2022-04-30 19:21:07 +02:00
Michael Eischer
cf5cb673fb repository: Use existing method to collect pack ids 2022-04-30 19:14:21 +02:00
Michael Eischer
b335cb6285 repository: Refactor index IDs collection 2022-04-30 19:14:21 +02:00
MichaelEischer
9c047f170a Merge pull request #3419 from DanielG/relax-file-modes
Enable admin to decide group access to repository files
2022-04-30 16:39:37 +02:00
Daniel Gröber
f31b4f29c1 Use config file modes to derive new dir/file modes
Fixes #2351
2022-04-30 15:59:51 +02:00
Alexander Neumann
71c653f9e0 Merge pull request #3727 from MichaelEischer/changelog-3475
Add changelog for local/sftp connection limit
2022-04-30 14:05:21 +02:00
Michael Eischer
29a8f92967 Add changelog for local/sftp connection limit 2022-04-30 13:36:47 +02:00
MichaelEischer
ac9324aeaf Merge pull request #3666 from MichaelEischer/compression
Implement compression support
2022-04-30 11:49:05 +02:00
Alexander Neumann
dc5adef255 Add documentation for --repository-version 2022-04-30 11:34:10 +02:00
Michael Eischer
4b01b06f2f repository: Test compressed blobs in StreamPack 2022-04-30 11:34:10 +02:00
Michael Eischer
bcab548617 pack: slightly expand testing of compressed blobs 2022-04-30 11:34:10 +02:00
Michael Eischer
ec2b25565a repository: test uncompressedLength field and index example 2022-04-30 11:34:10 +02:00
Michael Eischer
9ffb8920f1 repository: run blackbox tests using old and new repo version 2022-04-30 11:34:10 +02:00
Michael Eischer
abe5935693 repository: unify repository version-specific initialization
Mark the master index as compressed also when initializing a new
repository. This is only relevant for testing.
2022-04-30 11:34:10 +02:00
Alexander Neumann
8776031f96 Leave allocating slices to the decompress code 2022-04-30 11:34:10 +02:00
Alexander Neumann
5eb05a0afe Configure zstd encoder/decoder 2022-04-30 11:34:10 +02:00
Michael Eischer
2f36e044db Cleanup pack header check 2022-04-30 11:34:10 +02:00
Alexander Neumann
94dc9a0fa7 Amend changelog 2022-04-30 11:34:10 +02:00
Alexander Neumann
8b11b86383 Add option global --compression 2022-04-30 11:34:10 +02:00
Michael Eischer
f38f457a64 Add basic changelog for compression support 2022-04-30 11:34:10 +02:00
Michael Eischer
ba27d29d58 Print repository version when opening a repo 2022-04-30 11:34:10 +02:00
Michael Eischer
7132df529e repository: Increase index size for repo version 2
A compressed index is only about one third the size of an uncompressed
one. Thus increase the number of entries in an index to avoid cluttering
the repository with small indexes.
2022-04-30 11:34:10 +02:00
Michael Eischer
2535524132 debug: Add support for compressed blobs 2022-04-30 11:34:10 +02:00
Michael Eischer
fda7bb0f09 debug: Reduce code duplication 2022-04-30 11:34:10 +02:00
Michael Eischer
66f9048bce repository: Alloc zstd encoder/decoder on demand 2022-04-30 11:34:10 +02:00
Michael Eischer
fd05037e1a repository: recalibrate index batch allocation size 2022-04-30 11:34:10 +02:00
Michael Eischer
6fb408d90e repository: implement pack compression 2022-04-30 11:34:10 +02:00
Michael Eischer
362ab06023 init: Add flag to specify created repository version 2022-04-30 10:07:42 +02:00
Michael Eischer
4b957e7373 repository: Implement index/snapshot/lock compression
The config file is not compressed as it should remain readable by older
restic versions such that these can return a proper error.

As the old format for unpacked data does not include a version header,
make use of a trick: The old data is always encoded as JSON. Thus it can
only start with '{' or '['. For any other value the first byte indicates
a versioned format. The version is set to 2 for now. Then the zstd
compressed data follows.
2022-04-30 10:07:42 +02:00
Michael Eischer
0957b74887 Misc design.rst cleanups 2022-04-30 10:07:42 +02:00
Alexander Neumann
270ed00d1f doc: Add repository compression support documentation
Co-authored-by: Michael Eischer <michael.eischer@fau.de>
2022-04-30 10:07:42 +02:00
Alexander Neumann
4e1ef7804a Merge pull request #3717 from MichaelEischer/fix-stuck-repack
Fix stuck repack step
2022-04-30 09:50:43 +02:00
Alexander Neumann
e4780d3956 Merge pull request #3718 from MichaelEischer/sftp-docs-fix
doc: sftp with password actually works
2022-04-23 20:27:27 +02:00
Alexander Neumann
c183e35b5a Merge pull request #3719 from MichaelEischer/read-write-order
Describe repository read/write order required by repository format
2022-04-23 15:23:00 +02:00
Michael Eischer
6f9e20a1bb doc: Describe repository read/write order 2022-04-23 15:21:02 +02:00
Michael Eischer
f9219e8608 doc: sftp with password actually works 2022-04-23 11:57:36 +02:00
Michael Eischer
3b630d9998 add missing streamPacks changelog 2022-04-23 11:50:19 +02:00
Michael Eischer
566ac11c65 fix changelog name 2022-04-23 11:37:00 +02:00
Michael Eischer
f5609d1d3c prune: Fail early if too few backend connections 2022-04-23 11:32:52 +02:00
Michael Eischer
e597b99b55 repository: Reduce repack workers to prevent deadlock
As repack streams packs these occupy one backend connection. Uploading a
new pack also requires a backend connection. To prevent a deadlock
during repack when reaching the backend connections limit, simply limit
the repackWorker count to always leave one connection for uploading.
2022-04-23 11:28:18 +02:00
Michael Eischer
ee627cd832 backend/mem: Actually enforce connection limit
This will allow tests to detect deadlocks related to the connections
limit.
2022-04-23 11:22:00 +02:00
Michael Eischer
4f97492d28 Backend: Expose connections parameter 2022-04-23 11:13:08 +02:00
rawtaz
07a565e6f7 Merge pull request #3716 from MichaelEischer/password-error-on-stderr
Print password error message on stderr
2022-04-21 01:40:54 +02:00
Michael Eischer
bf7da7ff10 Print password error message on stderr
The password prompt itself is already printed on stderr.
2022-04-20 22:22:09 +02:00
Alexander Neumann
dba47d29d5 Merge pull request #3711 from restic/doc-grouping
doc: Clarify and make grouping in forget more noticeable
2022-04-16 10:08:11 +02:00
Leo R. Lundgren
8ac7519fd5 doc: Clarify and make grouping in forget more noticeable 2022-04-16 01:13:13 +02:00
Alexander Neumann
edc1a24a90 Merge pull request #3707 from duracell/patch-1
doc: fix missing "init" in rest-server example
2022-04-12 20:39:25 +02:00
Michael
9563e2f75c doc: fix missing "init" in rest-server example 2022-04-12 18:21:53 +02:00
Alexander Neumann
7f133a28b2 Update VERSION file 2022-04-11 20:34:14 +02:00
Alexander Neumann
4f3b1f19cb Set development version for 0.13.1 2022-04-10 20:41:00 +02:00
Alexander Neumann
89ee1cf9ee Merge pull request #3610 from MichaelEischer/windows-temp-files
Improve handling of temporary files on windows
2022-04-10 20:29:04 +02:00
Alexander Neumann
a059ef90f8 Merge pull request #3702 from MichaelEischer/extend-config-error
Print used key name if config fails to load
2022-04-10 20:25:24 +02:00
Michael Eischer
4077a81b34 Add simple test for fs.TempFile on windows 2022-04-09 23:37:58 +02:00
Michael Eischer
9a3f1a9703 Simplify and comment TempFile implementation for windows 2022-04-09 23:37:58 +02:00
Michael Eischer
c2aabb2686 Print used key name if config fails to load 2022-04-09 22:38:18 +02:00
MichaelEischer
c60a5f00c9 Merge pull request #3675 from ItsMattL/update
Refactor file handing for self-update.
2022-04-09 21:55:56 +02:00
Matt LaPlante
0ba9d4ced7 Refactor file handing for self-update.
* Write new file payload to a temp file before touching the original
binary. Minimizes the possibility of failing mid-write and corrupting
the binary.
* On Windows, move the original binary out to a temp file rather than
removing it as the running binary is locked. Fixes issue #2248.
2022-04-09 21:40:33 +02:00
Alexander Neumann
04e054465a Merge pull request #3475 from MichaelEischer/local-sftp-conn-limit
Limit concurrent operations for local / sftp backend
2022-04-09 21:33:00 +02:00
Alexander Neumann
1519e9f911 Merge pull request #3570 from MichaelEischer/list-snapshots-before-index
List snapshots before index
2022-04-09 21:15:49 +02:00
Michael Eischer
ebab35581c Check in integration test that snapshots are listed before the index
As an exception prune is still allowed to load the index before
snapshots, as it uses exclusive locks. In case of problems with locking
it is also better to load snapshots created after loading the index, as
this will lead to a prune sanity check failure instead of a broken snapshot.
2022-04-09 12:27:27 +02:00
Michael Eischer
7b9ae91e04 copy: Load snapshots before indexes 2022-04-09 12:27:25 +02:00
Michael Eischer
47243176fa diff: list snapshots only once 2022-04-09 12:26:31 +02:00
Michael Eischer
5af828e3e6 add changelogs 2022-04-09 12:26:31 +02:00
Michael Eischer
4636c20397 test that TestFindListOnce calls List only once 2022-04-09 12:26:31 +02:00
Michael Eischer
9e12159230 Fix O(n) backend list calls in FindFilteredSnapshots
When resolving snapshotIDs in FindFilteredSnapshots either
FindLatestSnapshot or FindSnapshot is called. Both operations issue a
list operation to the backend. When for example passing a long list of
snapshot ids to `forget` this could lead to a large number of list
operations.
2022-04-09 12:26:31 +02:00
Michael Eischer
3d29083e60 copy/find/ls/recover/stats: Memorize snapshot listing before index
These commands filter the snapshots according to some criteria which
essentially requires loading the index before filtering the snapshots.
Thus create a copy of the snapshots list beforehand and use it later on.
2022-04-09 12:26:30 +02:00
Michael Eischer
2ec0f3303a backup/diff/dump/restore/stats: List snapshots before index
During a backup the index is written before the corresponding snapshots.
To ensure that a concurrent/later restic run can read a snapshot's data,
restic thus must first load the snapshots and only afterwards the index.
Otherwise it is not possible to ensure that the loaded index is recent
enough to cover all of the snapshot's data.
2022-04-09 12:24:09 +02:00
Michael Eischer
ece06f125e sftp: Limit concurrent backend operations 2022-04-09 12:21:38 +02:00
Michael Eischer
cd783358d3 local: Limit concurrent backend operations
Use a limit of 2 similar to the filereader concurrency in the archiver.
2022-04-09 12:21:38 +02:00
Michael Eischer
0b258cc054 backends: clean reader closing 2022-04-09 12:21:38 +02:00
Alex Duchesne
9e34c791c9 Better temp file cleanup on Windows. 2022-04-09 12:00:22 +02:00
Alexander Neumann
7d55b4f95e Merge pull request #3701 from restic/rawtaz-doc-exclude-cachedir
doc: Link to CACHEDIR.TAG specification
2022-04-08 09:35:08 +02:00
rawtaz
de4e3117eb doc: Link to CACHEDIR.TAG specification 2022-04-08 03:07:27 +02:00
MichaelEischer
500079d265 Merge pull request #3689 from brightdroid/patch-1
added documentation for zsh autocompletion
2022-04-03 21:49:47 +02:00
Alexander Neumann
192288bc9c Merge pull request #3696 from cqjjjzr/fix-win-procgrp
Fix rclone (scoop shim) and sftp issue due to detached console on Windows
2022-04-03 13:46:28 +02:00
Charlie Jiang
d9c9415cfd Fix rclone (scoop shim) and sftp issue due to detached console on Windows 2022-04-03 17:53:17 +08:00
Alexander Neumann
59370b6062 Merge pull request #3695 from jernej-9/typo-fix
Fix a typo in the docs
2022-04-01 20:10:40 +02:00
Jernej Debevc
6e5731bf2f Fix a typo in the docs 2022-04-01 17:16:55 +02:00
Alexander Neumann
305cd1e730 Merge pull request #3693 from greatroar/cast-btrfs-super-magic
Cast unix.Statfs_t.Type to int64 when checking for btrfs
2022-04-01 08:09:46 +02:00
greatroar
c23c0f7c14 Cast unix.Statfs_t.Type to int64 when checking for btrfs
Fixes #3687. Uses the cast suggested by @MichaelEischer, except that the
contant isn't cast along, because it's untyped and will be converted by
the compiler as necessary.
2022-03-31 22:30:45 +02:00
Alexander Neumann
774c2e75ca Merge pull request #3680 from restic/update-deps
Update all dependencies (except fuse), require Go 1.15
2022-03-30 21:36:08 +02:00
Alexander Neumann
66d50b72e3 Require Go 1.15 or later
The library github.com/golang-jwt/jwt/v4 requires the FillByte() method
of *big.Int, so we're raising the minimum Go version to 1.15.
2022-03-30 21:11:17 +02:00
Alexander Neumann
89d86a7933 Update all dependencies (except fuse) 2022-03-30 21:11:13 +02:00
Christoph Roeder
7510bdc247 added documentation for zsh autocompletion 2022-03-30 10:45:52 +02:00
Alexander Neumann
f190d2e60e Merge pull request #3686 from restic/fix-diff
Fix diff
2022-03-30 07:43:07 +02:00
Alexander Neumann
206550a246 Fix diff
Nodes in trees were always printed with a `+` in diff, regardless of
whether or not a dir was added or removed. Let's use the mode we were
passed in printDir().

Closes #3685
2022-03-29 21:05:11 +02:00
Alexander Neumann
db8a958991 Merge pull request #3683 from MichaelEischer/fix-golangci-lint-warnings
Fix golangci lint warnings
2022-03-29 11:45:10 +02:00
Alexander Neumann
724ace0e99 Merge pull request #3682 from MichaelEischer/refactor-code
Pack size calculation cleanup and misc other changes
2022-03-29 11:06:04 +02:00
Michael Eischer
af31266b7d golangci-lint: replace deprecated golint with revive 2022-03-28 22:33:17 +02:00
Michael Eischer
2f81af6afa bloblru: Fix comment for New function 2022-03-28 22:25:25 +02:00
Michael Eischer
61e179ee78 switch to golang.org/x/term 2022-03-28 22:24:15 +02:00
Michael Eischer
c60540b196 add go:build headers everywhere 2022-03-28 22:23:47 +02:00
Michael Eischer
fefe9f5c0e pack: Hide more implementation details 2022-03-28 22:12:16 +02:00
Michael Eischer
a773cb6527 pack: cleanup header size calculation 2022-03-28 22:09:49 +02:00
Michael Eischer
6408686973 repository: Simplify Blob equality check 2022-03-28 22:09:49 +02:00
Michael Eischer
243698680a crypto: Use helpers for size calculations 2022-03-28 22:09:49 +02:00
Michael Eischer
d6db5a1fc2 archiver: Fix test
The test relied on an undeocumented sideeffect of the LoadBlob implementation
2022-03-28 22:09:49 +02:00
Michael Eischer
f78bd14e28 repository: Remove pack implementation details from MasterIndex 2022-03-28 22:09:49 +02:00
Michael Eischer
dc3d77dacc repository: make saveAndEncrypt private 2022-03-28 22:09:49 +02:00
Michael Eischer
6877e7edbb repository: Rename LoadAndDecrypt to LoadUnpacked
The method is the complement for SaveUnpacked and not for
SaveAndEncrypt. The latter assembles blobs into pack files.
2022-03-28 22:09:49 +02:00
Michael Eischer
2e1613d4c6 errors: Ensure that errors.IsFatal(errors.Fatal("err")) == true
This fixes a few cases where restic output "Fatal: Fatal: [...]"
2022-03-28 22:09:49 +02:00
Alexander Neumann
a08b95c497 Merge pull request #3513 from MichaelEischer/fast-copy
Speed-up copy command
2022-03-28 20:18:43 +02:00
Michael Eischer
537b4c310a copy: Implement by reusing repack
The repack operation copies all selected blobs from a set of pack files
into new pack files. For prune the source and destination repositories
are identical. To implement copy, just use a different source and
destination repository.
2022-03-26 20:47:15 +01:00
Alexander Neumann
4d5db61bd0 Merge pull request #3484 from MichaelEischer/stream-check-repack
Stream packs in `check --read-data` and during repacking
2022-03-26 20:46:17 +01:00
Alexander Neumann
4ab12f59a5 Set development version for 0.13.0 2022-03-26 20:10:07 +01:00
Alexander Neumann
e682f7c0d6 Add tests for StreamPack 2022-03-21 21:15:03 +01:00
Michael Eischer
27524979e8 restorer: Remove dead code 2022-02-13 11:43:09 +01:00
Michael Eischer
bba8ba7a5b repository: cancel streampack context after error 2022-02-12 20:18:25 +01:00
Michael Eischer
47554a3428 repository: Fix error handling in repack
When storing a blob fails, this is a fatal error which must not be
retried.
2022-02-12 20:18:25 +01:00
Michael Eischer
4b3dc415ef checker: cleanup header extraction 2022-02-12 20:18:25 +01:00
Michael Eischer
930a00ad54 checker: reuse bufio reader 2022-02-12 20:18:25 +01:00
Michael Eischer
34ebafb8b6 repository: don't crash if blob size is too short 2022-02-12 20:18:25 +01:00
Michael Eischer
becebf5d88 repository: remove unused DownloadAndHash 2022-02-12 20:18:25 +01:00
Michael Eischer
f1e58e7c7f checker: rewrite ReadData to stream packs 2022-02-12 20:18:25 +01:00
Michael Eischer
f40abd92fa restorer: convert to use StreamPack 2022-02-12 20:18:25 +01:00
Michael Eischer
f00f690658 repository: stream packs during repacking 2022-02-12 20:18:25 +01:00
Michael Eischer
c4a2bfcb39 repository: Add StreamPacks function
The function supports efficiently loading a specified list of blobs from
a single pack in a streaming fashion. That is there's no need for
temporary files independent of the pack size.
2022-02-12 20:18:25 +01:00
Michael Eischer
153e2ba859 repository: Implement lisiting blobs per pack file 2022-02-12 20:18:24 +01:00
921 changed files with 68195 additions and 30635 deletions

View File

@@ -1,12 +0,0 @@
# Folders
.git/
.github/
changelog/
doc/
docker/
helpers/
# Files
.gitignore
.golangci.yml
*.md

2
.gitattributes vendored Normal file
View File

@@ -0,0 +1,2 @@
# Workaround for https://github.com/golang/go/issues/52268.
**/testdata/fuzz/*/* eol=lf

View File

@@ -32,23 +32,30 @@ Output of `restic version`
--------------------------
How did you run restic exactly?
-------------------------------
What backend/service did you use to store the repository?
---------------------------------------------------------
Problem description / Steps to reproduce
----------------------------------------
<!--
This section should include at least:
* A description of the problem you are having with restic.
* The complete command line and any environment variables you used to
configure restic's backend access. Make sure to replace sensitive values!
* The output of the commands, what restic prints gives may give us much
information to diagnose the problem!
* The more time you spend describing an easy way to reproduce the behavior (if
this is possible), the easier it is for the project developers to fix it!
-->
What backend/server/service did you use to store the repository?
----------------------------------------------------------------
Expected behavior
-----------------
@@ -65,22 +72,12 @@ In this section, please try to concentrate on observations, so only describe
what you observed directly.
-->
Steps to reproduce the behavior
-------------------------------
<!--
The more time you spend describing an easy way to reproduce the behavior (if
this is possible), the easier it is for the project developers to fix it!
-->
Do you have any idea what may have caused this?
-----------------------------------------------
Do you have an idea how to solve the issue?
-------------------------------------------
<!--
Did something noteworthy happen on your system, Internet connection, backend services, etc?
-->
Did restic help you today? Did it make you happy in any way?

View File

@@ -28,13 +28,15 @@ Checklist
You do not need to check all the boxes below all at once. Feel free to take
your time and add more commits. If you're done and ready for review, please
check the last box. Enable a checkbox by replacing [ ] with [x].
Please always follow these steps:
- Read the [contribution guidelines](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#providing-patches).
- Enable [maintainer edits](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/allowing-changes-to-a-pull-request-branch-created-from-a-fork).
- Run `gofmt` on the code in all commits.
- Format all commit messages in the same style as [the other commits in the repository](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#git-commits).
-->
- [ ] I have read the [contribution guidelines](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#providing-patches).
- [ ] I have [enabled maintainer edits](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/allowing-changes-to-a-pull-request-branch-created-from-a-fork).
- [ ] I have added tests for all code changes.
- [ ] I have added documentation for relevant changes (in the manual).
- [ ] There's a new file in `changelog/unreleased/` that describes the changes for our users (see [template](https://github.com/restic/restic/blob/master/changelog/TEMPLATE)).
- [ ] I have run `gofmt` on the code in all commits.
- [ ] All commit messages are formatted in the same style as [the other commits in the repo](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#git-commits).
- [ ] I'm done! This pull request is ready for review.

13
.github/dependabot.yml vendored Normal file
View File

@@ -0,0 +1,13 @@
version: 2
updates:
# Dependencies listed in go.mod
- package-ecosystem: "gomod"
directory: "/" # Location of package manifests
schedule:
interval: "monthly"
# Dependencies listed in .github/workflows/*.yml
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "monthly"

95
.github/workflows/docker.yml vendored Normal file
View File

@@ -0,0 +1,95 @@
name: Create and publish a Docker image
on:
push:
tags:
- 'v*'
branches:
- 'master'
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
jobs:
build-and-push-image:
if: github.repository == 'restic/restic'
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
outputs:
image: ${{ steps.image.outputs.image }}
digest: ${{ steps.build-and-push.outputs.digest }}
steps:
- name: Checkout repository
uses: actions/checkout@v5
- name: Log in to the Container registry
uses: docker/login-action@184bdaa0721073962dff0199f1fb9940f07167d1
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@8e5442c4ef9f78752691e2d8f8d19755c6f78e81
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=sha
type=ref,event=branch
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
- name: Set up QEMU
uses: docker/setup-qemu-action@68827325e0b33c7199eb31dd4e31fbe9023e06e3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@2b51285047da1547ffb1b2203d8be4c0af6b1f20
- name: Ensure consistent binaries
run: |
echo "removing git directory for consistency with release binaries"
rm -rf .git
# remove VCS information from release builds, keep VCS for nightly builds on master
if: github.ref != 'refs/heads/master'
- name: Build and push Docker image
id: build-and-push
uses: docker/build-push-action@15560696de535e4014efeff63c48f16952e52dd1
with:
push: true
context: .
file: docker/Dockerfile.release
platforms: linux/386,linux/amd64,linux/arm,linux/arm64
pull: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
- name: Output image
id: image
run: |
# NOTE: Set the image as an output because the `env` context is not
# available to the inputs of a reusable workflow call.
image_name="${REGISTRY}/${IMAGE_NAME}"
echo "image=$image_name" >> "$GITHUB_OUTPUT"
provenance:
needs: [build-and-push-image]
permissions:
actions: read # for detecting the Github Actions environment.
id-token: write # for creating OIDC tokens for signing.
packages: write # for uploading attestations.
if: github.repository == 'restic/restic'
uses: slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@v2.1.0
with:
image: ${{ needs.build-and-push-image.outputs.image }}
digest: ${{ needs.build-and-push-image.outputs.digest }}
registry-username: ${{ github.actor }}
secrets:
registry-password: ${{ secrets.GITHUB_TOKEN }}

View File

@@ -7,9 +7,13 @@ on:
# run tests for all pull requests
pull_request:
merge_group:
permissions:
contents: read
env:
latest_go: "1.18.x"
latest_go: "1.24.x"
GO111MODULE: on
jobs:
@@ -19,47 +23,31 @@ jobs:
# list of jobs to run:
include:
- job_name: Windows
go: 1.18.x
go: 1.24.x
os: windows-latest
install_verb: install
- job_name: macOS
go: 1.18.x
go: 1.24.x
os: macOS-latest
test_fuse: false
install_verb: install
- job_name: Linux
go: 1.18.x
go: 1.24.x
os: ubuntu-latest
test_cloud_backends: true
test_fuse: true
check_changelog: true
install_verb: install
- job_name: Linux
go: 1.17.x
- job_name: Linux (race)
go: 1.24.x
os: ubuntu-latest
test_fuse: true
install_verb: install
test_opts: "-race"
- job_name: Linux
go: 1.16.x
go: 1.23.x
os: ubuntu-latest
test_fuse: true
install_verb: get
- job_name: Linux
go: 1.15.x
os: ubuntu-latest
test_fuse: true
install_verb: get
- job_name: Linux
go: 1.14.x
os: ubuntu-latest
test_fuse: true
install_verb: get
name: ${{ matrix.job_name }} Go ${{ matrix.go }}
runs-on: ${{ matrix.os }}
@@ -68,15 +56,18 @@ jobs:
GOPROXY: https://proxy.golang.org
steps:
- name: Check out code
uses: actions/checkout@v5
- name: Set up Go ${{ matrix.go }}
uses: actions/setup-go@v2
uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go }}
- name: Get programs (Linux/macOS)
run: |
echo "build Go tools"
go ${{ matrix.install_verb }} github.com/restic/rest-server/cmd/rest-server@latest
go install github.com/restic/rest-server/cmd/rest-server@master
echo "install minio server"
mkdir $HOME/bin
@@ -98,7 +89,7 @@ jobs:
chmod 755 $HOME/bin/rclone
rm -rf rclone*
# add $HOME/bin to path ($GOBIN was already added to the path by setup-go@v2)
# add $HOME/bin to path ($GOBIN was already added to the path by setup-go@v3)
echo $HOME/bin >> $GITHUB_PATH
if: matrix.os == 'ubuntu-latest' || matrix.os == 'macOS-latest'
@@ -108,7 +99,7 @@ jobs:
$ProgressPreference = 'SilentlyContinue'
echo "build Go tools"
go ${{ matrix.install_verb }} github.com/restic/rest-server/...
go install github.com/restic/rest-server/cmd/rest-server@master
echo "install minio server"
mkdir $Env:USERPROFILE/bin
@@ -120,7 +111,7 @@ jobs:
unzip rclone.zip
copy rclone*/rclone.exe $Env:USERPROFILE/bin
# add $USERPROFILE/bin to path ($GOBIN was already added to the path by setup-go@v2)
# add $USERPROFILE/bin to path ($GOBIN was already added to the path by setup-go@v3)
echo $Env:USERPROFILE\bin >> $Env:GITHUB_PATH
echo "install tar"
@@ -141,18 +132,23 @@ jobs:
echo $Env:USERPROFILE\tar\bin >> $Env:GITHUB_PATH
if: matrix.os == 'windows-latest'
- name: Check out code
uses: actions/checkout@v2
- name: Build with build.go
run: |
go run build.go
- name: Minimal test
run: |
./restic init
./restic backup .
env:
RESTIC_REPOSITORY: ../testrepo
RESTIC_PASSWORD: password
- name: Run local Tests
env:
RESTIC_TEST_FUSE: ${{ matrix.test_fuse }}
run: |
go test -cover ./...
go test -cover ${{matrix.test_opts}} ./...
- name: Test cloud backends
env:
@@ -189,11 +185,13 @@ jobs:
# prepare credentials for Google Cloud Storage tests in a temp file
export GOOGLE_APPLICATION_CREDENTIALS=$(mktemp --tmpdir restic-gcs-auth-XXXXXXX)
echo $RESTIC_TEST_GS_APPLICATION_CREDENTIALS_B64 | base64 -d > $GOOGLE_APPLICATION_CREDENTIALS
go test -cover -parallel 4 ./internal/backend/...
go test -cover -parallel 5 -timeout 15m ./internal/backend/...
# only run cloud backend tests for pull requests from and pushes to our
# own repo, otherwise the secrets are not available
if: (github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository) && matrix.test_cloud_backends
# Skip for Dependabot pull requests as these are run without secrets
# https://docs.github.com/en/code-security/dependabot/working-with-dependabot/automating-dependabot-with-github-actions#responding-to-events
if: ((github.repository == 'restic/restic' && github.event_name == 'push') || github.event.pull_request.head.repo.full_name == github.repository) && (github.actor != 'dependabot[bot]') && matrix.test_cloud_backends
- name: Check changelog files with calens
run: |
@@ -206,71 +204,57 @@ jobs:
cross_compile:
strategy:
# ATTENTION: the list of architectures must be in sync with helpers/build-release-binaries/main.go!
matrix:
# run cross-compile in two batches parallel so the overall tests run faster
targets:
- "linux/386 linux/amd64 linux/arm linux/arm64 linux/ppc64le linux/mips linux/mipsle linux/mips64 linux/mips64le linux/s390x \
openbsd/386 openbsd/amd64"
- "freebsd/386 freebsd/amd64 freebsd/arm \
aix/ppc64 \
darwin/amd64 darwin/arm64 \
netbsd/386 netbsd/amd64 \
windows/386 windows/amd64 \
solaris/amd64"
# run cross-compile in three batches parallel so the overall tests run faster
subset:
- "0/3"
- "1/3"
- "2/3"
env:
GOPROXY: https://proxy.golang.org
runs-on: ubuntu-latest
name: Cross Compile for ${{ matrix.targets }}
name: Cross Compile for subset ${{ matrix.subset }}
steps:
- name: Check out code
uses: actions/checkout@v5
- name: Set up Go ${{ env.latest_go }}
uses: actions/setup-go@v2
uses: actions/setup-go@v5
with:
go-version: ${{ env.latest_go }}
- name: Install gox
- name: Cross-compile for subset ${{ matrix.subset }}
run: |
go install github.com/mitchellh/gox@latest
- name: Check out code
uses: actions/checkout@v2
- name: Cross-compile with gox for ${{ matrix.targets }}
env:
GOFLAGS: "-trimpath"
GOX_ARCHS: "${{ matrix.targets }}"
run: |
mkdir build-output
gox -parallel 2 -verbose -osarch "$GOX_ARCHS" -output "build-output/{{.Dir}}_{{.OS}}_{{.Arch}}" ./cmd/restic
gox -parallel 2 -verbose -osarch "$GOX_ARCHS" -tags debug -output "build-output/{{.Dir}}_{{.OS}}_{{.Arch}}_debug" ./cmd/restic
mkdir build-output build-output-debug
go run ./helpers/build-release-binaries/main.go -o build-output -s . --platform-subset ${{ matrix.subset }}
go run ./helpers/build-release-binaries/main.go -o build-output-debug -s . --platform-subset ${{ matrix.subset }} --tags debug
lint:
name: lint
runs-on: ubuntu-latest
permissions:
contents: read
# allow annotating code in the PR
checks: write
steps:
- name: Check out code
uses: actions/checkout@v5
- name: Set up Go ${{ env.latest_go }}
uses: actions/setup-go@v2
uses: actions/setup-go@v5
with:
go-version: ${{ env.latest_go }}
- name: Check out code
uses: actions/checkout@v2
- name: golangci-lint
uses: golangci/golangci-lint-action@v2
uses: golangci/golangci-lint-action@v6
with:
# Required: the version of golangci-lint is required and must be specified without patch version: we always use the latest patch version.
version: v1.45
# Optional: show only new issues if it's a pull request. The default value is `false`.
only-new-issues: true
version: v1.64.8
args: --verbose --timeout 5m
skip-go-installation: true
# only run golangci-lint for pull requests, otherwise ALL hints get
# reported. We need to slowly address all issues until we can enable
@@ -283,16 +267,31 @@ jobs:
go mod tidy
git diff --exit-code go.mod go.sum
analyze:
name: Analyze results
needs: [test, cross_compile, lint]
if: always()
permissions: # no need to access code
contents: none
runs-on: ubuntu-latest
steps:
- name: Decide whether the needed jobs succeeded or failed
uses: re-actors/alls-green@05ac9388f0aebcb5727afa17fcccfecd6f8ec5fe
with:
jobs: ${{ toJSON(needs) }}
docker:
name: docker
runs-on: ubuntu-latest
steps:
- name: Check out code
uses: actions/checkout@v2
uses: actions/checkout@v5
- name: Docker meta
id: meta
uses: docker/metadata-action@v3
uses: docker/metadata-action@v5
with:
# list of Docker images to use as base name for tags
images: |
@@ -308,14 +307,14 @@ jobs:
type=sha
- name: Set up QEMU
uses: docker/setup-qemu-action@v1
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
uses: docker/setup-buildx-action@v3
- name: Build and push
id: docker_build
uses: docker/build-push-action@v2
uses: docker/build-push-action@v6
with:
push: false
context: .

2
.gitignore vendored
View File

@@ -1,3 +1,5 @@
/.idea
/restic
/restic.exe
/.vagrant
/.vscode

View File

@@ -10,13 +10,10 @@ linters:
# make sure all errors returned by functions are handled
- errcheck
# find unused code
- deadcode
# show how code can be simplified
- gosimple
# # make sure code is formatted
# make sure code is formatted
- gofmt
# examine code and report suspicious constructs, such as Printf calls whose
@@ -24,7 +21,7 @@ linters:
- govet
# make sure names and comments are used according to the conventions
- golint
- revive
# detect when assignments to existing variables are not used
- ineffassign
@@ -35,15 +32,14 @@ linters:
# find unused variables, functions, structs, types, etc.
- unused
# find unused struct fields
- structcheck
# find unused global variables
- varcheck
# parse and typecheck code
- typecheck
# ensure that http response bodies are closed
- bodyclose
- importas
issues:
# don't use the default exclude rules, this hides (among others) ignored
# errors from Close() calls
@@ -51,7 +47,24 @@ issues:
# list of things to not warn about
exclude:
# golint: do not warn about missing comments for exported stuff
- exported (function|method|var|type|const) `.*` should have comment or be unexported
# golint: ignore constants in all caps
# revive: do not warn about missing comments for exported stuff
- exported (function|method|var|type|const) .* should have comment or be unexported
# revive: ignore constants in all caps
- don't use ALL_CAPS in Go names; use CamelCase
# revive: lots of packages don't have such a comment
- "package-comments: should have a package comment"
# staticcheck: there's no easy way to replace these packages
- "SA1019: \"golang.org/x/crypto/poly1305\" is deprecated"
- "SA1019: \"golang.org/x/crypto/openpgp\" is deprecated"
- "redefines-builtin-id:"
exclude-rules:
# revive: ignore unused parameters in tests
- path: (_test\.go|testing\.go|backend/.*/tests\.go)
text: "unused-parameter:"
linters-settings:
importas:
alias:
- pkg: github.com/restic/restic/internal/test
alias: rtest

22
.readthedocs.yaml Normal file
View File

@@ -0,0 +1,22 @@
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
version: 2
build:
os: ubuntu-22.04
tools:
python: "3.11"
# Build HTMLZip
formats:
- htmlzip
# Build documentation in the docs/ directory with Sphinx
sphinx:
configuration: doc/conf.py
# https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
python:
install:
- requirements: doc/requirements.txt

File diff suppressed because it is too large Load Diff

View File

@@ -6,7 +6,8 @@ Ways to Help Out
Thank you for your contribution! Please **open an issue first** (or add a
comment to an existing issue) if you plan to work on any code or add a new
feature. This way, duplicate work is prevented and we can discuss your ideas
and design first.
and design first. Small bugfixes are an exception to this rule, just open a
pull request in this case.
There are several ways you can help us out. First of all code contributions and
bug fixes are most welcome. However even "minor" details as fixing spelling
@@ -48,9 +49,8 @@ environment was used and so on. Please tell us at least the following things:
Remember, the easier it is for us to reproduce the bug, the earlier it will be
corrected!
In addition, you can compile restic with debug support by running
`go run build.go -tags debug` and instructing it to create a debug
log by setting the environment variable `DEBUG_LOG` to a file, e.g. like this:
In addition, you can instruct restic to create a debug log by setting the
environment variable `DEBUG_LOG` to a file, e.g. like this:
$ export DEBUG_LOG=/tmp/restic-debug.log
$ restic backup ~/work
@@ -59,6 +59,19 @@ Please be aware that the debug log file will contain potentially sensitive
things like file and directory names, so please either redact it before
uploading it somewhere or post only the parts that are really relevant.
If restic gets stuck, please also include a stacktrace in the description.
On non-Windows systems, you can send a SIGQUIT signal to restic or press
`Ctrl-\` to achieve the same result. This causes restic to print a stacktrace
and then exit immediately. This will not damage your repository, however,
it might be necessary to manually clean up stale lock files using
`restic unlock`.
On Windows, please set the environment variable `RESTIC_DEBUG_STACKTRACE_SIGINT`
to `true` and press `Ctrl-C` to create a stacktrace.
If you think restic uses too much memory or a too large cache directory, then
please include the output of `restic stats --mode debug`.
Development Environment
=======================
@@ -66,8 +79,8 @@ Development Environment
The repository contains the code written for restic in the directories
`cmd/` and `internal/`.
Restic requires Go version 1.14 or later for compiling. Clone the repo (without
having `$GOPATH` set) and `cd` into the directory:
Make sure you have the minimum required Go version installed. Clone the repo
(without having `$GOPATH` set) and `cd` into the directory:
$ unset GOPATH
$ git clone https://github.com/restic/restic
@@ -77,12 +90,42 @@ Then use the `go` tool to build restic:
$ go build ./cmd/restic
$ ./restic version
restic 0.10.0-dev (compiled manually) compiled with go1.15.2 on linux/amd64
restic 0.14.0-dev (compiled manually) compiled with go1.19 on linux/amd64
To create a debug build use:
$ go build -tags debug ./cmd/restic
You can run all tests with the following command:
$ go test ./...
Performance and Memory Usage Issues
===================================
Debug builds of restic support the `--block-profile`, `--cpu-profile`,
`--mem-profile`, and `--trace-profile` options which collect performance data
that later on can be analyzed using the go tools:
$ restic --cpu-profile . [...]
$ go tool pprof -http localhost:12345 cpu.pprof
To analyze a trace profile use `go tool trace -http=localhost:12345 trace.out`.
As the memory usage of restic changes over time, it may be useful to capture a
snapshot of the current heap. This is possible using then `--listen-profile`
option. Then while restic runs you can query and afterwards analyze the heap statistics.
$ restic --listen-profile localhost:12345 [...]
$ curl http://localhost:12345/debug/pprof/heap -o heap.pprof
$ go tool pprof -http localhost:12345 heap.pprof
Further useful tools are setting the environment variable `GODEBUG=gctrace=1`,
which provides information about garbage collector runs. For a graphical variant
combine this with gcvis.
Providing Patches
=================

View File

@@ -10,8 +10,7 @@ For detailed usage and installation instructions check out the [documentation](h
You can ask questions in our [Discourse forum](https://forum.restic.net).
Quick start
-----------
## Quick start
Once you've [installed](https://restic.readthedocs.io/en/latest/020_installation.html) restic, start
off with creating a repository for your backups:
@@ -59,7 +58,7 @@ Therefore, restic supports the following backends for storing backups natively:
Restic is a program that does backups right and was designed with the
following principles in mind:
- **Easy:** Doing backups should be a frictionless process, otherwise
- **Easy**: Doing backups should be a frictionless process, otherwise
you might be tempted to skip it. Restic should be easy to configure
and use, so that, in the event of a data loss, you can just restore
it. Likewise, restoring data should not be complicated.
@@ -92,20 +91,17 @@ reproduce a byte identical version from the source code for that
release. Instructions on how to do that are contained in the
[builder repository](https://github.com/restic/builder).
News
----
## News
You can follow the restic project on Twitter [@resticbackup](https://twitter.com/resticbackup) or by subscribing to
You can follow the restic project on Mastodon [@resticbackup](https://fosstodon.org/@restic) or subscribe to
the [project blog](https://restic.net/blog/).
License
-------
## License
Restic is licensed under [BSD 2-Clause License](https://opensource.org/licenses/BSD-2-Clause). You can find the
complete text in [``LICENSE``](LICENSE).
complete text in [`LICENSE`](LICENSE).
Sponsorship
-----------
## Sponsorship
Backend integration tests for Google Cloud Storage and Microsoft Azure Blob
Storage are sponsored by [AppsCode](https://appscode.com)!

View File

@@ -1 +1 @@
0.13.0
0.18.0-dev

View File

@@ -3,8 +3,8 @@
// This program aims to make building Go programs for end users easier by just
// calling it with `go run`, without having to setup a GOPATH.
//
// This program needs Go >= 1.12. It'll use Go modules for compilation. It
// builds the package configured as Main in the Config struct.
// This program checks for a minimum Go version. It will use Go modules for
// compilation. It builds the package configured as Main in the Config struct.
// BSD 2-Clause License
//
@@ -43,7 +43,6 @@ package main
import (
"fmt"
"io"
"io/ioutil"
"os"
"os/exec"
"path/filepath"
@@ -54,12 +53,14 @@ import (
// config contains the configuration for the program to build.
var config = Config{
Name: "restic", // name of the program executable and directory
Namespace: "github.com/restic/restic", // subdir of GOPATH, e.g. "github.com/foo/bar"
Main: "./cmd/restic", // package name for the main package
DefaultBuildTags: []string{"selfupdate"}, // specify build tags which are always used
Tests: []string{"./..."}, // tests to run
MinVersion: GoVersion{Major: 1, Minor: 14, Patch: 0}, // minimum Go version supported
Name: "restic", // name of the program executable and directory
Namespace: "github.com/restic/restic", // subdir of GOPATH, e.g. "github.com/foo/bar"
Main: "./cmd/restic", // package name for the main package
// disable_grpc_modules is necessary to reduce the binary size since cloud.google.com/go/storage v1.44.0
// see https://github.com/googleapis/google-cloud-go/issues/11448
DefaultBuildTags: []string{"selfupdate", "disable_grpc_modules"}, // specify build tags which are always used
Tests: []string{"./..."}, // tests to run
MinVersion: GoVersion{Major: 1, Minor: 23, Patch: 0}, // minimum Go version supported
}
// Config configures the build.
@@ -179,7 +180,7 @@ func test(cwd string, env map[string]string, args ...string) error {
// getVersion returns the version string from the file VERSION in the current
// directory.
func getVersionFromFile() string {
buf, err := ioutil.ReadFile("VERSION")
buf, err := os.ReadFile("VERSION")
if err != nil {
verbosePrintf("error reading file VERSION: %v\n", err)
return ""
@@ -299,19 +300,21 @@ func (v GoVersion) AtLeast(other GoVersion) bool {
return true
}
if v.Major > other.Major {
return true
}
if v.Major < other.Major {
return false
}
if v.Minor > other.Minor {
return true
}
if v.Minor < other.Minor {
return false
}
if v.Patch < other.Patch {
return false
}
return true
return v.Patch >= other.Patch
}
func (v GoVersion) String() string {
@@ -319,12 +322,8 @@ func (v GoVersion) String() string {
}
func main() {
if !goVersion.AtLeast(GoVersion{1, 12, 0}) {
die("Go version (%v) is too old, restic requires Go >= 1.12\n", goVersion)
}
if !goVersion.AtLeast(config.MinVersion) {
fmt.Fprintf(os.Stderr, "%s detected, this program requires at least %s\n", goVersion, config.MinVersion)
fmt.Fprintf(os.Stderr, "Detected version %s is too old, restic requires at least %s\n", goVersion, config.MinVersion)
os.Exit(1)
}

View File

@@ -10,7 +10,7 @@ https://github.com/restic/restic/issues/2244
NOTE: This new implementation does not guarantee order in which blobs
are written to the target files and, for example, the last blob of a
file can be written to the file before any of the preceeding file blobs.
file can be written to the file before any of the preceding file blobs.
It is therefore possible to have gaps in the data written to the target
files if restore fails or interrupted by the user.

View File

@@ -1,6 +1,6 @@
Bugfix: Don't abort the stats command when data blobs are missing
Runing the stats command in the blobs-per-file mode on a repository with
Running the stats command in the blobs-per-file mode on a repository with
missing data blobs previously resulted in a crash.
https://github.com/restic/restic/pull/2668

View File

@@ -1,6 +1,6 @@
Bugfix: Mark repository files as read-only when using the local backend
Files stored in a local repository were marked as writeable on the
Files stored in a local repository were marked as writable on the
filesystem for non-Windows systems, which did not prevent accidental file
modifications outside of restic. In addition, the local backend did not work
with certain filesystems and network mounts which do not permit modifications

View File

@@ -5,7 +5,7 @@ another process using an exclusive lock through a filesystem snapshot. Restic
was unable to backup those files before. This update enables backing up these
files.
This needs to be enabled explicitely using the --use-fs-snapshot option of the
This needs to be enabled explicitly using the --use-fs-snapshot option of the
backup command.
https://github.com/restic/restic/issues/340

View File

@@ -1,7 +1,7 @@
Bugfix: Correct statistics for overlapping targets
Bugfix: Correct statistics for overlapping backup sources
A user reported that restic's statistics and progress information during backup
was not correctly calculated when the backup targets (files/dirs to save)
was not correctly calculated when the backup sources (files/dirs to save)
overlap. For example, consider a directory `foo` which contains (among others)
a file `foo/bar`. When `restic backup foo foo/bar` was run, restic counted the
size of the file `foo/bar` twice, so the completeness percentage as well as the

View File

@@ -2,7 +2,7 @@ Enhancement: Parallelize scan of snapshot content in `copy` and `prune`
The `copy` and `prune` commands used to traverse the directories of
snapshots one by one to find used data. This snapshot traversal is
now parallized which can speed up this step several times.
now parallelized which can speed up this step several times.
In addition the `check` command now reports how many snapshots have
already been processed.

View File

@@ -0,0 +1,9 @@
Enhancement: Support pruning even when the disk is full
When running out of disk space it was no longer possible to add or remove
data from a repository. To help with recovering from such a deadlock, the
prune command now supports an `--unsafe-recover-no-free-space` option to
recover from these situations. Make sure to read the documentation first!
https://github.com/restic/restic/issues/1153
https://github.com/restic/restic/pull/3481

View File

@@ -0,0 +1,8 @@
Change: Support debug log creation in release builds
Creating a debug log was only possible in debug builds which required users to
manually build restic. We changed the release builds to allow creating debug
logs by simply setting the environment variable `DEBUG_LOG=logname.log`.
https://github.com/restic/restic/issues/1842
https://github.com/restic/restic/pull/3826

View File

@@ -0,0 +1,28 @@
Enhancement: Add compression support
We've added compression support to the restic repository format. To create a
repository using the new format run `init --repository-version 2`. Please note
that the repository cannot be read by restic versions prior to 0.14.0.
You can configure whether data is compressed with the option `--compression`. It
can be set to `auto` (the default, which will compress very fast), `max` (which
will trade backup speed and CPU usage for better compression), or `off` (which
disables compression). Each setting is only applied for the current run of restic
and does *not* apply to future runs. The option can also be set via the
environment variable `RESTIC_COMPRESSION`.
To upgrade in place run `migrate upgrade_repo_v2` followed by `prune`. See the
documentation for more details. The migration checks the repository integrity
and upgrades the repository format, but will not change any data. Afterwards,
prune will rewrite the metadata to make use of compression.
As an alternative you can use the `copy` command to migrate snapshots; First
create a new repository using
`init --repository-version 2 --copy-chunker-params --repo2 path/to/old/repo`,
and then use the `copy` command to copy all snapshots to the new repository.
https://github.com/restic/restic/issues/21
https://github.com/restic/restic/issues/3779
https://github.com/restic/restic/pull/3666
https://github.com/restic/restic/pull/3704
https://github.com/restic/restic/pull/3733

View File

@@ -0,0 +1,18 @@
Enhancement: Adaptive IO concurrency based on backend connections
Many commands used hard-coded limits for the number of concurrent operations.
This prevented speed improvements by increasing the number of connections used
by a backend.
These limits have now been replaced by using the configured number of backend
connections instead, which can be controlled using the
`-o <backend-name>.connections=5` option. Commands will then automatically
scale their parallelism accordingly.
To limit the number of CPU cores used by restic, you can set the environment
variable `GOMAXPROCS` accordingly. For example to use a single CPU core, use
`GOMAXPROCS=1`.
https://github.com/restic/restic/issues/2162
https://github.com/restic/restic/issues/1467
https://github.com/restic/restic/pull/3611

View File

@@ -0,0 +1,8 @@
Bugfix: Support `self-update` on Windows
Restic `self-update` would fail in situations where the operating system
locks running binaries, including Windows. The new behavior works around
this by renaming the running file and swapping the updated file in place.
https://github.com/restic/restic/issues/2248
https://github.com/restic/restic/pull/3675

View File

@@ -0,0 +1,12 @@
Enhancement: Allow pack size customization
Restic now uses a target pack size of 16 MiB by default. This can be customized
using the `--pack-size size` option. Supported pack sizes range between 4 and
128 MiB.
It is possible to migrate an existing repository to _larger_ pack files using
`prune --repack-small`. This will rewrite every pack file which is
significantly smaller than the target size.
https://github.com/restic/restic/issues/2291
https://github.com/restic/restic/pull/3731

View File

@@ -0,0 +1,14 @@
Enhancement: Allow use of SAS token to authenticate to Azure
Previously restic only supported AccountKeys to authenticate to Azure
storage accounts, which necessitates giving a significant amount of
access.
We added support for Azure SAS tokens which are a more fine-grained
and time-limited manner of granting access. Set the `AZURE_ACCOUNT_NAME`
and `AZURE_ACCOUNT_SAS` environment variables to use a SAS token for
authentication. Note that if `AZURE_ACCOUNT_KEY` is set, it will take
precedence.
https://github.com/restic/restic/issues/2295
https://github.com/restic/restic/pull/3661

View File

@@ -0,0 +1,13 @@
Enhancement: Improve backup speed with many small files
We have restructured the backup pipeline to continue reading files while all
upload connections are busy. This allows the backup to already prepare the next
data file such that the upload can continue as soon as a connection becomes
available. This can especially improve the backup performance for high latency
backends.
The upload concurrency is now controlled using the `-o <backend-name>.connections=5`
option.
https://github.com/restic/restic/issues/2696
https://github.com/restic/restic/pull/3489

View File

@@ -0,0 +1,15 @@
Enhancement: Make snapshot directory structure of `mount` command customizable
We've added the possibility to customize the snapshot directory structure of
the `mount` command using templates passed to the `--snapshot-template` option.
The formatting of snapshots' timestamps is now controlled using `--time-template`
and supports subdirectories to for example group snapshots by year. Please
see `restic help mount` for further details.
Characters in tag names which are not allowed in a filename are replaced by
underscores `_`. For example a tag `foo/bar` will result in a directory name
of `foo_bar`.
https://github.com/restic/restic/issues/2907
https://github.com/restic/restic/pull/2913
https://github.com/restic/restic/pull/3691

View File

@@ -0,0 +1,12 @@
Enhancement: Optimize handling of duplicate blobs in `prune`
Restic `prune` always used to repack all data files containing duplicate
blobs. This effectively removed all duplicates during prune. However, as a
consequence all these data files were repacked even if the unused repository
space threshold could be reached with less work.
This is now changed and `prune` works nice and fast even when there are lots
of duplicate blobs.
https://github.com/restic/restic/issues/3114
https://github.com/restic/restic/pull/3290

View File

@@ -0,0 +1,13 @@
Change: Deprecate `check --check-unused` and add further checks
Since restic 0.12.0, it is expected to still have unused blobs after running
`prune`. This made the `--check-unused` option of the `check` command rather
useless and tended to confuse users. This option has been deprecated and is
now ignored.
The `check` command now also warns if a repository is using either the legacy
S3 layout or mixed pack files with both tree and data blobs. The latter is
known to cause performance problems.
https://github.com/restic/restic/issues/3295
https://github.com/restic/restic/pull/3730

View File

@@ -0,0 +1,14 @@
Bugfix: List snapshots in backend at most once to resolve snapshot IDs
Many commands support specifying a list of snapshot IDs which are then used to
determine the snapshots to be processed by the command. To resolve snapshot IDs
or `latest`, and check that these exist, restic previously listed all snapshots
stored in the repository. Depending on the backend this could be a slow and/or
expensive operation.
Restic now lists the snapshots only once and remembers the result in order to
resolve all further snapshot IDs swiftly.
https://github.com/restic/restic/issues/3428
https://github.com/restic/restic/pull/3570
https://github.com/restic/restic/pull/3395

View File

@@ -0,0 +1,14 @@
Bugfix: Fix rare 'not found in repository' error for `copy` command
In rare cases `copy` (and other commands) would report that `LoadTree(...)`
returned an `id [...] not found in repository` error. This could be caused by
a backup or copy command running concurrently. The error was only temporary;
running the failed restic command a second time as a workaround did resolve the
error.
This issue has now been fixed by correcting the order in which restic reads data
from the repository. It is now guaranteed that restic only loads snapshots for
which all necessary data is already available.
https://github.com/restic/restic/issues/3432
https://github.com/restic/restic/pull/3570

View File

@@ -0,0 +1,10 @@
Enhancement: Improve handling of temporary files on Windows
In some cases restic failed to delete temporary files, causing the current
command to fail. This has now been fixed by ensuring that Windows automatically
deletes the file. In addition, temporary files are only written to disk when
necessary, reducing disk writes.
https://github.com/restic/restic/issues/3465
https://github.com/restic/restic/issues/1551
https://github.com/restic/restic/pull/3610

View File

@@ -0,0 +1,7 @@
Bugfix: The `diff` command incorrectly listed some files as added
There was a bug in the `diff` command, causing it to always show files in a
removed directory as added. This has now been fixed.
https://github.com/restic/restic/issues/3685
https://github.com/restic/restic/pull/3686

View File

@@ -0,0 +1,13 @@
Bugfix: Fix rclone (shimmed by Scoop) and sftp not working on Windows
In #3602 a fix was introduced to address the problem of `rclone` prematurely
exiting when Ctrl+C is pressed on Windows. The solution was to create the
subprocess with its console detached from the restic console.
However, this solution failed when using `rclone` installed by Scoop or using
`sftp` with a passphrase-protected private key. We've now fixed this by using
a different approach to prevent Ctrl-C from passing down too early.
https://github.com/restic/restic/issues/3681
https://github.com/restic/restic/issues/3692
https://github.com/restic/restic/pull/3696

View File

@@ -0,0 +1,11 @@
Enhancement: Validate exclude patterns before backing up
Exclude patterns provided via `--exclude`, `--iexclude`, `--exclude-file` or
`--iexclude-file` previously weren't validated. As a consequence, invalid
patterns resulted in files that were meant to be excluded being backed up.
Restic now validates all patterns before running the backup and aborts with
a fatal error if an invalid pattern is detected.
https://github.com/restic/restic/issues/3709
https://github.com/restic/restic/pull/3734

View File

@@ -0,0 +1,13 @@
Bugfix: Directory sync errors for repositories accessed via SMB
On Linux and macOS, accessing a repository via a SMB/CIFS mount resulted in
restic failing to save the lock file, yielding the following errors:
Save(<lock/071fe833f0>) returned error, retrying after 552.330144ms: sync /repo/locks: no such file or directory
Save(<lock/bf789d7343>) returned error, retrying after 552.330144ms: sync /repo/locks: invalid argument
This has now been fixed by ignoring the relevant error codes.
https://github.com/restic/restic/issues/3720
https://github.com/restic/restic/issues/3751
https://github.com/restic/restic/pull/3752

View File

@@ -0,0 +1,8 @@
Bugfix: The `stats` command miscalculated restore size for multiple snapshots
Since restic 0.10.0 the restore size calculated by the `stats` command for
multiple snapshots was too low. The hardlink detection was accidentally applied
across multiple snapshots and thus ignored many files. This has now been fixed.
https://github.com/restic/restic/issues/3736
https://github.com/restic/restic/pull/3740

View File

@@ -0,0 +1,8 @@
Enhancement: Improve SFTP repository initialization over slow links
The `init` command, when used on an SFTP backend, now sends multiple `mkdir`
commands to the backend concurrently. This reduces the waiting times when
creating a repository over a very slow connection.
https://github.com/restic/restic/issues/3837
https://github.com/restic/restic/pull/3840

View File

@@ -0,0 +1,9 @@
Bugfix: Yield error on invalid policy to `forget`
The `forget` command previously silently ignored invalid/unsupported
units in the duration options, such as e.g. `--keep-within-daily 2w`.
Specifying an invalid/unsupported duration unit now results in an error.
https://github.com/restic/restic/issues/3861
https://github.com/restic/restic/pull/3862

View File

@@ -0,0 +1,21 @@
Enhancement: Use config file permissions to control file group access
Previously files in a local/SFTP repository would always end up with very
restrictive access permissions, allowing access only to the owner. This
prevented a number of valid use-cases involving groups and ACLs.
We now use the permissions of the config file in the repository to decide
whether group access should be given to newly created repository files or
not. We arrange for repository files to be created group readable exactly
when the repository config file is group readable.
To opt-in to group readable repositories, a simple `chmod -R g+r` or
equivalent on the config file can be used. For repositories that should
be writable by group members a tad more setup is required, see the docs.
Posix ACLs can also be used now that the group permissions being forced to
zero no longer masks the effect of ACL entries.
https://github.com/restic/restic/issues/2351
https://github.com/restic/restic/pull/3419
https://forum.restic.net/t/1391

View File

@@ -0,0 +1,9 @@
Enhancement: Allow limiting IO concurrency for local and SFTP backend
Restic did not support limiting the IO concurrency / number of connections for
accessing repositories stored using the local or SFTP backends. The number of
connections is now limited as for other backends, and can be configured via the
`-o local.connections=2` and `-o sftp.connections=5` options. This ensures that
restic does not overwhelm the backend with concurrent IO operations.
https://github.com/restic/restic/pull/3475

View File

@@ -0,0 +1,13 @@
Enhancement: Stream data in `check` and `prune` commands
The commands `check --read-data` and `prune` previously downloaded data files
into temporary files which could end up being written to disk. This could cause
a large amount of data being written to disk.
The pack files are now instead streamed, which removes the need for temporary
files. Please note that *uploads* during `backup` and `prune` still require
temporary files.
https://github.com/restic/restic/pull/3484
https://github.com/restic/restic/issues/3710
https://github.com/restic/restic/pull/3717

View File

@@ -0,0 +1,10 @@
Enhancement: Improve speed of `copy` command
The `copy` command could require a long time to copy snapshots for non-local
backends. This has been improved to provide a throughput comparable to the
`restore` command.
Additionally, `copy` now displays a progress bar.
https://github.com/restic/restic/issues/2923
https://github.com/restic/restic/pull/3513

View File

@@ -0,0 +1,8 @@
Change: Update dependencies and require Go 1.15 or newer
We've updated most dependencies. Since some libraries require newer language
features we're dropping support for Go 1.14, which means that restic now
requires at least Go 1.15 to build.
https://github.com/restic/restic/issues/3680
https://github.com/restic/restic/issues/3883

View File

@@ -0,0 +1,7 @@
Bugfix: Print "wrong password" to stderr instead of stdout
If an invalid password was entered, the error message was printed on stdout and
not on stderr as intended. This has now been fixed.
https://github.com/restic/restic/pull/3716
https://forum.restic.net/t/4965

View File

@@ -0,0 +1,8 @@
Enhancement: Display full IDs in `check` warnings
When running commands to inspect or repair a damaged repository, it is often
necessary to supply the full IDs of objects stored in the repository.
The output of `check` now includes full IDs instead of their shortened variant.
https://github.com/restic/restic/pull/3729

View File

@@ -0,0 +1,19 @@
Change: Replace `--repo2` option used by `init`/`copy` with `--from-repo`
The `init` and `copy` commands can read data from another repository.
However, confusingly `--repo2` referred to the repository *from* which the
`init` command copies parameters, but for the `copy` command `--repo2`
referred to the copy *destination*.
We've introduced a new option, `--from-repo`, which always refers to the
source repository for both commands. The old parameter names have been
deprecated but still work. To create a new repository and copy all snapshots
to it, the commands are now as follows:
```
restic -r /srv/restic-repo-copy init --from-repo /srv/restic-repo --copy-chunker-params
restic -r /srv/restic-repo-copy copy --from-repo /srv/restic-repo
```
https://github.com/restic/restic/pull/3742
https://forum.restic.net/t/5017

View File

@@ -0,0 +1,14 @@
Bugfix: Correctly rebuild index for legacy repositories
After running `rebuild-index` on a legacy repository containing mixed pack
files (that is, pack files which store both metadata and file data), `check`
printed warnings like `pack 12345678 contained in several indexes: ...`.
This warning was not critical, but has now nonetheless been fixed by properly
handling mixed pack files while rebuilding the index.
Running `prune` for such legacy repositories will also fix the warning by
reorganizing the pack files which caused it.
https://github.com/restic/restic/pull/3772
https://github.com/restic/restic/pull/3884
https://forum.restic.net/t/5044/13

View File

@@ -0,0 +1,7 @@
Enhancement: Optimize memory usage for directories with many files
Backing up a directory with hundreds of thousands or more files caused restic
to require large amounts of memory. We've now optimized the `backup` command
such that it requires up to 30% less memory.
https://github.com/restic/restic/pull/3773

View File

@@ -0,0 +1,11 @@
Bugfix: Limit number of key files tested while opening a repository
Previously, restic tested the password against every key in the repository
when opening a repository. The more keys there were in the repository, the
slower this operation became.
Restic now tests the password against up to 20 key files in the repository.
Alternatively, you can use the `--key-hint=<key ID>` option to specify a
specific key file to use instead.
https://github.com/restic/restic/pull/3776

View File

@@ -0,0 +1,11 @@
Enhancement: Validate include/exclude patterns before restoring
Patterns provided to `restore` via `--exclude`, `--iexclude`, `--include`
and `--iinclude` weren't validated before running the restore. Invalid
patterns would result in error messages being printed repeatedly, and
possibly unwanted files being restored.
Restic now validates all patterns before running the restore, and aborts
with a fatal error if an invalid pattern is detected.
https://github.com/restic/restic/pull/3819

View File

@@ -0,0 +1,8 @@
Enhancement: Implement `rewrite` command
Restic now has a `rewrite` command which allows to rewrite existing snapshots
to remove unwanted files.
https://github.com/restic/restic/issues/14
https://github.com/restic/restic/pull/2731
https://github.com/restic/restic/pull/4079

View File

@@ -0,0 +1,15 @@
Enhancement: Inform about successful retries after errors
When a recoverable error is encountered, restic shows a warning message saying
that it's retrying, e.g.:
`Save(<data/956b9ced99>) returned error, retrying after 357.131936ms: ...`
This message can be confusing in that it never clearly states whether the retry
is successful or not. This has now been fixed such that restic follows up with
a message confirming a successful retry, e.g.:
`Save(<data/956b9ced99>) operation successful after 1 retries`
https://github.com/restic/restic/issues/1734
https://github.com/restic/restic/pull/2661

View File

@@ -0,0 +1,12 @@
Enhancement: Improve handling of directories with duplicate entries
If for some reason a directory contains a duplicate entry, the `backup` command
would previously fail with a `node "path/to/file" already present` or `nodes
are not ordered got "path/to/file", last "path/to/file"` error.
The error handling has been improved to only report a warning in this case. Make
sure to check that the filesystem in question is not damaged if you see this!
https://github.com/restic/restic/issues/1866
https://github.com/restic/restic/issues/3937
https://github.com/restic/restic/pull/3880

View File

@@ -0,0 +1,10 @@
Bugfix: Make `mount` return exit code 0 after receiving Ctrl-C / SIGINT
To stop the `mount` command, a user has to press Ctrl-C or send a SIGINT
signal to restic. This used to cause restic to exit with a non-zero exit code.
The exit code has now been changed to zero as the above is the expected way
to stop the `mount` command and should therefore be considered successful.
https://github.com/restic/restic/issues/2015
https://github.com/restic/restic/pull/3894

View File

@@ -0,0 +1,19 @@
Enhancement: Support B2 API keys restricted to hiding but not deleting files
When the B2 backend does not have the necessary permissions to permanently
delete files, it now automatically falls back to hiding files. This allows
using restic with an application key which is not allowed to delete files.
This can prevent an attacker from deleting backups with such an API key.
To use this feature create an application key without the `deleteFiles`
capability. It is recommended to restrict the key to just one bucket.
For example using the `b2` command line tool:
`b2 create-key --bucket <bucketName> <keyName> listBuckets,readFiles,writeFiles,listFiles`
Alternatively, you can use the S3 backend to access B2, as described
in the documentation. In this mode, files are also only hidden instead
of being deleted permanently.
https://github.com/restic/restic/issues/2134
https://github.com/restic/restic/pull/2398

View File

@@ -0,0 +1,11 @@
Enhancement: Make `init` open only one connection for the SFTP backend
The `init` command using the SFTP backend used to connect twice to the
repository. This could be inconvenient if the user must enter a password,
or cause `init` to fail if the server does not correctly close the first SFTP
connection.
This has now been fixed by reusing the first/initial SFTP connection opened.
https://github.com/restic/restic/issues/2152
https://github.com/restic/restic/pull/3882

View File

@@ -0,0 +1,13 @@
Enhancement: Handle cache corruption on disk and in downloads
In rare situations, like for example after a system crash, the data stored
in the cache might be corrupted. This could cause restic to fail and required
manually deleting the cache.
Restic now automatically removes broken data from the cache, allowing it
to recover from such a situation without user intervention. In addition,
restic retries downloads which return corrupt data in order to also handle
temporary download problems.
https://github.com/restic/restic/issues/2533
https://github.com/restic/restic/pull/3521

View File

@@ -0,0 +1,17 @@
Bugfix: Don't read password from stdin for `backup --stdin`
The `backup` command when used with `--stdin` previously tried to read first
the password, then the data to be backed up from standard input. This meant
it would often confuse part of the data for the password.
From now on, it will instead exit with the message `Fatal: cannot read both
password and data from stdin` unless the password is passed in some other
way (such as `--restic-password-file`, `RESTIC_PASSWORD`, etc).
To enter the password interactively a password command has to be used. For
example on Linux, `mysqldump somedatabase | restic backup --stdin
--password-command='sh -c "systemd-ask-password < /dev/tty"'` securely reads
the password from the terminal.
https://github.com/restic/restic/issues/2591
https://github.com/restic/restic/pull/4011

View File

@@ -0,0 +1,9 @@
Enhancement: Support restoring symbolic links on Windows
The `restore` command now supports restoring symbolic links on Windows. Because
of Windows specific restrictions this is only possible when running restic with
the `SeCreateSymbolicLinkPrivilege` privilege or as an administrator.
https://github.com/restic/restic/issues/1078
https://github.com/restic/restic/issues/2699
https://github.com/restic/restic/pull/2875

View File

@@ -0,0 +1,20 @@
Enhancement: Stricter repository lock handling
Previously, restic commands kept running even if they failed to refresh their
locks in time. This could be a problem e.g. in case the client system running
a backup entered the standby power mode while the backup was still in progress
(which would prevent the client from refreshing its lock), and after a short
delay another host successfully runs `unlock` and `prune` on the repository,
which would remove all data added by the in-progress backup. If the backup
client later continues its backup, even though its lock had expired in the
meantime, this would lead to an incomplete snapshot.
To address this, lock handling is now much stricter. Commands requiring a lock
are canceled if the lock is not refreshed successfully in time. In addition,
if a lock file is not readable restic will not allow starting a command. It may
be necessary to remove invalid lock files manually or use `unlock --remove-all`.
Please make sure that no other restic processes are running concurrently before
doing this, however.
https://github.com/restic/restic/issues/2715
https://github.com/restic/restic/pull/3569

View File

@@ -0,0 +1,9 @@
Change: Include full snapshot ID in JSON output of `backup`
We have changed the JSON output of the backup command to include the full
snapshot ID instead of just a shortened version, as the latter can be ambiguous
in some rare cases. To derive the short ID, please truncate the full ID down to
eight characters.
https://github.com/restic/restic/issues/2724
https://github.com/restic/restic/pull/3993

View File

@@ -0,0 +1,8 @@
Enhancement: Add support for `credential_process` to S3 backend
Restic now uses a newer library for the S3 backend, which adds support for the
`credential_process` option in the AWS credential configuration.
https://github.com/restic/restic/issues/3029
https://github.com/restic/restic/issues/4034
https://github.com/restic/restic/pull/4025

View File

@@ -0,0 +1,8 @@
Enhancement: Make `mount` command support macOS using macFUSE 4.x
Restic now uses a different FUSE library for mounting snapshots and making them
available as a FUSE filesystem using the `mount` command. This adds support for
macFUSE 4.x which can be used to make this work on recent macOS versions.
https://github.com/restic/restic/issues/3096
https://github.com/restic/restic/pull/4024

View File

@@ -0,0 +1,7 @@
Enhancement: Support JSON output for the `init` command
The `init` command used to ignore the `--json` option, but now outputs a JSON
message if the repository was created successfully.
https://github.com/restic/restic/issues/3124
https://github.com/restic/restic/pull/3132

View File

@@ -0,0 +1,14 @@
Bugfix: Delete files on Backblaze B2 more reliably
Restic used to only delete the latest version of files stored in B2. In most
cases this worked well as there was only a single version of the file. However,
due to retries while uploading it is possible for multiple file versions to be
stored at B2. This could lead to various problems for files that should have
been deleted but still existed.
The implementation has now been changed to delete all versions of files, which
doubles the amount of Class B transactions necessary to delete files, but
assures that no file versions are left behind.
https://github.com/restic/restic/issues/3161
https://github.com/restic/restic/pull/3885

View File

@@ -0,0 +1,12 @@
Bugfix: Make SFTP backend report no space left on device
Backing up to an SFTP backend would spew repeated SSH_FX_FAILURE messages when
the remote disk was full. Restic now reports "sftp: no space left on device"
and exits immediately when it detects this condition.
A fix for this issue was implemented in restic 0.12.1, but unfortunately the
fix itself contained a bug that prevented it from taking effect.
https://github.com/restic/restic/issues/3336
https://github.com/restic/restic/pull/3345
https://github.com/restic/restic/pull/4075

View File

@@ -0,0 +1,10 @@
Bugfix: Improve handling of interrupted syscalls in `mount` command
Accessing restic's FUSE mount could result in "input/output" errors when using
programs in which syscalls can be interrupted. This is for example the case for
Go programs. This has now been fixed by improved error handling of interrupted
syscalls.
https://github.com/restic/restic/issues/3567
https://github.com/restic/restic/issues/3694
https://github.com/restic/restic/pull/3875

View File

@@ -0,0 +1,7 @@
Bugfix: Fix stuck `copy` command when `-o <backend>.connections=1`
When running the `copy` command with `-o <backend>.connections=1` the
command would be infinitely stuck. This has now been fixed.
https://github.com/restic/restic/issues/3897
https://github.com/restic/restic/pull/3898

View File

@@ -0,0 +1,9 @@
Bugfix: Correct prune statistics for partially compressed repositories
In a partially compressed repository, one data blob can exist both in an
uncompressed and a compressed version. This caused the `prune` statistics to
become inaccurate and e.g. report a too high value for the unused size, such
as "unused size after prune: 16777215.991 TiB". This has now been fixed.
https://github.com/restic/restic/issues/3918
https://github.com/restic/restic/pull/3980

View File

@@ -0,0 +1,11 @@
Change: Make `unlock` display message only when locks were actually removed
The `unlock` command used to print the "successfully removed locks" message
whenever it was run, regardless of lock files having being removed or not.
This has now been changed such that it only prints the message if any lock
files were actually removed. In addition, it also reports the number of
removed lock files.
https://github.com/restic/restic/issues/3929
https://github.com/restic/restic/pull/3935

View File

@@ -0,0 +1,15 @@
Enhancement: Improve handling of ErrDot errors in rclone and sftp backends
Since Go 1.19, restic can no longer implicitly run relative executables which
are found in the current directory (e.g. `rclone` if found in `.`). This is a
security feature of Go to prevent against running unintended and possibly
harmful executables.
The error message for this was just "cannot run executable found relative to
current directory". This has now been improved to yield a more specific error
message, informing the user how to explicitly allow running the executable
using the `-o rclone.program` and `-o sftp.command` extended options with `./`.
https://github.com/restic/restic/issues/3932
https://pkg.go.dev/os/exec#hdr-Executables_in_the_current_directory
https://go.dev/blog/path-security

View File

@@ -0,0 +1,8 @@
Bugfix: Make `backup` no longer hang on Solaris when seeing a FIFO file
The `backup` command used to hang on Solaris whenever it encountered a FIFO
file (named pipe), due to a bug in the handling of extended attributes. This
bug has now been fixed.
https://github.com/restic/restic/issues/4003
https://github.com/restic/restic/pull/4053

View File

@@ -0,0 +1,8 @@
Bugfix: Support ExFAT-formatted local backends on macOS Ventura
ExFAT-formatted disks could not be used as local backends starting from macOS
Ventura. Restic commands would fail with an "inappropriate ioctl for device"
error. This has now been fixed.
https://github.com/restic/restic/issues/4016
https://github.com/restic/restic/pull/4021

View File

@@ -0,0 +1,11 @@
Change: Don't print skipped snapshots by default in `copy` command
The `copy` command used to print each snapshot that was skipped because it
already existed in the target repository. The amount of this output could
practically bury the list of snapshots that were actually copied.
From now on, the skipped snapshots are by default not printed at all, but
this can be re-enabled by increasing the verbosity level of the command.
https://github.com/restic/restic/issues/4033
https://github.com/restic/restic/pull/4066

View File

@@ -0,0 +1,10 @@
Bugfix: Make `init` ignore "Access Denied" errors when creating S3 buckets
In restic 0.9.0 through 0.13.0, the `init` command ignored some permission
errors from S3 backends when trying to check for bucket existence, so that
manually created buckets with custom permissions could be used for backups.
This feature became broken in 0.14.0, but has now been restored again.
https://github.com/restic/restic/issues/4085
https://github.com/restic/restic/pull/4086

View File

@@ -0,0 +1,10 @@
Bugfix: Don't generate negative UIDs and GIDs in tar files from `dump`
When using a 32-bit build of restic, the `dump` command could in some cases
create tar files containing negative UIDs and GIDs, which cannot be read by
GNU tar. This corner case especially applies to backups from stdin on Windows.
This is now fixed such that `dump` creates valid tar files in these cases too.
https://github.com/restic/restic/issues/4103
https://github.com/restic/restic/pull/4104

View File

@@ -0,0 +1,17 @@
Enhancement: Restore files with long runs of zeros as sparse files
When using `restore --sparse`, the restorer may now write files containing long
runs of zeros as sparse files (also called files with holes), where the zeros
are not actually written to disk.
How much space is saved by writing sparse files depends on the operating
system, file system and the distribution of zeros in the file.
During backup restic still reads the whole file including sparse regions, but
with optimized processing speed of sparse regions.
https://github.com/restic/restic/issues/79
https://github.com/restic/restic/issues/3903
https://github.com/restic/restic/pull/2601
https://github.com/restic/restic/pull/3854
https://forum.restic.net/t/sparse-file-support/1264

View File

@@ -0,0 +1,7 @@
Enhancement: Make backup file read concurrency configurable
The `backup` command now supports a `--read-concurrency` option which allows
tuning restic for very fast storage like NVMe disks by controlling the number
of concurrent file reads during the backup process.
https://github.com/restic/restic/pull/2750

View File

@@ -0,0 +1,8 @@
Bugfix: Make `restore` replace existing symlinks
When restoring a symlink, restic used to report an error if the target path
already existed. This has now been fixed such that the potentially existing
target path is first removed before the symlink is restored.
https://github.com/restic/restic/issues/2578
https://github.com/restic/restic/pull/3780

View File

@@ -0,0 +1,6 @@
Enhancement: Optimize prune memory usage
The `prune` command needs large amounts of memory in order to determine what to
keep and what to remove. This is now optimized to use up to 30% less memory.
https://github.com/restic/restic/pull/3899

View File

@@ -0,0 +1,6 @@
Enhancement: Improve speed of parent snapshot detection in `backup` command
Backing up a large number of files using `--files-from-verbatim` or `--files-from-raw`
options could require a long time to find the parent snapshot. This has been improved.
https://github.com/restic/restic/pull/3905

View File

@@ -0,0 +1,12 @@
Enhancement: Add compression statistics to the `stats` command
When executed with `--mode raw-data` on a repository that supports compression,
the `stats` command now calculates and displays, for the selected repository or
snapshots: the uncompressed size of the data; the compression progress
(percentage of data that has been compressed); the compression ratio of the
compressed data; the total space saving.
It also takes into account both the compressed and uncompressed data if the
repository is only partially compressed.
https://github.com/restic/restic/pull/3915

View File

@@ -0,0 +1,6 @@
Enhancement: Provide command completion for PowerShell
Restic already provided generation of completion files for bash, fish and zsh.
Now powershell is supported, too.
https://github.com/restic/restic/pull/3925/files

View File

@@ -0,0 +1,10 @@
Enhancement: Allow `backup` file tree scanner to be disabled
The `backup` command walks the file tree in a separate scanner process to find
the total size and file/directory count, and uses this to provide an ETA. This
can slow down backups, especially of network filesystems.
The command now has a new option `--no-scan` which can be used to disable this
scanning in order to speed up backups when needed.
https://github.com/restic/restic/pull/3931

View File

@@ -0,0 +1,9 @@
Enhancement: Ignore additional/unknown files in repository
If a restic repository had additional files in it (not created by restic),
commands like `find` and `restore` could become confused and fail with an
`multiple IDs with prefix "12345678" found` error. These commands now
ignore such additional files.
https://github.com/restic/restic/pull/3943
https://forum.restic.net/t/which-protocol-should-i-choose-for-remote-linux-backups/5446/17

View File

@@ -0,0 +1,7 @@
Bugfix: Make `ls` return exit code 1 if snapshot cannot be loaded
The `ls` command used to show a warning and return exit code 0 when failing
to load a snapshot. This has now been fixed such that it instead returns exit
code 1 (still showing a warning).
https://github.com/restic/restic/pull/3951

View File

@@ -0,0 +1,9 @@
Enhancement: Improve `backup` performance for small files
When backing up small files restic was slower than it could be. In particular
this affected backups using maximum compression.
This has been fixed by reworking the internal parallelism of the backup
command, making it back up small files around two times faster.
https://github.com/restic/restic/pull/3955

View File

@@ -0,0 +1,7 @@
Change: Update dependencies and require Go 1.18 or newer
Most dependencies have been updated. Since some libraries require newer language
features, support for Go 1.15-1.17 has been dropped, which means that restic now
requires at least Go 1.18 to build.
https://github.com/restic/restic/pull/4041

View File

@@ -0,0 +1,11 @@
Bugfix: Make `self-update` enabled by default only in release builds
The `self-update` command was previously included by default in all builds of
restic as opposed to only in official release builds, even if the `selfupdate`
tag was not explicitly enabled when building.
This has now been corrected, and the `self-update` command is only available
if restic was built with `-tags selfupdate` (as done for official release
builds by `build.go`).
https://github.com/restic/restic/pull/4100

View File

@@ -0,0 +1,10 @@
Bugfix: Remove `b2_download_file_by_name: 404` warning from B2 backend
In some cases the B2 backend could print `b2_download_file_by_name: 404: :
b2.b2err` warnings. These are only debug messages and can be safely ignored.
Restic now uses an updated library for accessing B2, which removes the warning.
https://github.com/restic/restic/issues/3750
https://github.com/restic/restic/issues/4144
https://github.com/restic/restic/pull/4146

View File

@@ -0,0 +1,7 @@
Bugfix: Make `prune --quiet` not print progress bar
A regression in restic 0.15.0 caused `prune --quiet` to show a progress bar
while deciding how to process each pack files. This has now been fixed.
https://github.com/restic/restic/issues/4147
https://github.com/restic/restic/pull/4153

View File

@@ -0,0 +1,19 @@
Enhancement: Ignore empty lock files
With restic 0.15.0 the checks for stale locks became much stricter than before.
In particular, empty or unreadable locks were no longer silently ignored. This
made restic to complain with `Load(<lock/1234567812>, 0, 0) returned error,
retrying after 552.330144ms: load(<lock/1234567812>): invalid data returned`
and fail in the end.
The error message is now clarified and the implementation changed to ignore
empty lock files which are sometimes created as the result of a failed uploads
on some backends.
Please note that unreadable lock files still have to cleaned up manually. To do
so, you can run `restic unlock --remove-all` which removes all existing lock
files. But first make sure that no other restic process is currently using the
repository.
https://github.com/restic/restic/issues/4143
https://github.com/restic/restic/pull/4152

View File

@@ -0,0 +1,13 @@
Bugfix: Make `self-update --output` work with new filename on Windows
Since restic 0.14.0 the `self-update` command did not work when a custom output
filename was specified via the `--output` option. This has now been fixed.
As a workaround, either use an older restic version to run the self-update or
create an empty file with the output filename before updating e.g. using CMD:
`type nul > new-file.exe`
`restic self-update --output new-file.exe`
https://github.com/restic/restic/pull/4163
https://forum.restic.net/t/self-update-windows-started-failing-after-release-of-0-15/5836

View File

@@ -0,0 +1,6 @@
Bugfix: Add missing ETA in `backup` progress bar
A regression in restic 0.15.0 caused the ETA to be missing from the progress
bar displayed by the `backup` command. This has now been fixed.
https://github.com/restic/restic/pull/4167

View File

@@ -0,0 +1,12 @@
Bugfix: Sanitize filenames printed by `backup` during processing
The `backup` command would previously not sanitize the filenames it printed
during processing, potentially causing newlines or terminal control characters
to mangle the status output or even change the state of a terminal.
Filenames are now checked and quoted if they contain non-printable or
non-Unicode characters.
https://github.com/restic/restic/issues/2260
https://github.com/restic/restic/issues/4191
https://github.com/restic/restic/pull/4192

View File

@@ -0,0 +1,8 @@
Bugfix: Make `dump` interpret `--host` and `--path` correctly
A regression in restic 0.15.0 caused `dump` to confuse its `--host=<host>` and
`--path=<path>` options: it looked for snapshots with paths called `<host>`
from hosts called `<path>`. It now treats the options as intended.
https://github.com/restic/restic/issues/4211
https://github.com/restic/restic/pull/4212

Some files were not shown because too many files have changed in this diff Show More