Commit Graph
599 Commits
Author SHA1 Message Date
Rob MartinandMichael Eischer 2c9b353cbf repository: do not delete config on atomic backends during upgrade rollback (#22136)
In UpgradeRepo, if upgradeRepository fails, the error handler attempts
contingency rollback by re-uploading the original config file.

However, the contingency path unconditionally executed repo.be.Remove(ctx, h)
before calling repo.be.Save. On backends supporting atomic replacement (such
as local filesystems), a failed upgrade (e.g. out-of-disk-space or permissions
error while writing the temporary file) leaves the original, valid version 1
configuration file completely intact on disk.

Unconditionally removing the config file during rollback actively deletes
the working configuration. If the subsequent Save call also fails, the
repository is left permanently corrupted without a config file.

Condition the pre-removal on !repo.be.Properties().HasAtomicReplace, matching
the forward upgrade path in upgradeRepository. Add a unit test asserting that
aborted upgrades on atomic backends preserve the intact original configuration.

---------

Co-authored-by: Michael Eischer <michael.eischer@fau.de>
2026-11-15 18:57:01 +00:00
Michael Eischer f515b862de prune: simplify test setup 2026-11-05 21:15:09 +01:00
Pujitha Paladugu bba1b68082 prune: report failure to list pack files as a fatal error
Motivation: when listing pack files fails during prune (for example due
to a request timeout), the error was returned unwrapped and main printed
it with "%+v", which dumps a stack trace. As discussed in the issue, the
error needs one more level of wrapping before it can be treated as a
fatal error.

Approach: in decidePackAction, wrap an error from listing pack files
with errors.Fatalf("failed to list pack files: %v"). Errors that are
already fatal (e.g. ErrSizeNotMatching) and errors after context
cancellation are returned unchanged. The underlying error is preserved
via Unwrap. Retry log messages are unchanged; only the final error
output is shortened and says where the failure occurred.

Validation: `go test ./internal/repository/ -run Prune` and
`go test ./cmd/restic/ -run Prune` pass. The new test
TestPrunePackListError (backend whose pack list fails) fails without
the change and passes with it. I did not reproduce a real REST backend
timeout.

Report: https://github.com/restic/restic/issues/22120
Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
Assisted-by: claude-sonnet-5-5 (via Claude Code)
2026-10-18 12:58:49 -07:00
MikeeI adc0284113 copy: avoid a full index scan for disabled debug output (#22103) 2026-10-18 13:50:05 +00:00
JayandMichael Eischer 519a979107 ci: update golangci-lint to v2.14 (#22111)
Co-authored-by: Michael Eischer <michael.eischer@fau.de>
2026-10-11 16:33:41 +00:00
Michael Eischer 818dd21a4f index: hash blob IDs with maphash.Bytes (#22070)
Avoid constructing a maphash.Hash on every lookup and insert.
2026-09-25 22:05:12 +02:00
Michael Eischer 6adedec6b4 repository: use synctest for lock tests (#22038)
Replace select based on context/timer with deterministic pattern.
The tests depend on timing. Thus, use synctest which provides
deterministic clock updates and also speeds up test execution by
skipping sleep times.
2026-09-20 22:05:21 +02:00
Michael Eischer c6074f82fb Merge pull request #21978 from MichaelEischer/go-fix2
Run `go fix ./...`
2026-08-01 22:17:53 +02:00
Arpit Jain 8baffc4027 repository: reject key and config files that are too short (#21977)
Signed-off-by: Arpit Jain <arpitjain099@gmail.com>
2026-07-22 20:38:07 +00:00
Michael Eischer 14e523a1d9 Apply go fix -testingcontext ./... 2026-07-22 22:27:15 +02:00
Michael Eischer 2c169edad4 Apply go fix -rangeint ./... 2026-07-22 22:26:19 +02:00
Michael Eischer 8ed263c6dd Apply go fix -minmax ./... 2026-07-22 22:25:56 +02:00
Michael Eischer da4d8f5851 Apply go fix -fmtappendf ./... 2026-07-22 22:25:26 +02:00
Michael Eischer 32e9869cc8 Apply go fix -any ./... 2026-07-22 22:25:18 +02:00
Matthias Dötsch 905ca56ee8 Address review feedback on #21937:
- addToPacks now returns uint32 directly and the value is passed through
  store() to indexMap.add unchanged, instead of casting between int and
  uint32.
- merge() appends idx2.packs to idx.packs without going through
  addToPacks, so it now checks the pack count against the uint32 limit
  itself + error if exceeded.
- optimize changelog message
2026-07-13 07:25:08 +02:00
Matthias Dötsch e428de5f84 Reduce in-memory index entry size by storing packIndex as uint32
Each blob in a repository requires one index entry in memory, which is one
of the largest contributors to restic's memory usage. The indexEntry.packIndex
field, an index into the per-Index packs slice, was stored as a machine-word
int (8 bytes on amd64) although the value is always a small array index.

Store it as a uint32 instead. This shrinks indexEntry from 64 to 56 bytes on
64-bit platforms (56 % 8 == 0, so no padding is added), reducing index memory
by roughly 12% -- about 100 MB for a repository with ~13 million blobs, as
confirmed by profiling. uint32 allows 4.29 billion packs (>30 billion blobs at
the assumed 8 blobs/pack), far beyond any real repository; addToPacks guards
against overflow defensively. The on-disk repository format is unaffected.
2026-07-08 06:01:15 +00:00
Winfried PlappertandMichael Eischer 75de8b54e6 restic check and restic repair packs: treat missing packfiles the same as damaged and truncated packfiles (#21845)
Co-authored-by: Michael Eischer <michael.eischer@fau.de>
2026-06-28 12:09:42 +02:00
Michael Eischer 8cc69782b5 repository: expose ZeroChunk via chunker factory 2026-06-27 22:23:59 +02:00
Michael Eischer ccb828e87b repository: add ChunkerFactory and decouple archiver from chunker
Introduce restic.Chunker and ChunkerFactory interfaces with a
repository-backed implementation. The archiver obtains chunkers via
ChunkerFactory() from the repository.
2026-06-27 22:13:41 +02:00
Gilbert Gilb's db03aed816 feat(backends/s3): add warmup support for check command (#5248) 2026-06-25 20:46:55 +02:00
Michael Eischer 072305c5b3 repository: run version sub-tests of TestAllVersions in parallel
TestRepositoryIncrementalIndex no longer modified index.Full as this
modified global state and also had no effect on the test result.
2026-06-20 19:59:37 +02:00
Michael Eischer a1420ea6c7 repository: use separate rand instance per test
Use separate instances to prevent data races when executing tests in
parallel.
2026-06-20 19:50:10 +02:00
Michael Eischer b24d05bcb4 skip key decryption in tests 2026-06-20 19:46:22 +02:00
Michael Eischer f91afb2fa8 repository: run TestPruneSmall in parallel to other tests 2026-06-20 19:46:22 +02:00
Michael Eischer 2148508050 repository/checker: speed up test repo creation
data.TestCreateSnapshot is much faster than archiver.TestSnapshot
2026-06-20 19:46:22 +02:00
Michael Eischer 37e8f50aeb repository: decrease test data size for streamPack 2026-06-20 19:46:22 +02:00
Michael Eischer 7e5eaf1ead crypto: decrease parameters for calibrate test 2026-06-20 19:46:22 +02:00
Michael Eischer aae0985673 index: decrease depth of test repos 2026-06-20 19:46:22 +02:00
Michael Eischer f9d953d4cd index: test ForAllIndex with generated repo 2026-06-20 19:46:22 +02:00
Michael Eischer 69a3ba5b33 index: speed up oversized test data generation 2026-06-20 19:46:22 +02:00
Michael Eischer 1174e97b30 restic: speed up repository config initialization in tests
Avoid redundant polynomial generation, which takes a millisecond or so.
2026-06-20 19:46:22 +02:00
Michael Eischer 440d925467 repository: default to fastest compression in tests 2026-06-20 19:46:22 +02:00
Michael Eischer 9ab5fc59c2 restic: split FileType for backend.FileType
Equality of constants is enforced via internal/repository/filetype.go
using compile time checks.
2026-06-20 19:15:40 +02:00
Michael Eischer 6c509f7ac1 Correctly pair backend/restic.FileType with APIs
Use backend.FileType to interact with the backend. And restic.FileType
to interact with the repository.
2026-06-20 19:15:40 +02:00
Michael Eischer d9d54a505e restic: move Printer interface from internal/ui/progress
Move Printer and NewNoopPrinter to internal/restic so repository does
not have to import the ui packages.
2026-06-20 17:49:20 +02:00
Michael Eischer 0f4236cb39 repository: return unlock func from LockRepo
Drop the Unlocker interface and return the unlock callback directly
from LockRepo, simplifying callers that only need to defer unlock().
2026-06-20 17:48:35 +02:00
Michael Eischer 4d1b9cef63 internal/fileio: extract low-level file I/O from internal/fs
Move TempFile and PreallocateFile into internal/fileio. internal/fs
primarily focuses on converting between data.Node and the actual
filesystem state. Extract the two methods to not pull in unnecessary
dependencies.
2026-06-20 17:48:35 +02:00
Michael Eischer 1a9decf03e repository: show correct error if zstd decoder cannot be initialized 2026-06-14 16:52:40 +02:00
Michael Eischer 82708ff7f7 prune/repair pack: document ignored errors 2026-06-14 16:51:56 +02:00
Michael Eischer 7d36449ea8 repository: change LoadBlob, LookupBlob and LookupBlobSize to BlobHandles (#21857) 2026-06-14 15:30:00 +02:00
Michael Eischer 71f38d266c ensure reliable cleanup of test repository (#21880) 2026-06-14 14:25:00 +02:00
Michael Eischer 650f658244 repository/index: ensure progress bar cancelation on early index load failure (#21881) 2026-06-14 13:56:24 +02:00
Michael Eischer 406dec29a7 repository: remove redundant mutex in checkForOtherLocks
forAllLocks already serializes the callback calls. Still has to create a
copy of checkedIDs to prevent a data race between forAllLocks and the
callback.
2026-06-13 21:37:24 +02:00
Michael Eischer 6fe10a72eb repository: drop mutex from lockHandle 2026-06-13 21:37:06 +02:00
Michael Eischer 1a6d43286a repository: comment cleanups 2026-06-13 21:36:36 +02:00
Michael Eischer 151903d4d4 repository: fix goroutine leak due to delayedCancelContext in tests 2026-06-13 21:35:51 +02:00
Michael Eischer b86568b790 repository: merge Unlocker and lockContext 2026-06-13 21:35:41 +02:00
Michael Eischer a5f3256626 repository: partially deduplicate refresh and refreshStaleLock 2026-06-13 21:33:58 +02:00
Michael Eischer 5989671f87 repository: deduplicate error handling in checkForOtherLocks 2026-06-13 21:33:43 +02:00
Michael Eischer b7f3a1367a repository: add SIGHUP handler only on Unix systems 2026-06-13 21:33:27 +02:00