TestParallelSaveWithDuplication has been reworked to provoke
unreferenced packs using fewer goroutines than before and create
only one bytes.Reader per blob. This reduces memory usage
significantly.
The following actions have been taken to keep the chance of provoking
unreferenced packs due to #358 high:
* Interweaved processing of subsequent chunks
* Delaying each goroutine by a few pseudo-randomly chosen nanoseconds
(depending on the platform this will most probably only make the os
yield execution to another thread): together with the interweaved
processing of subsequent chunks, this ensures a minimalistic delay
between processing of (some) duplicated chunks
* Repeating the test 5 times with different seeds
On my test machine, the modified test provoked unreferenced packs 60
times in a row.
This commit adds an integration test, that calls Archiver.Save from
many goroutines processing several duplicated chunks concurrently.
The test asserts, that after all chunks have been saved, there are no
unreferenced packs in the repository.
The test has been checked to give the expected results:
1) Running the test with maxParallel=1 (all chunks are processed
sequentially) has been verified not to produce any unreferenced
packs. Consequently the test passes.
2) Running the test with unbounded parallelism (maxParallel=
math.MaxInt32) has been verified to produce unreferenced packs
all the time (at least 25 test runs). Consequently the test fails
due to #358.
references: #358
... by first adding a preliminary index entry and making this fail if
an index entry for the same blob already exists.
A preliminary index entry is characterized by not yet being associated
with a pack. Until now, these entries where added to the index just
like final index entries using index.Store, which silently overwrites
existing index entries.
This commit adds a new method index.StoreInProgress which refuses to
overwrite existing index entries and allows for creating preliminary
index entries only. The existing method index.Store has not been
changed and continues to silently overwrite existing index entries.
This distinction is important, as otherwise, it would be impossible to
update a preliminary index entry after the blob has been written to a
pack.
Resolves: restic#292
Philipp Serr