- addToPacks now returns uint32 directly and the value is passed through
store() to indexMap.add unchanged, instead of casting between int and
uint32.
- merge() appends idx2.packs to idx.packs without going through
addToPacks, so it now checks the pack count against the uint32 limit
itself + error if exceeded.
- optimize changelog message
Each blob in a repository requires one index entry in memory, which is one
of the largest contributors to restic's memory usage. The indexEntry.packIndex
field, an index into the per-Index packs slice, was stored as a machine-word
int (8 bytes on amd64) although the value is always a small array index.
Store it as a uint32 instead. This shrinks indexEntry from 64 to 56 bytes on
64-bit platforms (56 % 8 == 0, so no padding is added), reducing index memory
by roughly 12% -- about 100 MB for a repository with ~13 million blobs, as
confirmed by profiling. uint32 allows 4.29 billion packs (>30 billion blobs at
the assumed 8 blobs/pack), far beyond any real repository; addToPacks guards
against overflow defensively. The on-disk repository format is unaffected.
This removes them from the public interface. The latter now only
provides the PackBlob interface, without being bound to the type used
internally by the pack package.
In restic <0.10.0, it was possible that the blobs of a pack file were
split across multiple indexes. `MasterIndex.Rewrite` however assumed
that each an index always contains the full description of a pack file.
Therefore, further index entries for a pack were filtered out as
duplicates. Now, the code also checks the blobs contained in the index
entry while filtering out duplicates.
Those methods now only allow modifying snapshots. Internal data types
used by the repository are now read-only. The repository-internal code
can bypass the restrictions by wrapping the repository in an
`internalRepository` type.
The restriction itself is implemented by using a new datatype
WriteableFileType in the SaveUnpacked and RemoveUnpacked methods. This
statically ensures that code cannot bypass the access restrictions.
The test changes are somewhat noisy as some of them modify repository
internals and therefore require some way to bypass the access
restrictions. This works by capturing an `internalRepository` or
`Backend` when creating the Repository using a test helper function.
Matthias Dötsch
Michael Eischer
Martin Smith