Each blob in a repository requires one index entry in memory, which is one
of the largest contributors to restic's memory usage. The indexEntry.packIndex
field, an index into the per-Index packs slice, was stored as a machine-word
int (8 bytes on amd64) although the value is always a small array index.
Store it as a uint32 instead. This shrinks indexEntry from 64 to 56 bytes on
64-bit platforms (56 % 8 == 0, so no padding is added), reducing index memory
by roughly 12% -- about 100 MB for a repository with ~13 million blobs, as
confirmed by profiling. uint32 allows 4.29 billion packs (>30 billion blobs at
the assumed 8 blobs/pack), far beyond any real repository; addToPacks guards
against overflow defensively. The on-disk repository format is unaffected.
Introduce restic.Chunker and ChunkerFactory interfaces with a
repository-backed implementation. The archiver obtains chunkers via
ChunkerFactory() from the repository.
Update the chunker dependency. Move the chunk memory handling into the
fileSaver. This minimizes the logic required in a chunker
implementation.
readNextChunk is a direct derivative of Chunker.Next.
The treeCache in SnapshotsDir was never cleared when snapshots were
reloaded. This caused the "latest" symlink to keep pointing to the
previous snapshot even after new snapshots were added.
Add a generation counter to SnapshotsDirStructure that is incremented
whenever the directory structure is rebuilt (in makeDirs). The
treeCache checks this generation on each lookup and resets itself
when the generation changes, ensuring cached nodes (including symlinks)
are refreshed after a snapshot reload.
The treeCache in SnapshotsDir was never cleared when snapshots were
reloaded. This caused the "latest" symlink to keep pointing to the
previous snapshot even after new snapshots were added.
Add a generation counter to SnapshotsDirStructure that is incremented
whenever the directory structure is rebuilt (in makeDirs). The
treeCache checks this generation on each lookup and resets itself
when the generation changes, ensuring cached nodes (including symlinks)
are refreshed after a snapshot reload.
Modern filesystems provide microsecond granularity or better. The old
special case for macOS is removed as Go 1.25 (minimum for restic)
requires at least macOS 12, which uses APFS instead of the old HFS+.
Each time a local backend is created, the local backend also creates all
shard subdirectories. For the integration tests this has the downside
that this results in ~120 (number of test) * 256 (number of directories)
= 30k directories that are created unnecessarily. This significantly
slows down test execution and cleanup.
Matthias Dötsch
Michael Eischer
Alexander Neumann
Gilbert Gilb'sandGitHub
Michael EischerandGitHub
Ricardo SawirandMichael Eischer