Binary writers: zero only a window ahead of the vector sink's position

output_vector_sink grew the vector by resizing it to its whole new
capacity. resize() zeroes the characters it adds, so every doubling
zeroed megabytes of a fresh allocation at once: that faults in all its
pages and evicts them from the cache before the writer reaches them,
and the writer's stores then miss again. The sink now reserves the same
capacity as before but resizes the vector only to 64 KiB
(vector_sink_window) ahead of the position, which is still cached when
the writer's stores come.

to_cbor(), to_msgpack(), to_ubjson(), to_bjdata() and to_bon8() of an
array of one million numbers into a fresh vector: 4-26 % faster on x86
(Skylake laptop with Apple clang 13, Haswell VPS with GCC 16 and
Clang 23), 1-2.5 % on Apple M1 Max; other documents unchanged (within
0.4 % in reruns). The capacity and growth of the vector are as before.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

include/nlohmann/detail/output/output_adapters.hpp

@@ -116,6 +116,10 @@ inline void copy_short(CharType* dst, const CharType* src, const std::size_t len
 /// the number of characters an empty vector is grown to by output_vector_sink
 constexpr std::size_t min_vector_sink_capacity = 64;
 
+/// the number of characters output_vector_sink makes writable ahead of its
+/// position at a time: 64 KiB of the vector's capacity, not all of it
+constexpr std::size_t vector_sink_window = 65536;
+
 /// @brief non-virtual output sink writing into a std::vector
 ///
 /// This sink is not part of the virtual output_adapter_protocol hierarchy: it is
@@ -129,7 +133,8 @@ constexpr std::size_t min_vector_sink_capacity = 64;
 /// encoding them: insert() is an out-of-line call into a routine that can insert
 /// anywhere, and resize() is one as well with libstdc++. The sink therefore keeps
 /// its own write position: it grows the vector geometrically ahead of the
-/// position and copies each write to it, so a write is a capacity check and a
+/// position (making @ref vector_sink_window characters of it writable at a
+/// time) and copies each write to it, so a write is a capacity check and a
 /// short copy. The destructor shrinks the vector to the bytes actually written,
 /// also when a write throws; shrinking a vector never allocates. The sink owns
 /// that trim, so it can be moved but not copied, and the vector must not be used
@@ -277,9 +282,17 @@ class output_vector_sink
         // reallocate for every few bytes
         const std::size_t written = position();
         const std::size_t needed = written + length;
-        v->resize((std::max)(min_vector_sink_capacity, needed <= v->capacity()
+        const std::size_t capacity = (std::max)(min_vector_sink_capacity, needed <= v->capacity()
-                             ? v->capacity()
+                                                ? v->capacity()
-                             : (std::max)(needed, v->capacity() * 2)));
+                                                : (std::max)(needed, v->capacity() * 2));
+        v->reserve(capacity);
+        // resize() zeroes the characters it adds. Zeroing all of a doubled
+        // capacity, megabytes of a fresh allocation, at once faults in its
+        // pages and evicts them from the cache before the writer gets there,
+        // so its stores miss again; a window ahead of the position is still
+        // cached when they come (the binary writers on a long array of
+        // numbers: 4-26 % faster on x86, 1-2.5 % on Apple M1)
+        v->resize((std::min)(capacity, (std::max)(needed, written + vector_sink_window)));
         cursor = v->data() + written;
         limit = v->data() + v->size();
     }

single_include/nlohmann/json.hpp

@@ -27491,6 +27491,10 @@ inline void copy_short(CharType* dst, const CharType* src, const std::size_t len
 /// the number of characters an empty vector is grown to by output_vector_sink
 constexpr std::size_t min_vector_sink_capacity = 64;
 
+/// the number of characters output_vector_sink makes writable ahead of its
+/// position at a time: 64 KiB of the vector's capacity, not all of it
+constexpr std::size_t vector_sink_window = 65536;
+
 /// @brief non-virtual output sink writing into a std::vector
 ///
 /// This sink is not part of the virtual output_adapter_protocol hierarchy: it is
@@ -27504,7 +27508,8 @@ constexpr std::size_t min_vector_sink_capacity = 64;
 /// encoding them: insert() is an out-of-line call into a routine that can insert
 /// anywhere, and resize() is one as well with libstdc++. The sink therefore keeps
 /// its own write position: it grows the vector geometrically ahead of the
-/// position and copies each write to it, so a write is a capacity check and a
+/// position (making @ref vector_sink_window characters of it writable at a
+/// time) and copies each write to it, so a write is a capacity check and a
 /// short copy. The destructor shrinks the vector to the bytes actually written,
 /// also when a write throws; shrinking a vector never allocates. The sink owns
 /// that trim, so it can be moved but not copied, and the vector must not be used
@@ -27652,9 +27657,17 @@ class output_vector_sink
         // reallocate for every few bytes
         const std::size_t written = position();
         const std::size_t needed = written + length;
-        v->resize((std::max)(min_vector_sink_capacity, needed <= v->capacity()
+        const std::size_t capacity = (std::max)(min_vector_sink_capacity, needed <= v->capacity()
-                             ? v->capacity()
+                                                ? v->capacity()
-                             : (std::max)(needed, v->capacity() * 2)));
+                                                : (std::max)(needed, v->capacity() * 2));
+        v->reserve(capacity);
+        // resize() zeroes the characters it adds. Zeroing all of a doubled
+        // capacity, megabytes of a fresh allocation, at once faults in its
+        // pages and evicts them from the cache before the writer gets there,
+        // so its stores miss again; a window ahead of the position is still
+        // cached when they come (the binary writers on a long array of
+        // numbers: 4-26 % faster on x86, 1-2.5 % on Apple M1)
+        v->resize((std::min)(capacity, (std::max)(needed, written + vector_sink_window)));
         cursor = v->data() + written;
         limit = v->data() + v->size();
     }

tests/src/unit-binary_writer_sinks.cpp

@@ -50,6 +50,14 @@ std::vector<json> test_values()
         large_payloads.push_back(json::binary(std::vector<std::uint8_t>(static_cast<std::size_t>(100 * i), static_cast<std::uint8_t>(i))));
     }
 
+    // several times vector_sink_window characters in every format: the vector
+    // sink makes the capacity it grows to writable a window at a time
+    json many_windows = json::array();
+    for (std::uint64_t i = 0; i < 30000; ++i)
+    {
+        many_windows.push_back((i * 0x9E3779B97F4A7C15u) >> 1u);
+    }
+
     return
     {
         json(nullptr), json(true), json(false),
@@ -61,7 +69,7 @@ std::vector<json> test_values()
         json::array(), json::object(),
         json::array({1, 2, 3}), json({{"a", 1}, {"b", nullptr}}),
         json({{"nested", {{"deep", json::array({1, "two", 3.0, nullptr})}}}}),
-        big_array, big_object, large_payloads,
+        big_array, big_object, large_payloads, many_windows,
         json::binary(std::vector<std::uint8_t>(5000, 0x42))
     };
 }