Add benchmarks for the binary readers

The benchmark suite covered parsing JSON text, dumping, and serializing to
CBOR, but only one binary read: FromMsgpack. Nothing measured from_cbor,
from_ubjson, from_bjdata or from_bson, so a change to binary_reader.hpp had
no baseline to be compared against.

Add read benchmarks for every format, in the two shapes that matter: from a
contiguous buffer, which is what most callers pass, and from a FILE*, which
is what FromMsgpack already measures and which compiles to different code.
FromMsgpack itself is left untouched so its numbers stay comparable across
releases. The input is derived at setup time by serializing a parsed test
file, because the test data repository ships JSON only.

The test files are wide and shallow, but the readers' cost is per container,
so add three value shapes they do not cover -- deeply nested containers, many
sibling containers, and one flat array of numbers -- plus an indefinite-length
CBOR string, a form the writer never emits and which therefore has to be
assembled by hand. UBJSON and BJData are also captured in their size- and
type-annotated form, which the readers handle in a separate code path.

BSON requires an object at the top level, so it cannot reuse the array-rooted
test files; it is captured on the object-rooted ones, and the shapes are
wrapped in an object so every format measures the same value. The setup marks
the benchmark as skipped rather than letting the exception escape if that
requirement is ever violated.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

tests/benchmarks/src/benchmarks.cpp

@@ -214,4 +214,323 @@ static void BinaryToCbor(benchmark::State& state)
 }
 BENCHMARK(BinaryToCbor)->RangeMultiplier(2)->Range(8, 8 << 12);
 
+//////////////////////////////////////////////////////////////////////////////
+// parse binary formats
+//////////////////////////////////////////////////////////////////////////////
+
+// Only MessagePack had a read benchmark (FromMsgpack above, left untouched so
+// its numbers stay comparable across releases). The benchmarks below cover the
+// other formats, and read from a contiguous buffer as well as from a FILE*:
+// most callers pass a container, and the two adapters compile to different
+// code. The test data repository ships JSON only, so the input for each is
+// derived at setup time by serializing a parsed test file.
+
+/// binary format to benchmark; the _optimized variants add UBJSON/BJData size
+/// and type annotations, which the readers handle in a separate code path
+enum class binary_format
+{
+    cbor,
+    msgpack,
+    ubjson,
+    ubjson_optimized,
+    bjdata,
+    bjdata_optimized,
+    bson
+};
+
+static std::vector<std::uint8_t> to_binary(const json& j, const binary_format format)
+{
+    switch (format)
+    {
+        case binary_format::cbor:
+            return json::to_cbor(j);
+        case binary_format::msgpack:
+            return json::to_msgpack(j);
+        case binary_format::ubjson:
+            return json::to_ubjson(j);
+        case binary_format::ubjson_optimized:
+            return json::to_ubjson(j, true, true);
+        case binary_format::bjdata:
+            return json::to_bjdata(j);
+        case binary_format::bjdata_optimized:
+            return json::to_bjdata(j, true, true);
+        case binary_format::bson:
+        default:
+            return json::to_bson(j);
+    }
+}
+
+static json from_binary(const std::vector<std::uint8_t>& bytes, const binary_format format)
+{
+    switch (format)
+    {
+        case binary_format::cbor:
+            return json::from_cbor(bytes);
+        case binary_format::msgpack:
+            return json::from_msgpack(bytes);
+        case binary_format::ubjson:
+        case binary_format::ubjson_optimized:
+            return json::from_ubjson(bytes);
+        case binary_format::bjdata:
+        case binary_format::bjdata_optimized:
+            return json::from_bjdata(bytes);
+        case binary_format::bson:
+        default:
+            return json::from_bson(bytes);
+    }
+}
+
+static json from_binary(std::FILE* file, const binary_format format)
+{
+    switch (format)
+    {
+        case binary_format::cbor:
+            return json::from_cbor(file);
+        case binary_format::msgpack:
+            return json::from_msgpack(file);
+        case binary_format::ubjson:
+        case binary_format::ubjson_optimized:
+            return json::from_ubjson(file);
+        case binary_format::bjdata:
+        case binary_format::bjdata_optimized:
+            return json::from_bjdata(file);
+        case binary_format::bson:
+        default:
+            return json::from_bson(file);
+    }
+}
+
+/*!
+@brief serialize a parsed test file to @a format
+
+Returns an empty vector and marks the benchmark as skipped if the file cannot
+be represented in the format, rather than letting the exception escape: BSON
+requires an object at the top level, and several test files are arrays.
+*/
+static std::vector<std::uint8_t> binary_input(benchmark::State& state, const char* filename, const binary_format format)
+{
+    std::ifstream f(filename);
+    std::string const str((std::istreambuf_iterator<char>(f)), std::istreambuf_iterator<char>());
+    const json j = json::parse(str);
+
+    if (format == binary_format::bson && !j.is_object())
+    {
+        state.SkipWithError("BSON requires an object at the top level");
+        return {};
+    }
+
+    return to_binary(j, format);
+}
+
+static void FromBinaryBuffer(benchmark::State& state, const char* filename, const binary_format format)
+{
+    const std::vector<std::uint8_t> bytes = binary_input(state, filename, format);
+    if (bytes.empty())
+    {
+        return;
+    }
+
+    for (auto _ : state)
+    {
+        // the value is destroyed outside the timed section, because destroying
+        // a large DOM is not what this benchmark measures
+        state.PauseTiming();
+        auto* j = new json();
+        state.ResumeTiming();
+
+        *j = from_binary(bytes, format);
+
+        state.PauseTiming();
+        delete j;
+        state.ResumeTiming();
+    }
+
+    state.SetBytesProcessed(state.iterations() * bytes.size());
+}
+
+BENCHMARK_CAPTURE(FromBinaryBuffer, cbor / jeopardy, TEST_DATA_DIRECTORY "/jeopardy/jeopardy.json", binary_format::cbor);
+BENCHMARK_CAPTURE(FromBinaryBuffer, cbor / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::cbor);
+BENCHMARK_CAPTURE(FromBinaryBuffer, cbor / citm_catalog, TEST_DATA_DIRECTORY "/nativejson-benchmark/citm_catalog.json", binary_format::cbor);
+BENCHMARK_CAPTURE(FromBinaryBuffer, cbor / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::cbor);
+BENCHMARK_CAPTURE(FromBinaryBuffer, cbor / floats, TEST_DATA_DIRECTORY "/regression/floats.json", binary_format::cbor);
+BENCHMARK_CAPTURE(FromBinaryBuffer, cbor / signed_ints, TEST_DATA_DIRECTORY "/regression/signed_ints.json", binary_format::cbor);
+BENCHMARK_CAPTURE(FromBinaryBuffer, msgpack / jeopardy, TEST_DATA_DIRECTORY "/jeopardy/jeopardy.json", binary_format::msgpack);
+BENCHMARK_CAPTURE(FromBinaryBuffer, msgpack / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::msgpack);
+BENCHMARK_CAPTURE(FromBinaryBuffer, msgpack / citm_catalog, TEST_DATA_DIRECTORY "/nativejson-benchmark/citm_catalog.json", binary_format::msgpack);
+BENCHMARK_CAPTURE(FromBinaryBuffer, msgpack / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::msgpack);
+BENCHMARK_CAPTURE(FromBinaryBuffer, ubjson / jeopardy, TEST_DATA_DIRECTORY "/jeopardy/jeopardy.json", binary_format::ubjson);
+BENCHMARK_CAPTURE(FromBinaryBuffer, ubjson / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::ubjson);
+BENCHMARK_CAPTURE(FromBinaryBuffer, ubjson / citm_catalog, TEST_DATA_DIRECTORY "/nativejson-benchmark/citm_catalog.json", binary_format::ubjson);
+BENCHMARK_CAPTURE(FromBinaryBuffer, ubjson / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::ubjson);
+BENCHMARK_CAPTURE(FromBinaryBuffer, ubjson_optimized / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::ubjson_optimized);
+BENCHMARK_CAPTURE(FromBinaryBuffer, ubjson_optimized / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::ubjson_optimized);
+BENCHMARK_CAPTURE(FromBinaryBuffer, bjdata / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::bjdata);
+BENCHMARK_CAPTURE(FromBinaryBuffer, bjdata / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::bjdata);
+BENCHMARK_CAPTURE(FromBinaryBuffer, bjdata_optimized / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::bjdata_optimized);
+BENCHMARK_CAPTURE(FromBinaryBuffer, bjdata_optimized / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::bjdata_optimized);
+// BSON requires an object at the top level, so the array-rooted test files
+// (jeopardy and the regression files) cannot be captured here
+BENCHMARK_CAPTURE(FromBinaryBuffer, bson / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::bson);
+BENCHMARK_CAPTURE(FromBinaryBuffer, bson / citm_catalog, TEST_DATA_DIRECTORY "/nativejson-benchmark/citm_catalog.json", binary_format::bson);
+BENCHMARK_CAPTURE(FromBinaryBuffer, bson / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::bson);
+
+static void FromBinaryFile(benchmark::State& state, const char* filename, const binary_format format)
+{
+    const std::vector<std::uint8_t> bytes = binary_input(state, filename, format);
+    if (bytes.empty())
+    {
+        return;
+    }
+
+    const char* tmp = "benchmark_input.bin";
+    std::ofstream o(tmp, std::ios::binary);
+    o.write(reinterpret_cast<const char*>(bytes.data()), static_cast<std::streamsize>(bytes.size()));
+    o.flush();
+    o.close();
+
+    for (auto _ : state)
+    {
+        state.PauseTiming();
+        auto* j = new json();
+        auto* file = std::fopen(tmp, "rb");
+        state.ResumeTiming();
+
+        *j = from_binary(file, format);
+
+        state.PauseTiming();
+        std::fclose(file);
+        delete j;
+        state.ResumeTiming();
+    }
+
+    state.SetBytesProcessed(state.iterations() * bytes.size());
+}
+
+BENCHMARK_CAPTURE(FromBinaryFile, cbor / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::cbor);
+BENCHMARK_CAPTURE(FromBinaryFile, cbor / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::cbor);
+BENCHMARK_CAPTURE(FromBinaryFile, ubjson / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::ubjson);
+BENCHMARK_CAPTURE(FromBinaryFile, ubjson / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::ubjson);
+BENCHMARK_CAPTURE(FromBinaryFile, bjdata / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::bjdata);
+BENCHMARK_CAPTURE(FromBinaryFile, bson / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::bson);
+
+//////////////////////////////////////////////////////////////////////////////
+// parse binary formats: value shapes
+//////////////////////////////////////////////////////////////////////////////
+
+// The test files above are wide and shallow, but the readers' cost is per
+// container, so these cover the shapes that stress the container handling
+// itself. Every shape is wrapped in an object so that BSON, which requires an
+// object at the top level, measures the same value as the other formats.
+
+/// deeply nested arrays: one container per level, no other work
+static json make_nested()
+{
+    json nested = json::array();
+    json* p = &nested;
+    for (std::size_t i = 1; i < 1000; ++i)
+    {
+        p->push_back(json::array());
+        p = &p->operator[](0);
+    }
+
+    json j = json::object();
+    j["data"] = std::move(nested);
+    return j;
+}
+
+/// many sibling containers: maximum container churn, minimum nesting
+static json make_containers()
+{
+    json data = json::array();
+    for (std::size_t i = 0; i < 100000; ++i)
+    {
+        data.push_back(json::array({1, 2}));
+    }
+
+    json j = json::object();
+    j["data"] = std::move(data);
+    return j;
+}
+
+/// one flat array of numbers: the scalar decoding path, which must not move
+static json make_scalars()
+{
+    json data = json::array();
+    for (std::size_t i = 0; i < 1000000; ++i)
+    {
+        data.push_back(i);
+    }
+
+    json j = json::object();
+    j["data"] = std::move(data);
+    return j;
+}
+
+static void FromBinaryShape(benchmark::State& state, json (*build)(), const binary_format format)
+{
+    const std::vector<std::uint8_t> bytes = to_binary(build(), format);
+
+    for (auto _ : state)
+    {
+        state.PauseTiming();
+        auto* j = new json();
+        state.ResumeTiming();
+
+        *j = from_binary(bytes, format);
+
+        state.PauseTiming();
+        delete j;
+        state.ResumeTiming();
+    }
+
+    state.SetBytesProcessed(state.iterations() * bytes.size());
+}
+
+BENCHMARK_CAPTURE(FromBinaryShape, nested / cbor, make_nested, binary_format::cbor);
+BENCHMARK_CAPTURE(FromBinaryShape, nested / msgpack, make_nested, binary_format::msgpack);
+BENCHMARK_CAPTURE(FromBinaryShape, nested / ubjson, make_nested, binary_format::ubjson);
+BENCHMARK_CAPTURE(FromBinaryShape, nested / bjdata, make_nested, binary_format::bjdata);
+BENCHMARK_CAPTURE(FromBinaryShape, nested / bson, make_nested, binary_format::bson);
+BENCHMARK_CAPTURE(FromBinaryShape, containers / cbor, make_containers, binary_format::cbor);
+BENCHMARK_CAPTURE(FromBinaryShape, containers / msgpack, make_containers, binary_format::msgpack);
+BENCHMARK_CAPTURE(FromBinaryShape, containers / ubjson, make_containers, binary_format::ubjson);
+BENCHMARK_CAPTURE(FromBinaryShape, containers / ubjson_optimized, make_containers, binary_format::ubjson_optimized);
+BENCHMARK_CAPTURE(FromBinaryShape, containers / bjdata, make_containers, binary_format::bjdata);
+BENCHMARK_CAPTURE(FromBinaryShape, containers / bson, make_containers, binary_format::bson);
+// BSON names every array element, so a large array measures key generation
+// rather than scalar decoding and is left out here
+BENCHMARK_CAPTURE(FromBinaryShape, scalars / cbor, make_scalars, binary_format::cbor);
+BENCHMARK_CAPTURE(FromBinaryShape, scalars / msgpack, make_scalars, binary_format::msgpack);
+BENCHMARK_CAPTURE(FromBinaryShape, scalars / ubjson, make_scalars, binary_format::ubjson);
+BENCHMARK_CAPTURE(FromBinaryShape, scalars / bjdata, make_scalars, binary_format::bjdata);
+
+/*!
+@brief parse an indefinite-length CBOR string
+
+The writer never emits this form, so the input is assembled by hand: 0x7F
+opens the string, each chunk is a one-character string, and 0xFF closes it.
+*/
+static void FromCborChunkedString(benchmark::State& state, const std::size_t chunks)
+{
+    std::vector<std::uint8_t> bytes;
+    bytes.reserve(2 * chunks + 2);
+    bytes.push_back(0x7F);
+    for (std::size_t i = 0; i < chunks; ++i)
+    {
+        bytes.push_back(0x61); // string of length 1
+        bytes.push_back(0x61); // 'a'
+    }
+    bytes.push_back(0xFF);
+
+    for (auto _ : state)
+    {
+        json j = json::from_cbor(bytes);
+        benchmark::DoNotOptimize(j);
+    }
+
+    state.SetBytesProcessed(state.iterations() * bytes.size());
+}
+
+BENCHMARK_CAPTURE(FromCborChunkedString, 10000 chunks, 10000);
+
 BENCHMARK_MAIN();

tests/src/unit-bson.cpp

@@ -38,54 +38,6 @@ class huge_binary_t : public std::vector<std::uint8_t>
 using huge_binary_json = nlohmann::basic_json <
                          std::map, std::vector, std::string, bool, std::int64_t, std::uint64_t,
                          double, std::allocator, nlohmann::adl_serializer, huge_binary_t, void >;
-
-// a string type that can be made to report a size beyond INT32_MAX without
-// allocating that much memory, so BSON length overflow can be tested for
-// strings and (embedded) documents as well, following the same idea as
-// huge_binary_t.
-//
-// Unlike huge_binary_t (which is only ever used as the BSON *value* type),
-// this type doubles as basic_json's StringType and is therefore also used
-// for *object keys* (e.g. "s" or "nested" below). Only the designated test
-// value is meant to lie about its size - if every huge_string_t (including
-// keys) reported a huge size, the running totals computed while walking the
-// BSON document (see calc_bson_object_size & friends in binary_writer.hpp)
-// would need more than 32 bits, and on platforms where std::size_t is only
-// 32 bits wide that arithmetic would silently wrap around, producing wrong
-// (or even unguarded) lengths. The fake size is therefore opt-in via
-// as_huge(), and plain strings - in particular object keys - keep reporting
-// their real, small size.
-class huge_string_t : public std::string
-{
-  public:
-    using std::string::string;
-    huge_string_t(const std::string& s) : std::string(s) {} // NOLINT(google-explicit-constructor,hicpp-explicit-conversions)
-
-    // returns a copy of @a s whose size() pretends to be huge
-    static huge_string_t as_huge(const std::string& s)
-    {
-        huge_string_t result(s);
-        result.pretend_huge = true;
-        return result;
-    }
-
-    size_type size() const noexcept
-    {
-        if (pretend_huge)
-        {
-            // one byte more than the BSON length field can represent
-            return static_cast<size_type>((std::numeric_limits<std::int32_t>::max)()) + 1;
-        }
-        return std::string::size();
-    }
-
-  private:
-    bool pretend_huge = false;
-};
-
-using huge_string_json = nlohmann::basic_json <
-                         std::map, std::vector, huge_string_t, bool, std::int64_t, std::uint64_t,
-                         double, std::allocator, nlohmann::adl_serializer, std::vector<std::uint8_t>, void >;
 } // namespace
 
 TEST_CASE("BSON")
@@ -152,11 +104,6 @@ TEST_CASE("BSON")
     }
 
     SECTION("lengths exceeding INT32_MAX cannot be serialized to BSON")
-    {
-        // out_of_range.412 is thrown from a single shared helper
-        // (to_bson_length) that guards the BSON length fields of binary
-        // values, strings, and (embedded) documents alike
-        SECTION("binary")
     {
         huge_binary_json j;
         j["b"] = huge_binary_json::binary(huge_binary_t{});
@@ -164,27 +111,6 @@ TEST_CASE("BSON")
         CHECK_THROWS_WITH_AS(huge_binary_json::to_bson(j), "[json.exception.out_of_range.412] BSON length 2147483661 exceeds maximum of 2147483647", huge_binary_json::out_of_range&);
     }
 
-        SECTION("string")
-        {
-            huge_string_json j;
-            j["s"] = huge_string_t::as_huge("value");
-
-            CHECK_THROWS_WITH_AS(huge_string_json::to_bson(j), "[json.exception.out_of_range.412] BSON length 2147483661 exceeds maximum of 2147483647", huge_string_json::out_of_range&);
-        }
-
-        SECTION("document")
-        {
-            // an oversized string nested one level deep makes the
-            // *embedded* document's own length exceed INT32_MAX as well
-            huge_string_json nested;
-            nested["s"] = huge_string_t::as_huge("value");
-            huge_string_json j;
-            j["nested"] = nested;
-
-            CHECK_THROWS_WITH_AS(huge_string_json::to_bson(j), "[json.exception.out_of_range.412] BSON length 2147483674 exceeds maximum of 2147483647", huge_string_json::out_of_range&);
-        }
-    }
-
     SECTION("string length must be at least 1")
     {
         // from https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=11175
@@ -267,23 +193,6 @@ TEST_CASE("BSON")
             CHECK(json::from_bson(result, true, false) == j);
         }
 
-        SECTION("non-empty object with bool from a non-0/1 byte (lenient parsing)")
-        {
-            // documented lenient behavior (see gh-5333): any non-zero byte
-            // is accepted as `true`, not just 0x01
-            std::vector<std::uint8_t> const input =
-            {
-                0x0D, 0x00, 0x00, 0x00, // size (little endian)
-                0x08,               // entry: boolean
-                'e', 'n', 't', 'r', 'y', '\x00',
-                0x02,           // value = 0x02 (neither 0x00 nor 0x01)
-                0x00                    // end marker
-            };
-
-            const json expected = { { "entry", true } };
-            CHECK(json::from_bson(input) == expected);
-        }
-
         SECTION("non-empty object with double")
         {
             json const j =
@@ -590,29 +499,6 @@ TEST_CASE("BSON")
             CHECK(json::from_bson(result, true, false) == j);
         }
 
-        SECTION("array elements with non-conforming keys (lenient parsing)")
-        {
-            // documented lenient behavior (see gh-5333): BSON array element
-            // keys are not checked against the required decimal sequence
-            // "0", "1", "2", ... - elements are taken in encoded order
-            std::vector<std::uint8_t> const input =
-            {
-                0x26, 0x00, 0x00, 0x00, // size (little endian)
-                0x04, 'e', 'n', 't', 'r', 'y', '\x00', // entry: embedded array
-
-                0x1A, 0x00, 0x00, 0x00, // size (little endian)
-                0x10, '5', 0x00, 0x0A, 0x00, 0x00, 0x00, // key "5" (bogus)      -> 10
-                0x10, 'x', 0x00, 0x14, 0x00, 0x00, 0x00, // key "x" (non-numeric) -> 20
-                0x10, '1', 0x00, 0x1E, 0x00, 0x00, 0x00, // key "1" (out of order) -> 30
-                0x00, // end marker (embedded array)
-
-                0x00 // end marker
-            };
-
-            const json expected = { { "entry", json::array({10, 20, 30}) } };
-            CHECK(json::from_bson(input) == expected);
-        }
-
         SECTION("non-empty object with binary member")
         {
             const size_t N = 10;
@@ -708,31 +594,6 @@ TEST_CASE("BSON")
             CHECK(json::from_bson(result, true, false) == j);
         }
 
-        SECTION("binary member with subtype 0x02 (old binary) keeps its inner length prefix (lenient parsing)")
-        {
-            // documented lenient behavior (see gh-5333): the payload for
-            // binary subtype 0x02 ("old binary") is returned as-is,
-            // including its own inner 4-byte length prefix; it is not
-            // stripped or reinterpreted
-            std::vector<std::uint8_t> const input =
-            {
-                0x17, 0x00, 0x00, 0x00, // size (little endian)
-                0x05, 'e', 'n', 't', 'r', 'y', '\x00', // entry: binary
-
-                0x06, 0x00, 0x00, 0x00, // size of binary (little endian)
-                0x02, // "old binary" subtype
-                0x02, 0x00, 0x00, 0x00, // inner length prefix (part of the old-binary payload)
-                0x68, 0x69, // payload ('h', 'i')
-
-                0x00 // end marker
-            };
-
-            // the inner length prefix is part of the (unmodified) payload
-            const std::vector<std::uint8_t> expected_payload = {0x02, 0x00, 0x00, 0x00, 0x68, 0x69};
-            const json expected = { { "entry", json::binary(expected_payload, 0x02) } };
-            CHECK(json::from_bson(input) == expected);
-        }
-
         SECTION("Some more complex document")
         {
             json const j =