Compare commits

...
Author SHA1 Message Date
Niels LohmannandClaude ac58c6b3c5 Route the -Wduplicated-branches pragma through Hedley
Match #5485, which moved the binary writer's hand-rolled diagnostic
pragmas onto JSON_HEDLEY_PRAGMA (merged into develop while this branch
was open). The devirtualization's -Wduplicated-branches suppression in
write_compact_float was the one raw '#pragma GCC diagnostic' left; it
now uses JSON_HEDLEY_PRAGMA like the adjacent -Wfloat-equal line, still
guarded to GCC >= 7 and non-clang (the warning exists only there).

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-11 18:10:40 +02:00
Niels Lohmann 966657ca49 Address review findings on the binary writer output sinks
- binary_reserve_hint(): the 4-bytes-per-element estimate over-reserved by up
  to 4x for arrays of small scalars (CBOR encodes 0..23 in one byte), and the
  returned vector kept that capacity. Make the hint a strict lower bound on the
  encoded size instead, which also removes the 1 MiB clamp whose branch no test
  could reach (the largest container in the suite has 65793 elements).

- Guard the -Wduplicated-branches pragma with __GNUC__ >= 7. The warning does
  not exist before GCC 7, so naming it made GCC 4.8/4.9/5/6 - which the CI
  matrix still builds - warn under -Wpragmas on every including translation
  unit, breaking downstream -Werror builds.

- Constrain the adapter constructor of binary_writer with the enable_if its
  documentation already claimed, so a writer over some other sink type is no
  longer advertised as constructible from an output adapter.

- Let output_vector_adapter wrap output_vector_sink rather than duplicating the
  append logic, so the type-erased and templated paths share one implementation.

- Collapse the three copies of the memcpy/byte_swap/memcpy dance into a single
  byte_swap_buffer() helper, and add the MSVC _byteswap_* intrinsics so MSVC no
  longer falls back to the scalar shuffle this change exists to eliminate.

- Add a vector_writer() helper for the five vector-returning to_* overloads
  instead of spelling out the writer type at each call site, and drop a dead
  default member initializer on output_adapter_sink.

- New tests: the vector sink and the adapter sink must produce identical bytes
  for every format (the two to_* overloads no longer delegate to each other and
  could otherwise drift), and binary_reserve_hint() must never exceed the size
  actually written.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-11 18:10:40 +02:00
Niels LohmannandClaude Opus 4.8 b949334c2c Reserve output capacity up front for binary serialization
The vector-returning to_cbor/to_msgpack/to_ubjson/to_bjdata/to_bson grew
the output buffer purely by geometric reallocation. Reserving an estimate
up front avoids the early reallocations, which is the dominant per-byte
cost for array/object-heavy output.

The estimate (binary_reserve_hint) is deliberately conservative and safe
against untrusted input: it consults only the top-level element count
(O(1), no walk of the DOM), guards the multiplication against overflow,
and clamps the result to a fixed 1 MiB ceiling, so a large or hostile DOM
can never force an oversized allocation here. The buffer still grows
geometrically past the hint, so an underestimate only costs a few later
reallocations; scalars/strings/binary are written in one shot and get no
hint. Reserving capacity does not change the bytes produced.

Throughput (g++/clang -O3, vs the previous commit):
  cbor int array     +10% / +13%
  cbor object array  +20% / +38%

Output is byte-for-byte identical to develop across the binary
differential corpus.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-11 18:10:39 +02:00
Niels LohmannandClaude Opus 4.8 18293c7db1 Encode big-endian numbers with a byte swap instead of std::reverse
write_number() reordered multi-byte numbers for the big-endian formats
(CBOR/MessagePack/UBJSON) with std::reverse over the byte array. GCC
lowered only some sizes to a bswap; clang kept a scalar byte shuffle
(0 bswap instructions in the CBOR number path). Replace the reverse with
size-dispatched __builtin_bswap16/32/64 helpers (portable shift fallback
for other compilers; std::reverse retained for exotic sizes such as a
long double number_float_t).

Codegen: the CBOR number path now emits bswap on both compilers
(gcc 2 -> 16, clang 0 -> 4). Output is byte-for-byte identical to the
previous implementation across the binary differential corpus.

Throughput (isolated vs the std::reverse version, best of 9):
  CBOR int64 array   gcc +7%   clang +10%
  CBOR uint16 array  gcc +27%  clang flat

Modest but consistent on number-dense encodings; negligible on
string/blob-heavy output, as expected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-11 18:10:38 +02:00
Niels LohmannandClaude Opus 4.8 c7f3ef97da Fix CI failures from binary_writer output-sink change
Four CI jobs failed on the initial commit; all are addressed here without
changing any output (binary encodings remain byte-for-byte identical to
develop across the differential corpus):

1. ci_test_gcc / cuda (-Werror=duplicated-branches): for number_float_t ==
   float, static_cast<float>(n) is the identity, so write_compact_float's
   two branches are intentionally identical. Once the concrete vector sink
   is inlined, GCC constant-folds and diagnoses this (the type-erased path
   hid it behind a non-inlined virtual call). Silence -Wduplicated-branches
   for GCC (clang has no such warning) alongside the existing -Wfloat-equal
   pragma.

2. ci_static_analysis_clang (UBSan nonnull-attribute): binary_writer passes
   a null pointer with length 0 for empty strings/binary. output_vector_sink
   / output_adapter_sink declared write_characters JSON_HEDLEY_NON_NULL, so
   the sanitizer flagged the (harmless) zero-length call once the sink was
   called directly rather than through the attribute-free virtual base. Drop
   the attribute from both sinks, matching the pre-existing behavior.

3. ci_cpplint (build/include_what_you_use): output_adapter_sink uses
   std::move; add #include <utility>.

4. ci_cuda_example (nvcc 11.8): NVCC's front end rejects the default
   template argument on the binary_writer alias template. Revert the alias
   to its original single-parameter form (relying on binary_writer's own
   defaulted OutputSinkType) and spell out the full type in the vector-sink
   convenience functions.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-11 18:10:37 +02:00
Niels LohmannandClaude Opus 4.8 cb95492718 Devirtualize binary_writer via a value-type output sink
to_cbor/to_msgpack/to_ubjson/to_bjdata/to_bson wrote every byte through
output_adapter_t, a shared_ptr<output_adapter_protocol> whose
write_character/write_characters are virtual. Unlike the lexer (templated
on a concrete InputAdapterType), the binary writer never got that
treatment, so binary output paid a vtable lookup per byte and a
make_shared per call.

Template binary_writer on an OutputSinkType and give it two concrete,
non-virtual sinks:

- output_vector_sink: appends straight into a std::vector (push_back /
  insert), used by the vector-returning to_* convenience functions. No
  vtable, no shared_ptr; the writes inline.
- output_adapter_sink: forwards to a type-erased output_adapter_t, so the
  existing to_*(j, output_adapter) overloads (streams, strings, custom
  adapters) keep working exactly as before -- one virtual call each,
  unchanged.

binary_writer keeps a convenience constructor taking output_adapter_t
(building the default output_adapter_sink), so the adapter overloads are
untouched; only the convenience functions switch to the vector sink. The
friend declaration and the basic_json binary_writer alias gain the new
(defaulted) template parameter.

Output is byte-for-byte identical: verified across ~3000 randomized
values plus curated edge cases (all scalar widths, strings with invalid
UTF-8, binary, nested arrays/objects) for CBOR, MessagePack, UBJSON (both
size/type settings), BJData, and BSON, plus the output_adapter path, in
C++11/17/20. Warning-clean under clang -Weverything and the gcc pedantic
set; clang-tidy clean on the changed headers; make check-amalgamation
clean.

Throughput (g++ -O3, vs develop): scalar-dense binary output such as
integer arrays ~1.4x; many small to_cbor calls ~1.04x (DOM traversal
bound); string/blob-heavy output unchanged (already bulk-bound). No
workload regressed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-11 18:10:36 +02:00
Niels Lohmann aa391dc0a5 Fix two develop CI regressions: dump() nodiscard warning and binary-reader const-correctness (#5520)
* Discard dump()'s [[nodiscard]] return value in an exception-only check

CHECK_THROWS_WITH_AS(j.dump(), ...) called dump() only to trigger and
catch the exception, but never used the return value. dump() is
warn_unused_result, so GCC's pedantic build (-Werror --all-warnings)
rejected it as -Werror=unused-result, breaking ci_test_gcc. Wrapped in
utils::ignore_return_value(), matching every other such call in this
file.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Mark container_frame top as const in CBOR/UBJSON readers

clang-tidy's misc-const-correctness flagged these on PR #5520's CI: the
BSON sibling copy was already const, but these two were left mutable
even though only container_stack.back().remaining is ever written.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-11 17:55:46 +02:00
Niels Lohmann d2514a46f7 Add benchmarks for the binary readers (#5510)
The benchmark suite covered parsing JSON text, dumping, and serializing to
CBOR, but only one binary read: FromMsgpack. Nothing measured from_cbor,
from_ubjson, from_bjdata or from_bson, so a change to binary_reader.hpp had
no baseline to be compared against.

Add read benchmarks for every format, in the two shapes that matter: from a
contiguous buffer, which is what most callers pass, and from a FILE*, which
is what FromMsgpack already measures and which compiles to different code.
FromMsgpack itself is left untouched so its numbers stay comparable across
releases. The input is derived at setup time by serializing a parsed test
file, because the test data repository ships JSON only.

The test files are wide and shallow, but the readers' cost is per container,
so add three value shapes they do not cover -- deeply nested containers, many
sibling containers, and one flat array of numbers -- plus an indefinite-length
CBOR string, a form the writer never emits and which therefore has to be
assembled by hand. UBJSON and BJData are also captured in their size- and
type-annotated form, which the readers handle in a separate code path.

BSON requires an object at the top level, so it cannot reuse the array-rooted
test files; it is captured on the object-rooted ones, and the shapes are
wrapped in an object so every format measures the same value. The setup marks
the benchmark as skipped rather than letting the exception escape if that
requirement is ever violated.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-11 13:15:09 +02:00
8 changed files with 1312 additions and 341 deletions
@@ -1434,7 +1434,7 @@ class binary_reader
// a copy, not a reference: it must stay valid across the
// pop_back() below, which destroys the container_stack element
// it would otherwise alias
container_frame top = container_stack.back();
const container_frame top = container_stack.back();
bool at_end = false;
if (top.remaining != npos)
@@ -2211,7 +2211,7 @@ class binary_reader
// would otherwise alias.
for (;;)
{
container_frame top = container_stack.back();
const container_frame top = container_stack.back();
if (top.remaining != npos)
{
File diff suppressed because it is too large Load Diff
@@ -13,6 +13,7 @@
#include <iterator> // back_inserter
#include <memory> // shared_ptr, make_shared
#include <string> // basic_string
#include <utility> // move
#include <vector> // vector
#ifndef JSON_NO_IO
@@ -44,22 +45,32 @@ template<typename CharType> struct output_adapter_protocol
template<typename CharType>
using output_adapter_t = std::shared_ptr<output_adapter_protocol<CharType>>;
/// output adapter for byte vectors
/// @brief non-virtual output sink writing into a std::vector
///
/// This sink is not part of the virtual output_adapter_protocol hierarchy: it is
/// passed to binary_writer by value as a template parameter, so
/// write_character()/write_characters() are ordinary (inlinable) calls with no
/// vtable lookup and no shared_ptr. It is used for the common
/// `to_cbor`/`to_msgpack`/... into a std::vector. output_vector_adapter below
/// wraps this same sink to provide the virtual interface.
template<typename CharType, typename AllocatorType = std::allocator<CharType>>
class output_vector_adapter : public output_adapter_protocol<CharType>
class output_vector_sink
{
public:
explicit output_vector_adapter(std::vector<CharType, AllocatorType>& vec) noexcept
explicit output_vector_sink(std::vector<CharType, AllocatorType>& vec) noexcept
: v(vec)
{}
void write_character(CharType c) override
void write_character(CharType c)
{
v.push_back(c);
}
JSON_HEDLEY_NON_NULL(2)
void write_characters(const CharType* s, std::size_t length) override
// no JSON_HEDLEY_NON_NULL here: binary_writer legitimately passes a null
// pointer with length 0 for empty strings/binary values. Appending an empty
// range is a no-op; the type-erased path tolerates this via the (unattributed)
// virtual base, and the concrete sink must do the same.
void write_characters(const CharType* s, std::size_t length)
{
v.insert(v.end(), s, s + length);
}
@@ -68,6 +79,34 @@ class output_vector_adapter : public output_adapter_protocol<CharType>
std::vector<CharType, AllocatorType>& v;
};
/// output adapter for byte vectors
///
/// The appending itself lives in output_vector_sink; this class only adds the
/// virtual output_adapter_protocol interface on top of it, so both the
/// type-erased and the templated path share one implementation.
template<typename CharType, typename AllocatorType = std::allocator<CharType>>
class output_vector_adapter : public output_adapter_protocol<CharType>
{
public:
explicit output_vector_adapter(std::vector<CharType, AllocatorType>& vec) noexcept
: sink(vec)
{}
void write_character(CharType c) override
{
sink.write_character(c);
}
JSON_HEDLEY_NON_NULL(2)
void write_characters(const CharType* s, std::size_t length) override
{
sink.write_characters(s, length);
}
private:
output_vector_sink<CharType, AllocatorType> sink;
};
#ifndef JSON_NO_IO
/// output adapter for output streams
template<typename CharType>
@@ -118,6 +157,39 @@ class output_string_adapter : public output_adapter_protocol<CharType>
StringType& str;
};
/// @brief output sink forwarding to a type-erased output adapter
///
/// Wraps the polymorphic output_adapter_t so the same binary_writer template can
/// also target arbitrary adapters (output streams, strings, user-provided
/// adapters) via the `output_adapter`-based overloads. Each write still goes
/// through one virtual call, exactly as before; only the concrete sinks above
/// avoid it.
template<typename CharType>
class output_adapter_sink
{
public:
explicit output_adapter_sink(output_adapter_t<CharType> adapter)
: oa(std::move(adapter))
{
JSON_ASSERT(oa);
}
void write_character(CharType c)
{
oa->write_character(c);
}
// no JSON_HEDLEY_NON_NULL: forwards (null, 0) for empty payloads, exactly as
// the type-erased path already did before this sink existed
void write_characters(const CharType* s, std::size_t length)
{
oa->write_characters(s, length);
}
private:
output_adapter_t<CharType> oa;
};
template<typename CharType, typename StringType = std::basic_string<CharType>>
class output_adapter
{
+19 -6
View File
@@ -140,7 +140,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
friend ::nlohmann::detail::serializer<basic_json>;
template<typename BasicJsonType>
friend class ::nlohmann::detail::iter_impl;
template<typename BasicJsonType, typename CharType>
template<typename BasicJsonType, typename CharType, typename OutputSinkType>
friend class ::nlohmann::detail::binary_writer;
template<typename BasicJsonType, typename InputType, typename SAX>
friend class ::nlohmann::detail::binary_reader;
@@ -188,6 +188,14 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
template<typename InputType>
using binary_reader = ::nlohmann::detail::binary_reader<basic_json, InputType>;
template<typename CharType> using binary_writer = ::nlohmann::detail::binary_writer<basic_json, CharType>;
// binary_writer over a concrete (non-virtual) sink appending into a std::vector,
// used by the vector-returning to_* overloads
template<typename CharType> using vector_binary_writer =
::nlohmann::detail::binary_writer<basic_json, CharType, ::nlohmann::detail::output_vector_sink<CharType>>;
template<typename CharType> static vector_binary_writer<CharType> vector_writer(std::vector<CharType>& v)
{
return vector_binary_writer<CharType>(::nlohmann::detail::output_vector_sink<CharType>(v));
}
JSON_PRIVATE_UNLESS_TESTED:
using serializer = ::nlohmann::detail::serializer<basic_json>;
@@ -4374,7 +4382,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
static std::vector<std::uint8_t> to_cbor(const basic_json& j)
{
std::vector<std::uint8_t> result;
to_cbor(j, result);
result.reserve(detail::binary_reserve_hint(j));
vector_writer(result).write_cbor(j);
return result;
}
@@ -4397,7 +4406,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
static std::vector<std::uint8_t> to_msgpack(const basic_json& j)
{
std::vector<std::uint8_t> result;
to_msgpack(j, result);
result.reserve(detail::binary_reserve_hint(j));
vector_writer(result).write_msgpack(j);
return result;
}
@@ -4422,7 +4432,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
const bool use_type = false)
{
std::vector<std::uint8_t> result;
to_ubjson(j, result, use_size, use_type);
result.reserve(detail::binary_reserve_hint(j));
vector_writer(result).write_ubjson(j, use_size, use_type);
return result;
}
@@ -4450,7 +4461,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
const bjdata_version_t version = bjdata_version_t::draft2)
{
std::vector<std::uint8_t> result;
to_bjdata(j, result, use_size, use_type, version);
result.reserve(detail::binary_reserve_hint(j));
vector_writer(result).write_ubjson(j, use_size, use_type, true, true, version);
return result;
}
@@ -4477,7 +4489,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
static std::vector<std::uint8_t> to_bson(const basic_json& j)
{
std::vector<std::uint8_t> result;
to_bson(j, result);
result.reserve(detail::binary_reserve_hint(j));
vector_writer(result).write_bson(j);
return result;
}
File diff suppressed because it is too large Load Diff
+319
View File
@@ -252,4 +252,323 @@ static void BinaryToCbor(benchmark::State& state)
}
BENCHMARK(BinaryToCbor)->RangeMultiplier(2)->Range(8, 8 << 12);
//////////////////////////////////////////////////////////////////////////////
// parse binary formats
//////////////////////////////////////////////////////////////////////////////
// Only MessagePack had a read benchmark (FromMsgpack above, left untouched so
// its numbers stay comparable across releases). The benchmarks below cover the
// other formats, and read from a contiguous buffer as well as from a FILE*:
// most callers pass a container, and the two adapters compile to different
// code. The test data repository ships JSON only, so the input for each is
// derived at setup time by serializing a parsed test file.
/// binary format to benchmark; the _optimized variants add UBJSON/BJData size
/// and type annotations, which the readers handle in a separate code path
enum class binary_format
{
cbor,
msgpack,
ubjson,
ubjson_optimized,
bjdata,
bjdata_optimized,
bson
};
static std::vector<std::uint8_t> to_binary(const json& j, const binary_format format)
{
switch (format)
{
case binary_format::cbor:
return json::to_cbor(j);
case binary_format::msgpack:
return json::to_msgpack(j);
case binary_format::ubjson:
return json::to_ubjson(j);
case binary_format::ubjson_optimized:
return json::to_ubjson(j, true, true);
case binary_format::bjdata:
return json::to_bjdata(j);
case binary_format::bjdata_optimized:
return json::to_bjdata(j, true, true);
case binary_format::bson:
default:
return json::to_bson(j);
}
}
static json from_binary(const std::vector<std::uint8_t>& bytes, const binary_format format)
{
switch (format)
{
case binary_format::cbor:
return json::from_cbor(bytes);
case binary_format::msgpack:
return json::from_msgpack(bytes);
case binary_format::ubjson:
case binary_format::ubjson_optimized:
return json::from_ubjson(bytes);
case binary_format::bjdata:
case binary_format::bjdata_optimized:
return json::from_bjdata(bytes);
case binary_format::bson:
default:
return json::from_bson(bytes);
}
}
static json from_binary(std::FILE* file, const binary_format format)
{
switch (format)
{
case binary_format::cbor:
return json::from_cbor(file);
case binary_format::msgpack:
return json::from_msgpack(file);
case binary_format::ubjson:
case binary_format::ubjson_optimized:
return json::from_ubjson(file);
case binary_format::bjdata:
case binary_format::bjdata_optimized:
return json::from_bjdata(file);
case binary_format::bson:
default:
return json::from_bson(file);
}
}
/*!
@brief serialize a parsed test file to @a format
Returns an empty vector and marks the benchmark as skipped if the file cannot
be represented in the format, rather than letting the exception escape: BSON
requires an object at the top level, and several test files are arrays.
*/
static std::vector<std::uint8_t> binary_input(benchmark::State& state, const char* filename, const binary_format format)
{
std::ifstream f(filename);
std::string const str((std::istreambuf_iterator<char>(f)), std::istreambuf_iterator<char>());
const json j = json::parse(str);
if (format == binary_format::bson && !j.is_object())
{
state.SkipWithError("BSON requires an object at the top level");
return {};
}
return to_binary(j, format);
}
static void FromBinaryBuffer(benchmark::State& state, const char* filename, const binary_format format)
{
const std::vector<std::uint8_t> bytes = binary_input(state, filename, format);
if (bytes.empty())
{
return;
}
for (auto _ : state)
{
// the value is destroyed outside the timed section, because destroying
// a large DOM is not what this benchmark measures
state.PauseTiming();
auto* j = new json();
state.ResumeTiming();
*j = from_binary(bytes, format);
state.PauseTiming();
delete j;
state.ResumeTiming();
}
state.SetBytesProcessed(state.iterations() * bytes.size());
}
BENCHMARK_CAPTURE(FromBinaryBuffer, cbor / jeopardy, TEST_DATA_DIRECTORY "/jeopardy/jeopardy.json", binary_format::cbor);
BENCHMARK_CAPTURE(FromBinaryBuffer, cbor / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::cbor);
BENCHMARK_CAPTURE(FromBinaryBuffer, cbor / citm_catalog, TEST_DATA_DIRECTORY "/nativejson-benchmark/citm_catalog.json", binary_format::cbor);
BENCHMARK_CAPTURE(FromBinaryBuffer, cbor / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::cbor);
BENCHMARK_CAPTURE(FromBinaryBuffer, cbor / floats, TEST_DATA_DIRECTORY "/regression/floats.json", binary_format::cbor);
BENCHMARK_CAPTURE(FromBinaryBuffer, cbor / signed_ints, TEST_DATA_DIRECTORY "/regression/signed_ints.json", binary_format::cbor);
BENCHMARK_CAPTURE(FromBinaryBuffer, msgpack / jeopardy, TEST_DATA_DIRECTORY "/jeopardy/jeopardy.json", binary_format::msgpack);
BENCHMARK_CAPTURE(FromBinaryBuffer, msgpack / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::msgpack);
BENCHMARK_CAPTURE(FromBinaryBuffer, msgpack / citm_catalog, TEST_DATA_DIRECTORY "/nativejson-benchmark/citm_catalog.json", binary_format::msgpack);
BENCHMARK_CAPTURE(FromBinaryBuffer, msgpack / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::msgpack);
BENCHMARK_CAPTURE(FromBinaryBuffer, ubjson / jeopardy, TEST_DATA_DIRECTORY "/jeopardy/jeopardy.json", binary_format::ubjson);
BENCHMARK_CAPTURE(FromBinaryBuffer, ubjson / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::ubjson);
BENCHMARK_CAPTURE(FromBinaryBuffer, ubjson / citm_catalog, TEST_DATA_DIRECTORY "/nativejson-benchmark/citm_catalog.json", binary_format::ubjson);
BENCHMARK_CAPTURE(FromBinaryBuffer, ubjson / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::ubjson);
BENCHMARK_CAPTURE(FromBinaryBuffer, ubjson_optimized / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::ubjson_optimized);
BENCHMARK_CAPTURE(FromBinaryBuffer, ubjson_optimized / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::ubjson_optimized);
BENCHMARK_CAPTURE(FromBinaryBuffer, bjdata / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::bjdata);
BENCHMARK_CAPTURE(FromBinaryBuffer, bjdata / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::bjdata);
BENCHMARK_CAPTURE(FromBinaryBuffer, bjdata_optimized / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::bjdata_optimized);
BENCHMARK_CAPTURE(FromBinaryBuffer, bjdata_optimized / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::bjdata_optimized);
// BSON requires an object at the top level, so the array-rooted test files
// (jeopardy and the regression files) cannot be captured here
BENCHMARK_CAPTURE(FromBinaryBuffer, bson / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::bson);
BENCHMARK_CAPTURE(FromBinaryBuffer, bson / citm_catalog, TEST_DATA_DIRECTORY "/nativejson-benchmark/citm_catalog.json", binary_format::bson);
BENCHMARK_CAPTURE(FromBinaryBuffer, bson / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::bson);
static void FromBinaryFile(benchmark::State& state, const char* filename, const binary_format format)
{
const std::vector<std::uint8_t> bytes = binary_input(state, filename, format);
if (bytes.empty())
{
return;
}
const char* tmp = "benchmark_input.bin";
std::ofstream o(tmp, std::ios::binary);
o.write(reinterpret_cast<const char*>(bytes.data()), static_cast<std::streamsize>(bytes.size()));
o.flush();
o.close();
for (auto _ : state)
{
state.PauseTiming();
auto* j = new json();
auto* file = std::fopen(tmp, "rb");
state.ResumeTiming();
*j = from_binary(file, format);
state.PauseTiming();
std::fclose(file);
delete j;
state.ResumeTiming();
}
state.SetBytesProcessed(state.iterations() * bytes.size());
}
BENCHMARK_CAPTURE(FromBinaryFile, cbor / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::cbor);
BENCHMARK_CAPTURE(FromBinaryFile, cbor / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::cbor);
BENCHMARK_CAPTURE(FromBinaryFile, ubjson / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::ubjson);
BENCHMARK_CAPTURE(FromBinaryFile, ubjson / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::ubjson);
BENCHMARK_CAPTURE(FromBinaryFile, bjdata / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::bjdata);
BENCHMARK_CAPTURE(FromBinaryFile, bson / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::bson);
//////////////////////////////////////////////////////////////////////////////
// parse binary formats: value shapes
//////////////////////////////////////////////////////////////////////////////
// The test files above are wide and shallow, but the readers' cost is per
// container, so these cover the shapes that stress the container handling
// itself. Every shape is wrapped in an object so that BSON, which requires an
// object at the top level, measures the same value as the other formats.
/// deeply nested arrays: one container per level, no other work
static json make_nested()
{
json nested = json::array();
json* p = &nested;
for (std::size_t i = 1; i < 1000; ++i)
{
p->push_back(json::array());
p = &p->operator[](0);
}
json j = json::object();
j["data"] = std::move(nested);
return j;
}
/// many sibling containers: maximum container churn, minimum nesting
static json make_containers()
{
json data = json::array();
for (std::size_t i = 0; i < 100000; ++i)
{
data.push_back(json::array({1, 2}));
}
json j = json::object();
j["data"] = std::move(data);
return j;
}
/// one flat array of numbers: the scalar decoding path, which must not move
static json make_scalars()
{
json data = json::array();
for (std::size_t i = 0; i < 1000000; ++i)
{
data.push_back(i);
}
json j = json::object();
j["data"] = std::move(data);
return j;
}
static void FromBinaryShape(benchmark::State& state, json (*build)(), const binary_format format)
{
const std::vector<std::uint8_t> bytes = to_binary(build(), format);
for (auto _ : state)
{
state.PauseTiming();
auto* j = new json();
state.ResumeTiming();
*j = from_binary(bytes, format);
state.PauseTiming();
delete j;
state.ResumeTiming();
}
state.SetBytesProcessed(state.iterations() * bytes.size());
}
BENCHMARK_CAPTURE(FromBinaryShape, nested / cbor, make_nested, binary_format::cbor);
BENCHMARK_CAPTURE(FromBinaryShape, nested / msgpack, make_nested, binary_format::msgpack);
BENCHMARK_CAPTURE(FromBinaryShape, nested / ubjson, make_nested, binary_format::ubjson);
BENCHMARK_CAPTURE(FromBinaryShape, nested / bjdata, make_nested, binary_format::bjdata);
BENCHMARK_CAPTURE(FromBinaryShape, nested / bson, make_nested, binary_format::bson);
BENCHMARK_CAPTURE(FromBinaryShape, containers / cbor, make_containers, binary_format::cbor);
BENCHMARK_CAPTURE(FromBinaryShape, containers / msgpack, make_containers, binary_format::msgpack);
BENCHMARK_CAPTURE(FromBinaryShape, containers / ubjson, make_containers, binary_format::ubjson);
BENCHMARK_CAPTURE(FromBinaryShape, containers / ubjson_optimized, make_containers, binary_format::ubjson_optimized);
BENCHMARK_CAPTURE(FromBinaryShape, containers / bjdata, make_containers, binary_format::bjdata);
BENCHMARK_CAPTURE(FromBinaryShape, containers / bson, make_containers, binary_format::bson);
// BSON names every array element, so a large array measures key generation
// rather than scalar decoding and is left out here
BENCHMARK_CAPTURE(FromBinaryShape, scalars / cbor, make_scalars, binary_format::cbor);
BENCHMARK_CAPTURE(FromBinaryShape, scalars / msgpack, make_scalars, binary_format::msgpack);
BENCHMARK_CAPTURE(FromBinaryShape, scalars / ubjson, make_scalars, binary_format::ubjson);
BENCHMARK_CAPTURE(FromBinaryShape, scalars / bjdata, make_scalars, binary_format::bjdata);
/*!
@brief parse an indefinite-length CBOR string
The writer never emits this form, so the input is assembled by hand: 0x7F
opens the string, each chunk is a one-character string, and 0xFF closes it.
*/
static void FromCborChunkedString(benchmark::State& state, const std::size_t chunks)
{
std::vector<std::uint8_t> bytes;
bytes.reserve(2 * chunks + 2);
bytes.push_back(0x7F);
for (std::size_t i = 0; i < chunks; ++i)
{
bytes.push_back(0x61); // string of length 1
bytes.push_back(0x61); // 'a'
}
bytes.push_back(0xFF);
for (auto _ : state)
{
json j = json::from_cbor(bytes);
benchmark::DoNotOptimize(j);
}
state.SetBytesProcessed(state.iterations() * bytes.size());
}
BENCHMARK_CAPTURE(FromCborChunkedString, 10000 chunks, 10000);
BENCHMARK_MAIN();
+198
View File
@@ -0,0 +1,198 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++ (supporting code)
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
#include "doctest_compatibility.h"
#include <nlohmann/json.hpp>
using nlohmann::json;
#include <cstdint>
#include <string>
#include <vector>
namespace
{
// a spread of values exercising every writer path: scalars of each width, the
// float paths, strings, binary, and containers big enough to reallocate
std::vector<json> test_values()
{
json big_array = json::array();
for (int i = 0; i < 5000; ++i)
{
big_array.push_back(i);
}
json big_object = json::object();
for (int i = 0; i < 1000; ++i)
{
big_object[std::to_string(i)] = i;
}
return
{
json(nullptr), json(true), json(false),
json(0), json(-1), json(255), json(-129), json(65535), json(-32769),
json(4294967295U), json(-2147483649LL), json(18446744073709551615ULL),
json(0.0), json(-0.5), json(3.1415926535897932),
json(""), json("hello"), json(std::string(1000, 'x')),
json::binary({0x00, 0x01, 0x02}, 42),
json::array(), json::object(),
json::array({1, 2, 3}), json({{"a", 1}, {"b", nullptr}}),
json({{"nested", {{"deep", json::array({1, "two", 3.0, nullptr})}}}}),
big_array, big_object
};
}
// values to_bson() accepts: the document must be an object
std::vector<json> bson_values()
{
json big_object = json::object();
for (int i = 0; i < 1000; ++i)
{
big_object[std::to_string(i)] = i;
}
return
{
json::object(),
json({{"a", 1}, {"b", nullptr}, {"c", true}, {"d", 2.5}, {"e", "text"}}),
json({{"arr", json::array({1, 2, 3})}, {"obj", {{"k", "v"}}}}),
big_object
};
}
} // namespace
// The vector-returning to_*(j) overloads write through the non-virtual
// output_vector_sink, while to_*(j, adapter) goes through output_adapter_sink.
// The two are separate code paths that must stay byte-for-byte identical; these
// checks fail if either overload is ever changed without the other.
TEST_CASE("binary writer output sinks")
{
SECTION("vector sink and adapter sink agree")
{
// note: no SUBCASE inside these loops - doctest keys subcases by
// name/file/line, so a subcase in a loop body would only ever run for
// the first iteration
for (const auto& j : test_values())
{
CAPTURE(j.dump(-1, ' ', false, json::error_handler_t::replace));
std::vector<std::uint8_t> cbor;
json::to_cbor(j, cbor);
CHECK(json::to_cbor(j) == cbor);
std::vector<std::uint8_t> msgpack;
json::to_msgpack(j, msgpack);
CHECK(json::to_msgpack(j) == msgpack);
for (const bool use_size :
{
false, true
})
{
for (const bool use_type :
{
false, true
})
{
if (use_type && !use_size)
{
continue; // not a supported combination
}
CAPTURE(use_size);
CAPTURE(use_type);
std::vector<std::uint8_t> ubjson;
json::to_ubjson(j, ubjson, use_size, use_type);
CHECK(json::to_ubjson(j, use_size, use_type) == ubjson);
}
}
for (const auto version :
{
json::bjdata_version_t::draft2, json::bjdata_version_t::draft3
})
{
std::vector<std::uint8_t> bjdata;
json::to_bjdata(j, bjdata, false, false, version);
CHECK(json::to_bjdata(j, false, false, version) == bjdata);
}
}
for (const auto& j : bson_values())
{
CAPTURE(j.dump());
std::vector<std::uint8_t> bson;
json::to_bson(j, bson);
CHECK(json::to_bson(j) == bson);
}
}
SECTION("the char adapter produces the same bytes")
{
for (const auto& j : test_values())
{
CAPTURE(j.dump(-1, ' ', false, json::error_handler_t::replace));
const std::vector<std::uint8_t> expected = json::to_cbor(j);
std::vector<char> as_char;
json::to_cbor(j, as_char);
REQUIRE(as_char.size() == expected.size());
std::vector<std::uint8_t> as_bytes;
as_bytes.reserve(as_char.size());
for (const char c : as_char)
{
as_bytes.push_back(static_cast<std::uint8_t>(c));
}
CHECK(as_bytes == expected);
}
}
}
// binary_reserve_hint() is documented as a *lower* bound on the serialized size,
// so that reserving it up front can never leave the returned vector holding
// capacity beyond what the value actually needs.
TEST_CASE("binary_reserve_hint never over-reserves")
{
for (const auto& j : test_values())
{
CAPTURE(j.dump(-1, ' ', false, json::error_handler_t::replace));
const std::size_t hint = nlohmann::detail::binary_reserve_hint(j);
CHECK(hint <= json::to_cbor(j).size());
CHECK(hint <= json::to_msgpack(j).size());
CHECK(hint <= json::to_ubjson(j).size());
CHECK(hint <= json::to_ubjson(j, true, true).size());
CHECK(hint <= json::to_bjdata(j).size());
}
for (const auto& j : bson_values())
{
CAPTURE(j.dump());
CHECK(nlohmann::detail::binary_reserve_hint(j) <= json::to_bson(j).size());
}
SECTION("scalars get no hint")
{
CHECK(nlohmann::detail::binary_reserve_hint(json(nullptr)) == 0);
CHECK(nlohmann::detail::binary_reserve_hint(json(42)) == 0);
CHECK(nlohmann::detail::binary_reserve_hint(json("a string")) == 0);
CHECK(nlohmann::detail::binary_reserve_hint(json::binary({0x01})) == 0);
}
SECTION("containers are hinted from their element count")
{
CHECK(nlohmann::detail::binary_reserve_hint(json::array()) == 1);
CHECK(nlohmann::detail::binary_reserve_hint(json::array({1, 2, 3})) == 4);
CHECK(nlohmann::detail::binary_reserve_hint(json::object()) == 1);
CHECK(nlohmann::detail::binary_reserve_hint(json({{"a", 1}, {"b", 2}})) == 5);
}
}
+1 -1
View File
@@ -469,7 +469,7 @@ TEST_CASE("serialization of strings (bulk fast path)")
SECTION("invalid UTF-8 handling is unaffected by the fast path")
{
const json j = std::string("valid\xff" "more");
CHECK_THROWS_WITH_AS(j.dump(), "[json.exception.type_error.316] invalid UTF-8 byte at index 5: 0xFF", json::type_error&);
CHECK_THROWS_WITH_AS(utils::ignore_return_value(j.dump()), "[json.exception.type_error.316] invalid UTF-8 byte at index 5: 0xFF", json::type_error&);
CHECK(j.dump(-1, ' ', false, json::error_handler_t::replace) == "\"valid\xef\xbf\xbd" "more\"");
CHECK(j.dump(-1, ' ', true, json::error_handler_t::replace) == "\"valid\\ufffdmore\"");
CHECK(j.dump(-1, ' ', false, json::error_handler_t::ignore) == "\"validmore\"");