From 1054b2097e721e8a455285a1b528cc72caf859b7 Mon Sep 17 00:00:00 2001 From: Niels Lohmann Date: Wed, 23 Sep 2026 08:59:46 +0200 Subject: [PATCH] Speed up binary writing: value-type output sink + byte-swap number encoding (#5286) * Devirtualize binary_writer via a value-type output sink to_cbor/to_msgpack/to_ubjson/to_bjdata/to_bson wrote every byte through output_adapter_t, a shared_ptr whose write_character/write_characters are virtual. Unlike the lexer (templated on a concrete InputAdapterType), the binary writer never got that treatment, so binary output paid a vtable lookup per byte and a make_shared per call. Template binary_writer on an OutputSinkType and give it two concrete, non-virtual sinks: - output_vector_sink: appends straight into a std::vector (push_back / insert), used by the vector-returning to_* convenience functions. No vtable, no shared_ptr; the writes inline. - output_adapter_sink: forwards to a type-erased output_adapter_t, so the existing to_*(j, output_adapter) overloads (streams, strings, custom adapters) keep working exactly as before -- one virtual call each, unchanged. binary_writer keeps a convenience constructor taking output_adapter_t (building the default output_adapter_sink), so the adapter overloads are untouched; only the convenience functions switch to the vector sink. The friend declaration and the basic_json binary_writer alias gain the new (defaulted) template parameter. Output is byte-for-byte identical: verified across ~3000 randomized values plus curated edge cases (all scalar widths, strings with invalid UTF-8, binary, nested arrays/objects) for CBOR, MessagePack, UBJSON (both size/type settings), BJData, and BSON, plus the output_adapter path, in C++11/17/20. Warning-clean under clang -Weverything and the gcc pedantic set; clang-tidy clean on the changed headers; make check-amalgamation clean. Throughput (g++ -O3, vs develop): scalar-dense binary output such as integer arrays ~1.4x; many small to_cbor calls ~1.04x (DOM traversal bound); string/blob-heavy output unchanged (already bulk-bound). No workload regressed. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann * Fix CI failures from binary_writer output-sink change Four CI jobs failed on the initial commit; all are addressed here without changing any output (binary encodings remain byte-for-byte identical to develop across the differential corpus): 1. ci_test_gcc / cuda (-Werror=duplicated-branches): for number_float_t == float, static_cast(n) is the identity, so write_compact_float's two branches are intentionally identical. Once the concrete vector sink is inlined, GCC constant-folds and diagnoses this (the type-erased path hid it behind a non-inlined virtual call). Silence -Wduplicated-branches for GCC (clang has no such warning) alongside the existing -Wfloat-equal pragma. 2. ci_static_analysis_clang (UBSan nonnull-attribute): binary_writer passes a null pointer with length 0 for empty strings/binary. output_vector_sink / output_adapter_sink declared write_characters JSON_HEDLEY_NON_NULL, so the sanitizer flagged the (harmless) zero-length call once the sink was called directly rather than through the attribute-free virtual base. Drop the attribute from both sinks, matching the pre-existing behavior. 3. ci_cpplint (build/include_what_you_use): output_adapter_sink uses std::move; add #include . 4. ci_cuda_example (nvcc 11.8): NVCC's front end rejects the default template argument on the binary_writer alias template. Revert the alias to its original single-parameter form (relying on binary_writer's own defaulted OutputSinkType) and spell out the full type in the vector-sink convenience functions. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann * Encode big-endian numbers with a byte swap instead of std::reverse write_number() reordered multi-byte numbers for the big-endian formats (CBOR/MessagePack/UBJSON) with std::reverse over the byte array. GCC lowered only some sizes to a bswap; clang kept a scalar byte shuffle (0 bswap instructions in the CBOR number path). Replace the reverse with size-dispatched __builtin_bswap16/32/64 helpers (portable shift fallback for other compilers; std::reverse retained for exotic sizes such as a long double number_float_t). Codegen: the CBOR number path now emits bswap on both compilers (gcc 2 -> 16, clang 0 -> 4). Output is byte-for-byte identical to the previous implementation across the binary differential corpus. Throughput (isolated vs the std::reverse version, best of 9): CBOR int64 array gcc +7% clang +10% CBOR uint16 array gcc +27% clang flat Modest but consistent on number-dense encodings; negligible on string/blob-heavy output, as expected. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann * Reserve output capacity up front for binary serialization The vector-returning to_cbor/to_msgpack/to_ubjson/to_bjdata/to_bson grew the output buffer purely by geometric reallocation. Reserving an estimate up front avoids the early reallocations, which is the dominant per-byte cost for array/object-heavy output. The estimate (binary_reserve_hint) is deliberately conservative and safe against untrusted input: it consults only the top-level element count (O(1), no walk of the DOM), guards the multiplication against overflow, and clamps the result to a fixed 1 MiB ceiling, so a large or hostile DOM can never force an oversized allocation here. The buffer still grows geometrically past the hint, so an underestimate only costs a few later reallocations; scalars/strings/binary are written in one shot and get no hint. Reserving capacity does not change the bytes produced. Throughput (g++/clang -O3, vs the previous commit): cbor int array +10% / +13% cbor object array +20% / +38% Output is byte-for-byte identical to develop across the binary differential corpus. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann * Address review findings on the binary writer output sinks - binary_reserve_hint(): the 4-bytes-per-element estimate over-reserved by up to 4x for arrays of small scalars (CBOR encodes 0..23 in one byte), and the returned vector kept that capacity. Make the hint a strict lower bound on the encoded size instead, which also removes the 1 MiB clamp whose branch no test could reach (the largest container in the suite has 65793 elements). - Guard the -Wduplicated-branches pragma with __GNUC__ >= 7. The warning does not exist before GCC 7, so naming it made GCC 4.8/4.9/5/6 - which the CI matrix still builds - warn under -Wpragmas on every including translation unit, breaking downstream -Werror builds. - Constrain the adapter constructor of binary_writer with the enable_if its documentation already claimed, so a writer over some other sink type is no longer advertised as constructible from an output adapter. - Let output_vector_adapter wrap output_vector_sink rather than duplicating the append logic, so the type-erased and templated paths share one implementation. - Collapse the three copies of the memcpy/byte_swap/memcpy dance into a single byte_swap_buffer() helper, and add the MSVC _byteswap_* intrinsics so MSVC no longer falls back to the scalar shuffle this change exists to eliminate. - Add a vector_writer() helper for the five vector-returning to_* overloads instead of spelling out the writer type at each call site, and drop a dead default member initializer on output_adapter_sink. - New tests: the vector sink and the adapter sink must produce identical bytes for every format (the two to_* overloads no longer delegate to each other and could otherwise drift), and binary_reserve_hint() must never exceed the size actually written. Signed-off-by: Niels Lohmann * Route the -Wduplicated-branches pragma through Hedley Match #5485, which moved the binary writer's hand-rolled diagnostic pragmas onto JSON_HEDLEY_PRAGMA (merged into develop while this branch was open). The devirtualization's -Wduplicated-branches suppression in write_compact_float was the one raw '#pragma GCC diagnostic' left; it now uses JSON_HEDLEY_PRAGMA like the adjacent -Wfloat-equal line, still guarded to GCC >= 7 and non-clang (the warning exists only there). Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann --------- Signed-off-by: Niels Lohmann Co-authored-by: Claude Opus 4.8 --- .../nlohmann/detail/output/binary_writer.hpp | 458 +++++++++----- .../detail/output/output_adapters.hpp | 84 ++- include/nlohmann/json.hpp | 25 +- single_include/nlohmann/json.hpp | 567 ++++++++++++------ tests/src/unit-binary_writer_sinks.cpp | 198 ++++++ 5 files changed, 992 insertions(+), 340 deletions(-) create mode 100644 tests/src/unit-binary_writer_sinks.cpp diff --git a/include/nlohmann/detail/output/binary_writer.hpp b/include/nlohmann/detail/output/binary_writer.hpp index fe88e7f27..a355f1c15 100644 --- a/include/nlohmann/detail/output/binary_writer.hpp +++ b/include/nlohmann/detail/output/binary_writer.hpp @@ -16,9 +16,14 @@ #include // memcpy #include // numeric_limits #include // string +#include // enable_if, is_constructible #include // move #include // vector +#ifdef _MSC_VER + #include // _byteswap_ushort, _byteswap_ulong, _byteswap_uint64 +#endif + #include #include #include @@ -39,10 +44,41 @@ enum class bjdata_version_t // binary writer // /////////////////// +/*! +@brief capacity hint for binary serialization into a std::vector + +Returns a *lower* bound on the number of bytes the serialization will produce, +so that writing an array/object of many elements does not start reallocating +from an empty buffer. Every array element occupies at least one byte in every +supported binary format, and every object entry at least two (a key of at least +one byte plus a value of at least one), plus one byte for the container header, +so the hint can never exceed the final size and the returned vector is never +left holding capacity the caller did not ask for. The buffer still grows +geometrically past the hint, so under-reserving only costs a few later +reallocations. Only the top-level element count is consulted (O(1), no walk of +the DOM); a single scalar, string, or binary value is written in one shot and +needs no hint. +*/ +template +std::size_t binary_reserve_hint(const BasicJsonType& j) +{ + if (j.is_array()) + { + return j.size() + 1; + } + + if (j.is_object()) + { + return (j.size() * 2) + 1; + } + + return 0; +} + /*! @brief serialization to CBOR and MessagePack values */ -template +template> class binary_writer { using string_t = typename BasicJsonType::string_t; @@ -53,12 +89,28 @@ class binary_writer /*! @brief create a binary writer + @param[in] sink output sink to write to (a value-type sink such as + output_vector_sink, or output_adapter_sink wrapping a + type-erased output adapter) + */ + explicit binary_writer(OutputSinkType sink) : oa(std::move(sink)) + {} + + /*! + @brief create a binary writer from a type-erased output adapter + + Convenience constructor for the default (output_adapter_sink) sink so the + `output_adapter`-based overloads keep constructing the writer directly from + an adapter. Constrained to sinks that can actually be built from an adapter, + so that a writer over some other sink type is not advertised as constructible + from one. + @param[in] adapter output adapter to write to */ - explicit binary_writer(output_adapter_t adapter) : oa(std::move(adapter)) - { - JSON_ASSERT(oa); - } + template < typename SinkType = OutputSinkType, + typename std::enable_if < std::is_constructible>::value, int >::type = 0 > + explicit binary_writer(output_adapter_t adapter) : oa(SinkType(std::move(adapter))) + {} /*! @param[in] j JSON value to serialize @@ -99,15 +151,15 @@ class binary_writer { case value_t::null: { - oa->write_character(to_char_type(0xF6)); + oa.write_character(to_char_type(0xF6)); break; } case value_t::boolean: { - oa->write_character(j.m_data.m_value.boolean - ? to_char_type(0xF5) - : to_char_type(0xF4)); + oa.write_character(j.m_data.m_value.boolean + ? to_char_type(0xF5) + : to_char_type(0xF4)); break; } @@ -124,22 +176,22 @@ class binary_writer } else if (j.m_data.m_value.number_integer <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x18)); + oa.write_character(to_char_type(0x18)); write_number(static_cast(j.m_data.m_value.number_integer)); } else if (j.m_data.m_value.number_integer <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x19)); + oa.write_character(to_char_type(0x19)); write_number(static_cast(j.m_data.m_value.number_integer)); } else if (j.m_data.m_value.number_integer <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x1A)); + oa.write_character(to_char_type(0x1A)); write_number(static_cast(j.m_data.m_value.number_integer)); } else { - oa->write_character(to_char_type(0x1B)); + oa.write_character(to_char_type(0x1B)); write_number(static_cast(j.m_data.m_value.number_integer)); } } @@ -154,22 +206,22 @@ class binary_writer } else if (positive_number <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x38)); + oa.write_character(to_char_type(0x38)); write_number(static_cast(positive_number)); } else if (positive_number <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x39)); + oa.write_character(to_char_type(0x39)); write_number(static_cast(positive_number)); } else if (positive_number <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x3A)); + oa.write_character(to_char_type(0x3A)); write_number(static_cast(positive_number)); } else { - oa->write_character(to_char_type(0x3B)); + oa.write_character(to_char_type(0x3B)); write_number(static_cast(positive_number)); } } @@ -184,22 +236,22 @@ class binary_writer } else if (j.m_data.m_value.number_unsigned <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x18)); + oa.write_character(to_char_type(0x18)); write_number(static_cast(j.m_data.m_value.number_unsigned)); } else if (j.m_data.m_value.number_unsigned <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x19)); + oa.write_character(to_char_type(0x19)); write_number(static_cast(j.m_data.m_value.number_unsigned)); } else if (j.m_data.m_value.number_unsigned <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x1A)); + oa.write_character(to_char_type(0x1A)); write_number(static_cast(j.m_data.m_value.number_unsigned)); } else { - oa->write_character(to_char_type(0x1B)); + oa.write_character(to_char_type(0x1B)); write_number(static_cast(j.m_data.m_value.number_unsigned)); } break; @@ -210,16 +262,16 @@ class binary_writer if (std::isnan(j.m_data.m_value.number_float)) { // NaN is 0xf97e00 in CBOR - oa->write_character(to_char_type(0xF9)); - oa->write_character(to_char_type(0x7E)); - oa->write_character(to_char_type(0x00)); + oa.write_character(to_char_type(0xF9)); + oa.write_character(to_char_type(0x7E)); + oa.write_character(to_char_type(0x00)); } else if (std::isinf(j.m_data.m_value.number_float)) { // Infinity is 0xf97c00, -Infinity is 0xf9fc00 - oa->write_character(to_char_type(0xf9)); - oa->write_character(j.m_data.m_value.number_float > 0 ? to_char_type(0x7C) : to_char_type(0xFC)); - oa->write_character(to_char_type(0x00)); + oa.write_character(to_char_type(0xf9)); + oa.write_character(j.m_data.m_value.number_float > 0 ? to_char_type(0x7C) : to_char_type(0xFC)); + oa.write_character(to_char_type(0x00)); } else { @@ -238,31 +290,31 @@ class binary_writer } else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x78)); + oa.write_character(to_char_type(0x78)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x79)); + oa.write_character(to_char_type(0x79)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x7A)); + oa.write_character(to_char_type(0x7A)); write_number(static_cast(N)); } // LCOV_EXCL_START else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x7B)); + oa.write_character(to_char_type(0x7B)); write_number(static_cast(N)); } // LCOV_EXCL_STOP // step 2: write the string - oa->write_characters( - reinterpret_cast(j.m_data.m_value.string->data()), - j.m_data.m_value.string->size()); + oa.write_characters( + reinterpret_cast(j.m_data.m_value.string->data()), + j.m_data.m_value.string->size()); break; } @@ -276,23 +328,23 @@ class binary_writer } else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x98)); + oa.write_character(to_char_type(0x98)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x99)); + oa.write_character(to_char_type(0x99)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x9A)); + oa.write_character(to_char_type(0x9A)); write_number(static_cast(N)); } // LCOV_EXCL_START else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x9B)); + oa.write_character(to_char_type(0x9B)); write_number(static_cast(N)); } // LCOV_EXCL_STOP @@ -339,31 +391,31 @@ class binary_writer } else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x58)); + oa.write_character(to_char_type(0x58)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x59)); + oa.write_character(to_char_type(0x59)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x5A)); + oa.write_character(to_char_type(0x5A)); write_number(static_cast(N)); } // LCOV_EXCL_START else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x5B)); + oa.write_character(to_char_type(0x5B)); write_number(static_cast(N)); } // LCOV_EXCL_STOP // step 2: write each element - oa->write_characters( - reinterpret_cast(j.m_data.m_value.binary->data()), - N); + oa.write_characters( + reinterpret_cast(j.m_data.m_value.binary->data()), + N); break; } @@ -378,23 +430,23 @@ class binary_writer } else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0xB8)); + oa.write_character(to_char_type(0xB8)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0xB9)); + oa.write_character(to_char_type(0xB9)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0xBA)); + oa.write_character(to_char_type(0xBA)); write_number(static_cast(N)); } // LCOV_EXCL_START else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0xBB)); + oa.write_character(to_char_type(0xBB)); write_number(static_cast(N)); } // LCOV_EXCL_STOP @@ -423,15 +475,15 @@ class binary_writer { case value_t::null: // nil { - oa->write_character(to_char_type(0xC0)); + oa.write_character(to_char_type(0xC0)); break; } case value_t::boolean: // true and false { - oa->write_character(j.m_data.m_value.boolean - ? to_char_type(0xC3) - : to_char_type(0xC2)); + oa.write_character(j.m_data.m_value.boolean + ? to_char_type(0xC3) + : to_char_type(0xC2)); break; } @@ -450,25 +502,25 @@ class binary_writer else if (j.m_data.m_value.number_unsigned <= (std::numeric_limits::max)()) { // uint 8 - oa->write_character(to_char_type(0xCC)); + oa.write_character(to_char_type(0xCC)); write_number(static_cast(j.m_data.m_value.number_integer)); } else if (j.m_data.m_value.number_unsigned <= (std::numeric_limits::max)()) { // uint 16 - oa->write_character(to_char_type(0xCD)); + oa.write_character(to_char_type(0xCD)); write_number(static_cast(j.m_data.m_value.number_integer)); } else if (j.m_data.m_value.number_unsigned <= (std::numeric_limits::max)()) { // uint 32 - oa->write_character(to_char_type(0xCE)); + oa.write_character(to_char_type(0xCE)); write_number(static_cast(j.m_data.m_value.number_integer)); } else if (j.m_data.m_value.number_unsigned <= (std::numeric_limits::max)()) { // uint 64 - oa->write_character(to_char_type(0xCF)); + oa.write_character(to_char_type(0xCF)); write_number(static_cast(j.m_data.m_value.number_integer)); } } @@ -483,28 +535,28 @@ class binary_writer j.m_data.m_value.number_integer <= (std::numeric_limits::max)()) { // int 8 - oa->write_character(to_char_type(0xD0)); + oa.write_character(to_char_type(0xD0)); write_number(static_cast(j.m_data.m_value.number_integer)); } else if (j.m_data.m_value.number_integer >= (std::numeric_limits::min)() && j.m_data.m_value.number_integer <= (std::numeric_limits::max)()) { // int 16 - oa->write_character(to_char_type(0xD1)); + oa.write_character(to_char_type(0xD1)); write_number(static_cast(j.m_data.m_value.number_integer)); } else if (j.m_data.m_value.number_integer >= (std::numeric_limits::min)() && j.m_data.m_value.number_integer <= (std::numeric_limits::max)()) { // int 32 - oa->write_character(to_char_type(0xD2)); + oa.write_character(to_char_type(0xD2)); write_number(static_cast(j.m_data.m_value.number_integer)); } else if (j.m_data.m_value.number_integer >= (std::numeric_limits::min)() && j.m_data.m_value.number_integer <= (std::numeric_limits::max)()) { // int 64 - oa->write_character(to_char_type(0xD3)); + oa.write_character(to_char_type(0xD3)); write_number(static_cast(j.m_data.m_value.number_integer)); } } @@ -521,25 +573,25 @@ class binary_writer else if (j.m_data.m_value.number_unsigned <= (std::numeric_limits::max)()) { // uint 8 - oa->write_character(to_char_type(0xCC)); + oa.write_character(to_char_type(0xCC)); write_number(static_cast(j.m_data.m_value.number_integer)); } else if (j.m_data.m_value.number_unsigned <= (std::numeric_limits::max)()) { // uint 16 - oa->write_character(to_char_type(0xCD)); + oa.write_character(to_char_type(0xCD)); write_number(static_cast(j.m_data.m_value.number_integer)); } else if (j.m_data.m_value.number_unsigned <= (std::numeric_limits::max)()) { // uint 32 - oa->write_character(to_char_type(0xCE)); + oa.write_character(to_char_type(0xCE)); write_number(static_cast(j.m_data.m_value.number_integer)); } else if (j.m_data.m_value.number_unsigned <= (std::numeric_limits::max)()) { // uint 64 - oa->write_character(to_char_type(0xCF)); + oa.write_character(to_char_type(0xCF)); write_number(static_cast(j.m_data.m_value.number_integer)); } break; @@ -563,26 +615,26 @@ class binary_writer else if (N <= (std::numeric_limits::max)()) { // str 8 - oa->write_character(to_char_type(0xD9)); + oa.write_character(to_char_type(0xD9)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) { // str 16 - oa->write_character(to_char_type(0xDA)); + oa.write_character(to_char_type(0xDA)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) { // str 32 - oa->write_character(to_char_type(0xDB)); + oa.write_character(to_char_type(0xDB)); write_number(static_cast(N)); } // step 2: write the string - oa->write_characters( - reinterpret_cast(j.m_data.m_value.string->data()), - j.m_data.m_value.string->size()); + oa.write_characters( + reinterpret_cast(j.m_data.m_value.string->data()), + j.m_data.m_value.string->size()); break; } @@ -598,13 +650,13 @@ class binary_writer else if (N <= (std::numeric_limits::max)()) { // array 16 - oa->write_character(to_char_type(0xDC)); + oa.write_character(to_char_type(0xDC)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) { // array 32 - oa->write_character(to_char_type(0xDD)); + oa.write_character(to_char_type(0xDD)); write_number(static_cast(N)); } @@ -660,7 +712,7 @@ class binary_writer fixed = false; } - oa->write_character(to_char_type(output_type)); + oa.write_character(to_char_type(output_type)); if (!fixed) { write_number(static_cast(N)); @@ -672,7 +724,7 @@ class binary_writer ? 0xC8 // ext 16 : 0xC5; // bin 16 - oa->write_character(to_char_type(output_type)); + oa.write_character(to_char_type(output_type)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) @@ -681,7 +733,7 @@ class binary_writer ? 0xC9 // ext 32 : 0xC6; // bin 32 - oa->write_character(to_char_type(output_type)); + oa.write_character(to_char_type(output_type)); write_number(static_cast(N)); } @@ -697,9 +749,9 @@ class binary_writer } // step 2: write the byte string - oa->write_characters( - reinterpret_cast(j.m_data.m_value.binary->data()), - N); + oa.write_characters( + reinterpret_cast(j.m_data.m_value.binary->data()), + N); break; } @@ -716,13 +768,13 @@ class binary_writer else if (N <= (std::numeric_limits::max)()) { // map 16 - oa->write_character(to_char_type(0xDE)); + oa.write_character(to_char_type(0xDE)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) { // map 32 - oa->write_character(to_char_type(0xDF)); + oa.write_character(to_char_type(0xDF)); write_number(static_cast(N)); } @@ -761,7 +813,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('Z')); + oa.write_character(to_char_type('Z')); } break; } @@ -770,9 +822,9 @@ class binary_writer { if (add_prefix) { - oa->write_character(j.m_data.m_value.boolean - ? to_char_type('T') - : to_char_type('F')); + oa.write_character(j.m_data.m_value.boolean + ? to_char_type('T') + : to_char_type('F')); } break; } @@ -799,12 +851,12 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('S')); + oa.write_character(to_char_type('S')); } write_number_with_ubjson_prefix(j.m_data.m_value.string->size(), true, use_bjdata); - oa->write_characters( - reinterpret_cast(j.m_data.m_value.string->data()), - j.m_data.m_value.string->size()); + oa.write_characters( + reinterpret_cast(j.m_data.m_value.string->data()), + j.m_data.m_value.string->size()); break; } @@ -812,7 +864,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('[')); + oa.write_character(to_char_type('[')); } bool prefix_required = true; @@ -844,14 +896,14 @@ class binary_writer && !(use_bjdata && std::find(bjdx.begin(), bjdx.end(), first_prefix) != bjdx.end())) { prefix_required = false; - oa->write_character(to_char_type('$')); - oa->write_character(first_prefix); + oa.write_character(to_char_type('$')); + oa.write_character(first_prefix); } } if (use_count) { - oa->write_character(to_char_type('#')); + oa.write_character(to_char_type('#')); write_number_with_ubjson_prefix(j.m_data.m_value.array->size(), true, use_bjdata); } @@ -862,7 +914,7 @@ class binary_writer if (!use_count) { - oa->write_character(to_char_type(']')); + oa.write_character(to_char_type(']')); } break; @@ -872,7 +924,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('[')); + oa.write_character(to_char_type('[')); } if (use_type && (bjdata_draft3 || !j.m_data.m_value.binary->empty())) @@ -881,36 +933,36 @@ class binary_writer { JSON_THROW(other_error::create(502, "use_type requires use_size = true", &j)); } - oa->write_character(to_char_type('$')); - oa->write_character(bjdata_draft3 ? 'B' : 'U'); + oa.write_character(to_char_type('$')); + oa.write_character(bjdata_draft3 ? 'B' : 'U'); } if (use_count) { - oa->write_character(to_char_type('#')); + oa.write_character(to_char_type('#')); write_number_with_ubjson_prefix(j.m_data.m_value.binary->size(), true, use_bjdata); } if (use_type) { - oa->write_characters( - reinterpret_cast(j.m_data.m_value.binary->data()), - j.m_data.m_value.binary->size()); + oa.write_characters( + reinterpret_cast(j.m_data.m_value.binary->data()), + j.m_data.m_value.binary->size()); } else { for (size_t i = 0; i < j.m_data.m_value.binary->size(); ++i) { - oa->write_character(to_char_type(bjdata_draft3 ? 'B' : 'U')); + oa.write_character(to_char_type(bjdata_draft3 ? 'B' : 'U')); // the cast is needed for binary types whose value type // is not an integer (e.g., std::byte) - oa->write_character(to_char_type(static_cast(j.m_data.m_value.binary->data()[i]))); + oa.write_character(to_char_type(static_cast(j.m_data.m_value.binary->data()[i]))); } } if (!use_count) { - oa->write_character(to_char_type(']')); + oa.write_character(to_char_type(']')); } break; @@ -928,7 +980,7 @@ class binary_writer if (add_prefix) { - oa->write_character(to_char_type('{')); + oa.write_character(to_char_type('{')); } bool prefix_required = true; @@ -950,29 +1002,29 @@ class binary_writer if (same_prefix && !(use_bjdata && std::find(bjdx.begin(), bjdx.end(), first_prefix) != bjdx.end())) { prefix_required = false; - oa->write_character(to_char_type('$')); - oa->write_character(first_prefix); + oa.write_character(to_char_type('$')); + oa.write_character(first_prefix); } } if (use_count) { - oa->write_character(to_char_type('#')); + oa.write_character(to_char_type('#')); write_number_with_ubjson_prefix(j.m_data.m_value.object->size(), true, use_bjdata); } for (const auto& el : *j.m_data.m_value.object) { write_number_with_ubjson_prefix(el.first.size(), true, use_bjdata); - oa->write_characters( - reinterpret_cast(el.first.data()), - el.first.size()); + oa.write_characters( + reinterpret_cast(el.first.data()), + el.first.size()); write_ubjson(el.second, use_count, use_type, prefix_required, use_bjdata, bjdata_version); } if (!use_count) { - oa->write_character(to_char_type('}')); + oa.write_character(to_char_type('}')); } break; @@ -1026,13 +1078,13 @@ class binary_writer void write_bson_entry_header(const string_t& name, const std::uint8_t element_type) { - oa->write_character(to_char_type(element_type)); - oa->write_characters( - reinterpret_cast(name.data()), - name.size()); + oa.write_character(to_char_type(element_type)); + oa.write_characters( + reinterpret_cast(name.data()), + name.size()); // the terminating null byte is written explicitly rather than taken // from the buffer, so that string_t::data() need not be null-terminated - oa->write_character(to_char_type(0x00)); + oa.write_character(to_char_type(0x00)); } /*! @@ -1042,7 +1094,7 @@ class binary_writer const bool value) { write_bson_entry_header(name, 0x08); - oa->write_character(value ? to_char_type(0x01) : to_char_type(0x00)); + oa.write_character(value ? to_char_type(0x01) : to_char_type(0x00)); } /*! @@ -1072,12 +1124,12 @@ class binary_writer write_bson_entry_header(name, 0x02); write_number(to_bson_length(value.size() + 1ul), true); - oa->write_characters( - reinterpret_cast(value.data()), - value.size()); + oa.write_characters( + reinterpret_cast(value.data()), + value.size()); // the terminating null byte is written explicitly rather than taken // from the buffer, so that string_t::data() need not be null-terminated - oa->write_character(to_char_type(0x00)); + oa.write_character(to_char_type(0x00)); } /*! @@ -1206,7 +1258,7 @@ class binary_writer write_bson_element(string_t(key.data(), key.size()), el); } - oa->write_character(to_char_type(0x00)); + oa.write_character(to_char_type(0x00)); } /*! @@ -1226,7 +1278,7 @@ class binary_writer write_number(value.has_subtype() ? static_cast(value.subtype()) : static_cast(0x00)); - oa->write_characters(reinterpret_cast(value.data()), value.size()); + oa.write_characters(reinterpret_cast(value.data()), value.size()); } /*! @@ -1352,7 +1404,7 @@ class binary_writer write_bson_element(el.first, el.second); } - oa->write_character(to_char_type(0x00)); + oa.write_character(to_char_type(0x00)); } ////////// @@ -1396,7 +1448,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(get_ubjson_float_prefix(n)); + oa.write_character(get_ubjson_float_prefix(n)); } write_number(n, use_bjdata); } @@ -1412,7 +1464,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('i')); // int8 + oa.write_character(to_char_type('i')); // int8 } write_number(static_cast(n), use_bjdata); } @@ -1420,7 +1472,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('U')); // uint8 + oa.write_character(to_char_type('U')); // uint8 } write_number(static_cast(n), use_bjdata); } @@ -1428,7 +1480,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('I')); // int16 + oa.write_character(to_char_type('I')); // int16 } write_number(static_cast(n), use_bjdata); } @@ -1436,7 +1488,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('u')); // uint16 - bjdata only + oa.write_character(to_char_type('u')); // uint16 - bjdata only } write_number(static_cast(n), use_bjdata); } @@ -1444,7 +1496,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('l')); // int32 + oa.write_character(to_char_type('l')); // int32 } write_number(static_cast(n), use_bjdata); } @@ -1452,7 +1504,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('m')); // uint32 - bjdata only + oa.write_character(to_char_type('m')); // uint32 - bjdata only } write_number(static_cast(n), use_bjdata); } @@ -1460,7 +1512,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('L')); // int64 + oa.write_character(to_char_type('L')); // int64 } write_number(static_cast(n), use_bjdata); } @@ -1468,7 +1520,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('M')); // uint64 - bjdata only + oa.write_character(to_char_type('M')); // uint64 - bjdata only } write_number(static_cast(n), use_bjdata); } @@ -1476,14 +1528,14 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('H')); // high-precision number + oa.write_character(to_char_type('H')); // high-precision number } const auto number = BasicJsonType(n).dump(); write_number_with_ubjson_prefix(number.size(), true, use_bjdata); for (std::size_t i = 0; i < number.size(); ++i) { - oa->write_character(to_char_type(static_cast(number[i]))); + oa.write_character(to_char_type(static_cast(number[i]))); } } } @@ -1500,7 +1552,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('i')); // int8 + oa.write_character(to_char_type('i')); // int8 } write_number(static_cast(n), use_bjdata); } @@ -1508,7 +1560,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('U')); // uint8 + oa.write_character(to_char_type('U')); // uint8 } write_number(static_cast(n), use_bjdata); } @@ -1516,7 +1568,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('I')); // int16 + oa.write_character(to_char_type('I')); // int16 } write_number(static_cast(n), use_bjdata); } @@ -1524,7 +1576,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('u')); // uint16 - bjdata only + oa.write_character(to_char_type('u')); // uint16 - bjdata only } write_number(static_cast(n), use_bjdata); } @@ -1532,7 +1584,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('l')); // int32 + oa.write_character(to_char_type('l')); // int32 } write_number(static_cast(n), use_bjdata); } @@ -1540,7 +1592,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('m')); // uint32 - bjdata only + oa.write_character(to_char_type('m')); // uint32 - bjdata only } write_number(static_cast(n), use_bjdata); } @@ -1548,7 +1600,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('L')); // int64 + oa.write_character(to_char_type('L')); // int64 } write_number(static_cast(n), use_bjdata); } @@ -1557,14 +1609,14 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('H')); // high-precision number + oa.write_character(to_char_type('H')); // high-precision number } const auto number = BasicJsonType(n).dump(); write_number_with_ubjson_prefix(number.size(), true, use_bjdata); for (std::size_t i = 0; i < number.size(); ++i) { - oa->write_character(to_char_type(static_cast(number[i]))); + oa.write_character(to_char_type(static_cast(number[i]))); } } // LCOV_EXCL_STOP @@ -1856,10 +1908,10 @@ class binary_writer } } - oa->write_character('['); - oa->write_character('$'); - oa->write_character(dtype); - oa->write_character('#'); + oa.write_character('['); + oa.write_character('$'); + oa.write_character(dtype); + oa.write_character('#'); key = "_ArraySize_"; write_ubjson(value.at(key), use_count, use_type, true, true, bjdata_version); @@ -1955,6 +2007,87 @@ class binary_writer On the other hand, BSON and BJData use little endian and should reorder on big endian systems. */ + // single-instruction byte swaps (compilers lower these to bswap/rev/movbe); + // used to emit big-endian numbers without a per-byte std::reverse loop + static std::uint16_t byte_swap(std::uint16_t x) noexcept + { +#if defined(__GNUC__) || defined(__clang__) + return __builtin_bswap16(x); +#elif defined(_MSC_VER) + return _byteswap_ushort(x); +#else + return static_cast((x >> 8) | (x << 8)); +#endif + } + + static std::uint32_t byte_swap(std::uint32_t x) noexcept + { +#if defined(__GNUC__) || defined(__clang__) + return __builtin_bswap32(x); +#elif defined(_MSC_VER) + return _byteswap_ulong(x); +#else + return ((x & 0x000000FFu) << 24) | ((x & 0x0000FF00u) << 8) + | ((x & 0x00FF0000u) >> 8) | ((x & 0xFF000000u) >> 24); +#endif + } + + static std::uint64_t byte_swap(std::uint64_t x) noexcept + { +#if defined(__GNUC__) || defined(__clang__) + return __builtin_bswap64(x); +#elif defined(_MSC_VER) + return _byteswap_uint64(x); +#else + x = ((x & 0x00000000FFFFFFFFull) << 32) | ((x & 0xFFFFFFFF00000000ull) >> 32); + x = ((x & 0x0000FFFF0000FFFFull) << 16) | ((x & 0xFFFF0000FFFF0000ull) >> 16); + x = ((x & 0x00FF00FF00FF00FFull) << 8) | ((x & 0xFF00FF00FF00FF00ull) >> 8); + return x; +#endif + } + + /*! + @brief reverse the bytes of a buffer by byte-swapping it as UIntType + + Loading the buffer into an unsigned integer of the same width and swapping + that is what lets the compiler emit a single bswap/rev/movbe; reversing the + buffer element by element does not reliably get there (clang keeps a scalar + shuffle). The two memcpy calls are the only portable way to reinterpret the + bytes and are folded away by every optimizer. + */ + template + static void byte_swap_buffer(std::array& a) noexcept + { + static_assert(sizeof(UIntType) == N, "swap width must match the buffer size"); + UIntType v{}; + std::memcpy(&v, a.data(), sizeof(v)); + v = byte_swap(v); + std::memcpy(a.data(), &v, sizeof(v)); + } + + // reverse the bytes of a fixed-size buffer; a single byte_swap() for the + // common 2/4/8-byte number payloads, std::reverse for any other size + static void reverse_bytes(std::array& a) noexcept + { + byte_swap_buffer(a); + } + + static void reverse_bytes(std::array& a) noexcept + { + byte_swap_buffer(a); + } + + static void reverse_bytes(std::array& a) noexcept + { + byte_swap_buffer(a); + } + + template + static void reverse_bytes(std::array& a) noexcept + { + std::reverse(a.begin(), a.end()); + } + template void write_number(const NumberType n, const bool OutputIsLittleEndian = false) { @@ -1966,10 +2099,10 @@ class binary_writer if (is_little_endian != OutputIsLittleEndian) { // reverse byte order prior to conversion if necessary - std::reverse(vec.begin(), vec.end()); + reverse_bytes(vec); } - oa->write_characters(vec.data(), sizeof(NumberType)); + oa.write_characters(vec.data(), sizeof(NumberType)); } void write_compact_float(const number_float_t n, detail::input_format_t format) @@ -1977,21 +2110,30 @@ class binary_writer #ifdef __GNUC__ JSON_HEDLEY_DIAGNOSTIC_PUSH JSON_HEDLEY_PRAGMA(GCC diagnostic ignored "-Wfloat-equal") +#endif + // When number_float_t is float, static_cast(n) is the identity and + // both branches below are intentionally identical (the "compact" float + // representation is the value itself). Only GCC diagnoses this, and only + // when the sink calls are inlined; clang has no such warning. + // (-Wduplicated-branches only exists from GCC 7 on; naming it on an older + // GCC would itself warn under -Wpragmas) +#if defined(__GNUC__) && !defined(__clang__) && (__GNUC__ >= 7) + JSON_HEDLEY_PRAGMA(GCC diagnostic ignored "-Wduplicated-branches") #endif if (!std::isfinite(n) || ((static_cast(n) >= static_cast(std::numeric_limits::lowest()) && static_cast(n) <= static_cast((std::numeric_limits::max)()) && static_cast(static_cast(n)) == static_cast(n)))) { - oa->write_character(format == detail::input_format_t::cbor - ? get_cbor_float_prefix(static_cast(n)) - : get_msgpack_float_prefix(static_cast(n))); + oa.write_character(format == detail::input_format_t::cbor + ? get_cbor_float_prefix(static_cast(n)) + : get_msgpack_float_prefix(static_cast(n))); write_number(static_cast(n)); } else { - oa->write_character(format == detail::input_format_t::cbor - ? get_cbor_float_prefix(n) - : get_msgpack_float_prefix(n)); + oa.write_character(format == detail::input_format_t::cbor + ? get_cbor_float_prefix(n) + : get_msgpack_float_prefix(n)); write_number(n); } #ifdef __GNUC__ @@ -2058,7 +2200,7 @@ class binary_writer const bool is_little_endian = little_endianness(); /// the output - output_adapter_t oa = nullptr; + OutputSinkType oa; }; } // namespace detail diff --git a/include/nlohmann/detail/output/output_adapters.hpp b/include/nlohmann/detail/output/output_adapters.hpp index 94763e64a..7eb73121c 100644 --- a/include/nlohmann/detail/output/output_adapters.hpp +++ b/include/nlohmann/detail/output/output_adapters.hpp @@ -13,6 +13,7 @@ #include // back_inserter #include // shared_ptr, make_shared #include // basic_string +#include // move #include // vector #ifndef JSON_NO_IO @@ -44,22 +45,32 @@ template struct output_adapter_protocol template using output_adapter_t = std::shared_ptr>; -/// output adapter for byte vectors +/// @brief non-virtual output sink writing into a std::vector +/// +/// This sink is not part of the virtual output_adapter_protocol hierarchy: it is +/// passed to binary_writer by value as a template parameter, so +/// write_character()/write_characters() are ordinary (inlinable) calls with no +/// vtable lookup and no shared_ptr. It is used for the common +/// `to_cbor`/`to_msgpack`/... into a std::vector. output_vector_adapter below +/// wraps this same sink to provide the virtual interface. template> -class output_vector_adapter : public output_adapter_protocol +class output_vector_sink { public: - explicit output_vector_adapter(std::vector& vec) noexcept + explicit output_vector_sink(std::vector& vec) noexcept : v(vec) {} - void write_character(CharType c) override + void write_character(CharType c) { v.push_back(c); } - JSON_HEDLEY_NON_NULL(2) - void write_characters(const CharType* s, std::size_t length) override + // no JSON_HEDLEY_NON_NULL here: binary_writer legitimately passes a null + // pointer with length 0 for empty strings/binary values. Appending an empty + // range is a no-op; the type-erased path tolerates this via the (unattributed) + // virtual base, and the concrete sink must do the same. + void write_characters(const CharType* s, std::size_t length) { v.insert(v.end(), s, s + length); } @@ -68,6 +79,34 @@ class output_vector_adapter : public output_adapter_protocol std::vector& v; }; +/// output adapter for byte vectors +/// +/// The appending itself lives in output_vector_sink; this class only adds the +/// virtual output_adapter_protocol interface on top of it, so both the +/// type-erased and the templated path share one implementation. +template> +class output_vector_adapter : public output_adapter_protocol +{ + public: + explicit output_vector_adapter(std::vector& vec) noexcept + : sink(vec) + {} + + void write_character(CharType c) override + { + sink.write_character(c); + } + + JSON_HEDLEY_NON_NULL(2) + void write_characters(const CharType* s, std::size_t length) override + { + sink.write_characters(s, length); + } + + private: + output_vector_sink sink; +}; + #ifndef JSON_NO_IO /// output adapter for output streams template @@ -118,6 +157,39 @@ class output_string_adapter : public output_adapter_protocol StringType& str; }; +/// @brief output sink forwarding to a type-erased output adapter +/// +/// Wraps the polymorphic output_adapter_t so the same binary_writer template can +/// also target arbitrary adapters (output streams, strings, user-provided +/// adapters) via the `output_adapter`-based overloads. Each write still goes +/// through one virtual call, exactly as before; only the concrete sinks above +/// avoid it. +template +class output_adapter_sink +{ + public: + explicit output_adapter_sink(output_adapter_t adapter) + : oa(std::move(adapter)) + { + JSON_ASSERT(oa); + } + + void write_character(CharType c) + { + oa->write_character(c); + } + + // no JSON_HEDLEY_NON_NULL: forwards (null, 0) for empty payloads, exactly as + // the type-erased path already did before this sink existed + void write_characters(const CharType* s, std::size_t length) + { + oa->write_characters(s, length); + } + + private: + output_adapter_t oa; +}; + template> class output_adapter { diff --git a/include/nlohmann/json.hpp b/include/nlohmann/json.hpp index d55efacc2..a63363fbf 100644 --- a/include/nlohmann/json.hpp +++ b/include/nlohmann/json.hpp @@ -140,7 +140,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec friend ::nlohmann::detail::serializer; template friend class ::nlohmann::detail::iter_impl; - template + template friend class ::nlohmann::detail::binary_writer; template friend class ::nlohmann::detail::binary_reader; @@ -188,6 +188,14 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec template using binary_reader = ::nlohmann::detail::binary_reader; template using binary_writer = ::nlohmann::detail::binary_writer; + // binary_writer over a concrete (non-virtual) sink appending into a std::vector, + // used by the vector-returning to_* overloads + template using vector_binary_writer = + ::nlohmann::detail::binary_writer>; + template static vector_binary_writer vector_writer(std::vector& v) + { + return vector_binary_writer(::nlohmann::detail::output_vector_sink(v)); + } JSON_PRIVATE_UNLESS_TESTED: using serializer = ::nlohmann::detail::serializer; @@ -4444,7 +4452,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec static std::vector to_cbor(const basic_json& j) { std::vector result; - to_cbor(j, result); + result.reserve(detail::binary_reserve_hint(j)); + vector_writer(result).write_cbor(j); return result; } @@ -4467,7 +4476,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec static std::vector to_msgpack(const basic_json& j) { std::vector result; - to_msgpack(j, result); + result.reserve(detail::binary_reserve_hint(j)); + vector_writer(result).write_msgpack(j); return result; } @@ -4492,7 +4502,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec const bool use_type = false) { std::vector result; - to_ubjson(j, result, use_size, use_type); + result.reserve(detail::binary_reserve_hint(j)); + vector_writer(result).write_ubjson(j, use_size, use_type); return result; } @@ -4520,7 +4531,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec const bjdata_version_t version = bjdata_version_t::draft2) { std::vector result; - to_bjdata(j, result, use_size, use_type, version); + result.reserve(detail::binary_reserve_hint(j)); + vector_writer(result).write_ubjson(j, use_size, use_type, true, true, version); return result; } @@ -4547,7 +4559,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec static std::vector to_bson(const basic_json& j) { std::vector result; - to_bson(j, result); + result.reserve(detail::binary_reserve_hint(j)); + vector_writer(result).write_bson(j); return result; } diff --git a/single_include/nlohmann/json.hpp b/single_include/nlohmann/json.hpp index f300cbdb1..4af4af1e0 100644 --- a/single_include/nlohmann/json.hpp +++ b/single_include/nlohmann/json.hpp @@ -18621,9 +18621,14 @@ NLOHMANN_JSON_NAMESPACE_END #include // memcpy #include // numeric_limits #include // string +#include // enable_if, is_constructible #include // move #include // vector +#ifdef _MSC_VER + #include // _byteswap_ushort, _byteswap_ulong, _byteswap_uint64 +#endif + // #include // #include @@ -18644,6 +18649,7 @@ NLOHMANN_JSON_NAMESPACE_END #include // back_inserter #include // shared_ptr, make_shared #include // basic_string +#include // move #include // vector #ifndef JSON_NO_IO @@ -18676,22 +18682,32 @@ template struct output_adapter_protocol template using output_adapter_t = std::shared_ptr>; -/// output adapter for byte vectors +/// @brief non-virtual output sink writing into a std::vector +/// +/// This sink is not part of the virtual output_adapter_protocol hierarchy: it is +/// passed to binary_writer by value as a template parameter, so +/// write_character()/write_characters() are ordinary (inlinable) calls with no +/// vtable lookup and no shared_ptr. It is used for the common +/// `to_cbor`/`to_msgpack`/... into a std::vector. output_vector_adapter below +/// wraps this same sink to provide the virtual interface. template> -class output_vector_adapter : public output_adapter_protocol +class output_vector_sink { public: - explicit output_vector_adapter(std::vector& vec) noexcept + explicit output_vector_sink(std::vector& vec) noexcept : v(vec) {} - void write_character(CharType c) override + void write_character(CharType c) { v.push_back(c); } - JSON_HEDLEY_NON_NULL(2) - void write_characters(const CharType* s, std::size_t length) override + // no JSON_HEDLEY_NON_NULL here: binary_writer legitimately passes a null + // pointer with length 0 for empty strings/binary values. Appending an empty + // range is a no-op; the type-erased path tolerates this via the (unattributed) + // virtual base, and the concrete sink must do the same. + void write_characters(const CharType* s, std::size_t length) { v.insert(v.end(), s, s + length); } @@ -18700,6 +18716,34 @@ class output_vector_adapter : public output_adapter_protocol std::vector& v; }; +/// output adapter for byte vectors +/// +/// The appending itself lives in output_vector_sink; this class only adds the +/// virtual output_adapter_protocol interface on top of it, so both the +/// type-erased and the templated path share one implementation. +template> +class output_vector_adapter : public output_adapter_protocol +{ + public: + explicit output_vector_adapter(std::vector& vec) noexcept + : sink(vec) + {} + + void write_character(CharType c) override + { + sink.write_character(c); + } + + JSON_HEDLEY_NON_NULL(2) + void write_characters(const CharType* s, std::size_t length) override + { + sink.write_characters(s, length); + } + + private: + output_vector_sink sink; +}; + #ifndef JSON_NO_IO /// output adapter for output streams template @@ -18750,6 +18794,39 @@ class output_string_adapter : public output_adapter_protocol StringType& str; }; +/// @brief output sink forwarding to a type-erased output adapter +/// +/// Wraps the polymorphic output_adapter_t so the same binary_writer template can +/// also target arbitrary adapters (output streams, strings, user-provided +/// adapters) via the `output_adapter`-based overloads. Each write still goes +/// through one virtual call, exactly as before; only the concrete sinks above +/// avoid it. +template +class output_adapter_sink +{ + public: + explicit output_adapter_sink(output_adapter_t adapter) + : oa(std::move(adapter)) + { + JSON_ASSERT(oa); + } + + void write_character(CharType c) + { + oa->write_character(c); + } + + // no JSON_HEDLEY_NON_NULL: forwards (null, 0) for empty payloads, exactly as + // the type-erased path already did before this sink existed + void write_characters(const CharType* s, std::size_t length) + { + oa->write_characters(s, length); + } + + private: + output_adapter_t oa; +}; + template> class output_adapter { @@ -18796,10 +18873,41 @@ enum class bjdata_version_t // binary writer // /////////////////// +/*! +@brief capacity hint for binary serialization into a std::vector + +Returns a *lower* bound on the number of bytes the serialization will produce, +so that writing an array/object of many elements does not start reallocating +from an empty buffer. Every array element occupies at least one byte in every +supported binary format, and every object entry at least two (a key of at least +one byte plus a value of at least one), plus one byte for the container header, +so the hint can never exceed the final size and the returned vector is never +left holding capacity the caller did not ask for. The buffer still grows +geometrically past the hint, so under-reserving only costs a few later +reallocations. Only the top-level element count is consulted (O(1), no walk of +the DOM); a single scalar, string, or binary value is written in one shot and +needs no hint. +*/ +template +std::size_t binary_reserve_hint(const BasicJsonType& j) +{ + if (j.is_array()) + { + return j.size() + 1; + } + + if (j.is_object()) + { + return (j.size() * 2) + 1; + } + + return 0; +} + /*! @brief serialization to CBOR and MessagePack values */ -template +template> class binary_writer { using string_t = typename BasicJsonType::string_t; @@ -18810,12 +18918,28 @@ class binary_writer /*! @brief create a binary writer + @param[in] sink output sink to write to (a value-type sink such as + output_vector_sink, or output_adapter_sink wrapping a + type-erased output adapter) + */ + explicit binary_writer(OutputSinkType sink) : oa(std::move(sink)) + {} + + /*! + @brief create a binary writer from a type-erased output adapter + + Convenience constructor for the default (output_adapter_sink) sink so the + `output_adapter`-based overloads keep constructing the writer directly from + an adapter. Constrained to sinks that can actually be built from an adapter, + so that a writer over some other sink type is not advertised as constructible + from one. + @param[in] adapter output adapter to write to */ - explicit binary_writer(output_adapter_t adapter) : oa(std::move(adapter)) - { - JSON_ASSERT(oa); - } + template < typename SinkType = OutputSinkType, + typename std::enable_if < std::is_constructible>::value, int >::type = 0 > + explicit binary_writer(output_adapter_t adapter) : oa(SinkType(std::move(adapter))) + {} /*! @param[in] j JSON value to serialize @@ -18856,15 +18980,15 @@ class binary_writer { case value_t::null: { - oa->write_character(to_char_type(0xF6)); + oa.write_character(to_char_type(0xF6)); break; } case value_t::boolean: { - oa->write_character(j.m_data.m_value.boolean - ? to_char_type(0xF5) - : to_char_type(0xF4)); + oa.write_character(j.m_data.m_value.boolean + ? to_char_type(0xF5) + : to_char_type(0xF4)); break; } @@ -18881,22 +19005,22 @@ class binary_writer } else if (j.m_data.m_value.number_integer <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x18)); + oa.write_character(to_char_type(0x18)); write_number(static_cast(j.m_data.m_value.number_integer)); } else if (j.m_data.m_value.number_integer <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x19)); + oa.write_character(to_char_type(0x19)); write_number(static_cast(j.m_data.m_value.number_integer)); } else if (j.m_data.m_value.number_integer <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x1A)); + oa.write_character(to_char_type(0x1A)); write_number(static_cast(j.m_data.m_value.number_integer)); } else { - oa->write_character(to_char_type(0x1B)); + oa.write_character(to_char_type(0x1B)); write_number(static_cast(j.m_data.m_value.number_integer)); } } @@ -18911,22 +19035,22 @@ class binary_writer } else if (positive_number <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x38)); + oa.write_character(to_char_type(0x38)); write_number(static_cast(positive_number)); } else if (positive_number <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x39)); + oa.write_character(to_char_type(0x39)); write_number(static_cast(positive_number)); } else if (positive_number <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x3A)); + oa.write_character(to_char_type(0x3A)); write_number(static_cast(positive_number)); } else { - oa->write_character(to_char_type(0x3B)); + oa.write_character(to_char_type(0x3B)); write_number(static_cast(positive_number)); } } @@ -18941,22 +19065,22 @@ class binary_writer } else if (j.m_data.m_value.number_unsigned <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x18)); + oa.write_character(to_char_type(0x18)); write_number(static_cast(j.m_data.m_value.number_unsigned)); } else if (j.m_data.m_value.number_unsigned <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x19)); + oa.write_character(to_char_type(0x19)); write_number(static_cast(j.m_data.m_value.number_unsigned)); } else if (j.m_data.m_value.number_unsigned <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x1A)); + oa.write_character(to_char_type(0x1A)); write_number(static_cast(j.m_data.m_value.number_unsigned)); } else { - oa->write_character(to_char_type(0x1B)); + oa.write_character(to_char_type(0x1B)); write_number(static_cast(j.m_data.m_value.number_unsigned)); } break; @@ -18967,16 +19091,16 @@ class binary_writer if (std::isnan(j.m_data.m_value.number_float)) { // NaN is 0xf97e00 in CBOR - oa->write_character(to_char_type(0xF9)); - oa->write_character(to_char_type(0x7E)); - oa->write_character(to_char_type(0x00)); + oa.write_character(to_char_type(0xF9)); + oa.write_character(to_char_type(0x7E)); + oa.write_character(to_char_type(0x00)); } else if (std::isinf(j.m_data.m_value.number_float)) { // Infinity is 0xf97c00, -Infinity is 0xf9fc00 - oa->write_character(to_char_type(0xf9)); - oa->write_character(j.m_data.m_value.number_float > 0 ? to_char_type(0x7C) : to_char_type(0xFC)); - oa->write_character(to_char_type(0x00)); + oa.write_character(to_char_type(0xf9)); + oa.write_character(j.m_data.m_value.number_float > 0 ? to_char_type(0x7C) : to_char_type(0xFC)); + oa.write_character(to_char_type(0x00)); } else { @@ -18995,31 +19119,31 @@ class binary_writer } else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x78)); + oa.write_character(to_char_type(0x78)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x79)); + oa.write_character(to_char_type(0x79)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x7A)); + oa.write_character(to_char_type(0x7A)); write_number(static_cast(N)); } // LCOV_EXCL_START else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x7B)); + oa.write_character(to_char_type(0x7B)); write_number(static_cast(N)); } // LCOV_EXCL_STOP // step 2: write the string - oa->write_characters( - reinterpret_cast(j.m_data.m_value.string->data()), - j.m_data.m_value.string->size()); + oa.write_characters( + reinterpret_cast(j.m_data.m_value.string->data()), + j.m_data.m_value.string->size()); break; } @@ -19033,23 +19157,23 @@ class binary_writer } else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x98)); + oa.write_character(to_char_type(0x98)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x99)); + oa.write_character(to_char_type(0x99)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x9A)); + oa.write_character(to_char_type(0x9A)); write_number(static_cast(N)); } // LCOV_EXCL_START else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x9B)); + oa.write_character(to_char_type(0x9B)); write_number(static_cast(N)); } // LCOV_EXCL_STOP @@ -19096,31 +19220,31 @@ class binary_writer } else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x58)); + oa.write_character(to_char_type(0x58)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x59)); + oa.write_character(to_char_type(0x59)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x5A)); + oa.write_character(to_char_type(0x5A)); write_number(static_cast(N)); } // LCOV_EXCL_START else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0x5B)); + oa.write_character(to_char_type(0x5B)); write_number(static_cast(N)); } // LCOV_EXCL_STOP // step 2: write each element - oa->write_characters( - reinterpret_cast(j.m_data.m_value.binary->data()), - N); + oa.write_characters( + reinterpret_cast(j.m_data.m_value.binary->data()), + N); break; } @@ -19135,23 +19259,23 @@ class binary_writer } else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0xB8)); + oa.write_character(to_char_type(0xB8)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0xB9)); + oa.write_character(to_char_type(0xB9)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0xBA)); + oa.write_character(to_char_type(0xBA)); write_number(static_cast(N)); } // LCOV_EXCL_START else if (N <= (std::numeric_limits::max)()) { - oa->write_character(to_char_type(0xBB)); + oa.write_character(to_char_type(0xBB)); write_number(static_cast(N)); } // LCOV_EXCL_STOP @@ -19180,15 +19304,15 @@ class binary_writer { case value_t::null: // nil { - oa->write_character(to_char_type(0xC0)); + oa.write_character(to_char_type(0xC0)); break; } case value_t::boolean: // true and false { - oa->write_character(j.m_data.m_value.boolean - ? to_char_type(0xC3) - : to_char_type(0xC2)); + oa.write_character(j.m_data.m_value.boolean + ? to_char_type(0xC3) + : to_char_type(0xC2)); break; } @@ -19207,25 +19331,25 @@ class binary_writer else if (j.m_data.m_value.number_unsigned <= (std::numeric_limits::max)()) { // uint 8 - oa->write_character(to_char_type(0xCC)); + oa.write_character(to_char_type(0xCC)); write_number(static_cast(j.m_data.m_value.number_integer)); } else if (j.m_data.m_value.number_unsigned <= (std::numeric_limits::max)()) { // uint 16 - oa->write_character(to_char_type(0xCD)); + oa.write_character(to_char_type(0xCD)); write_number(static_cast(j.m_data.m_value.number_integer)); } else if (j.m_data.m_value.number_unsigned <= (std::numeric_limits::max)()) { // uint 32 - oa->write_character(to_char_type(0xCE)); + oa.write_character(to_char_type(0xCE)); write_number(static_cast(j.m_data.m_value.number_integer)); } else if (j.m_data.m_value.number_unsigned <= (std::numeric_limits::max)()) { // uint 64 - oa->write_character(to_char_type(0xCF)); + oa.write_character(to_char_type(0xCF)); write_number(static_cast(j.m_data.m_value.number_integer)); } } @@ -19240,28 +19364,28 @@ class binary_writer j.m_data.m_value.number_integer <= (std::numeric_limits::max)()) { // int 8 - oa->write_character(to_char_type(0xD0)); + oa.write_character(to_char_type(0xD0)); write_number(static_cast(j.m_data.m_value.number_integer)); } else if (j.m_data.m_value.number_integer >= (std::numeric_limits::min)() && j.m_data.m_value.number_integer <= (std::numeric_limits::max)()) { // int 16 - oa->write_character(to_char_type(0xD1)); + oa.write_character(to_char_type(0xD1)); write_number(static_cast(j.m_data.m_value.number_integer)); } else if (j.m_data.m_value.number_integer >= (std::numeric_limits::min)() && j.m_data.m_value.number_integer <= (std::numeric_limits::max)()) { // int 32 - oa->write_character(to_char_type(0xD2)); + oa.write_character(to_char_type(0xD2)); write_number(static_cast(j.m_data.m_value.number_integer)); } else if (j.m_data.m_value.number_integer >= (std::numeric_limits::min)() && j.m_data.m_value.number_integer <= (std::numeric_limits::max)()) { // int 64 - oa->write_character(to_char_type(0xD3)); + oa.write_character(to_char_type(0xD3)); write_number(static_cast(j.m_data.m_value.number_integer)); } } @@ -19278,25 +19402,25 @@ class binary_writer else if (j.m_data.m_value.number_unsigned <= (std::numeric_limits::max)()) { // uint 8 - oa->write_character(to_char_type(0xCC)); + oa.write_character(to_char_type(0xCC)); write_number(static_cast(j.m_data.m_value.number_integer)); } else if (j.m_data.m_value.number_unsigned <= (std::numeric_limits::max)()) { // uint 16 - oa->write_character(to_char_type(0xCD)); + oa.write_character(to_char_type(0xCD)); write_number(static_cast(j.m_data.m_value.number_integer)); } else if (j.m_data.m_value.number_unsigned <= (std::numeric_limits::max)()) { // uint 32 - oa->write_character(to_char_type(0xCE)); + oa.write_character(to_char_type(0xCE)); write_number(static_cast(j.m_data.m_value.number_integer)); } else if (j.m_data.m_value.number_unsigned <= (std::numeric_limits::max)()) { // uint 64 - oa->write_character(to_char_type(0xCF)); + oa.write_character(to_char_type(0xCF)); write_number(static_cast(j.m_data.m_value.number_integer)); } break; @@ -19320,26 +19444,26 @@ class binary_writer else if (N <= (std::numeric_limits::max)()) { // str 8 - oa->write_character(to_char_type(0xD9)); + oa.write_character(to_char_type(0xD9)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) { // str 16 - oa->write_character(to_char_type(0xDA)); + oa.write_character(to_char_type(0xDA)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) { // str 32 - oa->write_character(to_char_type(0xDB)); + oa.write_character(to_char_type(0xDB)); write_number(static_cast(N)); } // step 2: write the string - oa->write_characters( - reinterpret_cast(j.m_data.m_value.string->data()), - j.m_data.m_value.string->size()); + oa.write_characters( + reinterpret_cast(j.m_data.m_value.string->data()), + j.m_data.m_value.string->size()); break; } @@ -19355,13 +19479,13 @@ class binary_writer else if (N <= (std::numeric_limits::max)()) { // array 16 - oa->write_character(to_char_type(0xDC)); + oa.write_character(to_char_type(0xDC)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) { // array 32 - oa->write_character(to_char_type(0xDD)); + oa.write_character(to_char_type(0xDD)); write_number(static_cast(N)); } @@ -19417,7 +19541,7 @@ class binary_writer fixed = false; } - oa->write_character(to_char_type(output_type)); + oa.write_character(to_char_type(output_type)); if (!fixed) { write_number(static_cast(N)); @@ -19429,7 +19553,7 @@ class binary_writer ? 0xC8 // ext 16 : 0xC5; // bin 16 - oa->write_character(to_char_type(output_type)); + oa.write_character(to_char_type(output_type)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) @@ -19438,7 +19562,7 @@ class binary_writer ? 0xC9 // ext 32 : 0xC6; // bin 32 - oa->write_character(to_char_type(output_type)); + oa.write_character(to_char_type(output_type)); write_number(static_cast(N)); } @@ -19454,9 +19578,9 @@ class binary_writer } // step 2: write the byte string - oa->write_characters( - reinterpret_cast(j.m_data.m_value.binary->data()), - N); + oa.write_characters( + reinterpret_cast(j.m_data.m_value.binary->data()), + N); break; } @@ -19473,13 +19597,13 @@ class binary_writer else if (N <= (std::numeric_limits::max)()) { // map 16 - oa->write_character(to_char_type(0xDE)); + oa.write_character(to_char_type(0xDE)); write_number(static_cast(N)); } else if (N <= (std::numeric_limits::max)()) { // map 32 - oa->write_character(to_char_type(0xDF)); + oa.write_character(to_char_type(0xDF)); write_number(static_cast(N)); } @@ -19518,7 +19642,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('Z')); + oa.write_character(to_char_type('Z')); } break; } @@ -19527,9 +19651,9 @@ class binary_writer { if (add_prefix) { - oa->write_character(j.m_data.m_value.boolean - ? to_char_type('T') - : to_char_type('F')); + oa.write_character(j.m_data.m_value.boolean + ? to_char_type('T') + : to_char_type('F')); } break; } @@ -19556,12 +19680,12 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('S')); + oa.write_character(to_char_type('S')); } write_number_with_ubjson_prefix(j.m_data.m_value.string->size(), true, use_bjdata); - oa->write_characters( - reinterpret_cast(j.m_data.m_value.string->data()), - j.m_data.m_value.string->size()); + oa.write_characters( + reinterpret_cast(j.m_data.m_value.string->data()), + j.m_data.m_value.string->size()); break; } @@ -19569,7 +19693,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('[')); + oa.write_character(to_char_type('[')); } bool prefix_required = true; @@ -19601,14 +19725,14 @@ class binary_writer && !(use_bjdata && std::find(bjdx.begin(), bjdx.end(), first_prefix) != bjdx.end())) { prefix_required = false; - oa->write_character(to_char_type('$')); - oa->write_character(first_prefix); + oa.write_character(to_char_type('$')); + oa.write_character(first_prefix); } } if (use_count) { - oa->write_character(to_char_type('#')); + oa.write_character(to_char_type('#')); write_number_with_ubjson_prefix(j.m_data.m_value.array->size(), true, use_bjdata); } @@ -19619,7 +19743,7 @@ class binary_writer if (!use_count) { - oa->write_character(to_char_type(']')); + oa.write_character(to_char_type(']')); } break; @@ -19629,7 +19753,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('[')); + oa.write_character(to_char_type('[')); } if (use_type && (bjdata_draft3 || !j.m_data.m_value.binary->empty())) @@ -19638,36 +19762,36 @@ class binary_writer { JSON_THROW(other_error::create(502, "use_type requires use_size = true", &j)); } - oa->write_character(to_char_type('$')); - oa->write_character(bjdata_draft3 ? 'B' : 'U'); + oa.write_character(to_char_type('$')); + oa.write_character(bjdata_draft3 ? 'B' : 'U'); } if (use_count) { - oa->write_character(to_char_type('#')); + oa.write_character(to_char_type('#')); write_number_with_ubjson_prefix(j.m_data.m_value.binary->size(), true, use_bjdata); } if (use_type) { - oa->write_characters( - reinterpret_cast(j.m_data.m_value.binary->data()), - j.m_data.m_value.binary->size()); + oa.write_characters( + reinterpret_cast(j.m_data.m_value.binary->data()), + j.m_data.m_value.binary->size()); } else { for (size_t i = 0; i < j.m_data.m_value.binary->size(); ++i) { - oa->write_character(to_char_type(bjdata_draft3 ? 'B' : 'U')); + oa.write_character(to_char_type(bjdata_draft3 ? 'B' : 'U')); // the cast is needed for binary types whose value type // is not an integer (e.g., std::byte) - oa->write_character(to_char_type(static_cast(j.m_data.m_value.binary->data()[i]))); + oa.write_character(to_char_type(static_cast(j.m_data.m_value.binary->data()[i]))); } } if (!use_count) { - oa->write_character(to_char_type(']')); + oa.write_character(to_char_type(']')); } break; @@ -19685,7 +19809,7 @@ class binary_writer if (add_prefix) { - oa->write_character(to_char_type('{')); + oa.write_character(to_char_type('{')); } bool prefix_required = true; @@ -19707,29 +19831,29 @@ class binary_writer if (same_prefix && !(use_bjdata && std::find(bjdx.begin(), bjdx.end(), first_prefix) != bjdx.end())) { prefix_required = false; - oa->write_character(to_char_type('$')); - oa->write_character(first_prefix); + oa.write_character(to_char_type('$')); + oa.write_character(first_prefix); } } if (use_count) { - oa->write_character(to_char_type('#')); + oa.write_character(to_char_type('#')); write_number_with_ubjson_prefix(j.m_data.m_value.object->size(), true, use_bjdata); } for (const auto& el : *j.m_data.m_value.object) { write_number_with_ubjson_prefix(el.first.size(), true, use_bjdata); - oa->write_characters( - reinterpret_cast(el.first.data()), - el.first.size()); + oa.write_characters( + reinterpret_cast(el.first.data()), + el.first.size()); write_ubjson(el.second, use_count, use_type, prefix_required, use_bjdata, bjdata_version); } if (!use_count) { - oa->write_character(to_char_type('}')); + oa.write_character(to_char_type('}')); } break; @@ -19783,13 +19907,13 @@ class binary_writer void write_bson_entry_header(const string_t& name, const std::uint8_t element_type) { - oa->write_character(to_char_type(element_type)); - oa->write_characters( - reinterpret_cast(name.data()), - name.size()); + oa.write_character(to_char_type(element_type)); + oa.write_characters( + reinterpret_cast(name.data()), + name.size()); // the terminating null byte is written explicitly rather than taken // from the buffer, so that string_t::data() need not be null-terminated - oa->write_character(to_char_type(0x00)); + oa.write_character(to_char_type(0x00)); } /*! @@ -19799,7 +19923,7 @@ class binary_writer const bool value) { write_bson_entry_header(name, 0x08); - oa->write_character(value ? to_char_type(0x01) : to_char_type(0x00)); + oa.write_character(value ? to_char_type(0x01) : to_char_type(0x00)); } /*! @@ -19829,12 +19953,12 @@ class binary_writer write_bson_entry_header(name, 0x02); write_number(to_bson_length(value.size() + 1ul), true); - oa->write_characters( - reinterpret_cast(value.data()), - value.size()); + oa.write_characters( + reinterpret_cast(value.data()), + value.size()); // the terminating null byte is written explicitly rather than taken // from the buffer, so that string_t::data() need not be null-terminated - oa->write_character(to_char_type(0x00)); + oa.write_character(to_char_type(0x00)); } /*! @@ -19963,7 +20087,7 @@ class binary_writer write_bson_element(string_t(key.data(), key.size()), el); } - oa->write_character(to_char_type(0x00)); + oa.write_character(to_char_type(0x00)); } /*! @@ -19983,7 +20107,7 @@ class binary_writer write_number(value.has_subtype() ? static_cast(value.subtype()) : static_cast(0x00)); - oa->write_characters(reinterpret_cast(value.data()), value.size()); + oa.write_characters(reinterpret_cast(value.data()), value.size()); } /*! @@ -20109,7 +20233,7 @@ class binary_writer write_bson_element(el.first, el.second); } - oa->write_character(to_char_type(0x00)); + oa.write_character(to_char_type(0x00)); } ////////// @@ -20153,7 +20277,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(get_ubjson_float_prefix(n)); + oa.write_character(get_ubjson_float_prefix(n)); } write_number(n, use_bjdata); } @@ -20169,7 +20293,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('i')); // int8 + oa.write_character(to_char_type('i')); // int8 } write_number(static_cast(n), use_bjdata); } @@ -20177,7 +20301,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('U')); // uint8 + oa.write_character(to_char_type('U')); // uint8 } write_number(static_cast(n), use_bjdata); } @@ -20185,7 +20309,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('I')); // int16 + oa.write_character(to_char_type('I')); // int16 } write_number(static_cast(n), use_bjdata); } @@ -20193,7 +20317,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('u')); // uint16 - bjdata only + oa.write_character(to_char_type('u')); // uint16 - bjdata only } write_number(static_cast(n), use_bjdata); } @@ -20201,7 +20325,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('l')); // int32 + oa.write_character(to_char_type('l')); // int32 } write_number(static_cast(n), use_bjdata); } @@ -20209,7 +20333,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('m')); // uint32 - bjdata only + oa.write_character(to_char_type('m')); // uint32 - bjdata only } write_number(static_cast(n), use_bjdata); } @@ -20217,7 +20341,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('L')); // int64 + oa.write_character(to_char_type('L')); // int64 } write_number(static_cast(n), use_bjdata); } @@ -20225,7 +20349,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('M')); // uint64 - bjdata only + oa.write_character(to_char_type('M')); // uint64 - bjdata only } write_number(static_cast(n), use_bjdata); } @@ -20233,14 +20357,14 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('H')); // high-precision number + oa.write_character(to_char_type('H')); // high-precision number } const auto number = BasicJsonType(n).dump(); write_number_with_ubjson_prefix(number.size(), true, use_bjdata); for (std::size_t i = 0; i < number.size(); ++i) { - oa->write_character(to_char_type(static_cast(number[i]))); + oa.write_character(to_char_type(static_cast(number[i]))); } } } @@ -20257,7 +20381,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('i')); // int8 + oa.write_character(to_char_type('i')); // int8 } write_number(static_cast(n), use_bjdata); } @@ -20265,7 +20389,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('U')); // uint8 + oa.write_character(to_char_type('U')); // uint8 } write_number(static_cast(n), use_bjdata); } @@ -20273,7 +20397,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('I')); // int16 + oa.write_character(to_char_type('I')); // int16 } write_number(static_cast(n), use_bjdata); } @@ -20281,7 +20405,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('u')); // uint16 - bjdata only + oa.write_character(to_char_type('u')); // uint16 - bjdata only } write_number(static_cast(n), use_bjdata); } @@ -20289,7 +20413,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('l')); // int32 + oa.write_character(to_char_type('l')); // int32 } write_number(static_cast(n), use_bjdata); } @@ -20297,7 +20421,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('m')); // uint32 - bjdata only + oa.write_character(to_char_type('m')); // uint32 - bjdata only } write_number(static_cast(n), use_bjdata); } @@ -20305,7 +20429,7 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('L')); // int64 + oa.write_character(to_char_type('L')); // int64 } write_number(static_cast(n), use_bjdata); } @@ -20314,14 +20438,14 @@ class binary_writer { if (add_prefix) { - oa->write_character(to_char_type('H')); // high-precision number + oa.write_character(to_char_type('H')); // high-precision number } const auto number = BasicJsonType(n).dump(); write_number_with_ubjson_prefix(number.size(), true, use_bjdata); for (std::size_t i = 0; i < number.size(); ++i) { - oa->write_character(to_char_type(static_cast(number[i]))); + oa.write_character(to_char_type(static_cast(number[i]))); } } // LCOV_EXCL_STOP @@ -20613,10 +20737,10 @@ class binary_writer } } - oa->write_character('['); - oa->write_character('$'); - oa->write_character(dtype); - oa->write_character('#'); + oa.write_character('['); + oa.write_character('$'); + oa.write_character(dtype); + oa.write_character('#'); key = "_ArraySize_"; write_ubjson(value.at(key), use_count, use_type, true, true, bjdata_version); @@ -20712,6 +20836,87 @@ class binary_writer On the other hand, BSON and BJData use little endian and should reorder on big endian systems. */ + // single-instruction byte swaps (compilers lower these to bswap/rev/movbe); + // used to emit big-endian numbers without a per-byte std::reverse loop + static std::uint16_t byte_swap(std::uint16_t x) noexcept + { +#if defined(__GNUC__) || defined(__clang__) + return __builtin_bswap16(x); +#elif defined(_MSC_VER) + return _byteswap_ushort(x); +#else + return static_cast((x >> 8) | (x << 8)); +#endif + } + + static std::uint32_t byte_swap(std::uint32_t x) noexcept + { +#if defined(__GNUC__) || defined(__clang__) + return __builtin_bswap32(x); +#elif defined(_MSC_VER) + return _byteswap_ulong(x); +#else + return ((x & 0x000000FFu) << 24) | ((x & 0x0000FF00u) << 8) + | ((x & 0x00FF0000u) >> 8) | ((x & 0xFF000000u) >> 24); +#endif + } + + static std::uint64_t byte_swap(std::uint64_t x) noexcept + { +#if defined(__GNUC__) || defined(__clang__) + return __builtin_bswap64(x); +#elif defined(_MSC_VER) + return _byteswap_uint64(x); +#else + x = ((x & 0x00000000FFFFFFFFull) << 32) | ((x & 0xFFFFFFFF00000000ull) >> 32); + x = ((x & 0x0000FFFF0000FFFFull) << 16) | ((x & 0xFFFF0000FFFF0000ull) >> 16); + x = ((x & 0x00FF00FF00FF00FFull) << 8) | ((x & 0xFF00FF00FF00FF00ull) >> 8); + return x; +#endif + } + + /*! + @brief reverse the bytes of a buffer by byte-swapping it as UIntType + + Loading the buffer into an unsigned integer of the same width and swapping + that is what lets the compiler emit a single bswap/rev/movbe; reversing the + buffer element by element does not reliably get there (clang keeps a scalar + shuffle). The two memcpy calls are the only portable way to reinterpret the + bytes and are folded away by every optimizer. + */ + template + static void byte_swap_buffer(std::array& a) noexcept + { + static_assert(sizeof(UIntType) == N, "swap width must match the buffer size"); + UIntType v{}; + std::memcpy(&v, a.data(), sizeof(v)); + v = byte_swap(v); + std::memcpy(a.data(), &v, sizeof(v)); + } + + // reverse the bytes of a fixed-size buffer; a single byte_swap() for the + // common 2/4/8-byte number payloads, std::reverse for any other size + static void reverse_bytes(std::array& a) noexcept + { + byte_swap_buffer(a); + } + + static void reverse_bytes(std::array& a) noexcept + { + byte_swap_buffer(a); + } + + static void reverse_bytes(std::array& a) noexcept + { + byte_swap_buffer(a); + } + + template + static void reverse_bytes(std::array& a) noexcept + { + std::reverse(a.begin(), a.end()); + } + template void write_number(const NumberType n, const bool OutputIsLittleEndian = false) { @@ -20723,10 +20928,10 @@ class binary_writer if (is_little_endian != OutputIsLittleEndian) { // reverse byte order prior to conversion if necessary - std::reverse(vec.begin(), vec.end()); + reverse_bytes(vec); } - oa->write_characters(vec.data(), sizeof(NumberType)); + oa.write_characters(vec.data(), sizeof(NumberType)); } void write_compact_float(const number_float_t n, detail::input_format_t format) @@ -20734,21 +20939,30 @@ class binary_writer #ifdef __GNUC__ JSON_HEDLEY_DIAGNOSTIC_PUSH JSON_HEDLEY_PRAGMA(GCC diagnostic ignored "-Wfloat-equal") +#endif + // When number_float_t is float, static_cast(n) is the identity and + // both branches below are intentionally identical (the "compact" float + // representation is the value itself). Only GCC diagnoses this, and only + // when the sink calls are inlined; clang has no such warning. + // (-Wduplicated-branches only exists from GCC 7 on; naming it on an older + // GCC would itself warn under -Wpragmas) +#if defined(__GNUC__) && !defined(__clang__) && (__GNUC__ >= 7) + JSON_HEDLEY_PRAGMA(GCC diagnostic ignored "-Wduplicated-branches") #endif if (!std::isfinite(n) || ((static_cast(n) >= static_cast(std::numeric_limits::lowest()) && static_cast(n) <= static_cast((std::numeric_limits::max)()) && static_cast(static_cast(n)) == static_cast(n)))) { - oa->write_character(format == detail::input_format_t::cbor - ? get_cbor_float_prefix(static_cast(n)) - : get_msgpack_float_prefix(static_cast(n))); + oa.write_character(format == detail::input_format_t::cbor + ? get_cbor_float_prefix(static_cast(n)) + : get_msgpack_float_prefix(static_cast(n))); write_number(static_cast(n)); } else { - oa->write_character(format == detail::input_format_t::cbor - ? get_cbor_float_prefix(n) - : get_msgpack_float_prefix(n)); + oa.write_character(format == detail::input_format_t::cbor + ? get_cbor_float_prefix(n) + : get_msgpack_float_prefix(n)); write_number(n); } #ifdef __GNUC__ @@ -20815,7 +21029,7 @@ class binary_writer const bool is_little_endian = little_endianness(); /// the output - output_adapter_t oa = nullptr; + OutputSinkType oa; }; } // namespace detail @@ -24156,7 +24370,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec friend ::nlohmann::detail::serializer; template friend class ::nlohmann::detail::iter_impl; - template + template friend class ::nlohmann::detail::binary_writer; template friend class ::nlohmann::detail::binary_reader; @@ -24204,6 +24418,14 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec template using binary_reader = ::nlohmann::detail::binary_reader; template using binary_writer = ::nlohmann::detail::binary_writer; + // binary_writer over a concrete (non-virtual) sink appending into a std::vector, + // used by the vector-returning to_* overloads + template using vector_binary_writer = + ::nlohmann::detail::binary_writer>; + template static vector_binary_writer vector_writer(std::vector& v) + { + return vector_binary_writer(::nlohmann::detail::output_vector_sink(v)); + } JSON_PRIVATE_UNLESS_TESTED: using serializer = ::nlohmann::detail::serializer; @@ -28460,7 +28682,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec static std::vector to_cbor(const basic_json& j) { std::vector result; - to_cbor(j, result); + result.reserve(detail::binary_reserve_hint(j)); + vector_writer(result).write_cbor(j); return result; } @@ -28483,7 +28706,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec static std::vector to_msgpack(const basic_json& j) { std::vector result; - to_msgpack(j, result); + result.reserve(detail::binary_reserve_hint(j)); + vector_writer(result).write_msgpack(j); return result; } @@ -28508,7 +28732,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec const bool use_type = false) { std::vector result; - to_ubjson(j, result, use_size, use_type); + result.reserve(detail::binary_reserve_hint(j)); + vector_writer(result).write_ubjson(j, use_size, use_type); return result; } @@ -28536,7 +28761,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec const bjdata_version_t version = bjdata_version_t::draft2) { std::vector result; - to_bjdata(j, result, use_size, use_type, version); + result.reserve(detail::binary_reserve_hint(j)); + vector_writer(result).write_ubjson(j, use_size, use_type, true, true, version); return result; } @@ -28563,7 +28789,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec static std::vector to_bson(const basic_json& j) { std::vector result; - to_bson(j, result); + result.reserve(detail::binary_reserve_hint(j)); + vector_writer(result).write_bson(j); return result; } diff --git a/tests/src/unit-binary_writer_sinks.cpp b/tests/src/unit-binary_writer_sinks.cpp new file mode 100644 index 000000000..f60e1bf51 --- /dev/null +++ b/tests/src/unit-binary_writer_sinks.cpp @@ -0,0 +1,198 @@ +// __ _____ _____ _____ +// __| | __| | | | JSON for Modern C++ (supporting code) +// | | |__ | | | | | | version 3.12.0 +// |_____|_____|_____|_|___| https://github.com/nlohmann/json +// +// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann +// SPDX-License-Identifier: MIT + +#include "doctest_compatibility.h" + +#include +using nlohmann::json; + +#include +#include +#include + +namespace +{ + +// a spread of values exercising every writer path: scalars of each width, the +// float paths, strings, binary, and containers big enough to reallocate +std::vector test_values() +{ + json big_array = json::array(); + for (int i = 0; i < 5000; ++i) + { + big_array.push_back(i); + } + + json big_object = json::object(); + for (int i = 0; i < 1000; ++i) + { + big_object[std::to_string(i)] = i; + } + + return + { + json(nullptr), json(true), json(false), + json(0), json(-1), json(255), json(-129), json(65535), json(-32769), + json(4294967295U), json(-2147483649LL), json(18446744073709551615ULL), + json(0.0), json(-0.5), json(3.1415926535897932), + json(""), json("hello"), json(std::string(1000, 'x')), + json::binary({0x00, 0x01, 0x02}, 42), + json::array(), json::object(), + json::array({1, 2, 3}), json({{"a", 1}, {"b", nullptr}}), + json({{"nested", {{"deep", json::array({1, "two", 3.0, nullptr})}}}}), + big_array, big_object + }; +} + +// values to_bson() accepts: the document must be an object +std::vector bson_values() +{ + json big_object = json::object(); + for (int i = 0; i < 1000; ++i) + { + big_object[std::to_string(i)] = i; + } + + return + { + json::object(), + json({{"a", 1}, {"b", nullptr}, {"c", true}, {"d", 2.5}, {"e", "text"}}), + json({{"arr", json::array({1, 2, 3})}, {"obj", {{"k", "v"}}}}), + big_object + }; +} + +} // namespace + +// The vector-returning to_*(j) overloads write through the non-virtual +// output_vector_sink, while to_*(j, adapter) goes through output_adapter_sink. +// The two are separate code paths that must stay byte-for-byte identical; these +// checks fail if either overload is ever changed without the other. +TEST_CASE("binary writer output sinks") +{ + SECTION("vector sink and adapter sink agree") + { + // note: no SUBCASE inside these loops - doctest keys subcases by + // name/file/line, so a subcase in a loop body would only ever run for + // the first iteration + for (const auto& j : test_values()) + { + CAPTURE(j.dump(-1, ' ', false, json::error_handler_t::replace)); + + std::vector cbor; + json::to_cbor(j, cbor); + CHECK(json::to_cbor(j) == cbor); + + std::vector msgpack; + json::to_msgpack(j, msgpack); + CHECK(json::to_msgpack(j) == msgpack); + + for (const bool use_size : + { + false, true + }) + { + for (const bool use_type : + { + false, true + }) + { + if (use_type && !use_size) + { + continue; // not a supported combination + } + CAPTURE(use_size); + CAPTURE(use_type); + std::vector ubjson; + json::to_ubjson(j, ubjson, use_size, use_type); + CHECK(json::to_ubjson(j, use_size, use_type) == ubjson); + } + } + + for (const auto version : + { + json::bjdata_version_t::draft2, json::bjdata_version_t::draft3 + }) + { + std::vector bjdata; + json::to_bjdata(j, bjdata, false, false, version); + CHECK(json::to_bjdata(j, false, false, version) == bjdata); + } + } + + for (const auto& j : bson_values()) + { + CAPTURE(j.dump()); + std::vector bson; + json::to_bson(j, bson); + CHECK(json::to_bson(j) == bson); + } + } + + SECTION("the char adapter produces the same bytes") + { + for (const auto& j : test_values()) + { + CAPTURE(j.dump(-1, ' ', false, json::error_handler_t::replace)); + + const std::vector expected = json::to_cbor(j); + std::vector as_char; + json::to_cbor(j, as_char); + + REQUIRE(as_char.size() == expected.size()); + std::vector as_bytes; + as_bytes.reserve(as_char.size()); + for (const char c : as_char) + { + as_bytes.push_back(static_cast(c)); + } + CHECK(as_bytes == expected); + } + } +} + +// binary_reserve_hint() is documented as a *lower* bound on the serialized size, +// so that reserving it up front can never leave the returned vector holding +// capacity beyond what the value actually needs. +TEST_CASE("binary_reserve_hint never over-reserves") +{ + for (const auto& j : test_values()) + { + CAPTURE(j.dump(-1, ' ', false, json::error_handler_t::replace)); + + const std::size_t hint = nlohmann::detail::binary_reserve_hint(j); + + CHECK(hint <= json::to_cbor(j).size()); + CHECK(hint <= json::to_msgpack(j).size()); + CHECK(hint <= json::to_ubjson(j).size()); + CHECK(hint <= json::to_ubjson(j, true, true).size()); + CHECK(hint <= json::to_bjdata(j).size()); + } + + for (const auto& j : bson_values()) + { + CAPTURE(j.dump()); + CHECK(nlohmann::detail::binary_reserve_hint(j) <= json::to_bson(j).size()); + } + + SECTION("scalars get no hint") + { + CHECK(nlohmann::detail::binary_reserve_hint(json(nullptr)) == 0); + CHECK(nlohmann::detail::binary_reserve_hint(json(42)) == 0); + CHECK(nlohmann::detail::binary_reserve_hint(json("a string")) == 0); + CHECK(nlohmann::detail::binary_reserve_hint(json::binary({0x01})) == 0); + } + + SECTION("containers are hinted from their element count") + { + CHECK(nlohmann::detail::binary_reserve_hint(json::array()) == 1); + CHECK(nlohmann::detail::binary_reserve_hint(json::array({1, 2, 3})) == 4); + CHECK(nlohmann::detail::binary_reserve_hint(json::object()) == 1); + CHECK(nlohmann::detail::binary_reserve_hint(json({{"a", 1}, {"b", 2}})) == 5); + } +}