Compare commits

..
Author SHA1 Message Date
Niels Lohmann 00e5d21041 Stop CBOR indefinite-length strings from recursing per chunk
get_cbor_string() and get_cbor_binary() handled the indefinite-length forms
(0x7F and 0x5F) by calling themselves once per chunk. Each chunk therefore
cost a native stack frame, and since a chunk may itself be an indefinite-
length string, an input of repeated 0x7F bytes reached one frame per input
byte: 200,000 of them crash the process with SIGSEGV before a single byte is
rejected. This is the same defect as #5104, in a path the container-level
work does not touch.

Count the open levels instead of recursing through them. That is enough here
because every chunk is appended to the same result -- get_bytes() writes at
result.size() -- so there is no per-level state to keep. The temporary chunk
string and its copy into the result go away with the recursion.

The definite-length cases move to get_cbor_string_chunk() and
get_cbor_binary_chunk() unchanged, including their error messages, which
still name 0x7F and 0x5F because those are handled one level up.

Behaviour is unchanged. Comparing against develop over the interesting byte
sequences -- empty, single-chunk, nested, over-closed and truncated forms,
both strings and byte arrays, and an indefinite-length map key -- produces
identical values, error codes, messages and byte offsets. The 200,000-level
input now reports parse_error.110 at byte 200001 instead of crashing.

Note that nesting these is not valid CBOR: RFC 8949, Section 3.2.3 forbids
it. This does not change that either way -- it has always been accepted, and
rejecting it is a separate decision (#5317, #5325). Should it be rejected
later, that is now one condition on the level counter rather than a change to
the control flow.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-07 07:50:27 +02:00
Niels Lohmann 755c547003 Return the parsed value by move from from_cbor() and friends
The binary entry points end with

    return res ? result : basic_json(value_t::discarded);

The condition operator's second operand is an lvalue, so this is not a case
where the return value can be elided or implicitly moved from: every
successful from_cbor(), from_msgpack(), from_ubjson(), from_bjdata() and
from_bson() call deep-copies the value it just parsed, and then destroys the
original.

The copy is not cheap, and it is not incidental: basic_json's copy
constructor walks the whole value. Parsing a 2 MB CBOR document with 60,000
objects, median of 25 runs, clang 17 -O3:

    from_cbor      26.99 ms  ->  14.65 ms
    from_msgpack   26.82 ms  ->  14.82 ms

Moving instead of copying is the entire change; the parsed value is not used
again after the return expression is evaluated.

There is a second reason to prefer the move. The copy constructor recurses
once per nesting level, so the copy is also a stack-overflow path on the
return side, on a value the reader has already accepted. That is currently
masked because the readers themselves recurse and overflow first (#5104), but
it has to be fixed for making them iterative to have any effect.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-07 07:50:09 +02:00
5 changed files with 469 additions and 300 deletions
+137 -51
View File
@@ -996,23 +996,21 @@ class binary_reader
}
/*!
@brief reads a CBOR string
@brief reads a definite-length CBOR string
This function first reads starting bytes to determine the expected
string length and then copies this number of bytes into a string.
Additionally, CBOR's strings with indefinite lengths are supported.
Reads everything @ref get_cbor_string accepts except the indefinite-length
form, which that function handles itself. The bytes are appended to @a
result, so consecutive chunks of an indefinite-length string can be read
into the same string.
@param[out] result created string
@param[out] result string the bytes are appended to
@return whether string creation completed
*/
bool get_cbor_string(string_t& result)
{
if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::cbor, "string")))
{
return false;
}
@pre @a current is not EOF
*/
bool get_cbor_string_chunk(string_t& result)
{
switch (current)
{
// UTF-8 string (0x00..0x17 bytes follow)
@@ -1068,20 +1066,6 @@ class binary_reader
return get_number(input_format_t::cbor, len) && get_string(input_format_t::cbor, len, result);
}
case 0x7F: // UTF-8 string (indefinite length)
{
while (get() != 0xFF)
{
string_t chunk;
if (!get_cbor_string(chunk))
{
return false;
}
result.append(chunk);
}
return true;
}
default:
{
auto last_token = get_token_string();
@@ -1092,23 +1076,82 @@ class binary_reader
}
/*!
@brief reads a CBOR byte array
@brief reads a CBOR string
This function first reads starting bytes to determine the expected
byte array length and then copies this number of bytes into the byte array.
Additionally, CBOR's byte arrays with indefinite lengths are supported.
string length and then copies this number of bytes into a string.
Additionally, CBOR's strings with indefinite lengths are supported.
@param[out] result created byte array
@param[out] result created string
@return whether string creation completed
*/
bool get_cbor_string(string_t& result)
{
// number of indefinite-length strings that have been opened and not
// closed yet. RFC 8949, Section 3.2.3 does not permit nesting them,
// but this reader has always accepted it, so the open levels are
// counted instead of recursed through, which overflowed the stack for
// an input of repeated 0x7F bytes (see #5104). Every chunk is appended
// to the same result, so no per-level state is needed.
std::size_t open = 0;
while (true)
{
if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::cbor, "string")))
{
return false;
}
if (current == 0x7F) // UTF-8 string (indefinite length)
{
++open;
get();
continue;
}
// a break marker closes the innermost indefinite-length string;
// outside of one it is not a string and falls through to the error
if (open != 0 && current == 0xFF)
{
if (--open == 0)
{
return true;
}
get();
continue;
}
if (JSON_HEDLEY_UNLIKELY(!get_cbor_string_chunk(result)))
{
return false;
}
if (open == 0)
{
return true;
}
get();
}
}
/*!
@brief reads a definite-length CBOR byte array
Reads everything @ref get_cbor_binary accepts except the indefinite-length
form, which that function handles itself. The bytes are appended to @a
result, so consecutive chunks of an indefinite-length byte array can be
read into the same byte array.
@param[out] result byte array the bytes are appended to
@return whether byte array creation completed
*/
bool get_cbor_binary(binary_t& result)
{
if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::cbor, "binary")))
{
return false;
}
@pre @a current is not EOF
*/
bool get_cbor_binary_chunk(binary_t& result)
{
switch (current)
{
// Binary data (0x00..0x17 bytes follow)
@@ -1168,20 +1211,6 @@ class binary_reader
get_binary(input_format_t::cbor, len, result);
}
case 0x5F: // Binary data (indefinite length)
{
while (get() != 0xFF)
{
binary_t chunk;
if (!get_cbor_binary(chunk))
{
return false;
}
result.insert(result.end(), chunk.begin(), chunk.end());
}
return true;
}
default:
{
auto last_token = get_token_string();
@@ -1191,6 +1220,63 @@ class binary_reader
}
}
/*!
@brief reads a CBOR byte array
This function first reads starting bytes to determine the expected
byte array length and then copies this number of bytes into the byte array.
Additionally, CBOR's byte arrays with indefinite lengths are supported.
@param[out] result created byte array
@return whether byte array creation completed
*/
bool get_cbor_binary(binary_t& result)
{
// the open indefinite-length byte arrays are counted rather than
// recursed through, for the reason given in @ref get_cbor_string
std::size_t open = 0;
while (true)
{
if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::cbor, "binary")))
{
return false;
}
if (current == 0x5F) // Binary data (indefinite length)
{
++open;
get();
continue;
}
// a break marker closes the innermost indefinite-length byte
// array; outside of one it falls through to the error below
if (open != 0 && current == 0xFF)
{
if (--open == 0)
{
return true;
}
get();
continue;
}
if (JSON_HEDLEY_UNLIKELY(!get_cbor_binary_chunk(result)))
{
return false;
}
if (open == 0)
{
return true;
}
get();
}
}
/*!
@brief narrow a definite CBOR array/map length to std::size_t
+70 -28
View File
@@ -4473,8 +4473,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
basic_json result;
auto ia = detail::input_adapter(std::forward<InputType>(i));
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
/// @brief create a JSON value from an input in CBOR format (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -4490,8 +4493,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
basic_json result;
auto ia = detail::input_adapter(std::move(first), std::move(last));
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
template<typename T>
@@ -4516,8 +4522,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
auto ia = i.get();
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
// NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg)
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
/// @brief create a JSON value from an input in MessagePack format
@@ -4531,8 +4540,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
basic_json result;
auto ia = detail::input_adapter(std::forward<InputType>(i));
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
/// @brief create a JSON value from an input in MessagePack format (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -4547,8 +4559,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
basic_json result;
auto ia = detail::input_adapter(std::move(first), std::move(last));
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
template<typename T>
@@ -4571,8 +4586,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
auto ia = i.get();
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
// NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg)
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
/// @brief create a JSON value from an input in UBJSON format
@@ -4586,8 +4604,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
basic_json result;
auto ia = detail::input_adapter(std::forward<InputType>(i));
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
/// @brief create a JSON value from an input in UBJSON format (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -4602,8 +4623,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
basic_json result;
auto ia = detail::input_adapter(std::move(first), std::move(last));
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
template<typename T>
@@ -4626,8 +4650,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
auto ia = i.get();
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
// NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg)
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
/// @brief create a JSON value from an input in BJData format
@@ -4641,8 +4668,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
basic_json result;
auto ia = detail::input_adapter(std::forward<InputType>(i));
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::bjdata).sax_parse(input_format_t::bjdata, &sdp, strict); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::bjdata).sax_parse(input_format_t::bjdata, &sdp, strict)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
/// @brief create a JSON value from an input in BJData format (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -4657,8 +4687,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
basic_json result;
auto ia = detail::input_adapter(std::move(first), std::move(last));
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::bjdata).sax_parse(input_format_t::bjdata, &sdp, strict); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::bjdata).sax_parse(input_format_t::bjdata, &sdp, strict)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
/// @brief create a JSON value from an input in BSON format
@@ -4672,8 +4705,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
basic_json result;
auto ia = detail::input_adapter(std::forward<InputType>(i));
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
/// @brief create a JSON value from an input in BSON format (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -4688,8 +4724,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
basic_json result;
auto ia = detail::input_adapter(std::move(first), std::move(last));
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
template<typename T>
@@ -4712,8 +4751,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
auto ia = i.get();
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
// NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg)
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
/// @}
+207 -79
View File
@@ -11683,23 +11683,21 @@ class binary_reader
}
/*!
@brief reads a CBOR string
@brief reads a definite-length CBOR string
This function first reads starting bytes to determine the expected
string length and then copies this number of bytes into a string.
Additionally, CBOR's strings with indefinite lengths are supported.
Reads everything @ref get_cbor_string accepts except the indefinite-length
form, which that function handles itself. The bytes are appended to @a
result, so consecutive chunks of an indefinite-length string can be read
into the same string.
@param[out] result created string
@param[out] result string the bytes are appended to
@return whether string creation completed
*/
bool get_cbor_string(string_t& result)
{
if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::cbor, "string")))
{
return false;
}
@pre @a current is not EOF
*/
bool get_cbor_string_chunk(string_t& result)
{
switch (current)
{
// UTF-8 string (0x00..0x17 bytes follow)
@@ -11755,20 +11753,6 @@ class binary_reader
return get_number(input_format_t::cbor, len) && get_string(input_format_t::cbor, len, result);
}
case 0x7F: // UTF-8 string (indefinite length)
{
while (get() != 0xFF)
{
string_t chunk;
if (!get_cbor_string(chunk))
{
return false;
}
result.append(chunk);
}
return true;
}
default:
{
auto last_token = get_token_string();
@@ -11779,23 +11763,82 @@ class binary_reader
}
/*!
@brief reads a CBOR byte array
@brief reads a CBOR string
This function first reads starting bytes to determine the expected
byte array length and then copies this number of bytes into the byte array.
Additionally, CBOR's byte arrays with indefinite lengths are supported.
string length and then copies this number of bytes into a string.
Additionally, CBOR's strings with indefinite lengths are supported.
@param[out] result created byte array
@param[out] result created string
@return whether string creation completed
*/
bool get_cbor_string(string_t& result)
{
// number of indefinite-length strings that have been opened and not
// closed yet. RFC 8949, Section 3.2.3 does not permit nesting them,
// but this reader has always accepted it, so the open levels are
// counted instead of recursed through, which overflowed the stack for
// an input of repeated 0x7F bytes (see #5104). Every chunk is appended
// to the same result, so no per-level state is needed.
std::size_t open = 0;
while (true)
{
if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::cbor, "string")))
{
return false;
}
if (current == 0x7F) // UTF-8 string (indefinite length)
{
++open;
get();
continue;
}
// a break marker closes the innermost indefinite-length string;
// outside of one it is not a string and falls through to the error
if (open != 0 && current == 0xFF)
{
if (--open == 0)
{
return true;
}
get();
continue;
}
if (JSON_HEDLEY_UNLIKELY(!get_cbor_string_chunk(result)))
{
return false;
}
if (open == 0)
{
return true;
}
get();
}
}
/*!
@brief reads a definite-length CBOR byte array
Reads everything @ref get_cbor_binary accepts except the indefinite-length
form, which that function handles itself. The bytes are appended to @a
result, so consecutive chunks of an indefinite-length byte array can be
read into the same byte array.
@param[out] result byte array the bytes are appended to
@return whether byte array creation completed
*/
bool get_cbor_binary(binary_t& result)
{
if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::cbor, "binary")))
{
return false;
}
@pre @a current is not EOF
*/
bool get_cbor_binary_chunk(binary_t& result)
{
switch (current)
{
// Binary data (0x00..0x17 bytes follow)
@@ -11855,20 +11898,6 @@ class binary_reader
get_binary(input_format_t::cbor, len, result);
}
case 0x5F: // Binary data (indefinite length)
{
while (get() != 0xFF)
{
binary_t chunk;
if (!get_cbor_binary(chunk))
{
return false;
}
result.insert(result.end(), chunk.begin(), chunk.end());
}
return true;
}
default:
{
auto last_token = get_token_string();
@@ -11878,6 +11907,63 @@ class binary_reader
}
}
/*!
@brief reads a CBOR byte array
This function first reads starting bytes to determine the expected
byte array length and then copies this number of bytes into the byte array.
Additionally, CBOR's byte arrays with indefinite lengths are supported.
@param[out] result created byte array
@return whether byte array creation completed
*/
bool get_cbor_binary(binary_t& result)
{
// the open indefinite-length byte arrays are counted rather than
// recursed through, for the reason given in @ref get_cbor_string
std::size_t open = 0;
while (true)
{
if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::cbor, "binary")))
{
return false;
}
if (current == 0x5F) // Binary data (indefinite length)
{
++open;
get();
continue;
}
// a break marker closes the innermost indefinite-length byte
// array; outside of one it falls through to the error below
if (open != 0 && current == 0xFF)
{
if (--open == 0)
{
return true;
}
get();
continue;
}
if (JSON_HEDLEY_UNLIKELY(!get_cbor_binary_chunk(result)))
{
return false;
}
if (open == 0)
{
return true;
}
get();
}
}
/*!
@brief narrow a definite CBOR array/map length to std::size_t
@@ -25901,8 +25987,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
basic_json result;
auto ia = detail::input_adapter(std::forward<InputType>(i));
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
/// @brief create a JSON value from an input in CBOR format (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -25918,8 +26007,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
basic_json result;
auto ia = detail::input_adapter(std::move(first), std::move(last));
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
template<typename T>
@@ -25944,8 +26036,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
auto ia = i.get();
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
// NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg)
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
/// @brief create a JSON value from an input in MessagePack format
@@ -25959,8 +26054,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
basic_json result;
auto ia = detail::input_adapter(std::forward<InputType>(i));
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
/// @brief create a JSON value from an input in MessagePack format (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -25975,8 +26073,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
basic_json result;
auto ia = detail::input_adapter(std::move(first), std::move(last));
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
template<typename T>
@@ -25999,8 +26100,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
auto ia = i.get();
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
// NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg)
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
/// @brief create a JSON value from an input in UBJSON format
@@ -26014,8 +26118,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
basic_json result;
auto ia = detail::input_adapter(std::forward<InputType>(i));
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
/// @brief create a JSON value from an input in UBJSON format (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -26030,8 +26137,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
basic_json result;
auto ia = detail::input_adapter(std::move(first), std::move(last));
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
template<typename T>
@@ -26054,8 +26164,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
auto ia = i.get();
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
// NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg)
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
/// @brief create a JSON value from an input in BJData format
@@ -26069,8 +26182,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
basic_json result;
auto ia = detail::input_adapter(std::forward<InputType>(i));
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::bjdata).sax_parse(input_format_t::bjdata, &sdp, strict); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::bjdata).sax_parse(input_format_t::bjdata, &sdp, strict)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
/// @brief create a JSON value from an input in BJData format (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -26085,8 +26201,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
basic_json result;
auto ia = detail::input_adapter(std::move(first), std::move(last));
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::bjdata).sax_parse(input_format_t::bjdata, &sdp, strict); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::bjdata).sax_parse(input_format_t::bjdata, &sdp, strict)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
/// @brief create a JSON value from an input in BSON format
@@ -26100,8 +26219,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
basic_json result;
auto ia = detail::input_adapter(std::forward<InputType>(i));
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
/// @brief create a JSON value from an input in BSON format (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -26116,8 +26238,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
basic_json result;
auto ia = detail::input_adapter(std::move(first), std::move(last));
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
template<typename T>
@@ -26140,8 +26265,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
auto ia = i.get();
detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
// NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg)
const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict); // cppcheck-suppress[accessMoved]
return res ? result : basic_json(value_t::discarded);
if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict)) // cppcheck-suppress[accessMoved]
{
result = value_t::discarded;
}
return result;
}
/// @}
+3 -142
View File
@@ -38,54 +38,6 @@ class huge_binary_t : public std::vector<std::uint8_t>
using huge_binary_json = nlohmann::basic_json <
std::map, std::vector, std::string, bool, std::int64_t, std::uint64_t,
double, std::allocator, nlohmann::adl_serializer, huge_binary_t, void >;
// a string type that can be made to report a size beyond INT32_MAX without
// allocating that much memory, so BSON length overflow can be tested for
// strings and (embedded) documents as well, following the same idea as
// huge_binary_t.
//
// Unlike huge_binary_t (which is only ever used as the BSON *value* type),
// this type doubles as basic_json's StringType and is therefore also used
// for *object keys* (e.g. "s" or "nested" below). Only the designated test
// value is meant to lie about its size - if every huge_string_t (including
// keys) reported a huge size, the running totals computed while walking the
// BSON document (see calc_bson_object_size & friends in binary_writer.hpp)
// would need more than 32 bits, and on platforms where std::size_t is only
// 32 bits wide that arithmetic would silently wrap around, producing wrong
// (or even unguarded) lengths. The fake size is therefore opt-in via
// as_huge(), and plain strings - in particular object keys - keep reporting
// their real, small size.
class huge_string_t : public std::string
{
public:
using std::string::string;
huge_string_t(const std::string& s) : std::string(s) {} // NOLINT(google-explicit-constructor,hicpp-explicit-conversions)
// returns a copy of @a s whose size() pretends to be huge
static huge_string_t as_huge(const std::string& s)
{
huge_string_t result(s);
result.pretend_huge = true;
return result;
}
size_type size() const noexcept
{
if (pretend_huge)
{
// one byte more than the BSON length field can represent
return static_cast<size_type>((std::numeric_limits<std::int32_t>::max)()) + 1;
}
return std::string::size();
}
private:
bool pretend_huge = false;
};
using huge_string_json = nlohmann::basic_json <
std::map, std::vector, huge_string_t, bool, std::int64_t, std::uint64_t,
double, std::allocator, nlohmann::adl_serializer, std::vector<std::uint8_t>, void >;
} // namespace
TEST_CASE("BSON")
@@ -153,36 +105,10 @@ TEST_CASE("BSON")
SECTION("lengths exceeding INT32_MAX cannot be serialized to BSON")
{
// out_of_range.412 is thrown from a single shared helper
// (to_bson_length) that guards the BSON length fields of binary
// values, strings, and (embedded) documents alike
SECTION("binary")
{
huge_binary_json j;
j["b"] = huge_binary_json::binary(huge_binary_t{});
huge_binary_json j;
j["b"] = huge_binary_json::binary(huge_binary_t{});
CHECK_THROWS_WITH_AS(huge_binary_json::to_bson(j), "[json.exception.out_of_range.412] BSON length 2147483661 exceeds maximum of 2147483647", huge_binary_json::out_of_range&);
}
SECTION("string")
{
huge_string_json j;
j["s"] = huge_string_t::as_huge("value");
CHECK_THROWS_WITH_AS(huge_string_json::to_bson(j), "[json.exception.out_of_range.412] BSON length 2147483661 exceeds maximum of 2147483647", huge_string_json::out_of_range&);
}
SECTION("document")
{
// an oversized string nested one level deep makes the
// *embedded* document's own length exceed INT32_MAX as well
huge_string_json nested;
nested["s"] = huge_string_t::as_huge("value");
huge_string_json j;
j["nested"] = nested;
CHECK_THROWS_WITH_AS(huge_string_json::to_bson(j), "[json.exception.out_of_range.412] BSON length 2147483674 exceeds maximum of 2147483647", huge_string_json::out_of_range&);
}
CHECK_THROWS_WITH_AS(huge_binary_json::to_bson(j), "[json.exception.out_of_range.412] BSON length 2147483661 exceeds maximum of 2147483647", huge_binary_json::out_of_range&);
}
SECTION("string length must be at least 1")
@@ -267,23 +193,6 @@ TEST_CASE("BSON")
CHECK(json::from_bson(result, true, false) == j);
}
SECTION("non-empty object with bool from a non-0/1 byte (lenient parsing)")
{
// documented lenient behavior (see gh-5333): any non-zero byte
// is accepted as `true`, not just 0x01
std::vector<std::uint8_t> const input =
{
0x0D, 0x00, 0x00, 0x00, // size (little endian)
0x08, // entry: boolean
'e', 'n', 't', 'r', 'y', '\x00',
0x02, // value = 0x02 (neither 0x00 nor 0x01)
0x00 // end marker
};
const json expected = { { "entry", true } };
CHECK(json::from_bson(input) == expected);
}
SECTION("non-empty object with double")
{
json const j =
@@ -590,29 +499,6 @@ TEST_CASE("BSON")
CHECK(json::from_bson(result, true, false) == j);
}
SECTION("array elements with non-conforming keys (lenient parsing)")
{
// documented lenient behavior (see gh-5333): BSON array element
// keys are not checked against the required decimal sequence
// "0", "1", "2", ... - elements are taken in encoded order
std::vector<std::uint8_t> const input =
{
0x26, 0x00, 0x00, 0x00, // size (little endian)
0x04, 'e', 'n', 't', 'r', 'y', '\x00', // entry: embedded array
0x1A, 0x00, 0x00, 0x00, // size (little endian)
0x10, '5', 0x00, 0x0A, 0x00, 0x00, 0x00, // key "5" (bogus) -> 10
0x10, 'x', 0x00, 0x14, 0x00, 0x00, 0x00, // key "x" (non-numeric) -> 20
0x10, '1', 0x00, 0x1E, 0x00, 0x00, 0x00, // key "1" (out of order) -> 30
0x00, // end marker (embedded array)
0x00 // end marker
};
const json expected = { { "entry", json::array({10, 20, 30}) } };
CHECK(json::from_bson(input) == expected);
}
SECTION("non-empty object with binary member")
{
const size_t N = 10;
@@ -708,31 +594,6 @@ TEST_CASE("BSON")
CHECK(json::from_bson(result, true, false) == j);
}
SECTION("binary member with subtype 0x02 (old binary) keeps its inner length prefix (lenient parsing)")
{
// documented lenient behavior (see gh-5333): the payload for
// binary subtype 0x02 ("old binary") is returned as-is,
// including its own inner 4-byte length prefix; it is not
// stripped or reinterpreted
std::vector<std::uint8_t> const input =
{
0x17, 0x00, 0x00, 0x00, // size (little endian)
0x05, 'e', 'n', 't', 'r', 'y', '\x00', // entry: binary
0x06, 0x00, 0x00, 0x00, // size of binary (little endian)
0x02, // "old binary" subtype
0x02, 0x00, 0x00, 0x00, // inner length prefix (part of the old-binary payload)
0x68, 0x69, // payload ('h', 'i')
0x00 // end marker
};
// the inner length prefix is part of the (unmodified) payload
const std::vector<std::uint8_t> expected_payload = {0x02, 0x00, 0x00, 0x00, 0x68, 0x69};
const json expected = { { "entry", json::binary(expected_payload, 0x02) } };
CHECK(json::from_bson(input) == expected);
}
SECTION("Some more complex document")
{
json const j =
+52
View File
@@ -2035,6 +2035,58 @@ TEST_CASE("CBOR definite length equal to the indefinite-length sentinel")
}
}
TEST_CASE("CBOR indefinite-length strings do not recurse per chunk")
{
// Reading an indefinite-length string or byte array used to call itself
// once per chunk, so a payload of repeated 0x7F (or 0x5F) bytes exhausted
// the call stack before any of the input was rejected. The open levels are
// counted now, and the levels below prove the reader still reads the same
// values and reports the same errors at the same byte offsets.
json _;
SECTION("many open levels are reported, not crashed on")
{
const std::vector<uint8_t> input(200000, 0x7F);
CHECK_THROWS_WITH_AS(_ = json::from_cbor(input), "[json.exception.parse_error.110] parse error at byte 200001: syntax error while parsing CBOR string: unexpected end of input", json::parse_error&);
CHECK(json::from_cbor(input, true, false).is_discarded());
}
SECTION("many open levels are reported, not crashed on (binary)")
{
const std::vector<uint8_t> input(200000, 0x5F);
CHECK_THROWS_WITH_AS(_ = json::from_cbor(input), "[json.exception.parse_error.110] parse error at byte 200001: syntax error while parsing CBOR binary: unexpected end of input", json::parse_error&);
CHECK(json::from_cbor(input, true, false).is_discarded());
}
SECTION("chunks are still concatenated")
{
CHECK(json::from_cbor(std::vector<uint8_t>({0x7F, 0xFF})) == json(""));
CHECK(json::from_cbor(std::vector<uint8_t>({0x7F, 0x61, 0x61, 0xFF})) == json("a"));
// nested indefinite-length strings are concatenated across levels
CHECK(json::from_cbor(std::vector<uint8_t>({0x7F, 0x7F, 0x61, 0x61, 0xFF, 0x61, 0x62, 0xFF})) == json("ab"));
CHECK(json::from_cbor(std::vector<uint8_t>({0x7F, 0x7F, 0x7F, 0x61, 0x7A, 0xFF, 0xFF, 0xFF})) == json("z"));
CHECK(json::from_cbor(std::vector<uint8_t>({0xA1, 0x7F, 0x61, 0x61, 0xFF, 0x01})) == json({{"a", 1}}));
}
SECTION("chunks are still concatenated (binary)")
{
CHECK(json::from_cbor(std::vector<uint8_t>({0x5F, 0x41, 0x61, 0xFF})) == json::binary({0x61}));
CHECK(json::from_cbor(std::vector<uint8_t>({0x5F, 0x5F, 0x41, 0x61, 0xFF, 0x41, 0x62, 0xFF})) == json::binary({0x61, 0x62}));
}
SECTION("a chunk that is not a string is still rejected")
{
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0x7F, 0x7F, 0x00})), "[json.exception.parse_error.113] parse error at byte 3: syntax error while parsing CBOR string: expected length specification (0x60-0x7B) or indefinite string type (0x7F); last byte: 0x00", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0x5F, 0x5F, 0x00})), "[json.exception.parse_error.113] parse error at byte 3: syntax error while parsing CBOR binary: expected length specification (0x40-0x5B) or indefinite binary array type (0x5F); last byte: 0x00", json::parse_error&);
}
SECTION("a break marker outside an indefinite-length string is not a string")
{
// 0xFF only closes a string that was opened; on its own it is not one
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0xA1, 0xFF, 0x01})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR string: expected length specification (0x60-0x7B) or indefinite string type (0x7F); last byte: 0xFF", json::parse_error&);
}
}
TEST_CASE("CBOR roundtrips" * doctest::skip())
{
SECTION("input from flynn")