amorfo77/json
1
0
Fork 0
mirror of https://github.com/nlohmann/json.git synced 2026-09-07 16:57:59 +00:00
Code Issues Packages Projects Releases Wiki Activity

Bound UBJSON optimized arrays of a valueless type

Browse Source 
An element of type 'Z' (null), 'T' (true) or 'F' (false) is encoded by its
type marker alone, so an optimized UBJSON array of one of those has no
payload: reading an element consumes no input at all. Its declared count is
therefore the only thing that decides how much is allocated, and nothing
bounded it. "[$Z#l" and a four-byte count is nine bytes of input describing
two billion values; #2793 reports 35 GB and 150 seconds from ten bytes, and
OSS-Fuzz has an out-of-memory and a timeout report for the same shape.

Every other type costs at least one byte per element, so the end of the input
bounds it. 'N' (no-op) is already skipped rather than stored. Objects are not
affected either: each element is preceded by its key, which costs bytes. And
BJData already refuses these markers as an optimized type, so this is a plain
UBJSON matter.

Reject a count above 1,048,576 elements for those three types with
out_of_range.408, the code this reader already uses for a declared size it
will not honour. The check runs before the SAX start event, so no container
is opened and then abandoned.

Rejecting on the read side alone would break the guarantee that anything
to_ubjson() writes can be read back, and would trip the round-trip assertion
in fuzzer-parse_ubjson.cpp. So the writer falls back to the unoptimized
encoding, one byte per element, for arrays of these types above the same
limit. Its decision depends only on the array's size, which is identical for
a value and for anything parsed back from it, so the round trip is stable.

No existing test changes: the largest such count in the test suite is 65,793.
The excessive-size test that already used this shape still passes, now
rejected a little earlier than by the max_size() check it used to reach.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann
2026-09-07 07:50:29 +02:00
parent 3970ecccd1
commit ee69490b28
6 changed files with 160 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/mkdocs/docs/features/binary_formats/ubjson.md
+6
View File
@@ -69,6 +69,12 @@ The library uses the following mapping from JSON values types to UBJSON types ac
Note that `use_size = true` alone may result in larger representations - the benefit of this parameter is that the
receiving side is immediately informed on the number of elements of the container.
An array whose type marker is `Z` (null), `T` (true) or `F` (false) stores no payload at all, because the marker
already is the value. Its declared count is therefore the only thing that decides how much memory the receiving side
allocates, and a handful of bytes can describe billions of elements. `from_ubjson` rejects such an array with
[`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408) when the count exceeds 1,048,576, and
`to_ubjson` writes longer arrays of these types without the annotation, so any value it produces can be read back.
!!! info "Binary values"
If the JSON data contains the binary type, the value stored is a list of integers, as suggested by the UBJSON
docs/mkdocs/docs/home/exceptions.md
+9
View File
@@ -868,6 +868,12 @@ The size of an array or object in a [binary format](../features/binary_formats/i
the size following `#` for [UBJSON](../features/binary_formats/ubjson.md)/[BJData](../features/binary_formats/bjdata.md),
or the encoded length for [CBOR](../features/binary_formats/cbor.md).
The exception is also thrown for a [UBJSON](../features/binary_formats/ubjson.md) array of a type that is encoded by its
marker alone (`Z`, `T` or `F`) whose declared count exceeds 1,048,576. Such an array has no payload, so its count alone
decides how much memory is allocated, and a handful of bytes would otherwise describe billions of values.
[`to_ubjson`](../api/basic_json/to_ubjson.md) writes longer arrays of these types without the size and type annotation,
so any value it produces can still be read back.
!!! failure "Example messages"
```
@@ -879,6 +885,9 @@ or the encoded length for [CBOR](../features/binary_formats/cbor.md).
```
[json.exception.out_of_range.408] syntax error while parsing CBOR size: excessive map size
```
```
[json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive array size
```
### json.exception.out_of_range.409
include/nlohmann/detail/input/binary_reader.hpp
+31
View File
@@ -58,6 +58,26 @@ inline bool little_endianness(int num = 1) noexcept
return *reinterpret_cast<char*>(&num) == 1;
}
/*!
@brief largest element count accepted for a UBJSON container of a valueless type
An element of type 'Z' (null), 'T' (true) or 'F' (false) is encoded by its
type marker alone, so an optimized container of one of those types has no
payload at all and its declared count is the only thing that decides how much
is allocated: `[$Z#L` followed by a large count turns some ten bytes of input
into that many values (see #2793, which reports 35 GB and 150 seconds). Every
other type costs at least one byte per element and is bounded by the end of
the input.
This is a sanity bound rather than a security boundary, and it is far above
any container met in practice. @ref binary_writer falls back to the
unoptimized encoding for longer containers, so that a value serialized by
this library can always be read back.
@sa https://github.com/nlohmann/json/issues/2793
*/
JSON_INLINE_VARIABLE constexpr std::size_t max_valueless_container_size = 1 << 20;
///////////////////
// binary reader //
///////////////////
@@ -2799,6 +2819,17 @@ class binary_reader
if (size_and_type.first != npos)
{
// reading an element of a valueless type consumes no input, so the
// declared count alone decides how much is allocated; the check is
// made before the start event so that no container is opened that
// is then abandoned. See @ref max_valueless_container_size.
if (JSON_HEDLEY_UNLIKELY((size_and_type.second == 'Z' || size_and_type.second == 'T' || size_and_type.second == 'F')
&& size_and_type.first > max_valueless_container_size))
{
return sax->parse_error(chars_read, get_token_string(), out_of_range::create(408,
exception_message(input_format, "excessive array size", "size"), nullptr));
}
if (JSON_HEDLEY_UNLIKELY(!sax->start_array(size_and_type.first)))
{
return false;
include/nlohmann/detail/output/binary_writer.hpp
+11 -1
View File
@@ -826,7 +826,17 @@ class binary_writer
std::vector<CharType> bjdx = {'[', '{', 'S', 'H', 'T', 'F', 'N', 'Z'}; // excluded markers in bjdata optimized type
if (same_prefix && !(use_bjdata && std::find(bjdx.begin(), bjdx.end(), first_prefix) != bjdx.end()))
// an optimized array of a valueless type carries no payload, so a
// reader has nothing but the declared count to bound the allocation
// by and refuses an excessive one. Write the unoptimized form for
// those, at one byte per element, so the result can be read back.
// Objects are not affected: every element is preceded by its key.
const bool valueless_type = (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');
const bool excessive_valueless = valueless_type
&& j.m_data.m_value.array->size() > detail::max_valueless_container_size;
if (same_prefix && !excessive_valueless
&& !(use_bjdata && std::find(bjdx.begin(), bjdx.end(), first_prefix) != bjdx.end()))
{
prefix_required = false;
oa->write_character(to_char_type('$'));
single_include/nlohmann/json.hpp
+42 -1
View File
@@ -10745,6 +10745,26 @@ inline bool little_endianness(int num = 1) noexcept
return *reinterpret_cast<char*>(&num) == 1;
}
/*!
@brief largest element count accepted for a UBJSON container of a valueless type
An element of type 'Z' (null), 'T' (true) or 'F' (false) is encoded by its
type marker alone, so an optimized container of one of those types has no
payload at all and its declared count is the only thing that decides how much
is allocated: `[$Z#L` followed by a large count turns some ten bytes of input
into that many values (see #2793, which reports 35 GB and 150 seconds). Every
other type costs at least one byte per element and is bounded by the end of
the input.
This is a sanity bound rather than a security boundary, and it is far above
any container met in practice. @ref binary_writer falls back to the
unoptimized encoding for longer containers, so that a value serialized by
this library can always be read back.
@sa https://github.com/nlohmann/json/issues/2793
*/
JSON_INLINE_VARIABLE constexpr std::size_t max_valueless_container_size = 1 << 20;
///////////////////
// binary reader //
///////////////////
@@ -13486,6 +13506,17 @@ class binary_reader
if (size_and_type.first != npos)
{
// reading an element of a valueless type consumes no input, so the
// declared count alone decides how much is allocated; the check is
// made before the start event so that no container is opened that
// is then abandoned. See @ref max_valueless_container_size.
if (JSON_HEDLEY_UNLIKELY((size_and_type.second == 'Z' || size_and_type.second == 'T' || size_and_type.second == 'F')
&& size_and_type.first > max_valueless_container_size))
{
return sax->parse_error(chars_read, get_token_string(), out_of_range::create(408,
exception_message(input_format, "excessive array size", "size"), nullptr));
}
if (JSON_HEDLEY_UNLIKELY(!sax->start_array(size_and_type.first)))
{
return false;
@@ -17923,7 +17954,17 @@ class binary_writer
std::vector<CharType> bjdx = {'[', '{', 'S', 'H', 'T', 'F', 'N', 'Z'}; // excluded markers in bjdata optimized type
if (same_prefix && !(use_bjdata && std::find(bjdx.begin(), bjdx.end(), first_prefix) != bjdx.end()))
// an optimized array of a valueless type carries no payload, so a
// reader has nothing but the declared count to bound the allocation
// by and refuses an excessive one. Write the unoptimized form for
// those, at one byte per element, so the result can be read back.
// Objects are not affected: every element is preceded by its key.
const bool valueless_type = (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');
const bool excessive_valueless = valueless_type
&& j.m_data.m_value.array->size() > detail::max_valueless_container_size;
if (same_prefix && !excessive_valueless
&& !(use_bjdata && std::find(bjdx.begin(), bjdx.end(), first_prefix) != bjdx.end()))
{
prefix_required = false;
oa->write_character(to_char_type('$'));
tests/src/unit-ubjson.cpp
+61
View File
@@ -2149,6 +2149,67 @@ TEST_CASE("UBJSON")
}
}
TEST_CASE("UBJSON optimized arrays of a valueless type are bounded")
{
// An element of type 'Z', 'T' or 'F' is encoded by its marker alone, so an
// optimized array of one of those has no payload and the declared count is
// the only thing deciding how much is allocated. Ten bytes used to produce
// billions of values (#2793); every other type costs at least one byte per
// element and is bounded by the end of the input.
json _;
SECTION("an excessive count is rejected")
{
// 'l' is a big-endian int32: 0x7FFFFFFF elements, about 34 GB of value
for (const auto marker :
{'Z', 'T', 'F'
})
{
const std::vector<uint8_t> input = {'[', '$', static_cast<uint8_t>(marker), '#', 'l', 0x7F, 0xFF, 0xFF, 0xFF};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(input), "[json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive array size", json::out_of_range&);
CHECK(json::from_ubjson(input, true, false).is_discarded());
}
}
SECTION("ordinary counts are unaffected")
{
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'Z', '#', 'i', 3})) == json({nullptr, nullptr, nullptr}));
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'T', '#', 'i', 2})) == json({true, true}));
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'F', '#', 'i', 2})) == json({false, false}));
// 'N' is a no-op rather than a value, and still yields an empty array
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'N', '#', 'i', 2})) == json::array());
}
SECTION("a type with a payload is unaffected")
{
// A count past the limit is not rejected for 'U', which costs a byte
// per element and is bounded by the end of the input instead. The
// count is kept just past the limit rather than made huge, because a
// count that also exceeds the array's max_size() is reported as
// out_of_range before the input runs out, and max_size() depends on
// the width of std::size_t.
const std::vector<uint8_t> input = {'[', '$', 'U', '#', 'l', 0x00, 0x10, 0x00, 0x01};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(input), "[json.exception.parse_error.110] parse error at byte 10: syntax error while parsing UBJSON number: unexpected end of input", json::parse_error&);
CHECK(json::from_ubjson(input, true, false).is_discarded());
}
SECTION("the writer stays within what the reader accepts")
{
// below the limit the optimized form is used and is tiny; above it the
// writer falls back so that the result can still be read back
json const at_limit(1048576, nullptr);
const auto v_at_limit = json::to_ubjson(at_limit, true, true);
CHECK(v_at_limit.size() == 9);
CHECK(v_at_limit.at(1) == '$');
CHECK(json::from_ubjson(v_at_limit) == at_limit);
json const above_limit(1048577, nullptr);
const auto v_above_limit = json::to_ubjson(above_limit, true, true);
CHECK(v_above_limit.at(1) != '$');
CHECK(json::from_ubjson(v_above_limit) == above_limit);
}
}
TEST_CASE("Universal Binary JSON Specification Examples 1")
{
SECTION("Null Value")