mirror of
https://github.com/nlohmann/json.git
synced 2026-09-13 11:47:58 +00:00
Reserve capped array capacity in json_sax_dom_parser::start_array() for definite-length binary arrays (#5476)
* Reserve capped array capacity for definite-length binary arrays CBOR, MessagePack, and the optimized [$type#count UBJSON/BJData form all pass an exact element count to sax->start_array(len), but json_sax_dom_parser::start_array() (and the callback variant) only used len for an overflow check against max_size() and never reserved the underlying vector, so each element triggered a reallocation cascade via emplace_back(). Reserve upfront, but cap the reservation at 16384 elements: max_size() for a std::vector is far larger than any realistic input, so an unbounded reserve(len) would let a crafted/truncated header (e.g. CBOR 0x9A + a huge uint32 count with no data) trigger a multi-gigabyte allocation attempt instead of the normal graceful parse_error. With the cap, a hostile length still fails fast with the existing parse_error, while realistic arrays get a single up-front allocation. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Make the huge-claimed-length DoS regression tests portable across size_t widths On a platform where size_t is narrower than 64 bits (e.g. 32-bit mingw/msvc x86), the previously-hardcoded huge test lengths either collide with that platform's unknown_size() sentinel (CBOR/MessagePack, both using exactly SIZE_MAX) or exceed the platform's smaller vector<json>::max_size() (UBJSON/BJData's 0x7FFFFFFF), so the header is now rejected outright (out_of_range.408) instead of being accepted and only found short of data (parse_error.110). Both are safe, bounded rejections of the hostile input; the property under test -- no attempt to allocate space for billions of elements -- holds either way. Accept both outcomes instead of pinning the 64-bit-only exact result. Also fixed an unrelated clang-tidy finding (google-readability-casting) on the functional-style std::size_t(...) casts in the neighboring "arrays of various sizes" section. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix remaining CI failures in the huge-claimed-length DoS regression tests - Apply the same google-readability-casting fix (std::size_t{N} instead of std::size_t(N)) to the "arrays of various sizes" section in unit-msgpack.cpp, unit-ubjson.cpp, and unit-bjdata.cpp; only unit-cbor.cpp had been fixed previously, since clang-tidy's build didn't get far enough to report the other three in the same pass. - json_sax_dom_parser::start_array()'s max_size() check calls JSON_THROW directly rather than going through sax->parse_error(), so unlike the scanner's own "not enough data" parse_error it is not gated by allow_exceptions=false. On a platform where a header's claimed count exceeds max_size() (e.g. 32-bit, for UBJSON/BJData's 0x7FFFFFFF test value), from_ubjson/from_bjdata(input, true, false) can therefore still throw instead of returning a discarded value. Make that assertion tolerant of either outcome, same as the main exception-catching check above it. - Guard all four "a huge claimed length..." SECTIONs with #if !defined(JSON_NOEXCEPTION), matching this test suite's existing convention for exception-dependent tests: under JSON_NOEXCEPTION, JSON_THROW never produces a catchable C++ exception at all (it aborts the process), so a section that relies on try/catch to distinguish between two acceptable outcomes cannot be expressed under that build configuration regardless of platform. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Use (std::min)(len, reserve_cap) instead of a ternary in start_array() Addresses review feedback from @gregmarr on PR #5476. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann
@@ -3551,6 +3551,111 @@ TEST_CASE("BJData")
|
||||
}
|
||||
}
|
||||
|
||||
TEST_CASE("issue #5405 - array reserve for definite-length BJData arrays")
|
||||
{
|
||||
#if !defined(JSON_NOEXCEPTION)
|
||||
// this SECTION relies on catching a thrown exception to distinguish
|
||||
// which of two acceptable, bounded rejections a hostile header took;
|
||||
// under JSON_NOEXCEPTION, JSON_THROW never produces a catchable C++
|
||||
// exception (it aborts instead), so this cannot be tested that way here
|
||||
SECTION("a huge claimed length with no element data must not over-allocate")
|
||||
{
|
||||
// optimized form [$type#count: type 'i' (int8), count as a four-byte
|
||||
// little-endian 'l' (int32) of 0x7FFFFFFF (2147483647), but no
|
||||
// element data at all. max_size() for a std::vector is far larger
|
||||
// than this count, so it does not reject the header outright; the
|
||||
// (capped) reservation must not attempt to allocate space for
|
||||
// billions of elements before the missing data is detected.
|
||||
json _;
|
||||
const std::vector<uint8_t> input = {'[', '$', 'i', '#', 'l', 0xFF, 0xFF, 0xFF, 0x7F};
|
||||
// On a platform where std::vector<json>::max_size() is smaller than
|
||||
// the claimed count (e.g. 32-bit, where max_size() is bounded by a
|
||||
// 32-bit SIZE_MAX divided by sizeof(json)), the SAX consumer's own
|
||||
// check rejects the header outright (out_of_range.408, with the
|
||||
// claimed count in the message) instead of accepting it and only
|
||||
// finding it short of data once the (capped) reservation looks for
|
||||
// element bytes that were never provided (parse_error.110). Either
|
||||
// is an acceptable, bounded rejection of the hostile header -- the
|
||||
// property under test is that no path attempts to allocate space
|
||||
// for billions of elements.
|
||||
bool threw = false;
|
||||
try
|
||||
{
|
||||
_ = json::from_bjdata(input);
|
||||
}
|
||||
catch (const json::parse_error& e)
|
||||
{
|
||||
threw = true;
|
||||
CHECK(e.id == 110);
|
||||
CHECK(std::string(e.what()) == "[json.exception.parse_error.110] parse error at byte 10: syntax error while parsing BJData number: unexpected end of input");
|
||||
}
|
||||
catch (const json::out_of_range& e)
|
||||
{
|
||||
threw = true;
|
||||
CHECK(e.id == 408);
|
||||
CHECK(std::string(e.what()).find("excessive array size") != std::string::npos);
|
||||
}
|
||||
CHECK(threw);
|
||||
|
||||
// json_sax_dom_parser::start_array()'s max_size() check (unlike the
|
||||
// scanner's own parse_error path) throws unconditionally via
|
||||
// JSON_THROW rather than going through sax->parse_error(), so it is
|
||||
// not gated by allow_exceptions=false on a platform where this
|
||||
// header hits that check (e.g. 32-bit, see above) -- allow either
|
||||
// a discarded result or the same out_of_range it throws with
|
||||
// exceptions enabled.
|
||||
try
|
||||
{
|
||||
CHECK(json::from_bjdata(input, true, false).is_discarded());
|
||||
}
|
||||
catch (const json::out_of_range& e)
|
||||
{
|
||||
CHECK(e.id == 408);
|
||||
}
|
||||
}
|
||||
#endif
|
||||
|
||||
SECTION("arrays of various sizes decode to the same value as before the reserve optimization")
|
||||
{
|
||||
for (const auto size :
|
||||
{
|
||||
std::size_t{0}, std::size_t{1}, std::size_t{5}, // small
|
||||
std::size_t{16384}, // exactly at the reserve cap
|
||||
std::size_t{20000} // above the reserve cap
|
||||
})
|
||||
{
|
||||
CAPTURE(size)
|
||||
json j = json::array();
|
||||
for (std::size_t i = 0; i < size; ++i)
|
||||
{
|
||||
j.push_back(static_cast<int>(i % 1000));
|
||||
}
|
||||
|
||||
// exercise both the plain and the optimized [$type#count encoding
|
||||
const auto packed_plain = json::to_bjdata(j);
|
||||
CHECK(json::from_bjdata(packed_plain) == j);
|
||||
|
||||
const auto packed_optimized = json::to_bjdata(j, true, true);
|
||||
CHECK(json::from_bjdata(packed_optimized) == j);
|
||||
}
|
||||
}
|
||||
|
||||
SECTION("a user-defined SAX consumer is unaffected by the internal DOM reserve optimization")
|
||||
{
|
||||
// the reserve() call is local to json_sax_dom_parser / json_sax_dom_callback_parser;
|
||||
// a custom SAX consumer that does not touch a DOM array sees identical events
|
||||
json j = json::array();
|
||||
for (int i = 0; i < 100; ++i)
|
||||
{
|
||||
j.push_back(i);
|
||||
}
|
||||
const auto packed = json::to_bjdata(j, true, true);
|
||||
|
||||
SaxCountdown scp(1000000); // large enough to never trigger an abort
|
||||
CHECK(json::sax_parse(packed, &scp, json::input_format_t::bjdata));
|
||||
}
|
||||
}
|
||||
|
||||
TEST_CASE("Universal Binary JSON Specification Examples 1")
|
||||
{
|
||||
SECTION("Null Value")
|
||||
|
||||
@@ -2174,6 +2174,92 @@ TEST_CASE("CBOR indefinite-length strings do not recurse per chunk")
|
||||
}
|
||||
}
|
||||
|
||||
TEST_CASE("issue #5405 - array reserve for definite-length CBOR arrays")
|
||||
{
|
||||
#if !defined(JSON_NOEXCEPTION)
|
||||
// this SECTION relies on catching a thrown exception to distinguish
|
||||
// which of two acceptable, bounded rejections a hostile header took;
|
||||
// under JSON_NOEXCEPTION, JSON_THROW never produces a catchable C++
|
||||
// exception (it aborts instead), so this cannot be tested that way here
|
||||
SECTION("a huge claimed length with no element data must not over-allocate")
|
||||
{
|
||||
// 0x9A: array with a four-byte length; claims 0xFFFFFFFF (4294967295)
|
||||
// elements but provides none. max_size() for a std::vector is far
|
||||
// larger than this count, so it does not reject the header outright;
|
||||
// the (capped) reservation must not attempt to allocate space for
|
||||
// billions of elements before the missing data is detected.
|
||||
json _;
|
||||
const std::vector<uint8_t> input = {0x9A, 0xFF, 0xFF, 0xFF, 0xFF};
|
||||
// On a platform where std::size_t is narrower than 64 bits (e.g.
|
||||
// 32-bit), the claimed count 0xFFFFFFFF coincides with that
|
||||
// platform's detail::unknown_size() sentinel (SIZE_MAX), so the
|
||||
// format-level size check rejects it outright (out_of_range.408,
|
||||
// "excessive ... size") before the SAX consumer's own max_size()
|
||||
// check would even run; on a 64-bit platform it passes both of
|
||||
// those checks and is only found short of data once the (capped)
|
||||
// reservation looks for element bytes that were never provided
|
||||
// (parse_error.110). Either is an acceptable, bounded rejection of
|
||||
// the hostile header -- the property under test is that no path
|
||||
// attempts to allocate space for billions of elements.
|
||||
bool threw = false;
|
||||
try
|
||||
{
|
||||
_ = json::from_cbor(input);
|
||||
}
|
||||
catch (const json::parse_error& e)
|
||||
{
|
||||
threw = true;
|
||||
CHECK(e.id == 110);
|
||||
CHECK(std::string(e.what()) == "[json.exception.parse_error.110] parse error at byte 6: syntax error while parsing CBOR value: unexpected end of input");
|
||||
}
|
||||
catch (const json::out_of_range& e)
|
||||
{
|
||||
threw = true;
|
||||
CHECK(e.id == 408);
|
||||
CHECK(std::string(e.what()).find("excessive") != std::string::npos);
|
||||
}
|
||||
CHECK(threw);
|
||||
CHECK(json::from_cbor(input, true, false).is_discarded());
|
||||
}
|
||||
#endif
|
||||
|
||||
SECTION("arrays of various sizes decode to the same value as before the reserve optimization")
|
||||
{
|
||||
for (const auto size :
|
||||
{
|
||||
std::size_t{0}, std::size_t{1}, std::size_t{5}, // small
|
||||
std::size_t{16384}, // exactly at the reserve cap
|
||||
std::size_t{20000} // above the reserve cap
|
||||
})
|
||||
{
|
||||
CAPTURE(size)
|
||||
json j = json::array();
|
||||
for (std::size_t i = 0; i < size; ++i)
|
||||
{
|
||||
j.push_back(static_cast<int>(i % 1000));
|
||||
}
|
||||
|
||||
const auto packed = json::to_cbor(j);
|
||||
CHECK(json::from_cbor(packed) == j);
|
||||
}
|
||||
}
|
||||
|
||||
SECTION("a user-defined SAX consumer is unaffected by the internal DOM reserve optimization")
|
||||
{
|
||||
// the reserve() call is local to json_sax_dom_parser / json_sax_dom_callback_parser;
|
||||
// a custom SAX consumer that does not touch a DOM array sees identical events
|
||||
json j = json::array();
|
||||
for (int i = 0; i < 100; ++i)
|
||||
{
|
||||
j.push_back(i);
|
||||
}
|
||||
const auto packed = json::to_cbor(j);
|
||||
|
||||
SaxCountdown scp(1000000); // large enough to never trigger an abort
|
||||
CHECK(json::sax_parse(packed, &scp, json::input_format_t::cbor));
|
||||
}
|
||||
}
|
||||
|
||||
TEST_CASE("CBOR roundtrips" * doctest::skip())
|
||||
{
|
||||
SECTION("input from flynn")
|
||||
|
||||
@@ -1597,6 +1597,91 @@ TEST_CASE("MessagePack")
|
||||
}
|
||||
}
|
||||
|
||||
TEST_CASE("issue #5405 - array reserve for definite-length MessagePack arrays")
|
||||
{
|
||||
#if !defined(JSON_NOEXCEPTION)
|
||||
// this SECTION relies on catching a thrown exception to distinguish
|
||||
// which of two acceptable, bounded rejections a hostile header took;
|
||||
// under JSON_NOEXCEPTION, JSON_THROW never produces a catchable C++
|
||||
// exception (it aborts instead), so this cannot be tested that way here
|
||||
SECTION("a huge claimed length with no element data must not over-allocate")
|
||||
{
|
||||
// 0xdd: array 32 (four-byte length); claims 0xFFFFFFFF (4294967295)
|
||||
// elements but provides none. max_size() for a std::vector is far
|
||||
// larger than this count, so it does not reject the header outright;
|
||||
// the (capped) reservation must not attempt to allocate space for
|
||||
// billions of elements before the missing data is detected.
|
||||
json _;
|
||||
const std::vector<uint8_t> input = {0xdd, 0xFF, 0xFF, 0xFF, 0xFF};
|
||||
// On a platform where std::size_t is narrower than 64 bits (e.g.
|
||||
// 32-bit), the claimed count 0xFFFFFFFF coincides with that
|
||||
// platform's SIZE_MAX, which some size-narrowing checks treat the
|
||||
// same as detail::unknown_size(); it may then be rejected before
|
||||
// the SAX consumer's own max_size() check (out_of_range.408) rather
|
||||
// than being accepted and only found short of data once the
|
||||
// (capped) reservation looks for element bytes that were never
|
||||
// provided (parse_error.110). Either is an acceptable, bounded
|
||||
// rejection of the hostile header -- the property under test is
|
||||
// that no path attempts to allocate space for billions of elements.
|
||||
bool threw = false;
|
||||
try
|
||||
{
|
||||
_ = json::from_msgpack(input);
|
||||
}
|
||||
catch (const json::parse_error& e)
|
||||
{
|
||||
threw = true;
|
||||
CHECK(e.id == 110);
|
||||
CHECK(std::string(e.what()) == "[json.exception.parse_error.110] parse error at byte 6: syntax error while parsing MessagePack value: unexpected end of input");
|
||||
}
|
||||
catch (const json::out_of_range& e)
|
||||
{
|
||||
threw = true;
|
||||
CHECK(e.id == 408);
|
||||
CHECK(std::string(e.what()).find("excessive") != std::string::npos);
|
||||
}
|
||||
CHECK(threw);
|
||||
CHECK(json::from_msgpack(input, true, false).is_discarded());
|
||||
}
|
||||
#endif
|
||||
|
||||
SECTION("arrays of various sizes decode to the same value as before the reserve optimization")
|
||||
{
|
||||
for (const auto size :
|
||||
{
|
||||
std::size_t{0}, std::size_t{1}, std::size_t{5}, // small
|
||||
std::size_t{16384}, // exactly at the reserve cap
|
||||
std::size_t{20000} // above the reserve cap
|
||||
})
|
||||
{
|
||||
CAPTURE(size)
|
||||
json j = json::array();
|
||||
for (std::size_t i = 0; i < size; ++i)
|
||||
{
|
||||
j.push_back(static_cast<int>(i % 1000));
|
||||
}
|
||||
|
||||
const auto packed = json::to_msgpack(j);
|
||||
CHECK(json::from_msgpack(packed) == j);
|
||||
}
|
||||
}
|
||||
|
||||
SECTION("a user-defined SAX consumer is unaffected by the internal DOM reserve optimization")
|
||||
{
|
||||
// the reserve() call is local to json_sax_dom_parser / json_sax_dom_callback_parser;
|
||||
// a custom SAX consumer that does not touch a DOM array sees identical events
|
||||
json j = json::array();
|
||||
for (int i = 0; i < 100; ++i)
|
||||
{
|
||||
j.push_back(i);
|
||||
}
|
||||
const auto packed = json::to_msgpack(j);
|
||||
|
||||
SaxCountdown scp(1000000); // large enough to never trigger an abort
|
||||
CHECK(json::sax_parse(packed, &scp, json::input_format_t::msgpack));
|
||||
}
|
||||
}
|
||||
|
||||
// use this testcase outside [hide] to run it with Valgrind
|
||||
TEST_CASE("MessagePack nesting does not consume the call stack")
|
||||
{
|
||||
|
||||
@@ -2315,6 +2315,112 @@ TEST_CASE("UBJSON optimized arrays of a valueless type are bounded")
|
||||
}
|
||||
}
|
||||
|
||||
TEST_CASE("issue #5405 - array reserve for definite-length UBJSON arrays")
|
||||
{
|
||||
#if !defined(JSON_NOEXCEPTION)
|
||||
// this SECTION relies on catching a thrown exception to distinguish
|
||||
// which of two acceptable, bounded rejections a hostile header took;
|
||||
// under JSON_NOEXCEPTION, JSON_THROW never produces a catchable C++
|
||||
// exception (it aborts instead), so this cannot be tested that way here
|
||||
SECTION("a huge claimed length with no element data must not over-allocate")
|
||||
{
|
||||
// optimized form [$type#count: type 'i' (int8), count as a four-byte
|
||||
// 'l' (int32) of 0x7FFFFFFF (2147483647), but no element data at all.
|
||||
// max_size() for a std::vector is far larger than this count, so it
|
||||
// does not reject the header outright; the (capped) reservation must
|
||||
// not attempt to allocate space for billions of elements before the
|
||||
// missing data is detected.
|
||||
json _;
|
||||
const std::vector<uint8_t> input = {'[', '$', 'i', '#', 'l', 0x7F, 0xFF, 0xFF, 0xFF};
|
||||
// On a platform where std::vector<json>::max_size() is smaller than
|
||||
// the claimed count (e.g. 32-bit, where max_size() is bounded by a
|
||||
// 32-bit SIZE_MAX divided by sizeof(json)), the SAX consumer's own
|
||||
// check rejects the header outright (out_of_range.408, with the
|
||||
// claimed count in the message) instead of accepting it and only
|
||||
// finding it short of data once the (capped) reservation looks for
|
||||
// element bytes that were never provided (parse_error.110). Either
|
||||
// is an acceptable, bounded rejection of the hostile header -- the
|
||||
// property under test is that no path attempts to allocate space
|
||||
// for billions of elements.
|
||||
bool threw = false;
|
||||
try
|
||||
{
|
||||
_ = json::from_ubjson(input);
|
||||
}
|
||||
catch (const json::parse_error& e)
|
||||
{
|
||||
threw = true;
|
||||
CHECK(e.id == 110);
|
||||
CHECK(std::string(e.what()) == "[json.exception.parse_error.110] parse error at byte 10: syntax error while parsing UBJSON number: unexpected end of input");
|
||||
}
|
||||
catch (const json::out_of_range& e)
|
||||
{
|
||||
threw = true;
|
||||
CHECK(e.id == 408);
|
||||
CHECK(std::string(e.what()).find("excessive array size") != std::string::npos);
|
||||
}
|
||||
CHECK(threw);
|
||||
|
||||
// json_sax_dom_parser::start_array()'s max_size() check (unlike the
|
||||
// scanner's own parse_error path) throws unconditionally via
|
||||
// JSON_THROW rather than going through sax->parse_error(), so it is
|
||||
// not gated by allow_exceptions=false on a platform where this
|
||||
// header hits that check (e.g. 32-bit, see above) -- allow either
|
||||
// a discarded result or the same out_of_range it throws with
|
||||
// exceptions enabled.
|
||||
try
|
||||
{
|
||||
CHECK(json::from_ubjson(input, true, false).is_discarded());
|
||||
}
|
||||
catch (const json::out_of_range& e)
|
||||
{
|
||||
CHECK(e.id == 408);
|
||||
}
|
||||
}
|
||||
#endif
|
||||
|
||||
SECTION("arrays of various sizes decode to the same value as before the reserve optimization")
|
||||
{
|
||||
for (const auto size :
|
||||
{
|
||||
std::size_t{0}, std::size_t{1}, std::size_t{5}, // small
|
||||
std::size_t{16384}, // exactly at the reserve cap
|
||||
std::size_t{20000} // above the reserve cap
|
||||
})
|
||||
{
|
||||
CAPTURE(size)
|
||||
json j = json::array();
|
||||
for (std::size_t i = 0; i < size; ++i)
|
||||
{
|
||||
j.push_back(static_cast<int>(i % 1000));
|
||||
}
|
||||
|
||||
// exercise both the plain and the optimized [$type#count encoding
|
||||
const auto packed_plain = json::to_ubjson(j);
|
||||
CHECK(json::from_ubjson(packed_plain) == j);
|
||||
|
||||
const auto packed_optimized = json::to_ubjson(j, true, true);
|
||||
CHECK(json::from_ubjson(packed_optimized) == j);
|
||||
}
|
||||
}
|
||||
|
||||
SECTION("a user-defined SAX consumer is unaffected by the internal DOM reserve optimization")
|
||||
{
|
||||
// the reserve() call is local to json_sax_dom_parser / json_sax_dom_callback_parser;
|
||||
// a custom SAX consumer that does not touch a DOM array sees identical events
|
||||
json j = json::array();
|
||||
for (int i = 0; i < 100; ++i)
|
||||
{
|
||||
j.push_back(i);
|
||||
}
|
||||
const auto packed = json::to_ubjson(j, true, true);
|
||||
|
||||
SaxCountdown scp(1000000); // large enough to never trigger an abort
|
||||
CHECK(json::sax_parse(packed, &scp, json::input_format_t::ubjson));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
TEST_CASE("Universal Binary JSON Specification Examples 1")
|
||||
{
|
||||
SECTION("Null Value")
|
||||
|
||||
Reference in New Issue
Block a user