amorfo77/json
1
0 Fork 0
mirror of https://github.com/nlohmann/json.git synced 2026-11-13 15:35:23 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget (#6083)

Browse Source 
* Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget

The whole-input budget for the elements of optimized UBJSON arrays of
type Z, T and F (1048576 plus 4 per byte read) rejects files written by
3.12.0 that have long arrays of these types, such as 1100 rows of 1000
nulls with a `$Z` annotation, and there was no way to read them. The
per-byte rate is now the macro JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE.

Semantics:
- default 4, so the default build is unchanged (the generated code of
  from_ubjson is identical at -O2 up to source line numbers);
- the value must be a positive integer: 0, negative, non-integral, bool,
  character and wider-than-uintmax_t values fail to compile with a
  static_assert that names the macro;
- a value that does not fit into std::size_t is the largest size_t, so
  one spelling works for 32- and 64-bit targets;
- the limit base + rate * bytes_read saturates at the largest size_t
  instead of wrapping around, for any rate and input size; a rate of
  131072 (2^20 / 8) or more cannot be exceeded by any input (the
  shortest header of an array of more than 32767 elements is 8 bytes),
  which switches the whole-input budget off;
- the limit of one array (1048576) and the base (1048576) stay fixed
  constants, so the writer fallback still follows the base only and a
  reader with any rate reads everything the writer produces;
- the macro is not part of the ABI tag and must have the same value in
  every translation unit; it is documented that way.

The memory the arrays can take in a DOM is about 20 bytes per rate and
byte of input (measured, macOS, containers of 9 * rate elements:
25 bytes per input byte at rate 1, 81 at 4, 312 at 16, 1226 at 64, 5493
at 256) plus about 33 MB for the base. A 3.12.0 row of N values has a
header of b bytes (5 up to 255, 6 up to 32767, 8 above), so a table is
read completely if N <= rate * b, and otherwise up to
(1048576 + h * rate, h = outer header: 6, 7 or 9 bytes) / (N - rate * b) rows; 1100 rows of 1000 nulls need
a rate of 8. The new page of the macro documents the formulas, the
measurements, and two examples that are run by the documentation checks.

The tests of the budget move to tests/src/unit-ubjson_valueless.cpp, which
derives every boundary from the macro and is built with the rates 1,
1000000 and 0xFFFFFFFFFFFFFFFF in addition to the default (the last two
check that nothing is refused and that the limit does not wrap around);
they fail with the unchanged header for rate 1, 1000000 and the maximum.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Refuse char8_t for JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE and clarify its documentation

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann authored and GitHub committed 2026-11-11 22:37:35 +01:00
1 parent 8001ed67b9
commit 73aea7ce54
22 files changed
+1062 -278
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tests/src/unit-ubjson.cpp
+12 -201
View File
@@ -19,12 +19,8 @@
#include <nlohmann/json.hpp>
using nlohmann::json;
#include <cstdio>
#include <iostream>
#include <sstream>
#include <utility>
#include <fstream>
#include <limits>
#include <list>
#include <set>
#include "make_test_data_available.hpp"
@@ -33,6 +29,7 @@ using nlohmann::json;
#include "sax_countdown.hpp"
#include "allocation_counter.hpp"
#include "sax_event_loggers.hpp"
#include "ubjson_valueless.hpp"
using utils::SaxCountdown;
using utils::SaxEventLogger;
@@ -2441,208 +2438,22 @@ TEST_CASE("UBJSON optimized arrays of a valueless type are bounded")
}
}
namespace
TEST_CASE("UBJSON the writer keeps the valueless elements of an input within the base")
{
/// SAX consumer that counts the values and keeps the id of a parse error
struct ValuelessEventCounter : public nlohmann::json_sax<json>
{
bool null() override
{
return ++values < stop_after;
}
bool boolean(bool /*val*/) override
{
return ++values < stop_after;
}
bool number_integer(json::number_integer_t /*val*/) override
{
++values;
return true;
}
bool number_unsigned(json::number_unsigned_t /*val*/) override
{
++values;
return true;
}
bool number_float(json::number_float_t /*val*/, const std::string& /*s*/) override
{
++values;
return true;
}
bool string(std::string& /*val*/) override
{
++values;
return true;
}
bool binary(json::binary_t& /*val*/) override
{
++values;
return true;
}
bool start_object(std::size_t /*elements*/) override
{
return true;
}
bool key(std::string& /*val*/) override
{
return true;
}
bool end_object() override
{
return true;
}
bool start_array(std::size_t /*elements*/) override
{
return true;
}
bool end_array() override
{
return true;
}
bool parse_error(std::size_t /*position*/, const std::string& /*last_token*/, const json::exception& ex) override
{
error_id = ex.id;
return false;
}
/// the handler refuses the value that makes this many
std::size_t stop_after = (std::numeric_limits<std::size_t>::max)();
std::size_t values = 0;
int error_id = 0;
};
/// appends `[$<marker>#l<count>` (nine bytes) to @a v
void append_valueless_array(std::vector<std::uint8_t>& v, const char marker, const std::uint32_t count)
{
v.push_back('[');
v.push_back('$');
v.push_back(static_cast<std::uint8_t>(marker));
v.push_back('#');
v.push_back('l');
v.push_back(static_cast<std::uint8_t>(count >> 24));
v.push_back(static_cast<std::uint8_t>(count >> 16));
v.push_back(static_cast<std::uint8_t>(count >> 8));
v.push_back(static_cast<std::uint8_t>(count));
}
/// `[` followed by one `[$<marker>#l<count>` per entry of @a counts and `]`
std::vector<std::uint8_t> valueless_arrays(const char marker, const std::vector<std::uint32_t>& counts)
{
std::vector<std::uint8_t> v = {'['};
for (const auto count : counts)
{
append_valueless_array(v, marker, count);
}
v.push_back(']');
return v;
}
} // namespace
TEST_CASE("UBJSON the valueless elements of a whole input are bounded")
{
// The per-container limit alone let nine bytes (`[$Z#l` and a count of
// 1048576) announce a million values, and every further nine bytes another
// million. The elements of all containers of type 'Z', 'T' or 'F' in one
// input are now added up: they may not exceed 1048576 plus 4 for every
// byte read when the container's header has been read.
//
// The tests count events with a SAX handler where they can: building and
// comparing values of a million elements is what makes them slow under
// Valgrind, so there is one rejection and one round trip with a DOM.
// The budget of the reader is 1048576 plus a rate per byte read, see
// unit-ubjson_valueless.cpp, which tests it for several values of the rate.
// The writer does not use the rate and stays within the base
json _;
const char* const budget_message = "[json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive number of elements without payload";
// two arrays that exceed the limit: the second one, with its header read
// at byte 19, may hold 4 * 19 = 76 values on top of the base of 1048576
const std::uint32_t over_first = 524288;
const std::uint32_t over_second = 1048576 + 76 - over_first + 1;
SECTION("the input of the original report is rejected")
{
// 362 bytes (40 arrays of 1048576 values) used to allocate 661 MB; the
// second array is the one that is refused
const auto input = valueless_arrays('Z', {over_first, over_second});
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(input), budget_message, json::out_of_range&);
}
SECTION("every input adapter applies the budget")
{
const auto input = valueless_arrays('Z', {over_first, over_second});
// stream; not strict, which does not matter
std::stringstream stream;
stream.write(reinterpret_cast<const char*>(input.data()), static_cast<std::streamsize>(input.size()));
ValuelessEventCounter from_stream;
CHECK(!json::sax_parse(stream, &from_stream, json::input_format_t::ubjson, false));
CHECK(from_stream.error_id == 408);
// a container that is not contiguous; the counts of arrays of
// different types are added up
std::vector<std::uint8_t> mixed = {'['};
append_valueless_array(mixed, 'T', over_first);
append_valueless_array(mixed, 'F', over_second);
mixed.push_back(']');
const std::list<std::uint8_t> as_list(mixed.begin(), mixed.end());
ValuelessEventCounter from_list;
CHECK(!json::sax_parse(as_list.begin(), as_list.end(), &from_list, json::input_format_t::ubjson));
CHECK(from_list.error_id == 408);
CHECK(from_list.values == over_first);
// FILE*
FILE* const file = std::tmpfile(); // NOLINT(cppcoreguidelines-owning-memory)
REQUIRE(file != nullptr);
CHECK(std::fwrite(input.data(), 1, input.size(), file) == input.size());
std::rewind(file);
ValuelessEventCounter from_file;
CHECK(!json::sax_parse(file, &from_file, json::input_format_t::ubjson));
CHECK(from_file.error_id == 408);
std::fclose(file); // NOLINT(cert-err33-c,cppcoreguidelines-owning-memory)
}
SECTION("the limit is exact and the bytes read raise it")
{
// [ [$Z#l 524288] [$U#l 4096 + 4096 bytes] [$Z#l count] ]: chars_read
// is 1 + 9 + (9 + 4096) + 9 = 4124 when the last header has been read,
// so the last array may hold 4 * 4124 = 16496 values on top of the
// base of 1048576 less the 524288 of the first array
const std::uint32_t last_allowed = 1048576 + 16496 - 524288;
for (const std::uint32_t count :
{
last_allowed, last_allowed + 1
})
{
CAPTURE(count)
std::vector<std::uint8_t> input = {'['};
append_valueless_array(input, 'Z', 524288);
input.insert(input.end(), {'[', '$', 'U', '#', 'l', 0x00, 0x00, 0x10, 0x00});
input.insert(input.end(), 4096, 0x01);
append_valueless_array(input, 'Z', count);
input.push_back(']');
ValuelessEventCounter sax;
const bool accepted = count == last_allowed;
CHECK(json::sax_parse(input, &sax, json::input_format_t::ubjson) == accepted);
CHECK(sax.error_id == (accepted ? 0 : 408));
CHECK(sax.values == 524288u + 4096u + (accepted ? count : 0u));
}
// an array of 1048576 values on its own is accepted as before
const std::vector<std::uint8_t> single = {'[', '$', 'Z', '#', 'l', 0x00, 0x10, 0x00, 0x00};
// (the handler ends the parse at the first value, so that the million
// values are not read)
ValuelessEventCounter sax;
sax.stop_after = 1;
CHECK(!json::sax_parse(single, &sax, json::input_format_t::ubjson));
CHECK(sax.error_id == 0);
CHECK(sax.values == 1);
}
SECTION("to_ubjson writes the typed form up to the base only")
{
// 1048 rows of 1000 values fit under the base of 1048576, the 1049th
// would not: its elements are written one by one. Written all typed,
// 1100 rows of 6 bytes would be refused by a reader (1075 rows are the
// most it accepts). The rows hold null, true or false, and the
// fallback does not depend on which
// 1100 rows of 6 bytes would be refused by a reader with the default
// rate of the macro JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE (1074 rows
// are the most it accepts); the writer does not follow the macro, since
// the limit of a reader is never below the base. The rows hold null,
// true or false, and the fallback does not depend on which
std::vector<std::uint8_t> encoded;
{
json table = json::array();
@@ -2658,7 +2469,7 @@ TEST_CASE("UBJSON the valueless elements of a whole input are bounded")
CHECK(encoded.size() == 7 + (1048 * 6) + (52 * (4 + 1000)));
// the output is read back
ValuelessEventCounter sax;
utils::ValuelessEventCounter sax;
CHECK(json::sax_parse(encoded, &sax, json::input_format_t::ubjson));
CHECK(sax.error_id == 0);
CHECK(sax.values == 1100u * 1000u);
@@ -2685,7 +2496,7 @@ TEST_CASE("UBJSON the valueless elements of a whole input are bounded")
// an object member costs a key, so the count is bounded by the input
// and is not charged to the budget
const std::vector<std::uint8_t> input = {'{', '$', 'Z', '#', 'l', 0x00, 0x10, 0x00, 0x00};
ValuelessEventCounter sax;
utils::ValuelessEventCounter sax;
CHECK(!json::sax_parse(input, &sax, json::input_format_t::ubjson));
CHECK(sax.error_id == 110);
CHECK(sax.values == 0);