mirror of
https://github.com/nlohmann/json.git
synced 2026-09-29 03:00:31 +00:00
Add BON8 support (#2998)
* Add BON8 support Add to_bon8/from_bon8 and input_format_t::bon8 for BON8, a binary format that uses the byte values that cannot begin a UTF-8 character as type markers, so strings need no length prefix. It is the most compact of the supported binary formats on the benchmark files. The reader is non-recursive like the other binary readers. A string ends at the first byte that cannot continue it, so the reader hands the one or two bytes it reads past a string back to the value that follows. The writer produces the canonical representation of the specification, except for NFC normalization; its output is identical to that of the reference implementation (HikoGUI) on all files of the test data. The round-trip tests need the .bon8 files of json_test_data 3.2.0. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review comments - Reuse detail::validate_one_utf8 to check strings in to_bon8; the error now names the first byte of the invalid sequence. - Document that to_bon8 leaves bytes in the output adapter on an exception, and that string_open is only an output of write_bon8_marker. - Explain why the pushback buffer of the BON8 reader cannot overflow. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Select the BON8 float prefix by type get_bon8_float_prefix only depends on the type of its argument, so make the type a template parameter instead of passing an unused value. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Rename a test variable that Flawfinder mistakes for read() Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the BON8 CI failures - compare the float in write_bon8_float with number_float_t constants, so GCC does not warn about a float-to-double conversion - mark check_bon8_utf8's context as used when exceptions are disabled - choose the compact float prefix in a helper rather than with nested conditional operators (clang-tidy) - use auto for the cast in the BON8 integer reader (clang-tidy) - write the int32 minimum test values as long long literals (MSVC C4146) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Amalgamate Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Read BON8 strings in bulk from contiguous input - copy the valid UTF-8 of a string in one step when the input is contiguous (twitter.json is read in 1.68 instead of 2.52 ms, jeopardy.json in 196 instead of 297 ms, close to CBOR and MessagePack) - share the new valid_utf8_prefix() with the writer's UTF-8 check, which now skips ASCII 8 bytes at a time - let the fuzzer check that contiguous and stream input give the same value or error, and test both paths in the unit tests - clarify that a second 0xFF after a string is an empty string Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Link the BON8 functions from the other binary format pages Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Name the bulk scan flag after the input, not BON8 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Read BSON keys in bulk from contiguous input BSON keys (and array indices) are C-style strings, which were read byte by byte. For contiguous input they are now read up to their \x00-byte in one step, using the same bulk_scan flag as BON8 strings: twitter.json is read in 1.46 instead of 2.01 ms, citm_catalog.json in 2.93 instead of 3.33 ms, jeopardy.json in 182 instead of 207 ms. canada.json, whose keys are almost all one-digit array indices, takes 2 % longer. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the BON8 CI failures of the bulk-read tests - skip the contiguous-versus-stream tests of BON8 strings and BSON keys when exceptions are disabled: they catch the parse errors of invalid input, and without exceptions the library aborts instead - use static_cast for the int64 test value (google-readability-casting) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Move the explicit basic_json instantiation into its own test file Linking test-regression3_cpp20 with clang and MinGW failed with "relocation truncated to fit: IMAGE_REL_AMD64_REL32 against `.rdata'", as test-regression2 did before #5511. The explicit instantiation of basic_json<> for #4825 compiles every member function, including the BON8 reader and writer, into that object, and it was already close to the limit (2,226,104 bytes on develop, 2,234,960 with BON8; clang -O1, C++20). Give the instantiation a file of its own: unit-regression3 is now 1,594,736 bytes and unit-explicit_instantiation 1,095,064. The new file mentions JSON_HAS_CPP_17 and JSON_HAS_CPP_20 so it keeps being built for the C++17 standard the regression was about. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Convert the bytes of the BON8 test strings explicitly The str() helper constructed a std::string from a byte range, which converts each unsigned char implicitly; -fsanitize=integer reports that for bytes of 0x80 and above (ci_test_clang_sanitizer). Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann
@@ -112,7 +112,7 @@ endif()
|
||||
|
||||
if (CMAKE_CXX_COMPILER_ID STREQUAL "MSVC")
|
||||
# avoid stack overflow, see https://github.com/nlohmann/json/issues/2955
|
||||
json_test_set_test_options("test-cbor;test-msgpack;test-ubjson;test-bjdata;test-binary_formats" LINK_OPTIONS /STACK:4000000)
|
||||
json_test_set_test_options("test-bon8;test-cbor;test-msgpack;test-ubjson;test-bjdata;test-binary_formats" LINK_OPTIONS /STACK:4000000)
|
||||
endif()
|
||||
|
||||
# disable exceptions for test-disabled_exceptions
|
||||
|
||||
+4
-1
@@ -10,7 +10,7 @@ CXXFLAGS += -std=c++11
|
||||
CPPFLAGS += -I ../single_include
|
||||
|
||||
FUZZER_ENGINE = src/fuzzer-driver_afl.cpp
|
||||
FUZZERS = parse_afl_fuzzer parse_bson_fuzzer parse_cbor_fuzzer parse_msgpack_fuzzer parse_ubjson_fuzzer parse_bjdata_fuzzer
|
||||
FUZZERS = parse_afl_fuzzer parse_bson_fuzzer parse_cbor_fuzzer parse_msgpack_fuzzer parse_ubjson_fuzzer parse_bjdata_fuzzer parse_bon8_fuzzer
|
||||
fuzzers: $(FUZZERS)
|
||||
|
||||
parse_afl_fuzzer:
|
||||
@@ -30,3 +30,6 @@ parse_ubjson_fuzzer:
|
||||
|
||||
parse_bjdata_fuzzer:
|
||||
$(CXX) $(CXXFLAGS) $(CPPFLAGS) $(FUZZER_ENGINE) src/fuzzer-parse_bjdata.cpp -o $@
|
||||
|
||||
parse_bon8_fuzzer:
|
||||
$(CXX) $(CXXFLAGS) $(CPPFLAGS) $(FUZZER_ENGINE) src/fuzzer-parse_bon8.cpp -o $@
|
||||
|
||||
@@ -274,7 +274,8 @@ enum class binary_format
|
||||
ubjson_optimized,
|
||||
bjdata,
|
||||
bjdata_optimized,
|
||||
bson
|
||||
bson,
|
||||
bon8
|
||||
};
|
||||
|
||||
static std::vector<std::uint8_t> to_binary(const json& j, const binary_format format)
|
||||
@@ -293,6 +294,8 @@ static std::vector<std::uint8_t> to_binary(const json& j, const binary_format fo
|
||||
return json::to_bjdata(j);
|
||||
case binary_format::bjdata_optimized:
|
||||
return json::to_bjdata(j, true, true);
|
||||
case binary_format::bon8:
|
||||
return json::to_bon8(j);
|
||||
case binary_format::bson:
|
||||
default:
|
||||
return json::to_bson(j);
|
||||
@@ -313,6 +316,8 @@ static json from_binary(const std::vector<std::uint8_t>& bytes, const binary_for
|
||||
case binary_format::bjdata:
|
||||
case binary_format::bjdata_optimized:
|
||||
return json::from_bjdata(bytes);
|
||||
case binary_format::bon8:
|
||||
return json::from_bon8(bytes);
|
||||
case binary_format::bson:
|
||||
default:
|
||||
return json::from_bson(bytes);
|
||||
@@ -333,6 +338,8 @@ static json from_binary(std::FILE* file, const binary_format format)
|
||||
case binary_format::bjdata:
|
||||
case binary_format::bjdata_optimized:
|
||||
return json::from_bjdata(file);
|
||||
case binary_format::bon8:
|
||||
return json::from_bon8(file);
|
||||
case binary_format::bson:
|
||||
default:
|
||||
return json::from_bson(file);
|
||||
@@ -407,6 +414,10 @@ BENCHMARK_CAPTURE(FromBinaryBuffer, bjdata / canada, TEST_DATA_DIRECTORY "/nativ
|
||||
BENCHMARK_CAPTURE(FromBinaryBuffer, bjdata / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::bjdata);
|
||||
BENCHMARK_CAPTURE(FromBinaryBuffer, bjdata_optimized / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::bjdata_optimized);
|
||||
BENCHMARK_CAPTURE(FromBinaryBuffer, bjdata_optimized / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::bjdata_optimized);
|
||||
BENCHMARK_CAPTURE(FromBinaryBuffer, bon8 / jeopardy, TEST_DATA_DIRECTORY "/jeopardy/jeopardy.json", binary_format::bon8);
|
||||
BENCHMARK_CAPTURE(FromBinaryBuffer, bon8 / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::bon8);
|
||||
BENCHMARK_CAPTURE(FromBinaryBuffer, bon8 / citm_catalog, TEST_DATA_DIRECTORY "/nativejson-benchmark/citm_catalog.json", binary_format::bon8);
|
||||
BENCHMARK_CAPTURE(FromBinaryBuffer, bon8 / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::bon8);
|
||||
// BSON requires an object at the top level, so the array-rooted test files
|
||||
// (jeopardy and the regression files) cannot be captured here
|
||||
BENCHMARK_CAPTURE(FromBinaryBuffer, bson / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::bson);
|
||||
@@ -450,6 +461,8 @@ BENCHMARK_CAPTURE(FromBinaryFile, cbor / twitter, TEST_DATA_DIRECTORY "/nativejs
|
||||
BENCHMARK_CAPTURE(FromBinaryFile, ubjson / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::ubjson);
|
||||
BENCHMARK_CAPTURE(FromBinaryFile, ubjson / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::ubjson);
|
||||
BENCHMARK_CAPTURE(FromBinaryFile, bjdata / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::bjdata);
|
||||
BENCHMARK_CAPTURE(FromBinaryFile, bon8 / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::bon8);
|
||||
BENCHMARK_CAPTURE(FromBinaryFile, bon8 / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::bon8);
|
||||
BENCHMARK_CAPTURE(FromBinaryFile, bson / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::bson);
|
||||
|
||||
//////////////////////////////////////////////////////////////////////////////
|
||||
@@ -530,18 +543,21 @@ BENCHMARK_CAPTURE(FromBinaryShape, nested / msgpack, make_nested, binary_format:
|
||||
BENCHMARK_CAPTURE(FromBinaryShape, nested / ubjson, make_nested, binary_format::ubjson);
|
||||
BENCHMARK_CAPTURE(FromBinaryShape, nested / bjdata, make_nested, binary_format::bjdata);
|
||||
BENCHMARK_CAPTURE(FromBinaryShape, nested / bson, make_nested, binary_format::bson);
|
||||
BENCHMARK_CAPTURE(FromBinaryShape, nested / bon8, make_nested, binary_format::bon8);
|
||||
BENCHMARK_CAPTURE(FromBinaryShape, containers / cbor, make_containers, binary_format::cbor);
|
||||
BENCHMARK_CAPTURE(FromBinaryShape, containers / msgpack, make_containers, binary_format::msgpack);
|
||||
BENCHMARK_CAPTURE(FromBinaryShape, containers / ubjson, make_containers, binary_format::ubjson);
|
||||
BENCHMARK_CAPTURE(FromBinaryShape, containers / ubjson_optimized, make_containers, binary_format::ubjson_optimized);
|
||||
BENCHMARK_CAPTURE(FromBinaryShape, containers / bjdata, make_containers, binary_format::bjdata);
|
||||
BENCHMARK_CAPTURE(FromBinaryShape, containers / bson, make_containers, binary_format::bson);
|
||||
BENCHMARK_CAPTURE(FromBinaryShape, containers / bon8, make_containers, binary_format::bon8);
|
||||
// BSON names every array element, so a large array measures key generation
|
||||
// rather than scalar decoding and is left out here
|
||||
BENCHMARK_CAPTURE(FromBinaryShape, scalars / cbor, make_scalars, binary_format::cbor);
|
||||
BENCHMARK_CAPTURE(FromBinaryShape, scalars / msgpack, make_scalars, binary_format::msgpack);
|
||||
BENCHMARK_CAPTURE(FromBinaryShape, scalars / ubjson, make_scalars, binary_format::ubjson);
|
||||
BENCHMARK_CAPTURE(FromBinaryShape, scalars / bjdata, make_scalars, binary_format::bjdata);
|
||||
BENCHMARK_CAPTURE(FromBinaryShape, scalars / bon8, make_scalars, binary_format::bon8);
|
||||
|
||||
/*!
|
||||
@brief parse an indefinite-length CBOR string
|
||||
|
||||
+3
-3
@@ -1,6 +1,6 @@
|
||||
# Fuzz testing
|
||||
|
||||
Each parser of the library (JSON, BJData, BSON, CBOR, MessagePack, and UBJSON) can be fuzz tested. Currently,
|
||||
Each parser of the library (JSON, BJData, BON8, BSON, CBOR, MessagePack, and UBJSON) can be fuzz tested. Currently,
|
||||
[libFuzzer](https://llvm.org/docs/LibFuzzer.html) and [afl++](https://github.com/AFLplusplus/AFLplusplus) are supported.
|
||||
|
||||
## Corpus creation
|
||||
@@ -10,11 +10,11 @@ directory with some simple input files that cover several features of the parser
|
||||
for mutations.
|
||||
|
||||
```shell
|
||||
TEST_DATA_VERSION=3.1.0
|
||||
TEST_DATA_VERSION=3.2.0
|
||||
wget https://github.com/nlohmann/json_test_data/archive/refs/tags/v$TEST_DATA_VERSION.zip
|
||||
unzip v$TEST_DATA_VERSION.zip
|
||||
rm v$TEST_DATA_VERSION.zip
|
||||
for FORMAT in json bjdata bson cbor msgpack ubjson
|
||||
for FORMAT in json bjdata bon8 bson cbor msgpack ubjson
|
||||
do
|
||||
rm -fr corpus_$FORMAT
|
||||
mkdir corpus_$FORMAT
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
// __ _____ _____ _____
|
||||
// __| | __| | | | JSON for Modern C++ (supporting code)
|
||||
// | | |__ | | | | | | version 3.12.0
|
||||
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
|
||||
//
|
||||
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
|
||||
// SPDX-License-Identifier: MIT
|
||||
|
||||
/*
|
||||
This file implements a parser test suitable for fuzz testing. Given a byte
|
||||
array data, it performs the following steps:
|
||||
|
||||
- j1 = from_bon8(data)
|
||||
- vec = to_bon8(j1)
|
||||
- j2 = from_bon8(vec)
|
||||
- assert(j1 == j2)
|
||||
|
||||
It also checks that reading the data from a stream, which reads strings byte by
|
||||
byte, gives the same value or error as reading it from contiguous memory, which
|
||||
copies strings in bulk.
|
||||
|
||||
The provided function `LLVMFuzzerTestOneInput` can be used in different fuzzer
|
||||
drivers.
|
||||
*/
|
||||
|
||||
#include <cassert>
|
||||
#include <iostream>
|
||||
#include <sstream>
|
||||
#include <nlohmann/json.hpp>
|
||||
|
||||
// the round-trip checks below are assertions; NDEBUG would compile them away
|
||||
#ifdef NDEBUG
|
||||
#error "the fuzzer drivers must be built without NDEBUG"
|
||||
#endif
|
||||
|
||||
using json = nlohmann::json;
|
||||
|
||||
namespace
|
||||
{
|
||||
// the serialization of the value read from @a input, or the error message
|
||||
template<typename InputType>
|
||||
std::string read_bon8(InputType&& input)
|
||||
{
|
||||
try
|
||||
{
|
||||
const auto vec = json::to_bon8(json::from_bon8(std::forward<InputType>(input)));
|
||||
return {vec.begin(), vec.end()};
|
||||
}
|
||||
catch (const json::exception& e)
|
||||
{
|
||||
return e.what();
|
||||
}
|
||||
}
|
||||
} // namespace
|
||||
|
||||
// see http://llvm.org/docs/LibFuzzer.html
|
||||
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
|
||||
{
|
||||
// contiguous and stream input must be read alike
|
||||
{
|
||||
std::istringstream stream(std::string(reinterpret_cast<const char*>(data), size));
|
||||
assert(read_bon8(std::vector<uint8_t>(data, data + size)) == read_bon8(stream));
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
// step 1: parse input
|
||||
std::vector<uint8_t> const vec1(data, data + size);
|
||||
json const j1 = json::from_bon8(vec1);
|
||||
|
||||
try
|
||||
{
|
||||
// step 2: round trip
|
||||
std::vector<uint8_t> const vec2 = json::to_bon8(j1);
|
||||
|
||||
// parse serialization
|
||||
json const j2 = json::from_bon8(vec2);
|
||||
|
||||
// serializations must match
|
||||
assert(json::to_bon8(j2) == vec2);
|
||||
}
|
||||
catch (const json::parse_error&)
|
||||
{
|
||||
// parsing a BON8 serialization must not fail
|
||||
assert(false);
|
||||
}
|
||||
}
|
||||
catch (const json::parse_error&)
|
||||
{
|
||||
// parse errors are ok, because input may be random bytes
|
||||
}
|
||||
catch (const json::type_error&)
|
||||
{
|
||||
// type errors can occur during parsing, too
|
||||
}
|
||||
catch (const json::out_of_range&)
|
||||
{
|
||||
// out of range errors may happen if provided sizes are excessive
|
||||
}
|
||||
|
||||
// return 0 - non-zero return values are reserved for future use
|
||||
return 0;
|
||||
}
|
||||
@@ -185,6 +185,7 @@ TEST_CASE("alternative string type")
|
||||
|
||||
CHECK(alt_json::from_cbor(alt_json::to_cbor(doc)) == doc);
|
||||
CHECK(alt_json::from_msgpack(alt_json::to_msgpack(doc)) == doc);
|
||||
CHECK(alt_json::from_bon8(alt_json::to_bon8(doc)) == doc);
|
||||
// BSON is not covered: it additionally needs string_t::find(value_type),
|
||||
// which alt_string does not provide
|
||||
CHECK(alt_json::from_ubjson(alt_json::to_ubjson(doc)) == doc);
|
||||
|
||||
@@ -25,6 +25,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
const auto bjdata_1_size = json::to_bjdata(j).size();
|
||||
const auto bjdata_2_size = json::to_bjdata(j, true).size();
|
||||
const auto bjdata_3_size = json::to_bjdata(j, true, true).size();
|
||||
const auto bon8_size = json::to_bon8(j).size();
|
||||
const auto bson_size = json::to_bson(j).size();
|
||||
const auto cbor_size = json::to_cbor(j).size();
|
||||
const auto msgpack_size = json::to_msgpack(j).size();
|
||||
@@ -36,6 +37,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
CHECK(bjdata_1_size == 1112030);
|
||||
CHECK(bjdata_2_size == 1224148);
|
||||
CHECK(bjdata_3_size == 1224148);
|
||||
CHECK(bon8_size == 1055792);
|
||||
CHECK(bson_size == 1794522);
|
||||
CHECK(cbor_size == 1055552);
|
||||
CHECK(msgpack_size == 1056145);
|
||||
@@ -47,6 +49,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
CHECK((100.0 * double(bjdata_1_size) / double(json_size)) == Approx(53.199));
|
||||
CHECK((100.0 * double(bjdata_2_size) / double(json_size)) == Approx(58.563));
|
||||
CHECK((100.0 * double(bjdata_3_size) / double(json_size)) == Approx(58.563));
|
||||
CHECK((100.0 * double(bon8_size) / double(json_size)) == Approx(50.509));
|
||||
CHECK((100.0 * double(bson_size) / double(json_size)) == Approx(85.849));
|
||||
CHECK((100.0 * double(cbor_size) / double(json_size)) == Approx(50.497));
|
||||
CHECK((100.0 * double(msgpack_size) / double(json_size)) == Approx(50.526));
|
||||
@@ -64,6 +67,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
const auto bjdata_1_size = json::to_bjdata(j).size();
|
||||
const auto bjdata_2_size = json::to_bjdata(j, true).size();
|
||||
const auto bjdata_3_size = json::to_bjdata(j, true, true).size();
|
||||
const auto bon8_size = json::to_bon8(j).size();
|
||||
const auto bson_size = json::to_bson(j).size();
|
||||
const auto cbor_size = json::to_cbor(j).size();
|
||||
const auto msgpack_size = json::to_msgpack(j).size();
|
||||
@@ -75,6 +79,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
CHECK(bjdata_1_size == 425342);
|
||||
CHECK(bjdata_2_size == 429970);
|
||||
CHECK(bjdata_3_size == 429970);
|
||||
CHECK(bon8_size == 391396);
|
||||
CHECK(bson_size == 444568);
|
||||
CHECK(cbor_size == 402814);
|
||||
CHECK(msgpack_size == 401510);
|
||||
@@ -86,6 +91,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
CHECK((100.0 * double(bjdata_1_size) / double(json_size)) == Approx(91.097));
|
||||
CHECK((100.0 * double(bjdata_2_size) / double(json_size)) == Approx(92.089));
|
||||
CHECK((100.0 * double(bjdata_3_size) / double(json_size)) == Approx(92.089));
|
||||
CHECK((100.0 * double(bon8_size) / double(json_size)) == Approx(83.828));
|
||||
CHECK((100.0 * double(bson_size) / double(json_size)) == Approx(95.215));
|
||||
CHECK((100.0 * double(cbor_size) / double(json_size)) == Approx(86.273));
|
||||
CHECK((100.0 * double(msgpack_size) / double(json_size)) == Approx(85.993));
|
||||
@@ -103,6 +109,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
const auto bjdata_1_size = json::to_bjdata(j).size();
|
||||
const auto bjdata_2_size = json::to_bjdata(j, true).size();
|
||||
const auto bjdata_3_size = json::to_bjdata(j, true, true).size();
|
||||
const auto bon8_size = json::to_bon8(j).size();
|
||||
const auto bson_size = json::to_bson(j).size();
|
||||
const auto cbor_size = json::to_cbor(j).size();
|
||||
const auto msgpack_size = json::to_msgpack(j).size();
|
||||
@@ -114,6 +121,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
CHECK(bjdata_1_size == 390781);
|
||||
CHECK(bjdata_2_size == 433557);
|
||||
CHECK(bjdata_3_size == 432964);
|
||||
CHECK(bon8_size == 317879);
|
||||
CHECK(bson_size == 479430);
|
||||
CHECK(cbor_size == 342373);
|
||||
CHECK(msgpack_size == 342473);
|
||||
@@ -125,6 +133,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
CHECK((100.0 * double(bjdata_1_size) / double(json_size)) == Approx(78.109));
|
||||
CHECK((100.0 * double(bjdata_2_size) / double(json_size)) == Approx(86.659));
|
||||
CHECK((100.0 * double(bjdata_3_size) / double(json_size)) == Approx(86.541));
|
||||
CHECK((100.0 * double(bon8_size) / double(json_size)) == Approx(63.538));
|
||||
CHECK((100.0 * double(bson_size) / double(json_size)) == Approx(95.828));
|
||||
CHECK((100.0 * double(cbor_size) / double(json_size)) == Approx(68.433));
|
||||
CHECK((100.0 * double(msgpack_size) / double(json_size)) == Approx(68.453));
|
||||
@@ -142,6 +151,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
const auto bjdata_1_size = json::to_bjdata(j).size();
|
||||
const auto bjdata_2_size = json::to_bjdata(j, true).size();
|
||||
const auto bjdata_3_size = json::to_bjdata(j, true, true).size();
|
||||
const auto bon8_size = json::to_bon8(j).size();
|
||||
const auto bson_size = json::to_bson({{"", j}}).size(); // wrap array in object for BSON
|
||||
const auto cbor_size = json::to_cbor(j).size();
|
||||
const auto msgpack_size = json::to_msgpack(j).size();
|
||||
@@ -153,6 +163,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
CHECK(bjdata_1_size == 50710965);
|
||||
CHECK(bjdata_2_size == 51144830);
|
||||
CHECK(bjdata_3_size == 51144830);
|
||||
CHECK(bon8_size == 45942080);
|
||||
CHECK(bson_size == 56008520);
|
||||
CHECK(cbor_size == 46187320);
|
||||
CHECK(msgpack_size == 46158575);
|
||||
@@ -164,6 +175,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
CHECK((100.0 * double(bjdata_1_size) / double(json_size)) == Approx(96.576));
|
||||
CHECK((100.0 * double(bjdata_2_size) / double(json_size)) == Approx(97.402));
|
||||
CHECK((100.0 * double(bjdata_3_size) / double(json_size)) == Approx(97.402));
|
||||
CHECK((100.0 * double(bon8_size) / double(json_size)) == Approx(87.494));
|
||||
CHECK((100.0 * double(bson_size) / double(json_size)) == Approx(106.665));
|
||||
CHECK((100.0 * double(cbor_size) / double(json_size)) == Approx(87.961));
|
||||
CHECK((100.0 * double(msgpack_size) / double(json_size)) == Approx(87.906));
|
||||
@@ -181,6 +193,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
const auto bjdata_1_size = json::to_bjdata(j).size();
|
||||
const auto bjdata_2_size = json::to_bjdata(j, true).size();
|
||||
const auto bjdata_3_size = json::to_bjdata(j, true, true).size();
|
||||
const auto bon8_size = json::to_bon8(j).size();
|
||||
// BSON cannot process the file as it contains code point U+0000
|
||||
const auto cbor_size = json::to_cbor(j).size();
|
||||
const auto msgpack_size = json::to_msgpack(j).size();
|
||||
@@ -192,6 +205,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
CHECK(bjdata_1_size == 148695);
|
||||
CHECK(bjdata_2_size == 150569);
|
||||
CHECK(bjdata_3_size == 150569);
|
||||
CHECK(bon8_size == 144477);
|
||||
CHECK(cbor_size == 147095);
|
||||
CHECK(msgpack_size == 147017);
|
||||
CHECK(ubjson_1_size == 148695);
|
||||
@@ -202,6 +216,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
CHECK((100.0 * double(bjdata_1_size) / double(json_size)) == Approx(88.153));
|
||||
CHECK((100.0 * double(bjdata_2_size) / double(json_size)) == Approx(89.264));
|
||||
CHECK((100.0 * double(bjdata_3_size) / double(json_size)) == Approx(89.264));
|
||||
CHECK((100.0 * double(bon8_size) / double(json_size)) == Approx(85.653));
|
||||
CHECK((100.0 * double(cbor_size) / double(json_size)) == Approx(87.205));
|
||||
CHECK((100.0 * double(msgpack_size) / double(json_size)) == Approx(87.158));
|
||||
CHECK((100.0 * double(ubjson_1_size) / double(json_size)) == Approx(88.153));
|
||||
|
||||
@@ -12,6 +12,7 @@
|
||||
using nlohmann::json;
|
||||
|
||||
#include <cstdint>
|
||||
#include <limits>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
@@ -49,6 +50,12 @@ std::vector<json> test_values()
|
||||
};
|
||||
}
|
||||
|
||||
// BON8 has no integers above the int64 range, so to_bon8() rejects them
|
||||
bool bon8_representable(const json& j)
|
||||
{
|
||||
return !j.is_number_unsigned() || j.get<std::uint64_t>() <= static_cast<std::uint64_t>((std::numeric_limits<std::int64_t>::max)());
|
||||
}
|
||||
|
||||
// values to_bson() accepts: the document must be an object
|
||||
std::vector<json> bson_values()
|
||||
{
|
||||
@@ -92,6 +99,13 @@ TEST_CASE("binary writer output sinks")
|
||||
json::to_msgpack(j, msgpack);
|
||||
CHECK(json::to_msgpack(j) == msgpack);
|
||||
|
||||
if (bon8_representable(j))
|
||||
{
|
||||
std::vector<std::uint8_t> bon8;
|
||||
json::to_bon8(j, bon8);
|
||||
CHECK(json::to_bon8(j) == bon8);
|
||||
}
|
||||
|
||||
for (const bool use_size :
|
||||
{
|
||||
false, true
|
||||
@@ -172,6 +186,10 @@ TEST_CASE("binary_reserve_hint never over-reserves")
|
||||
CHECK(hint <= json::to_ubjson(j).size());
|
||||
CHECK(hint <= json::to_ubjson(j, true, true).size());
|
||||
CHECK(hint <= json::to_bjdata(j).size());
|
||||
if (bon8_representable(j))
|
||||
{
|
||||
CHECK(hint <= json::to_bon8(j).size());
|
||||
}
|
||||
}
|
||||
|
||||
for (const auto& j : bson_values())
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1066,6 +1066,45 @@ TEST_CASE("Incomplete BSON Input")
|
||||
}
|
||||
}
|
||||
|
||||
// the test catches the exceptions of invalid input
|
||||
#if !defined(JSON_NOEXCEPTION)
|
||||
TEST_CASE("BSON keys from contiguous and stream input")
|
||||
{
|
||||
// contiguous input reads a key up to its \x00-byte in one step, a stream
|
||||
// reads it byte by byte; both must give the same value or error for the
|
||||
// complete document and for every truncation of it
|
||||
const json j = {{"", true}, {"k", {1, 2, 3}}, {std::string(40, 'x'), {{"nested key", "value"}}}};
|
||||
const std::vector<std::uint8_t> bson = json::to_bson(j);
|
||||
CHECK(json::from_bson(bson) == j);
|
||||
|
||||
for (std::size_t length = 0; length <= bson.size(); ++length)
|
||||
{
|
||||
CAPTURE(length)
|
||||
const std::vector<std::uint8_t> input(bson.begin(), bson.begin() + static_cast<std::ptrdiff_t>(length));
|
||||
std::string from_vector;
|
||||
std::string from_stream;
|
||||
try
|
||||
{
|
||||
from_vector = json::from_bson(input).dump();
|
||||
}
|
||||
catch (const json::parse_error& e)
|
||||
{
|
||||
from_vector = e.what();
|
||||
}
|
||||
try
|
||||
{
|
||||
std::istringstream stream(std::string(input.begin(), input.end()));
|
||||
from_stream = json::from_bson(stream).dump();
|
||||
}
|
||||
catch (const json::parse_error& e)
|
||||
{
|
||||
from_stream = e.what();
|
||||
}
|
||||
CHECK(from_vector == from_stream);
|
||||
}
|
||||
}
|
||||
#endif
|
||||
|
||||
TEST_CASE("Negative size of binary value")
|
||||
{
|
||||
// invalid BSON: the size of the binary value is -1
|
||||
|
||||
@@ -2761,7 +2761,7 @@ TEST_CASE("diagnostic positions: value lifetime, input adapters, and SAX")
|
||||
|
||||
SECTION("binary formats have no text positions")
|
||||
{
|
||||
// binary formats (CBOR, MessagePack, UBJSON, BSON, BJData) are
|
||||
// binary formats (BJData, BON8, BSON, CBOR, MessagePack, UBJSON) are
|
||||
// parsed via detail::binary_reader, which never sets
|
||||
// start_position/end_position on the values it produces (they
|
||||
// have no notion of a text offset), so every value's position
|
||||
@@ -2778,6 +2778,10 @@ TEST_CASE("diagnostic positions: value lifetime, input adapters, and SAX")
|
||||
CHECK(from_msgpack.start_pos() == std::string::npos);
|
||||
CHECK(from_msgpack.end_pos() == std::string::npos);
|
||||
|
||||
const json from_bon8 = json::from_bon8(json::to_bon8(src));
|
||||
CHECK(from_bon8.start_pos() == std::string::npos);
|
||||
CHECK(from_bon8.end_pos() == std::string::npos);
|
||||
|
||||
const json from_ubjson = json::from_ubjson(json::to_ubjson(src));
|
||||
CHECK(from_ubjson.start_pos() == std::string::npos);
|
||||
CHECK(from_ubjson.end_pos() == std::string::npos);
|
||||
|
||||
@@ -84,6 +84,8 @@ TEST_CASE("binary type whose value type is not std::uint8_t")
|
||||
|
||||
// UBJSON has no binary type, so binary values are written as an array
|
||||
CHECK(byte_binary_json::from_ubjson(byte_binary_json::to_ubjson(j)) == byte_binary_json({0, 1, 255}));
|
||||
// the same holds for BON8
|
||||
CHECK(byte_binary_json::from_bon8(byte_binary_json::to_bon8(j)) == byte_binary_json({0, 1, 255}));
|
||||
}
|
||||
#endif
|
||||
}
|
||||
|
||||
@@ -297,6 +297,7 @@ TEST_CASE("object type without key_compare")
|
||||
const auto j = no_key_compare_json::parse(R"({"a":[1,2,3],"b":"x"})");
|
||||
CHECK(no_key_compare_json::from_cbor(no_key_compare_json::to_cbor(j)) == j);
|
||||
CHECK(no_key_compare_json::from_msgpack(no_key_compare_json::to_msgpack(j)) == j);
|
||||
CHECK(no_key_compare_json::from_bon8(no_key_compare_json::to_bon8(j)) == j);
|
||||
}
|
||||
|
||||
SECTION("flatten and unflatten")
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
// __ _____ _____ _____
|
||||
// __| | __| | | | JSON for Modern C++ (supporting code)
|
||||
// | | |__ | | | | | | version 3.12.0
|
||||
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
|
||||
//
|
||||
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
|
||||
// SPDX-License-Identifier: MIT
|
||||
|
||||
// cmake/test.cmake selects the C++ standard versions with which to build a
|
||||
// unit test based on the presence of JSON_HAS_CPP_<VERSION> macros.
|
||||
// The regression below only showed on C++17, so build this file for every
|
||||
// standard like the other regression tests:
|
||||
// JSON_HAS_CPP_17 JSON_HAS_CPP_20 (do not remove; see note at top of file)
|
||||
|
||||
#include "doctest_compatibility.h"
|
||||
|
||||
#include <nlohmann/json.hpp>
|
||||
using json = nlohmann::json;
|
||||
|
||||
/////////////////////////////////////////////////////////////////////
|
||||
// for #4825 - explicitly instantiating basic_json must compile; this
|
||||
// forces instantiation of binary_writer::write_bjdata_ndarray, whose
|
||||
// static_cast<string_t> was ambiguous under explicit instantiation on
|
||||
// C++17. Merely compiling this translation unit is the regression test.
|
||||
//
|
||||
// The instantiation compiles every member function, so it has a file of its
|
||||
// own: in unit-regression3.cpp it made the object too large for the MinGW
|
||||
// linker to relocate (see #5511).
|
||||
/////////////////////////////////////////////////////////////////////
|
||||
template class nlohmann::basic_json<>;
|
||||
|
||||
TEST_CASE("explicit instantiation of basic_json (#4825)")
|
||||
{
|
||||
const json j = {1, "two", 3.0};
|
||||
CHECK(j.size() == 3);
|
||||
CHECK(json::from_bjdata(json::to_bjdata(j)) == j);
|
||||
}
|
||||
@@ -326,6 +326,15 @@ TEST_CASE("ordered_json across binary formats")
|
||||
CHECK(collect_keys(restored) == original_keys);
|
||||
CHECK(collect_keys(restored["mango"]) == original_mango_keys);
|
||||
}
|
||||
|
||||
SECTION("BON8")
|
||||
{
|
||||
const auto bytes = ordered_json::to_bon8(original);
|
||||
const auto restored = ordered_json::from_bon8(bytes);
|
||||
CHECK(restored == original);
|
||||
CHECK(collect_keys(restored) == original_keys);
|
||||
CHECK(collect_keys(restored["mango"]) == original_mango_keys);
|
||||
}
|
||||
}
|
||||
|
||||
TEST_CASE("alt_json (custom string_t) across binary formats")
|
||||
@@ -353,6 +362,13 @@ TEST_CASE("alt_json (custom string_t) across binary formats")
|
||||
CHECK(restored == original);
|
||||
}
|
||||
|
||||
SECTION("BON8")
|
||||
{
|
||||
const auto bytes = alt_json::to_bon8(original);
|
||||
const auto restored = alt_json::from_bon8(bytes);
|
||||
CHECK(restored == original);
|
||||
}
|
||||
|
||||
SECTION("BSON")
|
||||
{
|
||||
const auto bytes = alt_json::to_bson(original);
|
||||
|
||||
@@ -332,6 +332,7 @@ TEST_CASE("regression tests 2")
|
||||
CHECK(float_json::from_cbor(float_json::to_cbor(j)) == j);
|
||||
CHECK(float_json::from_msgpack(float_json::to_msgpack(j)) == j);
|
||||
CHECK(float_json::from_ubjson(float_json::to_ubjson(j)) == j);
|
||||
CHECK(float_json::from_bon8(float_json::to_bon8(j)) == j);
|
||||
|
||||
float_json j2 = {1000.0, 2000.0, 3000.0};
|
||||
CHECK(float_json::from_ubjson(float_json::to_ubjson(j2, true, true)) == j2);
|
||||
|
||||
@@ -74,13 +74,7 @@ using ordered_json = nlohmann::ordered_json;
|
||||
#endif
|
||||
#endif
|
||||
|
||||
/////////////////////////////////////////////////////////////////////
|
||||
// for #4825 - explicitly instantiating basic_json must compile; this
|
||||
// forces instantiation of binary_writer::write_bjdata_ndarray, whose
|
||||
// static_cast<string_t> was ambiguous under explicit instantiation on
|
||||
// C++17. Merely compiling this translation unit is the regression test.
|
||||
/////////////////////////////////////////////////////////////////////
|
||||
template class nlohmann::basic_json<>;
|
||||
// the explicit instantiation for #4825 is in unit-explicit_instantiation.cpp
|
||||
|
||||
/////////////////////////////////////////////////////////////////////
|
||||
// for #4440
|
||||
@@ -894,6 +888,7 @@ TEST_CASE("regression test #5476 - array type without reserve()")
|
||||
// the binary formats pass a definite length to start_array()
|
||||
CHECK(deque_json::from_cbor(deque_json::to_cbor(j)) == j);
|
||||
CHECK(deque_json::from_msgpack(deque_json::to_msgpack(j)) == j);
|
||||
CHECK(deque_json::from_bon8(deque_json::to_bon8(j)) == j);
|
||||
|
||||
// parse() instantiates the callback parser as well, which reserves too
|
||||
const auto with_callback = deque_json::parse(R"([1,2,3])", [](int /*depth*/, deque_json::parse_event_t /*event*/, deque_json& /*parsed*/) noexcept
|
||||
|
||||
Reference in New Issue
Block a user