Files
json/tests/src/unit-msgpack.cpp
T
Niels Lohmann 253e4c3926 Bound what the DOM parsers reserve for arrays that announce a size (#6082)
* Bound the total number of valueless UBJSON elements per input

An optimized UBJSON array of type Z, T or F has no payload, so its
declared count is not bounded by the input. The limit of 1048576 applied
to each array on its own, so 9 bytes (`[$Z#l` and a count) produced
1048576 values and every further 9 bytes another 1048576: 362 bytes
allocated 661 MB in a DOM, and a SAX handler received 1M events per 9
bytes.

The reader now adds the counts of all such arrays of one input and
rejects the input with out_of_range.408 when the sum exceeds 1048576
plus 4 for every byte read so far. The base is the per-array limit, so
an array that was accepted on its own before is still accepted at the
start of an input. These arrays add about 80 bytes of memory per input
byte (measured 77; one `Z` per byte needs 32), plus the base; the memory
of a document as a whole is not bounded by this. The check runs once per
valueless array, before its start event, with plain size_t arithmetic
that cannot overflow for a 32-bit size_t. Containers with a payload type
and objects (every member needs a key) are bounded by the input already
and are unchanged. BJData rejects these types.

to_ubjson counts the elements of the typed arrays it writes in one call
and writes an array in the unoptimized form (`[#n Z Z ...`) once it would
bring the total above 1048576, which a reader accepts at any input size.
The check is made only for a typed array of a valueless type, so BJData
and other arrays do not pay for it. Everything to_ubjson writes in
version 3.13.0 can therefore be read back.

Two consequences:
- to_ubjson output changes for values with more than 1048576 elements in
  arrays of type Z, T or F: such an array is written at one byte per
  element, e.g. 270000 rows of 100 nulls take 26.8 MB instead of 1.35 MB.
- Output of version 3.12.0 and earlier (typed arrays without such a
  total) is rejected by from_ubjson when the arrays hold more than
  1048576 elements plus 4 per byte of input. 3.12.0 writes a row of up to 255 nulls in 5 bytes (6 above
  255), so a table is rejected from 1075 rows of 1000 nulls or 209723
  rows of 25 (accepted: 1000 rows of 1000, 209000 rows of 25); rows of
  20 or fewer are never rejected.

Measured on the test data: no file of json_test_data (converted with
to_ubjson in all three modes, 846 round trips) holds an array of this
kind with more than 0.17 values per byte. All of them are read back.

Reproducer (K arrays of 1048576 nulls), Apple clang -O2:
  K=40 (362 bytes): before 41943040 values, 664 MB; after rejected, 33 MB
  K=2 (20 bytes):   before 2097152 values, 53 MB;   after rejected, 33 MB

The new tests count events with a SAX handler where they can: building
and comparing million-element values made the first version take 176 s
under Valgrind, the current one about 22 s.

Behaviour change: an input that holds more than 1048576 elements in
arrays of type Z, T or F at more than 4 per byte is now rejected.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Report an empty array for UBJSON `[$N#<n>` instead of n elements

A no-op (N) is not a value, so an optimized array of type N holds no
element whatever count it declares. The reader skipped the elements but
still passed the declared count to the SAX parser: a handler was told
start_array(n) and no element followed, and the DOM parser reserved
min(n, 16384) values (256 KB) for an array that stays empty. Nine bytes
(`[$N#l 0x7FFFFFFF`) therefore committed 256 KB: 2000 of them in one
input, 18 KB, committed 512 MB of heap (capacity 163,840,000 values for
the 10000 arrays of the original probe: 2.6 GB for 70 KB of input).
Untouched pages keep the resident size small, but the commit charge and
a 32-bit address space are consumed.

The reader now announces 0 elements for an optimized array of type N.
The loop that skips the elements is unchanged (it never ran). A count
that does not fit into std::size_t is still rejected by the count
reader; a count that is too large for the array type of the DOM (before:
out_of_range.408 "excessive array size" from the DOM handler) is
accepted now, because the array is empty.

Behaviour change: sax_parse reports start_array(0) for `[$N#n` (before:
n); from_ubjson no longer throws out_of_range.408 for an N-typed array
whose count exceeds the maximum size of the array type.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Bound what the DOM parsers reserve for arrays that announce a size

Cause: #5476 made the DOM parsers reserve storage (up to 16384 elements)
for a counted array of CBOR, MessagePack, UBJSON and BJData when it
opens, before any input backs the count. The arrays that are open at the
same time keep their storage until the parse ends, so N nested headers
that announce 16384 elements and are never completed hold
N * 16384 * sizeof(json): 100,000 nested CBOR headers `99 40 00`
(300,000 bytes) peaked at 1.65 GiB, 3.12.0 at 12 MiB; the same for
MessagePack `DC 40 00`, UBJSON `[#I 16384` and BJData.

Fix: json_sax_dom_error_state, the base of the three DOM parsers, keeps a
credit of elements that may still be reserved: 65,536 at the start plus
2 for every byte the reader has read. An array that announces no more
than the credit left is reserved as before (one comparison and one
subtraction per counted array); a larger one is reserved what is left
and grows as its elements arrive. binary_reader tells a DOM parser (a
handler whose pointer converts to json_sax_dom_error_state*; a handler
that derives from one privately or protectedly, or any other SAX handler,
is not told anything and reserves as in the base of this change) where to
find its count of characters read for the duration of sax_parse(). An
RAII guard puts back the previous binding on every exit, also by
exception, so the parser never keeps a pointer to a reader that is gone
(unbound, only the per-array cap of 16384 applies); the public SAX
interface is unchanged, and start_array() still reports the announced number. A
bound per array by the remaining input would not help, since the same
bytes would back every open level; the bound has to be over all arrays.

The elements of an array take at least one byte of input each, so the
credit is more than honest input needs: none of the 19 inputs measured
(canada, citm_catalog, twitter, mesh, update-center, random, numbers,
animation, ik_rig, instruments, geo_response, jeopardy and 7 synthetic
ones, each as CBOR, MessagePack, UBJSON and BJData, with and without
size/type optimization) was given less than it announced. A value that
nests five or more arrays of 16384 elements, each with an array as its
first element, is the exception: the later ones grow as before.

Measurements (clang -O2, macOS arm64, peak memory footprint of
`/usr/bin/time -l`, reproducer: n nested headers):

              n = 20,000                n = 100,000
              develop   fix   3.12.0   develop    fix   3.12.0
  CBOR        338.7    5.9    8.2      1687.7    23.3   11.8   MiB
  MessagePack 338.9    6.0    7.3      1687.9    23.8   11.9   MiB
  UBJSON      339.0    7.6    8.0      1687.9    30.1   11.6   MiB
  BJData      339.0    7.3    7.9      1688.0    30.8   11.6   MiB

That is at most (65,536 + 2 per input byte) values: 1 MiB plus 32 bytes
per input byte with 16-byte values (a value is 24 bytes with
JSON_DIAGNOSTICS and up to 40 with JSON_DIAGNOSTIC_POSITIONS), as
documented in untrusted_input.md. Lower fixed caps do not get there: a
cap of 64 elements still peaks at 113 MiB (n = 100,000) and costs 4.7 %
on arrays of 300 elements (mat300.cbor) and 5.0 % on arrays of 10,000
(arrays_of_big.cbor).

Reader instructions retired (min of 7, 108 rows: 18 inputs in the 6 forms
CBOR, MessagePack, UBJSON and BJData plain and optimized; jeopardy, 46 MB,
is within 0.1 %), compared with the base of this change: -0.19 % to +0.37 % in the formats that announce
sizes (canada.cbor 81.9 M -> 82.1 M, many_small.cbor 333.5 M -> 334.2 M);
the benefit of #5476 is kept (without any reserve canada.cbor takes
116.8 M, +42.6 %, many_small.cbor +71.5 %, mat300.cbor +81.6 %). Rows of
plain UBJSON and BJData, whose arrays announce no size, move by -2.7 % to
+2.6 % without any code of theirs changing: the base alternates
between two instruction counts per process (452 M and 463 M for
many_pairs.ubjson, equal in cycles), and a struct with three unused
members added to the same class reproduces the 474 M mode.

Tests (CBOR, MessagePack, UBJSON, BJData): 200 nested headers that
announce 16384 elements peak at 52,433,600 bytes of DOM allocations on
the base and below 128 Ki values with the fix (scaled by sizeof(value),
so that it holds with the diagnostic macros) (counting allocator, no
process memory measured); an array that completes is still reserved in
one allocation; CBOR also checks that a DOM parser is not bound to a finished
parse (ASan: stack-use-after-scope without the guard), that wrappers
derived publicly (also with a parse inside the parse), privately and
protectedly compile and are bound or not as described, that an array that got less than it
announced decodes and that a SAX handler is told the announced size.
The new tests take 4.6 s under Valgrind at -O0 (gcc:12, all four files).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Clarify the reservation note for announced array counts

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Report an empty array for UBJSON [$N#<n>] instead of n elements (#6079)

* Report an empty array for UBJSON `[$N#<n>` instead of n elements

A no-op (N) is not a value, so an optimized array of type N holds no
element whatever count it declares. The reader skipped the elements but
still passed the declared count to the SAX parser: a handler was told
start_array(n) and no element followed, and the DOM parser reserved
min(n, 16384) values (256 KB) for an array that stays empty. Nine bytes
(`[$N#l 0x7FFFFFFF`) therefore committed 256 KB: 2000 of them in one
input, 18 KB, committed 512 MB of heap (capacity 163,840,000 values for
the 10000 arrays of the original probe: 2.6 GB for 70 KB of input).
Untouched pages keep the resident size small, but the commit charge and
a 32-bit address space are consumed.

The reader now announces 0 elements for an optimized array of type N.
The loop that skips the elements is unchanged (it never ran). A count
that does not fit into std::size_t is still rejected by the count
reader; a count that is too large for the array type of the DOM (before:
out_of_range.408 "excessive array size" from the DOM handler) is
accepted now, because the array is empty.

Behaviour change: sax_parse reports start_array(0) for `[$N#n` (before:
n); from_ubjson no longer throws out_of_range.408 for an N-typed array
whose count exceeds the maximum size of the array type.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Clarify the reservation note for announced array counts

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget (#6083)

* Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget

The whole-input budget for the elements of optimized UBJSON arrays of
type Z, T and F (1048576 plus 4 per byte read) rejects files written by
3.12.0 that have long arrays of these types, such as 1100 rows of 1000
nulls with a `$Z` annotation, and there was no way to read them. The
per-byte rate is now the macro JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE.

Semantics:
- default 4, so the default build is unchanged (the generated code of
  from_ubjson is identical at -O2 up to source line numbers);
- the value must be a positive integer: 0, negative, non-integral, bool,
  character and wider-than-uintmax_t values fail to compile with a
  static_assert that names the macro;
- a value that does not fit into std::size_t is the largest size_t, so
  one spelling works for 32- and 64-bit targets;
- the limit base + rate * bytes_read saturates at the largest size_t
  instead of wrapping around, for any rate and input size; a rate of
  131072 (2^20 / 8) or more cannot be exceeded by any input (the
  shortest header of an array of more than 32767 elements is 8 bytes),
  which switches the whole-input budget off;
- the limit of one array (1048576) and the base (1048576) stay fixed
  constants, so the writer fallback still follows the base only and a
  reader with any rate reads everything the writer produces;
- the macro is not part of the ABI tag and must have the same value in
  every translation unit; it is documented that way.

The memory the arrays can take in a DOM is about 20 bytes per rate and
byte of input (measured, macOS, containers of 9 * rate elements:
25 bytes per input byte at rate 1, 81 at 4, 312 at 16, 1226 at 64, 5493
at 256) plus about 33 MB for the base. A 3.12.0 row of N values has a
header of b bytes (5 up to 255, 6 up to 32767, 8 above), so a table is
read completely if N <= rate * b, and otherwise up to
(1048576 + h * rate, h = outer header: 6, 7 or 9 bytes) / (N - rate * b) rows; 1100 rows of 1000 nulls need
a rate of 8. The new page of the macro documents the formulas, the
measurements, and two examples that are run by the documentation checks.

The tests of the budget move to tests/src/unit-ubjson_valueless.cpp, which
derives every boundary from the macro and is built with the rates 1,
1000000 and 0xFFFFFFFFFFFFFFFF in addition to the default (the last two
check that nothing is refused and that the limit does not wrap around);
they fail with the unchanged header for rate 1, 1000000 and the maximum.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Refuse char8_t for JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE and clarify its documentation

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Use fseek instead of rewind in the UBJSON FILE* test

clang-tidy (bugprone-unsafe-functions) rejects rewind() because it has no
error detection. Check the result of fseek() instead.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Charge an announced array only what is reserved for it

An array that announced more than reserve_cap() elements but no more than
the credit left was charged the announced count, while reserve_array()
reserved at most reserve_cap() of them. An outer array of 20000 elements
therefore left a third nested array of 16384 with 12796 instead of the
full reservation. reserve_count() now caps the count first and charges
that, and the DOM parsers reserve it through reserve_array_capped(),
which does not cap again, so the fast path costs the same as before
(instructions retired within +-0.06 % on canada, citm_catalog and twitter
in CBOR, MessagePack and UBJSON).

Also re-wrap the comment of valueless_elements_base.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-11-15 17:02:05 +01:00

2384 lines
113 KiB
C++

// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++ (supporting code)
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
#include "doctest_compatibility.h"
// capture whether JSON_DELETE_DEPRECATED_FUNCTIONS was enabled on the command
// line *before* including json.hpp, since the library #undefs it once the header
// has been fully processed (see include/nlohmann/detail/macro_unscope.hpp); the
// tests of deprecated functions are skipped if these functions are deleted
#if defined(JSON_DELETE_DEPRECATED_FUNCTIONS) && (JSON_DELETE_DEPRECATED_FUNCTIONS == 1)
#define JSON_TEST_DEPRECATED_FUNCTIONS_DELETED
#endif
#include <nlohmann/json.hpp>
using nlohmann::json;
#ifdef JSON_TEST_NO_GLOBAL_UDLS
using namespace nlohmann::literals; // NOLINT(google-build-using-namespace)
#endif
#include <cstdint> // SIZE_MAX, UINT32_MAX
#include <fstream>
#include <sstream>
#include <iomanip>
#include <limits>
#include <list>
#include <set>
#include "make_test_data_available.hpp"
#include "round_trip_corpus.hpp"
#include "test_utils.hpp"
#include "sax_countdown.hpp"
#include "allocation_counter.hpp"
#include "sax_event_loggers.hpp"
using utils::SaxCountdown;
using utils::SaxEventLogger;
TEST_CASE("MessagePack")
{
SECTION("individual values")
{
SECTION("discarded")
{
// a discarded value cannot be serialized to MessagePack
json const j = json::value_t::discarded;
CHECK_THROWS_WITH_AS(json::to_msgpack(j), "[json.exception.type_error.321] cannot serialize discarded value to MessagePack", json::type_error&);
}
SECTION("discarded values nested in a container")
{
json const discarded = json::value_t::discarded;
SECTION("in an array")
{
json const j = {1, discarded, 2};
#if JSON_DIAGNOSTICS
CHECK_THROWS_WITH_AS(json::to_msgpack(j), "[json.exception.type_error.321] (/1) cannot serialize discarded value to MessagePack", json::type_error&);
#else
CHECK_THROWS_WITH_AS(json::to_msgpack(j), "[json.exception.type_error.321] cannot serialize discarded value to MessagePack", json::type_error&);
#endif
}
SECTION("as an object value")
{
json j;
j["a"] = 1;
j["b"] = discarded;
#if JSON_DIAGNOSTICS
CHECK_THROWS_WITH_AS(json::to_msgpack(j), "[json.exception.type_error.321] (/b) cannot serialize discarded value to MessagePack", json::type_error&);
#else
CHECK_THROWS_WITH_AS(json::to_msgpack(j), "[json.exception.type_error.321] cannot serialize discarded value to MessagePack", json::type_error&);
#endif
}
SECTION("nested deeper (array in object in array)")
{
json inner_array = {1, discarded};
json middle_object;
middle_object["x"] = inner_array;
json const j = {middle_object};
#if JSON_DIAGNOSTICS
CHECK_THROWS_WITH_AS(json::to_msgpack(j), "[json.exception.type_error.321] (/0/x/1) cannot serialize discarded value to MessagePack", json::type_error&);
#else
CHECK_THROWS_WITH_AS(json::to_msgpack(j), "[json.exception.type_error.321] cannot serialize discarded value to MessagePack", json::type_error&);
#endif
}
}
SECTION("null")
{
json const j = nullptr;
std::vector<uint8_t> const expected = {0xc0};
const auto result = json::to_msgpack(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
SECTION("boolean")
{
SECTION("true")
{
json const j = true;
std::vector<uint8_t> const expected = {0xc3};
const auto result = json::to_msgpack(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
SECTION("false")
{
json const j = false;
std::vector<uint8_t> const expected = {0xc2};
const auto result = json::to_msgpack(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
SECTION("number")
{
SECTION("signed")
{
SECTION("-32..-1 (negative fixnum)")
{
for (auto i = -32; i <= -1; ++i)
{
CAPTURE(i)
// create JSON value with integer number
json const j = i;
// check type
CHECK(j.is_number_integer());
// create expected byte vector
std::vector<uint8_t> const expected
{
static_cast<uint8_t>(i)
};
// compare result + size
const auto result = json::to_msgpack(j);
CHECK(result == expected);
CHECK(result.size() == 1);
// check individual bytes
CHECK(static_cast<int8_t>(result[0]) == i);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
SECTION("0..127 (positive fixnum)")
{
for (size_t i = 0; i <= 127; ++i)
{
CAPTURE(i)
// create JSON value with integer number
json j = -1;
j.get_ref<json::number_integer_t&>() = static_cast<json::number_integer_t>(i);
// check type
CHECK(j.is_number_integer());
// create expected byte vector
std::vector<uint8_t> const expected{static_cast<uint8_t>(i)};
// compare result + size
const auto result = json::to_msgpack(j);
CHECK(result == expected);
CHECK(result.size() == 1);
// check individual bytes
CHECK(result[0] == i);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
SECTION("128..255 (int 8)")
{
for (size_t i = 128; i <= 255; ++i)
{
CAPTURE(i)
// create JSON value with integer number
json j = -1;
j.get_ref<json::number_integer_t&>() = static_cast<json::number_integer_t>(i);
// check type
CHECK(j.is_number_integer());
// create expected byte vector
std::vector<uint8_t> const expected
{
0xcc,
static_cast<uint8_t>(i),
};
// compare result + size
const auto result = json::to_msgpack(j);
CHECK(result == expected);
CHECK(result.size() == 2);
// check individual bytes
CHECK(result[0] == 0xcc);
auto const restored = static_cast<uint8_t>(result[1]);
CHECK(restored == i);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
SECTION("256..65535 (int 16)")
{
for (size_t i = 256; i <= 65535; i = utils::next_integer_sample(i, static_cast<size_t>(65535), static_cast<size_t>(7)))
{
CAPTURE(i)
// create JSON value with integer number
json j = -1;
j.get_ref<json::number_integer_t&>() = static_cast<json::number_integer_t>(i);
// check type
CHECK(j.is_number_integer());
// create expected byte vector
std::vector<uint8_t> const expected
{
0xcd,
static_cast<uint8_t>((i >> 8) & 0xff),
static_cast<uint8_t>(i & 0xff),
};
// compare result + size
const auto result = json::to_msgpack(j);
CHECK(result == expected);
CHECK(result.size() == 3);
// check individual bytes
CHECK(result[0] == 0xcd);
auto const restored = static_cast<uint16_t>((static_cast<uint8_t>(result[1]) * 256) + static_cast<uint8_t>(result[2]));
CHECK(restored == i);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
SECTION("65536..4294967295 (int 32)")
{
for (uint32_t i :
{
65536u, 77777u, 1048576u, 4294967295u
})
{
CAPTURE(i)
// create JSON value with integer number
json j = -1;
j.get_ref<json::number_integer_t&>() = static_cast<json::number_integer_t>(i);
// check type
CHECK(j.is_number_integer());
// create expected byte vector
std::vector<uint8_t> const expected
{
0xce,
static_cast<uint8_t>((i >> 24) & 0xff),
static_cast<uint8_t>((i >> 16) & 0xff),
static_cast<uint8_t>((i >> 8) & 0xff),
static_cast<uint8_t>(i & 0xff),
};
// compare result + size
const auto result = json::to_msgpack(j);
CHECK(result == expected);
CHECK(result.size() == 5);
// check individual bytes
CHECK(result[0] == 0xce);
uint32_t const restored = (static_cast<uint32_t>(result[1]) << 030) +
(static_cast<uint32_t>(result[2]) << 020) +
(static_cast<uint32_t>(result[3]) << 010) +
static_cast<uint32_t>(result[4]);
CHECK(restored == i);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
SECTION("4294967296..9223372036854775807 (int 64)")
{
for (uint64_t i :
{
4294967296LU, 9223372036854775807LU
})
{
CAPTURE(i)
// create JSON value with integer number
json j = -1;
j.get_ref<json::number_integer_t&>() = static_cast<json::number_integer_t>(i);
// check type
CHECK(j.is_number_integer());
// create expected byte vector
std::vector<uint8_t> const expected
{
0xcf,
static_cast<uint8_t>((i >> 070) & 0xff),
static_cast<uint8_t>((i >> 060) & 0xff),
static_cast<uint8_t>((i >> 050) & 0xff),
static_cast<uint8_t>((i >> 040) & 0xff),
static_cast<uint8_t>((i >> 030) & 0xff),
static_cast<uint8_t>((i >> 020) & 0xff),
static_cast<uint8_t>((i >> 010) & 0xff),
static_cast<uint8_t>(i & 0xff),
};
// compare result + size
const auto result = json::to_msgpack(j);
CHECK(result == expected);
CHECK(result.size() == 9);
// check individual bytes
CHECK(result[0] == 0xcf);
uint64_t const restored = (static_cast<uint64_t>(result[1]) << 070) +
(static_cast<uint64_t>(result[2]) << 060) +
(static_cast<uint64_t>(result[3]) << 050) +
(static_cast<uint64_t>(result[4]) << 040) +
(static_cast<uint64_t>(result[5]) << 030) +
(static_cast<uint64_t>(result[6]) << 020) +
(static_cast<uint64_t>(result[7]) << 010) +
static_cast<uint64_t>(result[8]);
CHECK(restored == i);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
SECTION("-128..-33 (int 8)")
{
for (auto i = -128; i <= -33; ++i)
{
CAPTURE(i)
// create JSON value with integer number
json const j = i;
// check type
CHECK(j.is_number_integer());
// create expected byte vector
std::vector<uint8_t> const expected
{
0xd0,
static_cast<uint8_t>(i),
};
// compare result + size
const auto result = json::to_msgpack(j);
CHECK(result == expected);
CHECK(result.size() == 2);
// check individual bytes
CHECK(result[0] == 0xd0);
CHECK(static_cast<int8_t>(result[1]) == i);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
SECTION("-9263 (int 16)")
{
json const j = -9263;
std::vector<uint8_t> const expected = {0xd1, 0xdb, 0xd1};
const auto result = json::to_msgpack(j);
CHECK(result == expected);
auto const restored = static_cast<int16_t>((result[1] << 8) + result[2]);
CHECK(restored == -9263);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
SECTION("-32768..-129 (int 16)")
{
for (int16_t i = -32768; i <= static_cast<std::int16_t>(-129); i = utils::next_integer_sample(i, static_cast<int16_t>(-129), static_cast<int16_t>(7)))
{
CAPTURE(i)
// create JSON value with integer number
json const j = i;
// check type
CHECK(j.is_number_integer());
// create expected byte vector
std::vector<uint8_t> const expected
{
0xd1,
static_cast<uint8_t>((i >> 8) & 0xff),
static_cast<uint8_t>(i & 0xff),
};
// compare result + size
const auto result = json::to_msgpack(j);
CHECK(result == expected);
CHECK(result.size() == 3);
// check individual bytes
CHECK(result[0] == 0xd1);
auto const restored = static_cast<int16_t>((result[1] << 8) + result[2]);
CHECK(restored == i);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
SECTION("-32769..-2147483648")
{
std::vector<int32_t> const numbers
{
-32769,
-65536,
-77777,
-1048576,
-2147483648LL,
};
for (auto i : numbers)
{
CAPTURE(i)
// create JSON value with integer number
json const j = i;
// check type
CHECK(j.is_number_integer());
// create expected byte vector
std::vector<uint8_t> const expected
{
0xd2,
static_cast<uint8_t>((i >> 24) & 0xff),
static_cast<uint8_t>((i >> 16) & 0xff),
static_cast<uint8_t>((i >> 8) & 0xff),
static_cast<uint8_t>(i & 0xff),
};
// compare result + size
const auto result = json::to_msgpack(j);
CHECK(result == expected);
CHECK(result.size() == 5);
// check individual bytes
CHECK(result[0] == 0xd2);
uint32_t const restored = (static_cast<uint32_t>(result[1]) << 030) +
(static_cast<uint32_t>(result[2]) << 020) +
(static_cast<uint32_t>(result[3]) << 010) +
static_cast<uint32_t>(result[4]);
CHECK(static_cast<std::int32_t>(restored) == i);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
SECTION("-9223372036854775808..-2147483649 (int 64)")
{
std::vector<int64_t> const numbers
{
(std::numeric_limits<int64_t>::min)(),
-2147483649LL,
};
for (auto i : numbers)
{
CAPTURE(i)
// create JSON value with unsigned integer number
json const j = i;
// check type
CHECK(j.is_number_integer());
// create expected byte vector
std::vector<uint8_t> const expected
{
0xd3,
static_cast<uint8_t>((i >> 070) & 0xff),
static_cast<uint8_t>((i >> 060) & 0xff),
static_cast<uint8_t>((i >> 050) & 0xff),
static_cast<uint8_t>((i >> 040) & 0xff),
static_cast<uint8_t>((i >> 030) & 0xff),
static_cast<uint8_t>((i >> 020) & 0xff),
static_cast<uint8_t>((i >> 010) & 0xff),
static_cast<uint8_t>(i & 0xff),
};
// compare result + size
const auto result = json::to_msgpack(j);
CHECK(result == expected);
CHECK(result.size() == 9);
// check individual bytes
CHECK(result[0] == 0xd3);
int64_t const restored = (static_cast<int64_t>(result[1]) << 070) +
(static_cast<int64_t>(result[2]) << 060) +
(static_cast<int64_t>(result[3]) << 050) +
(static_cast<int64_t>(result[4]) << 040) +
(static_cast<int64_t>(result[5]) << 030) +
(static_cast<int64_t>(result[6]) << 020) +
(static_cast<int64_t>(result[7]) << 010) +
static_cast<int64_t>(result[8]);
CHECK(restored == i);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
}
SECTION("unsigned")
{
SECTION("0..127 (positive fixnum)")
{
for (size_t i = 0; i <= 127; ++i)
{
CAPTURE(i)
// create JSON value with unsigned integer number
json const j = i;
// check type
CHECK(j.is_number_unsigned());
// create expected byte vector
std::vector<uint8_t> const expected{static_cast<uint8_t>(i)};
// compare result + size
const auto result = json::to_msgpack(j);
CHECK(result == expected);
CHECK(result.size() == 1);
// check individual bytes
CHECK(result[0] == i);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
SECTION("128..255 (uint 8)")
{
for (size_t i = 128; i <= 255; ++i)
{
CAPTURE(i)
// create JSON value with unsigned integer number
json const j = i;
// check type
CHECK(j.is_number_unsigned());
// create expected byte vector
std::vector<uint8_t> const expected
{
0xcc,
static_cast<uint8_t>(i),
};
// compare result + size
const auto result = json::to_msgpack(j);
CHECK(result == expected);
CHECK(result.size() == 2);
// check individual bytes
CHECK(result[0] == 0xcc);
auto const restored = static_cast<uint8_t>(result[1]);
CHECK(restored == i);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
SECTION("256..65535 (uint 16)")
{
for (size_t i = 256; i <= 65535; i = utils::next_integer_sample(i, static_cast<size_t>(65535), static_cast<size_t>(7)))
{
CAPTURE(i)
// create JSON value with unsigned integer number
json const j = i;
// check type
CHECK(j.is_number_unsigned());
// create expected byte vector
std::vector<uint8_t> const expected
{
0xcd,
static_cast<uint8_t>((i >> 8) & 0xff),
static_cast<uint8_t>(i & 0xff),
};
// compare result + size
const auto result = json::to_msgpack(j);
CHECK(result == expected);
CHECK(result.size() == 3);
// check individual bytes
CHECK(result[0] == 0xcd);
auto const restored = static_cast<uint16_t>((static_cast<uint8_t>(result[1]) * 256) + static_cast<uint8_t>(result[2]));
CHECK(restored == i);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
SECTION("65536..4294967295 (uint 32)")
{
for (const uint32_t i :
{
65536u, 77777u, 1048576u, 4294967295u
})
{
CAPTURE(i)
// create JSON value with unsigned integer number
json const j = i;
// check type
CHECK(j.is_number_unsigned());
// create expected byte vector
std::vector<uint8_t> const expected
{
0xce,
static_cast<uint8_t>((i >> 24) & 0xff),
static_cast<uint8_t>((i >> 16) & 0xff),
static_cast<uint8_t>((i >> 8) & 0xff),
static_cast<uint8_t>(i & 0xff),
};
// compare result + size
const auto result = json::to_msgpack(j);
CHECK(result == expected);
CHECK(result.size() == 5);
// check individual bytes
CHECK(result[0] == 0xce);
uint32_t const restored = (static_cast<uint32_t>(result[1]) << 030) +
(static_cast<uint32_t>(result[2]) << 020) +
(static_cast<uint32_t>(result[3]) << 010) +
static_cast<uint32_t>(result[4]);
CHECK(restored == i);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
SECTION("4294967296..18446744073709551615 (uint 64)")
{
for (const uint64_t i :
{
4294967296LU, 18446744073709551615LU
})
{
CAPTURE(i)
// create JSON value with unsigned integer number
json const j = i;
// check type
CHECK(j.is_number_unsigned());
// create expected byte vector
std::vector<uint8_t> const expected
{
0xcf,
static_cast<uint8_t>((i >> 070) & 0xff),
static_cast<uint8_t>((i >> 060) & 0xff),
static_cast<uint8_t>((i >> 050) & 0xff),
static_cast<uint8_t>((i >> 040) & 0xff),
static_cast<uint8_t>((i >> 030) & 0xff),
static_cast<uint8_t>((i >> 020) & 0xff),
static_cast<uint8_t>((i >> 010) & 0xff),
static_cast<uint8_t>(i & 0xff),
};
// compare result + size
const auto result = json::to_msgpack(j);
CHECK(result == expected);
CHECK(result.size() == 9);
// check individual bytes
CHECK(result[0] == 0xcf);
uint64_t const restored = (static_cast<uint64_t>(result[1]) << 070) +
(static_cast<uint64_t>(result[2]) << 060) +
(static_cast<uint64_t>(result[3]) << 050) +
(static_cast<uint64_t>(result[4]) << 040) +
(static_cast<uint64_t>(result[5]) << 030) +
(static_cast<uint64_t>(result[6]) << 020) +
(static_cast<uint64_t>(result[7]) << 010) +
static_cast<uint64_t>(result[8]);
CHECK(restored == i);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
}
SECTION("float")
{
SECTION("3.1415925")
{
double const v = 3.1415925;
json const j = v;
std::vector<uint8_t> const expected =
{
0xcb, 0x40, 0x09, 0x21, 0xfb, 0x3f, 0xa6, 0xde, 0xfc
};
const auto result = json::to_msgpack(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result) == v);
CHECK(json::from_msgpack(result, true, false) == j);
}
SECTION("1.0")
{
double const v = 1.0;
json const j = v;
std::vector<uint8_t> const expected =
{
0xca, 0x3f, 0x80, 0x00, 0x00
};
const auto result = json::to_msgpack(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result) == v);
CHECK(json::from_msgpack(result, true, false) == j);
}
SECTION("128.128")
{
double const v = 128.1280059814453125;
json const j = v;
std::vector<uint8_t> const expected =
{
0xca, 0x43, 0x00, 0x20, 0xc5
};
const auto result = json::to_msgpack(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result) == v);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
}
SECTION("string")
{
SECTION("N = 0..31")
{
// explicitly enumerate the first byte for all 32 strings
const std::vector<uint8_t> first_bytes =
{
0xa0, 0xa1, 0xa2, 0xa3, 0xa4, 0xa5, 0xa6, 0xa7, 0xa8,
0xa9, 0xaa, 0xab, 0xac, 0xad, 0xae, 0xaf, 0xb0, 0xb1,
0xb2, 0xb3, 0xb4, 0xb5, 0xb6, 0xb7, 0xb8, 0xb9, 0xba,
0xbb, 0xbc, 0xbd, 0xbe, 0xbf
};
for (size_t N = 0; N < first_bytes.size(); ++N)
{
CAPTURE(N)
// create JSON value with string containing of N * 'x'
const auto s = std::string(N, 'x');
json const j = s;
// create expected byte vector
std::vector<uint8_t> expected;
expected.push_back(first_bytes[N]);
for (size_t i = 0; i < N; ++i)
{
expected.push_back('x');
}
// check first byte
CHECK((first_bytes[N] & 0x1f) == N);
// compare result + size
const auto result = json::to_msgpack(j);
CHECK(result == expected);
CHECK(result.size() == N + 1);
// check that no null byte is appended
if (N > 0)
{
CHECK(result.back() != '\x00');
}
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
SECTION("N = 32..255")
{
for (size_t N = 32; N <= 255; ++N)
{
CAPTURE(N)
// create JSON value with string containing of N * 'x'
const auto s = std::string(N, 'x');
json const j = s;
// create expected byte vector
std::vector<uint8_t> expected;
expected.push_back(0xd9);
expected.push_back(static_cast<uint8_t>(N));
for (size_t i = 0; i < N; ++i)
{
expected.push_back('x');
}
// compare result + size
const auto result = json::to_msgpack(j);
CHECK(result == expected);
CHECK(result.size() == N + 2);
// check that no null byte is appended
CHECK(result.back() != '\x00');
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
SECTION("N = 256..65535")
{
for (size_t N :
{
256u, 999u, 1025u, 3333u, 2048u, 65535u
})
{
CAPTURE(N)
// create JSON value with string containing of N * 'x'
const auto s = std::string(N, 'x');
json const j = s;
// create expected byte vector (hack: create string first)
std::vector<uint8_t> expected(N, 'x');
// reverse order of commands, because we insert at begin()
expected.insert(expected.begin(), static_cast<uint8_t>(N & 0xff));
expected.insert(expected.begin(), static_cast<uint8_t>((N >> 8) & 0xff));
expected.insert(expected.begin(), 0xda);
// compare result + size
const auto result = json::to_msgpack(j);
CHECK(result == expected);
CHECK(result.size() == N + 3);
// check that no null byte is appended
CHECK(result.back() != '\x00');
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
SECTION("N = 65536..4294967295")
{
for (size_t N :
{
65536u, 77777u, 1048576u
})
{
CAPTURE(N)
// create JSON value with string containing of N * 'x'
const auto s = std::string(N, 'x');
json const j = s;
// create expected byte vector (hack: create string first)
std::vector<uint8_t> expected(N, 'x');
// reverse order of commands, because we insert at begin()
expected.insert(expected.begin(), static_cast<uint8_t>(N & 0xff));
expected.insert(expected.begin(), static_cast<uint8_t>((N >> 8) & 0xff));
expected.insert(expected.begin(), static_cast<uint8_t>((N >> 16) & 0xff));
expected.insert(expected.begin(), static_cast<uint8_t>((N >> 24) & 0xff));
expected.insert(expected.begin(), 0xdb);
// compare result + size
const auto result = json::to_msgpack(j);
CHECK(result == expected);
CHECK(result.size() == N + 5);
// check that no null byte is appended
CHECK(result.back() != '\x00');
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
}
SECTION("array")
{
SECTION("empty")
{
json const j = json::array();
std::vector<uint8_t> const expected = {0x90};
const auto result = json::to_msgpack(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
SECTION("[null]")
{
json const j = {nullptr};
std::vector<uint8_t> const expected = {0x91, 0xc0};
const auto result = json::to_msgpack(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
SECTION("[1,2,3,4,5]")
{
json const j = json::parse("[1,2,3,4,5]");
std::vector<uint8_t> const expected = {0x95, 0x01, 0x02, 0x03, 0x04, 0x05};
const auto result = json::to_msgpack(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
SECTION("[[[[]]]]")
{
json const j = json::parse("[[[[]]]]");
std::vector<uint8_t> const expected = {0x91, 0x91, 0x91, 0x90};
const auto result = json::to_msgpack(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
SECTION("array 16")
{
json j(16, nullptr);
std::vector<uint8_t> expected(j.size() + 3, 0xc0); // all null
expected[0] = 0xdc; // array 16
expected[1] = 0x00; // size (0x0010), byte 0
expected[2] = 0x10; // size (0x0010), byte 1
const auto result = json::to_msgpack(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
SECTION("array 32")
{
json j(65536, nullptr);
std::vector<uint8_t> expected(j.size() + 5, 0xc0); // all null
expected[0] = 0xdd; // array 32
expected[1] = 0x00; // size (0x00100000), byte 0
expected[2] = 0x01; // size (0x00100000), byte 1
expected[3] = 0x00; // size (0x00100000), byte 2
expected[4] = 0x00; // size (0x00100000), byte 3
const auto result = json::to_msgpack(j);
//CHECK(result == expected);
CHECK(result.size() == expected.size());
for (size_t i = 0; i < expected.size(); ++i)
{
CAPTURE(i)
CHECK(result[i] == expected[i]);
}
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
SECTION("object")
{
SECTION("empty")
{
json const j = json::object();
std::vector<uint8_t> const expected = {0x80};
const auto result = json::to_msgpack(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
SECTION("{\"\":null}")
{
json const j = {{"", nullptr}};
std::vector<uint8_t> const expected = {0x81, 0xa0, 0xc0};
const auto result = json::to_msgpack(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
SECTION("{\"a\": {\"b\": {\"c\": {}}}}")
{
json const j = json::parse(R"({"a": {"b": {"c": {}}}})");
std::vector<uint8_t> const expected =
{
0x81, 0xa1, 0x61, 0x81, 0xa1, 0x62, 0x81, 0xa1, 0x63, 0x80
};
const auto result = json::to_msgpack(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
SECTION("map 16")
{
json const j = R"({"00": null, "01": null, "02": null, "03": null,
"04": null, "05": null, "06": null, "07": null,
"08": null, "09": null, "10": null, "11": null,
"12": null, "13": null, "14": null, "15": null})"_json;
const auto result = json::to_msgpack(j);
// Checking against an expected vector byte by byte is
// difficult, because no assumption on the order of key/value
// pairs are made. We therefore only check the prefix (type and
// size) and the overall size. The rest is then handled in the
// roundtrip check.
CHECK(result.size() == 67); // 1 type, 2 size, 16*4 content
CHECK(result[0] == 0xde); // map 16
CHECK(result[1] == 0x00); // byte 0 of size (0x0010)
CHECK(result[2] == 0x10); // byte 1 of size (0x0010)
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
SECTION("map 32")
{
json j;
for (auto i = 0; i < 65536; ++i)
{
// format i to a fixed width of 5
// each entry will need 7 bytes: 6 for fixstr, 1 for null
std::stringstream ss;
ss << std::setw(5) << std::setfill('0') << i;
j.emplace(ss.str(), nullptr);
}
const auto result = json::to_msgpack(j);
// Checking against an expected vector byte by byte is
// difficult, because no assumption on the order of key/value
// pairs are made. We therefore only check the prefix (type and
// size) and the overall size. The rest is then handled in the
// roundtrip check.
CHECK(result.size() == 458757); // 1 type, 4 size, 65536*7 content
CHECK(result[0] == 0xdf); // map 32
CHECK(result[1] == 0x00); // byte 0 of size (0x00010000)
CHECK(result[2] == 0x01); // byte 1 of size (0x00010000)
CHECK(result[3] == 0x00); // byte 2 of size (0x00010000)
CHECK(result[4] == 0x00); // byte 3 of size (0x00010000)
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
SECTION("extension")
{
SECTION("N = 0..255")
{
for (size_t N = 0; N <= 0xFF; ++N)
{
CAPTURE(N)
// create JSON value with byte array containing of N * 'x'
const auto s = std::vector<uint8_t>(N, 'x');
json j = json::binary(s);
std::uint8_t const subtype = 42;
j.get_binary().set_subtype(subtype);
// create expected byte vector
std::vector<uint8_t> expected;
switch (N)
{
case 1:
expected.push_back(static_cast<std::uint8_t>(0xD4));
break;
case 2:
expected.push_back(static_cast<std::uint8_t>(0xD5));
break;
case 4:
expected.push_back(static_cast<std::uint8_t>(0xD6));
break;
case 8:
expected.push_back(static_cast<std::uint8_t>(0xD7));
break;
case 16:
expected.push_back(static_cast<std::uint8_t>(0xD8));
break;
default:
expected.push_back(static_cast<std::uint8_t>(0xC7));
expected.push_back(static_cast<std::uint8_t>(N));
break;
}
expected.push_back(subtype);
for (size_t i = 0; i < N; ++i)
{
expected.push_back(0x78);
}
// compare result + size
const auto result = json::to_msgpack(j);
CHECK(result == expected);
switch (N)
{
case 1:
case 2:
case 4:
case 8:
case 16:
CHECK(result.size() == N + 2);
break;
default:
CHECK(result.size() == N + 3);
break;
}
// check that no null byte is appended
if (N > 0)
{
CHECK(result.back() != '\x00');
}
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
SECTION("N = 256..65535")
{
for (std::size_t N :
{
256u, 999u, 1025u, 3333u, 2048u, 65535u
})
{
CAPTURE(N)
// create JSON value with string containing of N * 'x'
const auto s = std::vector<uint8_t>(N, 'x');
json j = json::binary(s);
std::uint8_t const subtype = 42;
j.get_binary().set_subtype(subtype);
// create expected byte vector (hack: create string first)
std::vector<uint8_t> expected(N, 'x');
// reverse order of commands, because we insert at begin()
expected.insert(expected.begin(), subtype);
expected.insert(expected.begin(), static_cast<uint8_t>(N & 0xff));
expected.insert(expected.begin(), static_cast<uint8_t>((N >> 8) & 0xff));
expected.insert(expected.begin(), 0xC8);
// compare result + size
const auto result = json::to_msgpack(j);
CHECK(result == expected);
CHECK(result.size() == N + 4);
// check that no null byte is appended
CHECK(result.back() != '\x00');
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
SECTION("N = 65536..4294967295")
{
for (std::size_t N :
{
65536u, 77777u, 1048576u
})
{
CAPTURE(N)
// create JSON value with string containing of N * 'x'
const auto s = std::vector<uint8_t>(N, 'x');
json j = json::binary(s);
std::uint8_t const subtype = 42;
j.get_binary().set_subtype(subtype);
// create expected byte vector (hack: create string first)
std::vector<uint8_t> expected(N, 'x');
// reverse order of commands, because we insert at begin()
expected.insert(expected.begin(), subtype);
expected.insert(expected.begin(), static_cast<uint8_t>(N & 0xff));
expected.insert(expected.begin(), static_cast<uint8_t>((N >> 8) & 0xff));
expected.insert(expected.begin(), static_cast<uint8_t>((N >> 16) & 0xff));
expected.insert(expected.begin(), static_cast<uint8_t>((N >> 24) & 0xff));
expected.insert(expected.begin(), 0xC9);
// compare result + size
const auto result = json::to_msgpack(j);
CHECK(result == expected);
CHECK(result.size() == N + 6);
// check that no null byte is appended
CHECK(result.back() != '\x00');
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
}
SECTION("binary")
{
SECTION("N = 0..255")
{
for (std::size_t N = 0; N <= 0xFF; ++N)
{
CAPTURE(N)
// create JSON value with byte array containing of N * 'x'
const auto s = std::vector<uint8_t>(N, 'x');
json const j = json::binary(s);
// create expected byte vector
std::vector<std::uint8_t> expected;
expected.push_back(static_cast<std::uint8_t>(0xC4));
expected.push_back(static_cast<std::uint8_t>(N));
for (size_t i = 0; i < N; ++i)
{
expected.push_back(0x78);
}
// compare result + size
const auto result = json::to_msgpack(j);
CHECK(result == expected);
CHECK(result.size() == N + 2);
// check that no null byte is appended
if (N > 0)
{
CHECK(result.back() != '\x00');
}
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
SECTION("N = 256..65535")
{
for (std::size_t N :
{
256u, 999u, 1025u, 3333u, 2048u, 65535u
})
{
CAPTURE(N)
// create JSON value with string containing of N * 'x'
const auto s = std::vector<std::uint8_t>(N, 'x');
json const j = json::binary(s);
// create expected byte vector (hack: create string first)
std::vector<std::uint8_t> expected(N, 'x');
// reverse order of commands, because we insert at begin()
expected.insert(expected.begin(), static_cast<std::uint8_t>(N & 0xff));
expected.insert(expected.begin(), static_cast<std::uint8_t>((N >> 8) & 0xff));
expected.insert(expected.begin(), 0xC5);
// compare result + size
const auto result = json::to_msgpack(j);
CHECK(result == expected);
CHECK(result.size() == N + 3);
// check that no null byte is appended
CHECK(result.back() != '\x00');
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
SECTION("N = 65536..4294967295")
{
for (std::size_t N :
{
65536u, 77777u, 1048576u
})
{
CAPTURE(N)
// create JSON value with string containing of N * 'x'
const auto s = std::vector<std::uint8_t>(N, 'x');
json const j = json::binary(s);
// create expected byte vector (hack: create string first)
std::vector<uint8_t> expected(N, 'x');
// reverse order of commands, because we insert at begin()
expected.insert(expected.begin(), static_cast<std::uint8_t>(N & 0xff));
expected.insert(expected.begin(), static_cast<std::uint8_t>((N >> 8) & 0xff));
expected.insert(expected.begin(), static_cast<std::uint8_t>((N >> 16) & 0xff));
expected.insert(expected.begin(), static_cast<std::uint8_t>((N >> 24) & 0xff));
expected.insert(expected.begin(), 0xC6);
// compare result + size
const auto result = json::to_msgpack(j);
CHECK(result == expected);
CHECK(result.size() == N + 5);
// check that no null byte is appended
CHECK(result.back() != '\x00');
// roundtrip
CHECK(json::from_msgpack(result) == j);
CHECK(json::from_msgpack(result, true, false) == j);
}
}
}
}
SECTION("from float32")
{
auto given = std::vector<uint8_t>({0xca, 0x41, 0xc8, 0x00, 0x01});
json const j = json::from_msgpack(given);
CHECK(j.get<double>() == Approx(25.0000019073486));
}
SECTION("errors")
{
SECTION("empty byte vector")
{
json _;
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>()), "[json.exception.parse_error.110] parse error at byte 1: syntax error while parsing MessagePack value: unexpected end of input", json::parse_error&);
CHECK(json::from_msgpack(std::vector<uint8_t>(), true, false).is_discarded());
}
SECTION("too short byte vector")
{
json _;
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0x87})),
"[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing MessagePack string: unexpected end of input", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0xcc})),
"[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing MessagePack number: unexpected end of input", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0xcd})),
"[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing MessagePack number: unexpected end of input", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0xcd, 0x00})),
"[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing MessagePack number: unexpected end of input", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0xce})),
"[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing MessagePack number: unexpected end of input", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0xce, 0x00})),
"[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing MessagePack number: unexpected end of input", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0xce, 0x00, 0x00})),
"[json.exception.parse_error.110] parse error at byte 4: syntax error while parsing MessagePack number: unexpected end of input", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0xce, 0x00, 0x00, 0x00})),
"[json.exception.parse_error.110] parse error at byte 5: syntax error while parsing MessagePack number: unexpected end of input", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0xcf})),
"[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing MessagePack number: unexpected end of input", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0xcf, 0x00})),
"[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing MessagePack number: unexpected end of input", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0xcf, 0x00, 0x00})),
"[json.exception.parse_error.110] parse error at byte 4: syntax error while parsing MessagePack number: unexpected end of input", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0xcf, 0x00, 0x00, 0x00})),
"[json.exception.parse_error.110] parse error at byte 5: syntax error while parsing MessagePack number: unexpected end of input", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0xcf, 0x00, 0x00, 0x00, 0x00})),
"[json.exception.parse_error.110] parse error at byte 6: syntax error while parsing MessagePack number: unexpected end of input", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0xcf, 0x00, 0x00, 0x00, 0x00, 0x00})),
"[json.exception.parse_error.110] parse error at byte 7: syntax error while parsing MessagePack number: unexpected end of input", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0xcf, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00})),
"[json.exception.parse_error.110] parse error at byte 8: syntax error while parsing MessagePack number: unexpected end of input", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0xcf, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00})),
"[json.exception.parse_error.110] parse error at byte 9: syntax error while parsing MessagePack number: unexpected end of input", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0xa5, 0x68, 0x65})),
"[json.exception.parse_error.110] parse error at byte 4: syntax error while parsing MessagePack string: unexpected end of input", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0x92, 0x01})),
"[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing MessagePack value: unexpected end of input", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0x81, 0xa1, 0x61})),
"[json.exception.parse_error.110] parse error at byte 4: syntax error while parsing MessagePack value: unexpected end of input", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0xc4, 0x02})),
"[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing MessagePack binary: unexpected end of input", json::parse_error&);
CHECK(json::from_msgpack(std::vector<uint8_t>({0x87}), true, false).is_discarded());
CHECK(json::from_msgpack(std::vector<uint8_t>({0xcc}), true, false).is_discarded());
CHECK(json::from_msgpack(std::vector<uint8_t>({0xcd}), true, false).is_discarded());
CHECK(json::from_msgpack(std::vector<uint8_t>({0xcd, 0x00}), true, false).is_discarded());
CHECK(json::from_msgpack(std::vector<uint8_t>({0xce}), true, false).is_discarded());
CHECK(json::from_msgpack(std::vector<uint8_t>({0xce, 0x00}), true, false).is_discarded());
CHECK(json::from_msgpack(std::vector<uint8_t>({0xce, 0x00, 0x00}), true, false).is_discarded());
CHECK(json::from_msgpack(std::vector<uint8_t>({0xce, 0x00, 0x00, 0x00}), true, false).is_discarded());
CHECK(json::from_msgpack(std::vector<uint8_t>({0xcf}), true, false).is_discarded());
CHECK(json::from_msgpack(std::vector<uint8_t>({0xcf, 0x00}), true, false).is_discarded());
CHECK(json::from_msgpack(std::vector<uint8_t>({0xcf, 0x00, 0x00}), true, false).is_discarded());
CHECK(json::from_msgpack(std::vector<uint8_t>({0xcf, 0x00, 0x00, 0x00}), true, false).is_discarded());
CHECK(json::from_msgpack(std::vector<uint8_t>({0xcf, 0x00, 0x00, 0x00, 0x00}), true, false).is_discarded());
CHECK(json::from_msgpack(std::vector<uint8_t>({0xcf, 0x00, 0x00, 0x00, 0x00, 0x00}), true, false).is_discarded());
CHECK(json::from_msgpack(std::vector<uint8_t>({0xcf, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00}), true, false).is_discarded());
CHECK(json::from_msgpack(std::vector<uint8_t>({0xcf, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00}), true, false).is_discarded());
CHECK(json::from_msgpack(std::vector<uint8_t>({0xa5, 0x68, 0x65}), true, false).is_discarded());
CHECK(json::from_msgpack(std::vector<uint8_t>({0x92, 0x01}), true, false).is_discarded());
CHECK(json::from_msgpack(std::vector<uint8_t>({0x81, 0xA1, 0x61}), true, false).is_discarded());
CHECK(json::from_msgpack(std::vector<uint8_t>({0xc4, 0x02}), true, false).is_discarded());
CHECK(json::from_msgpack(std::vector<uint8_t>({0xc4}), true, false).is_discarded());
}
SECTION("unexpected end inside int with stream")
{
json _;
const std::string data = {static_cast<char>(0xd2u), static_cast<char>(0x12u), static_cast<char>(0x34u), static_cast<char>(0x56u)};
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::istringstream(data, std::ios::binary)),
"[json.exception.parse_error.110] parse error at byte 5: syntax error while parsing MessagePack number: unexpected end of input", json::parse_error&);
}
SECTION("misuse wchar for binary")
{
json _;
// creates 0xd2 after UTF-8 decoding, triggers get_elements in wide_string_input_adapter for code coverage
const std::u32string data = {static_cast<char32_t>(0x0280)};
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(data),
"[json.exception.parse_error.112] parse error at byte 1: wide string type cannot be interpreted as binary data", json::parse_error&);
}
SECTION("unsupported bytes")
{
SECTION("concrete examples")
{
json _;
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0xc1})), "[json.exception.parse_error.112] parse error at byte 1: syntax error while parsing MessagePack value: invalid byte: 0xC1", json::parse_error&);
}
SECTION("all unsupported bytes")
{
for (auto byte :
{
// never used
0xc1
})
{
json _;
CHECK_THROWS_AS(_ = json::from_msgpack(std::vector<uint8_t>({static_cast<uint8_t>(byte)})), json::parse_error&);
CHECK(json::from_msgpack(std::vector<uint8_t>({static_cast<uint8_t>(byte)}), true, false).is_discarded());
}
}
}
SECTION("invalid string in map")
{
json _;
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0x81, 0xff, 0x01})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing MessagePack object key: only string keys are supported, but found an integer; last byte: 0xFF", json::parse_error&);
CHECK(json::from_msgpack(std::vector<uint8_t>({0x81, 0xff, 0x01}), true, false).is_discarded());
}
SECTION("non-string key (see #3381)")
{
// only strings map to JSON object keys; any other key is rejected
// with a message naming its type
const std::vector<std::pair<std::vector<std::uint8_t>, std::string>> cases =
{
{{0x81, 0xC0, 0x01}, "nil; last byte: 0xC0"},
{{0x81, 0xC2, 0x01}, "a boolean; last byte: 0xC2"},
{{0x81, 0xC3, 0x01}, "a boolean; last byte: 0xC3"},
{{0x81, 0xCA, 0x3F, 0x80, 0x00, 0x00, 0x01}, "a float; last byte: 0xCA"},
{{0x81, 0xCB, 0x3F, 0xF0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01}, "a float; last byte: 0xCB"},
{{0x81, 0xC4, 0x00, 0x01}, "a bin; last byte: 0xC4"},
{{0x81, 0xC5, 0x00, 0x00, 0x01}, "a bin; last byte: 0xC5"},
{{0x81, 0xC6, 0x00, 0x00, 0x00, 0x00, 0x01}, "a bin; last byte: 0xC6"},
{{0x81, 0xC7, 0x00, 0x01, 0x01}, "an ext; last byte: 0xC7"},
{{0x81, 0xC8, 0x00, 0x00, 0x01, 0x01}, "an ext; last byte: 0xC8"},
{{0x81, 0xC9, 0x00, 0x00, 0x00, 0x00, 0x01, 0x01}, "an ext; last byte: 0xC9"},
{{0x81, 0xD4, 0x01, 0x00, 0x01}, "an ext; last byte: 0xD4"},
{{0x81, 0xD5, 0x01, 0x00, 0x00, 0x01}, "an ext; last byte: 0xD5"},
{{0x81, 0xD6, 0x01, 0x00, 0x00, 0x00, 0x00, 0x01}, "an ext; last byte: 0xD6"},
{{0x81, 0xD7, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01}, "an ext; last byte: 0xD7"},
{{0x81, 0xD8, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01}, "an ext; last byte: 0xD8"},
{{0x81, 0xCC, 0x01, 0x01}, "an integer; last byte: 0xCC"},
{{0x81, 0xCD, 0x00, 0x01, 0x01}, "an integer; last byte: 0xCD"},
{{0x81, 0xCE, 0x00, 0x00, 0x00, 0x01, 0x01}, "an integer; last byte: 0xCE"},
{{0x81, 0xCF, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x01}, "an integer; last byte: 0xCF"},
{{0x81, 0xD0, 0x01, 0x01}, "an integer; last byte: 0xD0"},
{{0x81, 0xD1, 0x00, 0x01, 0x01}, "an integer; last byte: 0xD1"},
{{0x81, 0xD2, 0x00, 0x00, 0x00, 0x01, 0x01}, "an integer; last byte: 0xD2"},
{{0x81, 0xD3, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x01}, "an integer; last byte: 0xD3"},
{{0x81, 0x00, 0x01}, "an integer; last byte: 0x00"},
{{0x81, 0x7F, 0x01}, "an integer; last byte: 0x7F"},
{{0x81, 0xE0, 0x01}, "an integer; last byte: 0xE0"},
{{0x81, 0x80, 0x01}, "a map; last byte: 0x80"},
{{0x81, 0x8F, 0x01}, "a map; last byte: 0x8F"},
{{0x81, 0xDE, 0x00, 0x00, 0x01}, "a map; last byte: 0xDE"},
{{0x81, 0xDF, 0x00, 0x00, 0x00, 0x00, 0x01}, "a map; last byte: 0xDF"},
{{0x81, 0x90, 0x01}, "an array; last byte: 0x90"},
{{0x81, 0x9F, 0x01}, "an array; last byte: 0x9F"},
{{0x81, 0xDC, 0x00, 0x00, 0x01}, "an array; last byte: 0xDC"},
{{0x81, 0xDD, 0x00, 0x00, 0x00, 0x00, 0x01}, "an array; last byte: 0xDD"},
};
for (const auto& c : cases)
{
CAPTURE(c.first)
const std::string expected = "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing MessagePack object key: only string keys are supported, but found " + c.second;
json _;
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(c.first), expected.c_str(), json::parse_error&);
CHECK(json::from_msgpack(c.first, true, false).is_discarded());
}
json _;
// the unused byte 0xC1 is still reported as a malformed string
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0x81, 0xC1, 0x01})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing MessagePack string: expected length specification (0xA0-0xBF, 0xD9-0xDB); last byte: 0xC1", json::parse_error&);
// a missing key is still reported as the end of input
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0x81})), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing MessagePack string: unexpected end of input", json::parse_error&);
}
SECTION("ill-formed UTF-8 in string (see #5529, #5651)")
{
// the MessagePack specification explicitly allows a str object to
// contain a byte sequence that is not valid UTF-8 and expects a
// deserializer to hand the original bytes back unchanged; this
// library follows that, unlike CBOR/UBJSON/BJData/BSON, whose
// specifications require text strings to be valid UTF-8
// a fixstr of length 2 (0xA0 | 2) whose bytes are not valid UTF-8
// (0xC0 0xAE is an overlong encoding of '.') round-trips byte for
// byte as a string value
const std::vector<uint8_t> ill_formed_value = {0xa2, 0xc0, 0xae};
json j_value;
CHECK_NOTHROW(j_value = json::from_msgpack(ill_formed_value));
REQUIRE(j_value.is_string());
CHECK(j_value.get_ref<const json::string_t&>() == std::string("\xc0\xae"));
CHECK(json::from_msgpack(json::to_msgpack(j_value)) == j_value);
// dump() still requires valid UTF-8 and throws for such a value,
// unless an error handler that replaces or ignores the bytes is
// passed
CHECK_THROWS_AS(utils::ignore_return_value(j_value.dump()), json::type_error&);
// the same bytes as an object key round-trip as well
const std::vector<uint8_t> ill_formed_key = {0x81, 0xa2, 0xc0, 0xae, 0x01};
json j_key;
CHECK_NOTHROW(j_key = json::from_msgpack(ill_formed_key));
REQUIRE(j_key.is_object());
CHECK(j_key.contains(std::string("\xc0\xae")));
CHECK(json::from_msgpack(json::to_msgpack(j_key)) == j_key);
// a MessagePack bin8 blob with the very same bytes is NOT text
// and must still be accepted as-is
json _;
CHECK_NOTHROW(_ = json::from_msgpack(std::vector<uint8_t>({0xc4, 0x02, 0xc0, 0xae})));
CHECK(_ == json::binary(std::vector<std::uint8_t>({0xc0, 0xae})));
// valid UTF-8 must still round-trip
const json j = "h\xc3\xa9llo, w\xc3\xb6rld! \xe6\x97\xa5\xe6\x9c\xac\xe8\xaa\x9e"; // héllo, wörld! 日本語
CHECK(json::from_msgpack(json::to_msgpack(j)) == j);
}
SECTION("strict mode")
{
std::vector<uint8_t> const vec = {0xc0, 0xc0};
SECTION("non-strict mode")
{
const auto result = json::from_msgpack(vec, false);
CHECK(result == json());
}
SECTION("strict mode")
{
json _;
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(vec), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing MessagePack value: expected end of input; last byte: 0xC0", json::parse_error&);
CHECK(json::from_msgpack(vec, true, false).is_discarded());
}
}
}
SECTION("SAX aborts")
{
SECTION("start_array(len)")
{
std::vector<uint8_t> const v = {0x93, 0x01, 0x02, 0x03};
SaxCountdown scp(0);
CHECK(!json::sax_parse(v, &scp, json::input_format_t::msgpack));
}
SECTION("start_object(len)")
{
std::vector<uint8_t> const v = {0x81, 0xa3, 0x66, 0x6F, 0x6F, 0xc2};
SaxCountdown scp(0);
CHECK(!json::sax_parse(v, &scp, json::input_format_t::msgpack));
}
SECTION("key()")
{
std::vector<uint8_t> const v = {0x81, 0xa3, 0x66, 0x6F, 0x6F, 0xc2};
SaxCountdown scp(1);
CHECK(!json::sax_parse(v, &scp, json::input_format_t::msgpack));
}
}
}
TEST_CASE("issue #5405 - array reserve for definite-length MessagePack arrays")
{
#if !defined(JSON_NOEXCEPTION)
// this SECTION relies on catching a thrown exception to distinguish
// which of two acceptable, bounded rejections a hostile header took;
// under JSON_NOEXCEPTION, JSON_THROW never produces a catchable C++
// exception (it aborts instead), so this cannot be tested that way here
SECTION("a huge claimed length with no element data must not over-allocate")
{
// 0xdd: array 32 (four-byte length); claims 0xFFFFFFFF (4294967295)
// elements but provides none. max_size() for a std::vector is far
// larger than this count, so it does not reject the header outright;
// the (capped) reservation must not attempt to allocate space for
// billions of elements before the missing data is detected.
json _;
const std::vector<uint8_t> input = {0xdd, 0xFF, 0xFF, 0xFF, 0xFF};
// On a platform where std::size_t is narrower than 64 bits (e.g.
// 32-bit), the claimed count 0xFFFFFFFF coincides with that
// platform's SIZE_MAX, which some size-narrowing checks treat the
// same as detail::unknown_size(); it may then be rejected before
// the SAX consumer's own max_size() check (out_of_range.408) rather
// than being accepted and only found short of data once the
// (capped) reservation looks for element bytes that were never
// provided (parse_error.110). Either is an acceptable, bounded
// rejection of the hostile header -- the property under test is
// that no path attempts to allocate space for billions of elements.
bool threw = false;
try
{
_ = json::from_msgpack(input);
}
catch (const json::parse_error& e)
{
threw = true;
CHECK(e.id == 110);
CHECK(std::string(e.what()) == "[json.exception.parse_error.110] parse error at byte 6: syntax error while parsing MessagePack value: unexpected end of input");
}
catch (const json::out_of_range& e)
{
threw = true;
CHECK(e.id == 408);
CHECK(std::string(e.what()).find("excessive") != std::string::npos);
}
CHECK(threw);
CHECK(json::from_msgpack(input, true, false).is_discarded());
}
#endif
SECTION("arrays of various sizes decode to the same value as before the reserve optimization")
{
for (const auto size :
{
std::size_t{0}, std::size_t{1}, std::size_t{5}, // small
std::size_t{16384}, // exactly at the reserve cap
std::size_t{20000} // above the reserve cap
})
{
CAPTURE(size)
json j = json::array();
for (std::size_t i = 0; i < size; ++i)
{
j.push_back(static_cast<int>(i % 1000));
}
const auto packed = json::to_msgpack(j);
CHECK(json::from_msgpack(packed) == j);
}
}
SECTION("a user-defined SAX consumer is unaffected by the internal DOM reserve optimization")
{
// the reserve() call is local to json_sax_dom_parser / json_sax_dom_callback_parser;
// a custom SAX consumer that does not touch a DOM array sees identical events
json j = json::array();
for (int i = 0; i < 100; ++i)
{
j.push_back(i);
}
const auto packed = json::to_msgpack(j);
SaxCountdown scp(1000000); // large enough to never trigger an abort
CHECK(json::sax_parse(packed, &scp, json::input_format_t::msgpack));
}
}
TEST_CASE("MessagePack - open arrays reserve only what the input has backed")
{
// an array that announces its size gets storage for (up to 16384 of) its
// elements when it opens; the arrays that are open at the same time must
// not hold more than the input read so far can back
SECTION("nested headers that announce 16384 elements and are never completed")
{
// 200 levels would hold 200 * 16384 * sizeof(json),
// if each of them reserved what it announces
const auto input = utils::repeated({0xDC, 0x40, 0x00}, 200);
utils::counts() = utils::allocation_counts{};
const auto j = utils::counted_json::from_msgpack(input, true, false);
CHECK(j.is_discarded());
CAPTURE(utils::counts().peak)
// the credit is 65,536 values; the base peaks at more than 3 million
CHECK(utils::counts().peak < sizeof(utils::counted_json) * 128 * 1024);
}
SECTION("a completed array is still reserved in one allocation")
{
json array = json::array();
for (int i = 0; i < 10000; ++i)
{
array.push_back(i % 100);
}
{
const auto packed = json::to_msgpack(array);
utils::counts() = utils::allocation_counts{};
const auto j = utils::counted_json::from_msgpack(packed);
CHECK(j.size() == 10000);
// the array and its storage; growing the storage as the elements
// arrive would take more than a dozen allocations
CAPTURE(utils::counts().allocations)
CHECK(utils::counts().allocations <= 4);
}
}
}
TEST_CASE("regression test - MessagePack ext type rejects a subtype that doesn't fit a single byte")
{
// subtype 0-255 must still round-trip correctly (regression guard, pre-existing behavior)
CHECK(json::from_msgpack(json::to_msgpack(json::binary({1, 2}, 0))).get_binary().subtype() == 0);
CHECK(json::from_msgpack(json::to_msgpack(json::binary({1, 2}, 200))).get_binary().subtype() == 200);
CHECK(json::from_msgpack(json::to_msgpack(json::binary({1, 2}, 255))).get_binary().subtype() == 255);
// a subtype > 255 must throw instead of silently truncating
CHECK_THROWS_AS(json::to_msgpack(json::binary({1, 2}, 256)), json::out_of_range);
CHECK_THROWS_WITH_AS(json::to_msgpack(json::binary({1, 2}, 70000)), "[json.exception.out_of_range.415] subtype 70000 is too large for the MessagePack ext type (max 255)", json::out_of_range);
// a binary value with no subtype at all must be unaffected
CHECK(json::from_msgpack(json::to_msgpack(json::binary({1, 2}))).get_binary().has_subtype() == false);
}
// use this testcase outside [hide] to run it with Valgrind
TEST_CASE("MessagePack nesting does not consume the call stack")
{
// Reading a container used to call back into the value reader once per
// element, so the native call stack grew with the nesting depth of the
// input: one frame per byte for repeated 0x91 (a one-element array), which
// crashes the process long before the input is exhausted (#5104). The
// containers are kept on a heap stack now.
//
// Note that deeply nested values must not be compared, copied or dumped
// here: those operations are still recursive, and would reintroduce the
// very crash this checks for. Depth is measured by descending instead.
SECTION("an unterminated chain is reported, not crashed on")
{
json _;
const std::vector<uint8_t> input(300000, 0x91);
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(input), "[json.exception.parse_error.110] parse error at byte 300001: syntax error while parsing MessagePack value: unexpected end of input", json::parse_error&);
CHECK(json::from_msgpack(input, true, false).is_discarded());
}
SECTION("a well-formed deep value is read through the SAX interface")
{
std::vector<uint8_t> input(300000, 0x91);
input.push_back(0x01); // innermost value
SaxCountdown accept_all(600001);
CHECK(json::sax_parse(input, &accept_all, json::input_format_t::msgpack));
}
SECTION("a well-formed deep value is read into a value")
{
const std::size_t depth = 10000;
std::vector<uint8_t> input(depth, 0x91);
input.push_back(0x01);
json j = json::from_msgpack(input);
std::size_t measured = 0;
const json* p = &j;
while (p->is_array() && !p->empty())
{
p = &p->front();
++measured;
}
CHECK(measured == depth);
CHECK(p->is_number());
}
SECTION("containers are still read the same way")
{
CHECK(json::from_msgpack(std::vector<uint8_t>({0x90})) == json::array());
CHECK(json::from_msgpack(std::vector<uint8_t>({0x80})) == json::object());
CHECK(json::from_msgpack(std::vector<uint8_t>({0x92, 0x90, 0x80})) == json({json::array(), json::object()}));
CHECK(json::from_msgpack(std::vector<uint8_t>({0x91, 0x91, 0x91, 0x90})) == json({{{json::array()}}}));
CHECK(json::from_msgpack(std::vector<uint8_t>({0x81, 0xA1, 'a', 0x81, 0xA1, 'b', 0x92, 0x01, 0x02})) == json({{"a", {{"b", {1, 2}}}}}));
// array 16 and map 32, i.e. the counted forms
CHECK(json::from_msgpack(std::vector<uint8_t>({0xDC, 0x00, 0x02, 0x01, 0x02})) == json({1, 2}));
CHECK(json::from_msgpack(std::vector<uint8_t>({0xDF, 0x00, 0x00, 0x00, 0x01, 0xA1, 'k', 0xC3})) == json({{"k", true}}));
}
}
TEST_CASE("MessagePack input that cannot be read is discarded by every overload")
{
std::vector<std::uint8_t> input = json::to_msgpack(json({{"a", {1, 2}}}));
input.pop_back();
json _;
CHECK_THROWS_AS(_ = json::from_msgpack(input.begin(), input.end()), json::parse_error&);
CHECK(json::from_msgpack(input, true, false).is_discarded());
CHECK(json::from_msgpack(input.begin(), input.end(), true, false).is_discarded());
#ifndef JSON_TEST_DEPRECATED_FUNCTIONS_DELETED
CHECK(json::from_msgpack(input.data(), input.size(), true, false).is_discarded());
CHECK(json::from_msgpack({input.data(), input.size()}, true, false).is_discarded());
#endif
}
TEST_CASE("MessagePack SAX parsing stops at every event")
{
// Containers are opened and closed by the loop that reads them; a SAX
// handler that rejects any event - including the end of a nested
// container - must stop the parse right there.
const auto count_events = [](const std::vector<std::uint8_t>& input)
{
int events = 0;
while (true)
{
SaxCountdown scp(events);
if (json::sax_parse(input, &scp, json::input_format_t::msgpack))
{
return events;
}
++events;
REQUIRE(events < 1000);
}
};
// 20 events: every container kind closes inside another one
const json j = json::parse(R"({"a": [1, {"b": []}], "c": {"d": [[2]]}})");
CHECK(count_events(json::to_msgpack(j)) == 20);
}
TEST_CASE("single MessagePack roundtrip")
{
SECTION("sample.json")
{
std::string const filename = TEST_DATA_DIRECTORY "/json_testsuite/sample.json";
// parse JSON file
std::ifstream f_json(filename);
const json j1 = json::parse(f_json);
// parse MessagePack file
auto packed = utils::read_binary_file(filename + ".msgpack");
json j2;
CHECK_NOTHROW(j2 = json::from_msgpack(packed));
// compare parsed JSON values
CHECK(j1 == j2);
SECTION("roundtrips")
{
SECTION("std::ostringstream")
{
std::basic_ostringstream<char> ss;
json::to_msgpack(j1, ss);
json j3 = json::from_msgpack(ss.str());
CHECK(j1 == j3);
}
SECTION("std::string")
{
std::string s;
json::to_msgpack(j1, s);
json j3 = json::from_msgpack(s);
CHECK(j1 == j3);
}
}
// check with different start index
packed.insert(packed.begin(), 5, 0xff);
CHECK(j1 == json::from_msgpack(packed.begin() + 5, packed.end()));
}
}
TEST_CASE("Parse MessagePack directly from a file using iterator and sentinel")
{
std::string const filename = TEST_DATA_DIRECTORY "/json_testsuite/sample.json.msgpack";
std::ifstream file(filename, std::ios::binary);
const std::istreambuf_iterator<char> first(file);
const json parsed = json::from_msgpack(first, utils::istreambuf_sentinel{});
CHECK((parsed.is_object() || parsed.is_array()));
}
TEST_CASE("MessagePack round-trip invariants")
{
// This checks what the parse_msgpack_fuzzer driver checks (see
// tests/src/fuzzer-parse_msgpack.cpp), so that a regression shows up in
// CI rather than as an OSS-Fuzz report: anything from_msgpack() returns
// (j1) can be serialized, parsed back (j2), and serialized again to
// reproduce the exact bytes.
for (const auto& j0 : utils::round_trip_corpus::values())
{
json j1;
try
{
// turn the corpus value into a value as from_msgpack() returns it
j1 = json::from_msgpack(json::to_msgpack(j0));
}
catch (const json::exception&)
{
// the fuzzer driver only ever sees values from_msgpack() actually
// produced, so skip corpus values that do not survive the
// round trip here, too
continue;
}
INFO("j1 = " << j1.dump());
const std::vector<std::uint8_t> vec = json::to_msgpack(j1);
json j2;
// anything the library writes must be parsable by the library
REQUIRE_NOTHROW(j2 = json::from_msgpack(vec));
CHECK(json::to_msgpack(j2) == vec);
}
}
TEST_CASE("MessagePack roundtrips" * doctest::skip())
{
SECTION("input from msgpack-python")
{
// most of these are excluded due to differences in key order (not a real problem)
std::set<std::string> exclude_packed;
exclude_packed.insert(TEST_DATA_DIRECTORY "/json.org/1.json");
exclude_packed.insert(TEST_DATA_DIRECTORY "/json.org/2.json");
exclude_packed.insert(TEST_DATA_DIRECTORY "/json.org/3.json");
exclude_packed.insert(TEST_DATA_DIRECTORY "/json.org/4.json");
exclude_packed.insert(TEST_DATA_DIRECTORY "/json.org/5.json");
exclude_packed.insert(TEST_DATA_DIRECTORY "/json_testsuite/sample.json"); // kills AppVeyor
exclude_packed.insert(TEST_DATA_DIRECTORY "/json_tests/pass1.json");
exclude_packed.insert(TEST_DATA_DIRECTORY "/regression/working_file.json");
exclude_packed.insert(TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_object.json");
exclude_packed.insert(TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_object_basic.json");
exclude_packed.insert(TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_object_duplicated_key.json");
exclude_packed.insert(TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_object_long_strings.json");
exclude_packed.insert(TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_object_simple.json");
exclude_packed.insert(TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_object_string_unicode.json");
for (std::string filename :
{
TEST_DATA_DIRECTORY "/json_nlohmann_tests/all_unicode.json",
TEST_DATA_DIRECTORY "/json.org/1.json",
TEST_DATA_DIRECTORY "/json.org/2.json",
TEST_DATA_DIRECTORY "/json.org/3.json",
TEST_DATA_DIRECTORY "/json.org/4.json",
TEST_DATA_DIRECTORY "/json.org/5.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip01.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip02.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip03.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip04.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip05.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip06.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip07.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip08.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip09.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip10.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip11.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip12.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip13.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip14.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip15.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip16.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip17.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip18.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip19.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip20.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip21.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip22.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip23.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip24.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip25.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip26.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip27.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip28.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip29.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip30.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip31.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip32.json",
TEST_DATA_DIRECTORY "/json_testsuite/sample.json", // kills AppVeyor
TEST_DATA_DIRECTORY "/json_tests/pass1.json",
TEST_DATA_DIRECTORY "/json_tests/pass2.json",
TEST_DATA_DIRECTORY "/json_tests/pass3.json",
TEST_DATA_DIRECTORY "/regression/floats.json",
TEST_DATA_DIRECTORY "/regression/signed_ints.json",
TEST_DATA_DIRECTORY "/regression/unsigned_ints.json",
TEST_DATA_DIRECTORY "/regression/working_file.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_array_arraysWithSpaces.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_array_empty-string.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_array_empty.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_array_ending_with_newline.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_array_false.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_array_heterogeneous.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_array_null.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_array_with_1_and_newline.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_array_with_leading_space.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_array_with_several_null.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_array_with_trailing_space.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_0e+1.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_0e1.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_after_space.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_double_close_to_zero.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_double_huge_neg_exp.json",
//TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_huge_exp.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_int_with_exp.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_minus_zero.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_negative_int.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_negative_one.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_negative_zero.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_real_capital_e.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_real_capital_e_neg_exp.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_real_capital_e_pos_exp.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_real_exponent.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_real_fraction_exponent.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_real_neg_exp.json",
//TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_real_neg_overflow.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_real_pos_exponent.json",
//TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_real_pos_overflow.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_real_underflow.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_simple_int.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_simple_real.json",
//TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_too_big_neg_int.json",
//TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_too_big_pos_int.json",
//TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_number_very_big_negative_int.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_object.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_object_basic.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_object_duplicated_key.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_object_duplicated_key_and_value.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_object_empty.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_object_empty_key.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_object_escaped_null_in_key.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_object_extreme_numbers.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_object_long_strings.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_object_simple.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_object_string_unicode.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_object_with_newlines.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_1_2_3_bytes_UTF-8_sequences.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_UTF-16_Surrogates_U+1D11E_MUSICAL_SYMBOL_G_CLEF.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_accepted_surrogate_pair.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_accepted_surrogate_pairs.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_allowed_escapes.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_backslash_and_u_escaped_zero.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_backslash_doublequotes.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_comments.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_double_escape_a.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_double_escape_n.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_escaped_control_character.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_escaped_noncharacter.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_in_array.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_in_array_with_leading_space.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_last_surrogates_1_and_2.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_newline_uescaped.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_nonCharacterInUTF-8_U+10FFFF.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_nonCharacterInUTF-8_U+1FFFF.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_nonCharacterInUTF-8_U+FFFF.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_null_escape.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_one-byte-utf-8.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_pi.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_simple_ascii.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_space.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_three-byte-utf-8.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_two-byte-utf-8.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_u+2028_line_sep.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_u+2029_par_sep.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_uEscape.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_unescaped_char_delete.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_unicode.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_unicodeEscapedBackslash.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_unicode_2.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_unicode_U+200B_ZERO_WIDTH_SPACE.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_unicode_U+2064_invisible_plus.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_unicode_escaped_double_quote.json",
// TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_utf16.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_utf8.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_string_with_del_character.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_structure_lonely_false.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_structure_lonely_int.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_structure_lonely_negative_real.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_structure_lonely_null.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_structure_lonely_string.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_structure_lonely_true.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_structure_string_empty.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_structure_trailing_newline.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_structure_true_in_array.json",
TEST_DATA_DIRECTORY "/nst_json_testsuite/test_parsing/y_structure_whitespace_array.json"
})
{
CAPTURE(filename)
std::ifstream f_json(filename);
const json j1 = json::parse(f_json);
auto packed = utils::read_binary_file(filename + ".msgpack");
{
INFO_WITH_TEMP(filename + ": std::vector<uint8_t>");
json j2;
CHECK_NOTHROW(j2 = json::from_msgpack(packed));
CHECK(j1 == j2);
}
{
INFO_WITH_TEMP(filename + ": std::ifstream");
std::ifstream f_msgpack(filename + ".msgpack", std::ios::binary);
json j2;
CHECK_NOTHROW(j2 = json::from_msgpack(f_msgpack));
CHECK(j1 == j2);
}
#ifndef JSON_TEST_DEPRECATED_FUNCTIONS_DELETED
{
INFO_WITH_TEMP(filename + ": uint8_t* and size");
json j2;
CHECK_NOTHROW(j2 = json::from_msgpack({packed.data(), packed.size()}));
CHECK(j1 == j2);
}
#endif
{
INFO_WITH_TEMP(filename + ": output to output adapters");
if (exclude_packed.count(filename) == 0u)
{
{
INFO_WITH_TEMP(filename + ": output adapters: std::vector<uint8_t>");
std::vector<uint8_t> vec;
json::to_msgpack(j1, vec);
CHECK(vec == packed);
}
}
}
}
}
}
TEST_CASE("MessagePack large strings and binaries (chunked reader)")
{
// get_msgpack_string()/get_msgpack_binary() both read through get_binary(),
// which reads in bounded chunks (binary_reader.hpp, chunk_size == 4096);
// make sure roundtripping is correct for lengths around and beyond that
// chunk size, for both vector (iterator) and pointer inputs.
for (const std::size_t len :
{
std::size_t{0}, std::size_t{1}, std::size_t{4095}, std::size_t{4096},
std::size_t{4097}, std::size_t{8192}, std::size_t{100000}
})
{
CAPTURE(len)
// string
const json j_string = std::string(len, 'x');
const std::vector<std::uint8_t> v_string = json::to_msgpack(j_string);
CHECK(json::from_msgpack(v_string) == j_string);
// pointer input exercises the std::memcpy fast path
CHECK(json::from_msgpack(reinterpret_cast<const char*>(v_string.data()),
reinterpret_cast<const char*>(v_string.data()) + v_string.size()) == j_string);
// binary
const json j_binary = json::binary(std::vector<std::uint8_t>(len, 0xCD));
const std::vector<std::uint8_t> v_binary = json::to_msgpack(j_binary);
CHECK(json::from_msgpack(v_binary) == j_binary);
CHECK(json::from_msgpack(reinterpret_cast<const char*>(v_binary.data()),
reinterpret_cast<const char*>(v_binary.data()) + v_binary.size()) == j_binary);
// a truncated payload must still be reported as an error
if (len > 16)
{
std::vector<std::uint8_t> truncated = v_string;
truncated.resize(truncated.size() - 8);
json _;
CHECK_THROWS_AS(_ = json::from_msgpack(truncated), json::parse_error);
}
}
}
TEST_CASE("MessagePack arrays of scalars from contiguous and non-contiguous input")
{
// contiguous input reads the integers, floats, booleans, and nils an
// array begins with in one loop, other input value by value; both must
// give the same events, value, or error for the complete document and for
// every truncation of it, and stop at every event a SAX handler rejects
const std::vector<std::uint8_t> msgpack =
{
0x83, // map(3)
0xA1, 'a', // "a"
0xDC, 0x00, 0x16, // array 16 (22)
0x00, 0x7F, 0xCC, 0xFF, 0xCD, 0x01, 0x00, // 0, 127, 255, 256
0xCE, 0x00, 0x01, 0x00, 0x00, // 65536
0xCF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, // 18446744073709551615
0xFF, 0xE0, 0xD0, 0x80, 0xD1, 0xFF, 0x00, // -1, -32, -128, -256
0xD2, 0xFF, 0xFE, 0xFF, 0xFF, // -65537
0xD3, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // -9223372036854775808
0xC2, 0xC3, 0xC0, // false, true, nil
0xCA, 0x3F, 0xC0, 0x00, 0x00, // 1.5 (float 32)
0xCB, 0xBF, 0xB9, 0x99, 0x99, 0x99, 0x99, 0x99, 0x9A, // -0.1
0xA1, 'x', // "x" (ends the loop)
0x01, // 1
0xCB, 0x40, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // 2.5
0xD9, 0x01, 'y', // "y" (str 8)
0xC0, // nil
0xA1, 'b', // "b"
0x93, // fixarray(3)
0x92, 0xCB, 0x3F, 0xF4, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // [1.25,
0xCB, 0xC0, 0x0E, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // -3.75]
0x92, 0xCB, 0x3F, 0xF4, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // [1.25,
0xCB, 0xC0, 0x0E, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // -3.75]
0x90, // []
0xA1, 'c', // "c"
0x94, 0x80, 0x01, 0xC4, 0x01, 0x05, 0x91, 0x03 // [{}, 1, bin 8, [3]]
};
const json j = json::from_msgpack(msgpack);
CHECK(j["a"].size() == 22);
CHECK(j["a"][5] == 18446744073709551615u);
CHECK(j["a"][7] == -32);
CHECK(j["a"][11] == (std::numeric_limits<std::int64_t>::min)());
CHECK(j["a"][15] == 1.5);
CHECK(j["a"][16] == -0.1);
CHECK(j["a"][17] == "x");
CHECK(j["b"] == json::parse("[[1.25, -3.75], [1.25, -3.75], []]"));
CHECK(j["c"][2].is_binary());
for (std::size_t length = 0; length <= msgpack.size(); ++length)
{
CAPTURE(length)
const std::vector<std::uint8_t> contiguous(msgpack.begin(), msgpack.begin() + static_cast<std::ptrdiff_t>(length));
const std::list<std::uint8_t> bytewise(contiguous.begin(), contiguous.end());
CHECK(json::from_msgpack(contiguous, true, false).dump() == json::from_msgpack(bytewise.begin(), bytewise.end(), true, false).dump());
#if !defined(JSON_NOEXCEPTION)
std::string contiguous_error;
std::string bytewise_error;
try
{
json _;
_ = json::from_msgpack(contiguous);
}
catch (const json::parse_error& e)
{
contiguous_error = e.what();
}
try
{
json _;
_ = json::from_msgpack(bytewise.begin(), bytewise.end());
}
catch (const json::parse_error& e)
{
bytewise_error = e.what();
}
CHECK(contiguous_error == bytewise_error);
#endif
SaxEventLogger from_contiguous;
SaxEventLogger from_bytewise;
CHECK(json::sax_parse(contiguous, &from_contiguous, json::input_format_t::msgpack)
== json::sax_parse(bytewise.begin(), bytewise.end(), &from_bytewise, json::input_format_t::msgpack));
CHECK(from_contiguous.events == from_bytewise.events);
}
for (int events = 0; events <= 60; ++events)
{
CAPTURE(events)
const std::list<std::uint8_t> bytewise(msgpack.begin(), msgpack.end());
SaxCountdown from_contiguous(events);
SaxCountdown from_bytewise(events);
CHECK(json::sax_parse(msgpack, &from_contiguous, json::input_format_t::msgpack)
== json::sax_parse(bytewise.begin(), bytewise.end(), &from_bytewise, json::input_format_t::msgpack));
}
}
TEST_CASE("MessagePack arrays of one element from contiguous and non-contiguous input")
{
// contiguous input leaves an array of one element to the general path
// rather than to the loop that reads arrays of scalars; both inputs must
// still give the same events, value, or error for the complete document
// and for every truncation of it, with enough bytes after each array that
// the loop could have been entered
const json j = json::parse(R"({"a": [[[[1.5]]]], "b": [7], "c": [{"x": 2.5}], "d": ["s"], "e": [[1, 2]], "f": [null], "g": 1.25})");
const std::vector<std::uint8_t> bytes = json::to_msgpack(j);
CHECK(json::from_msgpack(bytes) == j);
for (std::size_t length = 0; length <= bytes.size(); ++length)
{
CAPTURE(length)
const std::vector<std::uint8_t> contiguous(bytes.begin(), bytes.begin() + static_cast<std::ptrdiff_t>(length));
const std::list<std::uint8_t> bytewise(contiguous.begin(), contiguous.end());
CHECK(json::from_msgpack(contiguous, true, false).dump() == json::from_msgpack(bytewise.begin(), bytewise.end(), true, false).dump());
SaxEventLogger from_contiguous;
SaxEventLogger from_bytewise;
CHECK(json::sax_parse(contiguous, &from_contiguous, json::input_format_t::msgpack, true, false)
== json::sax_parse(bytewise.begin(), bytewise.end(), &from_bytewise, json::input_format_t::msgpack, true, false));
CHECK(from_contiguous.events == from_bytewise.events);
}
}