amorfo77/json
1
0 Fork 0
mirror of https://github.com/nlohmann/json.git synced 2026-10-27 08:18:55 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
0c3570d44a95f096b2c5131c0b952899084261c9
json/tests/src/unit-ubjson.cpp
T
Niels Lohmann 0c3570d44a Validate the UBJSON/BJData container type marker for count 0; fix UBJSON docs (#5929) 
* Validate the optimized-container type marker for count 0 in UBJSON and BJData

The '$' type marker of an optimized container was only checked when an
element was read, so containers with count 0 accepted any marker:
`[$x#i0`, `{$x#i0` and `[$E#i0` parsed as empty containers although a
non-empty container with the same marker is rejected and the UBJSON
specification requires a legal marker after '$'.

get_ubjson_size_type() now checks the marker right after '$' against the
allowed set (UBJSON: Z N T F i U I l L d D H C S [ {; BJData: B U i u I m
l M L h d D C, with the existing exclusion of T F Z N S H [ {) and reports
parse_error.112 "marker ... is not a permitted optimized array type".

Input that was accepted and is now rejected: an optimized container with
count 0 and an invalid '$' type marker. Valid input is unaffected. Two
existing tests that used an invalid marker incidentally ("[$#" and a
negative-count case) were adapted.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Correct and complete the UBJSON documentation

Fix statements that disagree with the behavior (checked with probes):
- `Z` is null; the unused marker is `N`.
- `$` requires `#`; use_type with use_size=false throws other_error.502 only
  for non-empty arrays, objects and binary values.
- NaN and infinity are written as IEEE 754 float64 and read back unchanged;
  the UBJSON specification encodes infinity as null.
- `H` is documented (semantics, parse_error.115, out_of_range.406); the
  "complete mapping" claim is qualified.
- use_type "same type" means same marker.
- Long Z/T/F arrays lose only the `$` annotation; the `#` count and the
  element markers are kept.
- from_ubjson lists the 112, 113, 115, 406 and 408 causes that occur for
  UBJSON, without the BJData-only ND-array wording.
- Duplicate object keys keep the last value; the `$` type marker is
  validated even for count 0.
- Smaller fixes: float32 row, "map" -> "object", https link, exception 110
  strict case, to_ubjson history of discarded values.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Regenerate single_include and apply astyle

Result of make amalgamate (including make pretty) for the preceding changes.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Address review of the UBJSON documentation

Clarify that empty containers carry no $ annotation of their own and note that a counted N object is not supported.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: format tables, use the Oxford comma, drop discarded-value remarks

Reformat the two UBJSON mapping tables that this PR changed (uniform column
widths, padded cells), use the serial comma in lists of three or more in the
added text, and restore the original type_error.321 bullet in to_ubjson.md.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix narrowing of the optimized-container count in BJData and UBJSON tests

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-26 18:25:45 +01:00

3403 lines
152 KiB
C++
Raw Blame History

// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++ (supporting code)
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
#include "doctest_compatibility.h"
// capture whether JSON_DELETE_DEPRECATED_FUNCTIONS was enabled on the command
// line *before* including json.hpp, since the library #undefs it once the header
// has been fully processed (see include/nlohmann/detail/macro_unscope.hpp); the
// tests of deprecated functions are skipped if these functions are deleted
#if defined(JSON_DELETE_DEPRECATED_FUNCTIONS) && (JSON_DELETE_DEPRECATED_FUNCTIONS == 1)
#define JSON_TEST_DEPRECATED_FUNCTIONS_DELETED
#endif
#include <nlohmann/json.hpp>
using nlohmann::json;
#include <iostream>
#include <fstream>
#include <set>
#include "make_test_data_available.hpp"
#include "round_trip_corpus.hpp"
#include "test_utils.hpp"
#include "sax_countdown.hpp"
using utils::SaxCountdown;
TEST_CASE("UBJSON")
{
SECTION("individual values")
{
SECTION("discarded")
{
// a discarded value cannot be serialized to UBJSON
json const j = json::value_t::discarded;
CHECK_THROWS_WITH_AS(json::to_ubjson(j), "[json.exception.type_error.321] cannot serialize discarded value to UBJSON", json::type_error&);
}
SECTION("discarded values nested in a container")
{
json const discarded = json::value_t::discarded;
SECTION("in an array")
{
json const j = {1, discarded, 2};
#if JSON_DIAGNOSTICS
CHECK_THROWS_WITH_AS(json::to_ubjson(j), "[json.exception.type_error.321] (/1) cannot serialize discarded value to UBJSON", json::type_error&);
#else
CHECK_THROWS_WITH_AS(json::to_ubjson(j), "[json.exception.type_error.321] cannot serialize discarded value to UBJSON", json::type_error&);
#endif
}
SECTION("as an object value")
{
json j;
j["a"] = 1;
j["b"] = discarded;
#if JSON_DIAGNOSTICS
CHECK_THROWS_WITH_AS(json::to_ubjson(j), "[json.exception.type_error.321] (/b) cannot serialize discarded value to UBJSON", json::type_error&);
#else
CHECK_THROWS_WITH_AS(json::to_ubjson(j), "[json.exception.type_error.321] cannot serialize discarded value to UBJSON", json::type_error&);
#endif
}
SECTION("nested deeper (array in object in array)")
{
json inner_array = {1, discarded};
json middle_object;
middle_object["x"] = inner_array;
json const j = {middle_object};
#if JSON_DIAGNOSTICS
CHECK_THROWS_WITH_AS(json::to_ubjson(j), "[json.exception.type_error.321] (/0/x/1) cannot serialize discarded value to UBJSON", json::type_error&);
#else
CHECK_THROWS_WITH_AS(json::to_ubjson(j), "[json.exception.type_error.321] cannot serialize discarded value to UBJSON", json::type_error&);
#endif
}
SECTION("optimized array of all-discarded elements")
{
json const j = {discarded, discarded};
#if JSON_DIAGNOSTICS
CHECK_THROWS_WITH_AS(json::to_ubjson(j, true, true), "[json.exception.type_error.321] (/0) cannot serialize discarded value to UBJSON", json::type_error&);
#else
CHECK_THROWS_WITH_AS(json::to_ubjson(j, true, true), "[json.exception.type_error.321] cannot serialize discarded value to UBJSON", json::type_error&);
#endif
}
}
SECTION("null")
{
json const j = nullptr;
std::vector<uint8_t> expected = {'Z'};
const auto result = json::to_ubjson(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
SECTION("boolean")
{
SECTION("true")
{
json const j = true;
std::vector<uint8_t> const expected = {'T'};
const auto result = json::to_ubjson(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
SECTION("false")
{
json const j = false;
std::vector<uint8_t> const expected = {'F'};
const auto result = json::to_ubjson(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
SECTION("number")
{
SECTION("signed")
{
SECTION("-9223372036854775808..-2147483649 (int64)")
{
std::vector<int64_t> const numbers
{
(std::numeric_limits<int64_t>::min)(),
-1000000000000000000LL,
-100000000000000000LL,
-10000000000000000LL,
-1000000000000000LL,
-100000000000000LL,
-10000000000000LL,
-1000000000000LL,
-100000000000LL,
-10000000000LL,
-2147483649LL,
};
for (auto i : numbers)
{
CAPTURE(i)
// create JSON value with integer number
json const j = i;
// check type
CHECK(j.is_number_integer());
// create expected byte vector
std::vector<uint8_t> const expected
{
static_cast<uint8_t>('L'),
static_cast<uint8_t>((i >> 56) & 0xff),
static_cast<uint8_t>((i >> 48) & 0xff),
static_cast<uint8_t>((i >> 40) & 0xff),
static_cast<uint8_t>((i >> 32) & 0xff),
static_cast<uint8_t>((i >> 24) & 0xff),
static_cast<uint8_t>((i >> 16) & 0xff),
static_cast<uint8_t>((i >> 8) & 0xff),
static_cast<uint8_t>(i & 0xff),
};
// compare result + size
const auto result = json::to_ubjson(j);
CHECK(result == expected);
CHECK(result.size() == 9);
// check individual bytes
CHECK(result[0] == 'L');
int64_t const restored = (static_cast<int64_t>(result[1]) << 070) +
(static_cast<int64_t>(result[2]) << 060) +
(static_cast<int64_t>(result[3]) << 050) +
(static_cast<int64_t>(result[4]) << 040) +
(static_cast<int64_t>(result[5]) << 030) +
(static_cast<int64_t>(result[6]) << 020) +
(static_cast<int64_t>(result[7]) << 010) +
static_cast<int64_t>(result[8]);
CHECK(restored == i);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
SECTION("-2147483648..-32769 (int32)")
{
std::vector<int32_t> numbers;
numbers.push_back(-32769);
numbers.push_back(-100000);
numbers.push_back(-1000000);
numbers.push_back(-10000000);
numbers.push_back(-100000000);
numbers.push_back(-1000000000);
numbers.push_back(-2147483647 - 1); // https://stackoverflow.com/a/29356002/266378
for (auto i : numbers)
{
CAPTURE(i)
// create JSON value with integer number
json const j = i;
// check type
CHECK(j.is_number_integer());
// create expected byte vector
std::vector<uint8_t> const expected
{
static_cast<uint8_t>('l'),
static_cast<uint8_t>((i >> 24) & 0xff),
static_cast<uint8_t>((i >> 16) & 0xff),
static_cast<uint8_t>((i >> 8) & 0xff),
static_cast<uint8_t>(i & 0xff),
};
// compare result + size
const auto result = json::to_ubjson(j);
CHECK(result == expected);
CHECK(result.size() == 5);
// check individual bytes
CHECK(result[0] == 'l');
int32_t const restored = (static_cast<int32_t>(result[1]) << 030) +
(static_cast<int32_t>(result[2]) << 020) +
(static_cast<int32_t>(result[3]) << 010) +
static_cast<int32_t>(result[4]);
CHECK(restored == i);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
SECTION("-32768..-129 (int16)")
{
for (int32_t i = -32768; i <= -129; i = utils::next_integer_sample(i, -129, 7))
{
CAPTURE(i)
// create JSON value with integer number
json const j = i;
// check type
CHECK(j.is_number_integer());
// create expected byte vector
std::vector<uint8_t> const expected
{
static_cast<uint8_t>('I'),
static_cast<uint8_t>((i >> 8) & 0xff),
static_cast<uint8_t>(i & 0xff),
};
// compare result + size
const auto result = json::to_ubjson(j);
CHECK(result == expected);
CHECK(result.size() == 3);
// check individual bytes
CHECK(result[0] == 'I');
auto const restored = static_cast<int16_t>(((result[1] << 8) + result[2]));
CHECK(restored == i);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
SECTION("-9263 (int16)")
{
json const j = -9263;
std::vector<uint8_t> expected = {'I', 0xdb, 0xd1};
// compare result + size
const auto result = json::to_ubjson(j);
CHECK(result == expected);
CHECK(result.size() == 3);
// check individual bytes
CHECK(result[0] == 'I');
auto const restored = static_cast<int16_t>(((result[1] << 8) + result[2]));
CHECK(restored == -9263);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
SECTION("-128..-1 (int8)")
{
for (auto i = -128; i <= -1; ++i)
{
CAPTURE(i)
// create JSON value with integer number
json const j = i;
// check type
CHECK(j.is_number_integer());
// create expected byte vector
std::vector<uint8_t> const expected
{
'i',
static_cast<uint8_t>(i),
};
// compare result + size
const auto result = json::to_ubjson(j);
CHECK(result == expected);
CHECK(result.size() == 2);
// check individual bytes
CHECK(result[0] == 'i');
CHECK(static_cast<int8_t>(result[1]) == i);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
SECTION("0..127 (int8)")
{
for (size_t i = 0; i <= 127; ++i)
{
CAPTURE(i)
// create JSON value with integer number
json j = -1;
j.get_ref<json::number_integer_t&>() = static_cast<json::number_integer_t>(i);
// check type
CHECK(j.is_number_integer());
// create expected byte vector
std::vector<uint8_t> const expected
{
static_cast<uint8_t>('i'),
static_cast<uint8_t>(i),
};
// compare result + size
const auto result = json::to_ubjson(j);
CHECK(result == expected);
CHECK(result.size() == 2);
// check individual bytes
CHECK(result[0] == 'i');
CHECK(result[1] == i);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
SECTION("128..255 (uint8)")
{
for (size_t i = 128; i <= 255; ++i)
{
CAPTURE(i)
// create JSON value with integer number
json j = -1;
j.get_ref<json::number_integer_t&>() = static_cast<json::number_integer_t>(i);
// check type
CHECK(j.is_number_integer());
// create expected byte vector
std::vector<uint8_t> const expected
{
static_cast<uint8_t>('U'),
static_cast<uint8_t>(i),
};
// compare result + size
const auto result = json::to_ubjson(j);
CHECK(result == expected);
CHECK(result.size() == 2);
// check individual bytes
CHECK(result[0] == 'U');
CHECK(result[1] == i);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
SECTION("256..32767 (int16)")
{
for (size_t i = 256; i <= 32767; i = utils::next_integer_sample(i, static_cast<size_t>(32767), static_cast<size_t>(7)))
{
CAPTURE(i)
// create JSON value with integer number
json j = -1;
j.get_ref<json::number_integer_t&>() = static_cast<json::number_integer_t>(i);
// check type
CHECK(j.is_number_integer());
// create expected byte vector
std::vector<uint8_t> const expected
{
static_cast<uint8_t>('I'),
static_cast<uint8_t>((i >> 8) & 0xff),
static_cast<uint8_t>(i & 0xff),
};
// compare result + size
const auto result = json::to_ubjson(j);
CHECK(result == expected);
CHECK(result.size() == 3);
// check individual bytes
CHECK(result[0] == 'I');
auto const restored = static_cast<uint16_t>((static_cast<uint8_t>(result[1]) * 256) + static_cast<uint8_t>(result[2]));
CHECK(restored == i);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
SECTION("65536..2147483647 (int32)")
{
for (uint32_t i :
{
65536u, 77777u, 1048576u
})
{
CAPTURE(i)
// create JSON value with integer number
json j = -1;
j.get_ref<json::number_integer_t&>() = static_cast<json::number_integer_t>(i);
// check type
CHECK(j.is_number_integer());
// create expected byte vector
std::vector<uint8_t> const expected
{
'l',
static_cast<uint8_t>((i >> 24) & 0xff),
static_cast<uint8_t>((i >> 16) & 0xff),
static_cast<uint8_t>((i >> 8) & 0xff),
static_cast<uint8_t>(i & 0xff),
};
// compare result + size
const auto result = json::to_ubjson(j);
CHECK(result == expected);
CHECK(result.size() == 5);
// check individual bytes
CHECK(result[0] == 'l');
uint32_t const restored = (static_cast<uint32_t>(result[1]) << 030) +
(static_cast<uint32_t>(result[2]) << 020) +
(static_cast<uint32_t>(result[3]) << 010) +
static_cast<uint32_t>(result[4]);
CHECK(restored == i);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
SECTION("2147483648..9223372036854775807 (int64)")
{
std::vector<uint64_t> const v = {2147483648ul, 9223372036854775807ul};
for (uint64_t i : v)
{
CAPTURE(i)
// create JSON value with integer number
json j = -1;
j.get_ref<json::number_integer_t&>() = static_cast<json::number_integer_t>(i);
// check type
CHECK(j.is_number_integer());
// create expected byte vector
std::vector<uint8_t> const expected
{
'L',
static_cast<uint8_t>((i >> 070) & 0xff),
static_cast<uint8_t>((i >> 060) & 0xff),
static_cast<uint8_t>((i >> 050) & 0xff),
static_cast<uint8_t>((i >> 040) & 0xff),
static_cast<uint8_t>((i >> 030) & 0xff),
static_cast<uint8_t>((i >> 020) & 0xff),
static_cast<uint8_t>((i >> 010) & 0xff),
static_cast<uint8_t>(i & 0xff),
};
// compare result + size
const auto result = json::to_ubjson(j);
CHECK(result == expected);
CHECK(result.size() == 9);
// check individual bytes
CHECK(result[0] == 'L');
uint64_t const restored = (static_cast<uint64_t>(result[1]) << 070) +
(static_cast<uint64_t>(result[2]) << 060) +
(static_cast<uint64_t>(result[3]) << 050) +
(static_cast<uint64_t>(result[4]) << 040) +
(static_cast<uint64_t>(result[5]) << 030) +
(static_cast<uint64_t>(result[6]) << 020) +
(static_cast<uint64_t>(result[7]) << 010) +
static_cast<uint64_t>(result[8]);
CHECK(restored == i);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
}
SECTION("unsigned")
{
SECTION("0..127 (int8)")
{
for (size_t i = 0; i <= 127; ++i)
{
CAPTURE(i)
// create JSON value with unsigned integer number
json const j = i;
// check type
CHECK(j.is_number_unsigned());
// create expected byte vector
std::vector<uint8_t> const expected
{
'i',
static_cast<uint8_t>(i),
};
// compare result + size
const auto result = json::to_ubjson(j);
CHECK(result == expected);
CHECK(result.size() == 2);
// check individual bytes
CHECK(result[0] == 'i');
auto const restored = static_cast<uint8_t>(result[1]);
CHECK(restored == i);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
SECTION("128..255 (uint8)")
{
for (size_t i = 128; i <= 255; ++i)
{
CAPTURE(i)
// create JSON value with unsigned integer number
json const j = i;
// check type
CHECK(j.is_number_unsigned());
// create expected byte vector
std::vector<uint8_t> const expected
{
'U',
static_cast<uint8_t>(i),
};
// compare result + size
const auto result = json::to_ubjson(j);
CHECK(result == expected);
CHECK(result.size() == 2);
// check individual bytes
CHECK(result[0] == 'U');
auto const restored = static_cast<uint8_t>(result[1]);
CHECK(restored == i);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
SECTION("256..32767 (int16)")
{
for (size_t i = 256; i <= 32767; i = utils::next_integer_sample(i, static_cast<size_t>(32767), static_cast<size_t>(7)))
{
CAPTURE(i)
// create JSON value with unsigned integer number
json const j = i;
// check type
CHECK(j.is_number_unsigned());
// create expected byte vector
std::vector<uint8_t> const expected
{
'I',
static_cast<uint8_t>((i >> 8) & 0xff),
static_cast<uint8_t>(i & 0xff),
};
// compare result + size
const auto result = json::to_ubjson(j);
CHECK(result == expected);
CHECK(result.size() == 3);
// check individual bytes
CHECK(result[0] == 'I');
auto const restored = static_cast<uint16_t>((static_cast<uint8_t>(result[1]) * 256) + static_cast<uint8_t>(result[2]));
CHECK(restored == i);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
SECTION("65536..2147483647 (int32)")
{
for (uint32_t i :
{
65536u, 77777u, 1048576u
})
{
CAPTURE(i)
// create JSON value with unsigned integer number
json const j = i;
// check type
CHECK(j.is_number_unsigned());
// create expected byte vector
std::vector<uint8_t> const expected
{
'l',
static_cast<uint8_t>((i >> 24) & 0xff),
static_cast<uint8_t>((i >> 16) & 0xff),
static_cast<uint8_t>((i >> 8) & 0xff),
static_cast<uint8_t>(i & 0xff),
};
// compare result + size
const auto result = json::to_ubjson(j);
CHECK(result == expected);
CHECK(result.size() == 5);
// check individual bytes
CHECK(result[0] == 'l');
uint32_t const restored = (static_cast<uint32_t>(result[1]) << 030) +
(static_cast<uint32_t>(result[2]) << 020) +
(static_cast<uint32_t>(result[3]) << 010) +
static_cast<uint32_t>(result[4]);
CHECK(restored == i);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
SECTION("2147483648..9223372036854775807 (int64)")
{
std::vector<uint64_t> const v = {2147483648ul, 9223372036854775807ul};
for (uint64_t i : v)
{
CAPTURE(i)
// create JSON value with integer number
json const j = i;
// check type
CHECK(j.is_number_unsigned());
// create expected byte vector
std::vector<uint8_t> const expected
{
'L',
static_cast<uint8_t>((i >> 070) & 0xff),
static_cast<uint8_t>((i >> 060) & 0xff),
static_cast<uint8_t>((i >> 050) & 0xff),
static_cast<uint8_t>((i >> 040) & 0xff),
static_cast<uint8_t>((i >> 030) & 0xff),
static_cast<uint8_t>((i >> 020) & 0xff),
static_cast<uint8_t>((i >> 010) & 0xff),
static_cast<uint8_t>(i & 0xff),
};
// compare result + size
const auto result = json::to_ubjson(j);
CHECK(result == expected);
CHECK(result.size() == 9);
// check individual bytes
CHECK(result[0] == 'L');
uint64_t const restored = (static_cast<uint64_t>(result[1]) << 070) +
(static_cast<uint64_t>(result[2]) << 060) +
(static_cast<uint64_t>(result[3]) << 050) +
(static_cast<uint64_t>(result[4]) << 040) +
(static_cast<uint64_t>(result[5]) << 030) +
(static_cast<uint64_t>(result[6]) << 020) +
(static_cast<uint64_t>(result[7]) << 010) +
static_cast<uint64_t>(result[8]);
CHECK(restored == i);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
}
SECTION("float64")
{
SECTION("3.1415925")
{
double v = 3.1415925;
json const j = v;
std::vector<uint8_t> expected =
{
'D', 0x40, 0x09, 0x21, 0xfb, 0x3f, 0xa6, 0xde, 0xfc
};
const auto result = json::to_ubjson(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result) == v);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
SECTION("high-precision number")
{
SECTION("unsigned integer number")
{
std::vector<uint8_t> const vec = {'H', 'i', 0x14, '1', '2', '3', '4', '5', '6', '7', '8', '9', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '0'};
const auto j = json::from_ubjson(vec);
CHECK(j.is_number_unsigned());
CHECK(j.dump() == "12345678901234567890");
}
SECTION("signed integer number")
{
std::vector<uint8_t> const vec = {'H', 'i', 0x13, '-', '1', '2', '3', '4', '5', '6', '7', '8', '9', '0', '1', '2', '3', '4', '5', '6', '7', '8'};
const auto j = json::from_ubjson(vec);
CHECK(j.is_number_integer());
CHECK(j.dump() == "-123456789012345678");
}
SECTION("floating-point number")
{
std::vector<uint8_t> const vec = {'H', 'i', 0x16, '3', '.', '1', '4', '1', '5', '9', '2', '6', '5', '3', '5', '8', '9', '7', '9', '3', '2', '3', '8', '4', '6'};
const auto j = json::from_ubjson(vec);
CHECK(j.is_number_float());
CHECK(j.dump() == "3.141592653589793");
}
SECTION("errors")
{
// error while parsing length
std::vector<uint8_t> const vec0 = {'H', 'i'};
CHECK(json::from_ubjson(vec0, true, false).is_discarded());
// error while parsing string
std::vector<uint8_t> const vec1 = {'H', 'i', '1'};
CHECK(json::from_ubjson(vec1, true, false).is_discarded());
json _;
std::vector<uint8_t> const vec2 = {'H', 'i', 2, '1', 'A', '3'};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vec2), "[json.exception.parse_error.115] parse error at byte 5: syntax error while parsing UBJSON high-precision number: invalid number text: 1A", json::parse_error);
std::vector<uint8_t> const vec3 = {'H', 'i', 2, '1', '.'};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vec3), "[json.exception.parse_error.115] parse error at byte 5: syntax error while parsing UBJSON high-precision number: invalid number text: 1.", json::parse_error);
// Reject NULs where they are read, including trailing NULs and payloads cut off after one.
SECTION("NUL in high-precision number (issue #5753)")
{
for (const auto& vec : std::vector<std::vector<uint8_t>>
{
{'H', 'i', 3, '1', 0, 'x'},
{'H', 'i', 2, '1', 0},
{'H', 'i', 3, '1', 0}
})
{
CAPTURE(vec)
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vec), "[json.exception.parse_error.115] parse error at byte 5: syntax error while parsing UBJSON high-precision number: invalid number text; last byte: 0x00", json::parse_error);
CHECK(json::from_ubjson(vec, true, false).is_discarded());
CHECK(json::from_ubjson(vec, false, false).is_discarded());
}
std::vector<uint8_t> const nested = {'[', 'H', 'i', 3, '1', 0, 'x', ']'};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(nested), "[json.exception.parse_error.115] parse error at byte 6: syntax error while parsing UBJSON high-precision number: invalid number text; last byte: 0x00", json::parse_error);
CHECK(json::from_ubjson(nested, true, false).is_discarded());
std::vector<uint8_t> const valid = {'H', 'i', 1, '1'};
const auto j = json::from_ubjson(valid);
CHECK(j.is_number_unsigned());
CHECK(j == json(1));
}
std::vector<uint8_t> const vec_overflow = {'H', 'i', 5, '1', 'e', '4', '0', '0'};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vec_overflow), "[json.exception.out_of_range.406] number overflow parsing '1e400'", json::out_of_range&);
std::vector<uint8_t> const vec4 = {'H', 2, '1', '0'};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vec4), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing UBJSON size: expected length type specification (U, i, I, l, L) after '#'; last byte: 0x02", json::parse_error);
}
SECTION("serialization")
{
// number that does not fit int64
json const j = 11111111111111111111ULL;
CHECK(j.is_number_unsigned());
// number will be serialized to high-precision number
const auto vec = json::to_ubjson(j);
std::vector<uint8_t> expected = {'H', 'i', 0x14, '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'};
CHECK(vec == expected);
// roundtrip
CHECK(json::from_ubjson(vec) == j);
}
}
}
SECTION("string")
{
SECTION("N = 0..127")
{
for (size_t N = 0; N <= 127; ++N)
{
CAPTURE(N)
// create JSON value with string containing of N * 'x'
const auto s = std::string(N, 'x');
json const j = s;
// create expected byte vector
std::vector<uint8_t> expected;
expected.push_back('S');
expected.push_back('i');
expected.push_back(static_cast<uint8_t>(N));
for (size_t i = 0; i < N; ++i)
{
expected.push_back('x');
}
// compare result + size
const auto result = json::to_ubjson(j);
CHECK(result == expected);
CHECK(result.size() == N + 3);
// check that no null byte is appended
if (N > 0)
{
CHECK(result.back() != '\x00');
}
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
SECTION("N = 128..255")
{
for (size_t N = 128; N <= 255; ++N)
{
CAPTURE(N)
// create JSON value with string containing of N * 'x'
const auto s = std::string(N, 'x');
json const j = s;
// create expected byte vector
std::vector<uint8_t> expected;
expected.push_back('S');
expected.push_back('U');
expected.push_back(static_cast<uint8_t>(N));
for (size_t i = 0; i < N; ++i)
{
expected.push_back('x');
}
// compare result + size
const auto result = json::to_ubjson(j);
CHECK(result == expected);
CHECK(result.size() == N + 3);
// check that no null byte is appended
CHECK(result.back() != '\x00');
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
SECTION("N = 256..32767")
{
for (size_t N :
{
256u, 999u, 1025u, 3333u, 2048u, 32767u
})
{
CAPTURE(N)
// create JSON value with string containing of N * 'x'
const auto s = std::string(N, 'x');
json const j = s;
// create expected byte vector (hack: create string first)
std::vector<uint8_t> expected(N, 'x');
// reverse order of commands, because we insert at begin()
expected.insert(expected.begin(), static_cast<uint8_t>(N & 0xff));
expected.insert(expected.begin(), static_cast<uint8_t>((N >> 8) & 0xff));
expected.insert(expected.begin(), 'I');
expected.insert(expected.begin(), 'S');
// compare result + size
const auto result = json::to_ubjson(j);
CHECK(result == expected);
CHECK(result.size() == N + 4);
// check that no null byte is appended
CHECK(result.back() != '\x00');
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
SECTION("N = 65536..2147483647")
{
for (size_t N :
{
65536u, 77777u, 1048576u
})
{
CAPTURE(N)
// create JSON value with string containing of N * 'x'
const auto s = std::string(N, 'x');
json const j = s;
// create expected byte vector (hack: create string first)
std::vector<uint8_t> expected(N, 'x');
// reverse order of commands, because we insert at begin()
expected.insert(expected.begin(), static_cast<uint8_t>(N & 0xff));
expected.insert(expected.begin(), static_cast<uint8_t>((N >> 8) & 0xff));
expected.insert(expected.begin(), static_cast<uint8_t>((N >> 16) & 0xff));
expected.insert(expected.begin(), static_cast<uint8_t>((N >> 24) & 0xff));
expected.insert(expected.begin(), 'l');
expected.insert(expected.begin(), 'S');
// compare result + size
const auto result = json::to_ubjson(j);
CHECK(result == expected);
CHECK(result.size() == N + 6);
// check that no null byte is appended
CHECK(result.back() != '\x00');
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
}
SECTION("binary")
{
SECTION("N = 0..127")
{
for (std::size_t N = 0; N <= 127; ++N)
{
CAPTURE(N)
// create JSON value with byte array containing of N * 'x'
const auto s = std::vector<std::uint8_t>(N, 'x');
json const j = json::binary(s);
// create expected byte vector
std::vector<std::uint8_t> expected;
expected.push_back(static_cast<std::uint8_t>('['));
if (N != 0)
{
expected.push_back(static_cast<std::uint8_t>('$'));
expected.push_back(static_cast<std::uint8_t>('U'));
}
expected.push_back(static_cast<std::uint8_t>('#'));
expected.push_back(static_cast<std::uint8_t>('i'));
expected.push_back(static_cast<std::uint8_t>(N));
for (size_t i = 0; i < N; ++i)
{
expected.push_back(0x78);
}
// compare result + size
const auto result = json::to_ubjson(j, true, true);
CHECK(result == expected);
if (N == 0)
{
CHECK(result.size() == N + 4);
}
else
{
CHECK(result.size() == N + 6);
}
// check that no null byte is appended
if (N > 0)
{
CHECK(result.back() != '\x00');
}
// roundtrip only works to an array of numbers
json j_out = s;
CHECK(json::from_ubjson(result) == j_out);
CHECK(json::from_ubjson(result, true, false) == j_out);
}
}
SECTION("N = 128..255")
{
for (std::size_t N = 128; N <= 255; ++N)
{
CAPTURE(N)
// create JSON value with byte array containing of N * 'x'
const auto s = std::vector<std::uint8_t>(N, 'x');
json const j = json::binary(s);
// create expected byte vector
std::vector<uint8_t> expected;
expected.push_back(static_cast<std::uint8_t>('['));
expected.push_back(static_cast<std::uint8_t>('$'));
expected.push_back(static_cast<std::uint8_t>('U'));
expected.push_back(static_cast<std::uint8_t>('#'));
expected.push_back(static_cast<std::uint8_t>('U'));
expected.push_back(static_cast<std::uint8_t>(N));
for (size_t i = 0; i < N; ++i)
{
expected.push_back(0x78);
}
// compare result + size
const auto result = json::to_ubjson(j, true, true);
CHECK(result == expected);
CHECK(result.size() == N + 6);
// check that no null byte is appended
CHECK(result.back() != '\x00');
// roundtrip only works to an array of numbers
json j_out = s;
CHECK(json::from_ubjson(result) == j_out);
CHECK(json::from_ubjson(result, true, false) == j_out);
}
}
SECTION("N = 256..32767")
{
for (std::size_t N :
{
256u, 999u, 1025u, 3333u, 2048u, 32767u
})
{
CAPTURE(N)
// create JSON value with byte array containing of N * 'x'
const auto s = std::vector<std::uint8_t>(N, 'x');
json const j = json::binary(s);
// create expected byte vector
std::vector<std::uint8_t> expected(N + 7, 'x');
expected[0] = '[';
expected[1] = '$';
expected[2] = 'U';
expected[3] = '#';
expected[4] = 'I';
expected[5] = static_cast<std::uint8_t>((N >> 8) & 0xFF);
expected[6] = static_cast<std::uint8_t>(N & 0xFF);
// compare result + size
const auto result = json::to_ubjson(j, true, true);
CHECK(result == expected);
CHECK(result.size() == N + 7);
// check that no null byte is appended
CHECK(result.back() != '\x00');
// roundtrip only works to an array of numbers
json j_out = s;
CHECK(json::from_ubjson(result) == j_out);
CHECK(json::from_ubjson(result, true, false) == j_out);
}
}
SECTION("N = 32768..2147483647")
{
for (std::size_t N :
{
32768u, 77777u, 1048576u
})
{
CAPTURE(N)
// create JSON value with byte array containing of N * 'x'
const auto s = std::vector<std::uint8_t>(N, 'x');
json const j = json::binary(s);
// create expected byte vector
std::vector<std::uint8_t> expected(N + 9, 'x');
expected[0] = '[';
expected[1] = '$';
expected[2] = 'U';
expected[3] = '#';
expected[4] = 'l';
expected[5] = static_cast<std::uint8_t>((N >> 24) & 0xFF);
expected[6] = static_cast<std::uint8_t>((N >> 16) & 0xFF);
expected[7] = static_cast<std::uint8_t>((N >> 8) & 0xFF);
expected[8] = static_cast<std::uint8_t>(N & 0xFF);
// compare result + size
const auto result = json::to_ubjson(j, true, true);
CHECK(result == expected);
CHECK(result.size() == N + 9);
// check that no null byte is appended
CHECK(result.back() != '\x00');
// roundtrip only works to an array of numbers
json j_out = s;
CHECK(json::from_ubjson(result) == j_out);
CHECK(json::from_ubjson(result, true, false) == j_out);
}
}
SECTION("Other Serializations")
{
const std::size_t N = 10;
const auto s = std::vector<std::uint8_t>(N, 'x');
json const j = json::binary(s);
SECTION("No Count No Type")
{
std::vector<uint8_t> expected;
expected.push_back(static_cast<std::uint8_t>('['));
for (std::size_t i = 0; i < N; ++i)
{
expected.push_back(static_cast<std::uint8_t>('U'));
expected.push_back(static_cast<std::uint8_t>(0x78));
}
expected.push_back(static_cast<std::uint8_t>(']'));
// compare result + size
const auto result = json::to_ubjson(j, false, false);
CHECK(result == expected);
CHECK(result.size() == N + 12);
// check that no null byte is appended
CHECK(result.back() != '\x00');
// roundtrip only works to an array of numbers
json j_out = s;
CHECK(json::from_ubjson(result) == j_out);
CHECK(json::from_ubjson(result, true, false) == j_out);
}
SECTION("Yes Count No Type")
{
std::vector<std::uint8_t> expected;
expected.push_back(static_cast<std::uint8_t>('['));
expected.push_back(static_cast<std::uint8_t>('#'));
expected.push_back(static_cast<std::uint8_t>('i'));
expected.push_back(static_cast<std::uint8_t>(N));
for (size_t i = 0; i < N; ++i)
{
expected.push_back(static_cast<std::uint8_t>('U'));
expected.push_back(static_cast<std::uint8_t>(0x78));
}
// compare result + size
const auto result = json::to_ubjson(j, true, false);
CHECK(result == expected);
CHECK(result.size() == N + 14);
// check that no null byte is appended
CHECK(result.back() != '\x00');
// roundtrip only works to an array of numbers
json j_out = s;
CHECK(json::from_ubjson(result) == j_out);
CHECK(json::from_ubjson(result, true, false) == j_out);
}
}
}
SECTION("array")
{
SECTION("empty")
{
SECTION("size=false type=false")
{
json const j = json::array();
std::vector<uint8_t> expected = {'[', ']'};
const auto result = json::to_ubjson(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
SECTION("size=true type=false")
{
json const j = json::array();
std::vector<uint8_t> expected = {'[', '#', 'i', 0};
const auto result = json::to_ubjson(j, true);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
SECTION("size=true type=true")
{
json const j = json::array();
std::vector<uint8_t> expected = {'[', '#', 'i', 0};
const auto result = json::to_ubjson(j, true, true);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
SECTION("[null]")
{
SECTION("size=false type=false")
{
json const j = {nullptr};
std::vector<uint8_t> expected = {'[', 'Z', ']'};
const auto result = json::to_ubjson(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
SECTION("size=true type=false")
{
json const j = {nullptr};
std::vector<uint8_t> expected = {'[', '#', 'i', 1, 'Z'};
const auto result = json::to_ubjson(j, true);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
SECTION("size=true type=true")
{
json const j = {nullptr};
std::vector<uint8_t> expected = {'[', '$', 'Z', '#', 'i', 1};
const auto result = json::to_ubjson(j, true, true);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
SECTION("[1,2,3,4,5]")
{
SECTION("size=false type=false")
{
json const j = json::parse("[1,2,3,4,5]");
std::vector<uint8_t> expected = {'[', 'i', 1, 'i', 2, 'i', 3, 'i', 4, 'i', 5, ']'};
const auto result = json::to_ubjson(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
SECTION("size=true type=false")
{
json const j = json::parse("[1,2,3,4,5]");
std::vector<uint8_t> expected = {'[', '#', 'i', 5, 'i', 1, 'i', 2, 'i', 3, 'i', 4, 'i', 5};
const auto result = json::to_ubjson(j, true);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
SECTION("size=true type=true")
{
json const j = json::parse("[1,2,3,4,5]");
std::vector<uint8_t> expected = {'[', '$', 'i', '#', 'i', 5, 1, 2, 3, 4, 5};
const auto result = json::to_ubjson(j, true, true);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
SECTION("[[[[]]]]")
{
SECTION("size=false type=false")
{
json const j = json::parse("[[[[]]]]");
std::vector<uint8_t> expected = {'[', '[', '[', '[', ']', ']', ']', ']'};
const auto result = json::to_ubjson(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
SECTION("size=true type=false")
{
json const j = json::parse("[[[[]]]]");
std::vector<uint8_t> expected = {'[', '#', 'i', 1, '[', '#', 'i', 1, '[', '#', 'i', 1, '[', '#', 'i', 0};
const auto result = json::to_ubjson(j, true);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
SECTION("size=true type=true")
{
json const j = json::parse("[[[[]]]]");
std::vector<uint8_t> expected = {'[', '$', '[', '#', 'i', 1, '$', '[', '#', 'i', 1, '$', '[', '#', 'i', 1, '#', 'i', 0};
const auto result = json::to_ubjson(j, true, true);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
SECTION("array with uint16_t elements")
{
SECTION("size=false type=false")
{
json j(257, nullptr);
std::vector<uint8_t> expected(j.size() + 2, 'Z'); // all null
expected[0] = '['; // opening array
expected[258] = ']'; // closing array
const auto result = json::to_ubjson(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
SECTION("size=true type=false")
{
json j(257, nullptr);
std::vector<uint8_t> expected(j.size() + 5, 'Z'); // all null
expected[0] = '['; // opening array
expected[1] = '#'; // array size
expected[2] = 'I'; // int16
expected[3] = 0x01; // 0x0101, first byte
expected[4] = 0x01; // 0x0101, second byte
const auto result = json::to_ubjson(j, true);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
SECTION("size=true type=true")
{
json j(257, nullptr);
std::vector<uint8_t> expected = {'[', '$', 'Z', '#', 'I', 0x01, 0x01};
const auto result = json::to_ubjson(j, true, true);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
SECTION("array with uint32_t elements")
{
SECTION("size=false type=false")
{
json j(65793, nullptr);
std::vector<uint8_t> expected(j.size() + 2, 'Z'); // all null
expected[0] = '['; // opening array
expected[65794] = ']'; // closing array
const auto result = json::to_ubjson(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
SECTION("size=true type=false")
{
json j(65793, nullptr);
std::vector<uint8_t> expected(j.size() + 7, 'Z'); // all null
expected[0] = '['; // opening array
expected[1] = '#'; // array size
expected[2] = 'l'; // int32
expected[3] = 0x00; // 0x00010101, first byte
expected[4] = 0x01; // 0x00010101, second byte
expected[5] = 0x01; // 0x00010101, third byte
expected[6] = 0x01; // 0x00010101, fourth byte
const auto result = json::to_ubjson(j, true);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
SECTION("size=true type=true")
{
json j(65793, nullptr);
std::vector<uint8_t> expected = {'[', '$', 'Z', '#', 'l', 0x00, 0x01, 0x01, 0x01};
const auto result = json::to_ubjson(j, true, true);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
}
SECTION("object")
{
SECTION("empty")
{
SECTION("size=false type=false")
{
json const j = json::object();
std::vector<uint8_t> expected = {'{', '}'};
const auto result = json::to_ubjson(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
SECTION("size=true type=false")
{
json const j = json::object();
std::vector<uint8_t> expected = {'{', '#', 'i', 0};
const auto result = json::to_ubjson(j, true);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
SECTION("size=true type=true")
{
json const j = json::object();
std::vector<uint8_t> expected = {'{', '#', 'i', 0};
const auto result = json::to_ubjson(j, true, true);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
SECTION("{\"\":null}")
{
SECTION("size=false type=false")
{
json const j = {{"", nullptr}};
std::vector<uint8_t> expected = {'{', 'i', 0, 'Z', '}'};
const auto result = json::to_ubjson(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
SECTION("size=true type=false")
{
json const j = {{"", nullptr}};
std::vector<uint8_t> expected = {'{', '#', 'i', 1, 'i', 0, 'Z'};
const auto result = json::to_ubjson(j, true);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
SECTION("size=true type=true")
{
json const j = {{"", nullptr}};
std::vector<uint8_t> expected = {'{', '$', 'Z', '#', 'i', 1, 'i', 0};
const auto result = json::to_ubjson(j, true, true);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
SECTION("{\"a\": {\"b\": {\"c\": {}}}}")
{
SECTION("size=false type=false")
{
json const j = json::parse(R"({"a": {"b": {"c": {}}}})");
std::vector<uint8_t> expected =
{
'{', 'i', 1, 'a', '{', 'i', 1, 'b', '{', 'i', 1, 'c', '{', '}', '}', '}', '}'
};
const auto result = json::to_ubjson(j);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
SECTION("size=true type=false")
{
json const j = json::parse(R"({"a": {"b": {"c": {}}}})");
std::vector<uint8_t> expected =
{
'{', '#', 'i', 1, 'i', 1, 'a', '{', '#', 'i', 1, 'i', 1, 'b', '{', '#', 'i', 1, 'i', 1, 'c', '{', '#', 'i', 0
};
const auto result = json::to_ubjson(j, true);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
SECTION("size=true type=true")
{
json const j = json::parse(R"({"a": {"b": {"c": {}}}})");
std::vector<uint8_t> expected =
{
'{', '$', '{', '#', 'i', 1, 'i', 1, 'a', '$', '{', '#', 'i', 1, 'i', 1, 'b', '$', '{', '#', 'i', 1, 'i', 1, 'c', '#', 'i', 0
};
const auto result = json::to_ubjson(j, true, true);
CHECK(result == expected);
// roundtrip
CHECK(json::from_ubjson(result) == j);
CHECK(json::from_ubjson(result, true, false) == j);
}
}
}
}
SECTION("errors")
{
SECTION("strict mode")
{
std::vector<uint8_t> const vec = {'Z', 'Z'};
SECTION("non-strict mode")
{
const auto result = json::from_ubjson(vec, false);
CHECK(result == json());
}
SECTION("strict mode")
{
json _;
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vec), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing UBJSON value: expected end of input; last byte: 0x5A", json::parse_error&);
}
}
SECTION("excessive size")
{
SECTION("array")
{
std::vector<uint8_t> const v_ubjson = {'[', '$', 'Z', '#', 'L', 0x78, 0x28, 0x00, 0x68, 0x28, 0x69, 0x69, 0x17};
json _;
CHECK_THROWS_AS(_ = json::from_ubjson(v_ubjson), json::out_of_range&);
json j;
nlohmann::detail::json_sax_dom_callback_parser<json, decltype(nlohmann::detail::input_adapter(v_ubjson))> scp(j, [](int /*unused*/, json::parse_event_t /*unused*/, const json& /*unused*/) noexcept
{
return true;
});
CHECK_THROWS_AS(_ = json::sax_parse(v_ubjson, &scp, json::input_format_t::ubjson), json::out_of_range&);
}
SECTION("object")
{
std::vector<uint8_t> const v_ubjson = {'{', '$', 'Z', '#', 'L', 0x78, 0x28, 0x00, 0x68, 0x28, 0x69, 0x69, 0x17};
json _;
CHECK_THROWS_AS(_ = json::from_ubjson(v_ubjson), json::out_of_range&);
json j;
nlohmann::detail::json_sax_dom_callback_parser<json, decltype(nlohmann::detail::input_adapter(v_ubjson))> scp(j, [](int /*unused*/, json::parse_event_t /*unused*/, const json& /*unused*/) noexcept
{
return true;
});
CHECK_THROWS_AS(_ = json::sax_parse(v_ubjson, &scp, json::input_format_t::ubjson), json::out_of_range&);
}
SECTION("array with a known size, read with a callback")
{
// a sized array announces its length to start_array()
std::vector<uint8_t> const v_ubjson = {'[', '#', 'i', 2, 'i', 1, 'i', 2};
json j;
nlohmann::detail::json_sax_dom_callback_parser<json, decltype(nlohmann::detail::input_adapter(v_ubjson))> scp(j, [](int /*unused*/, json::parse_event_t /*unused*/, const json& /*unused*/) noexcept
{
return true;
});
CHECK(json::sax_parse(v_ubjson, &scp, json::input_format_t::ubjson));
CHECK(j == json({1, 2}));
// the readers reject a size this large before they announce
// it, so it can only reach start_array() directly (the largest
// value stands for an unknown size and is never checked)
json k;
nlohmann::detail::json_sax_dom_callback_parser<json, decltype(nlohmann::detail::input_adapter(v_ubjson))> scp2(k, [](int /*unused*/, json::parse_event_t /*unused*/, const json& /*unused*/) noexcept
{
return true;
});
CHECK_THROWS_AS(scp2.start_array((std::numeric_limits<std::size_t>::max)() - 1), json::out_of_range&);
}
}
}
SECTION("SAX aborts")
{
SECTION("start_array()")
{
std::vector<uint8_t> const v = {'[', 'T', 'F', ']'};
SaxCountdown scp(0);
CHECK(!json::sax_parse(v, &scp, json::input_format_t::ubjson));
}
SECTION("start_object()")
{
std::vector<uint8_t> const v = {'{', 'i', 3, 'f', 'o', 'o', 'F', '}'};
SaxCountdown scp(0);
CHECK(!json::sax_parse(v, &scp, json::input_format_t::ubjson));
}
SECTION("key() in object")
{
std::vector<uint8_t> const v = {'{', 'i', 3, 'f', 'o', 'o', 'F', '}'};
SaxCountdown scp(1);
CHECK(!json::sax_parse(v, &scp, json::input_format_t::ubjson));
}
SECTION("start_array(len)")
{
std::vector<uint8_t> const v = {'[', '#', 'i', '2', 'T', 'F'};
SaxCountdown scp(0);
CHECK(!json::sax_parse(v, &scp, json::input_format_t::ubjson));
}
SECTION("start_object(len)")
{
std::vector<uint8_t> const v = {'{', '#', 'i', '1', 3, 'f', 'o', 'o', 'F'};
SaxCountdown scp(0);
CHECK(!json::sax_parse(v, &scp, json::input_format_t::ubjson));
}
SECTION("key() in object with length")
{
std::vector<uint8_t> const v = {'{', 'i', 3, 'f', 'o', 'o', 'F', '}'};
SaxCountdown scp(1);
CHECK(!json::sax_parse(v, &scp, json::input_format_t::ubjson));
}
}
SECTION("parsing values")
{
SECTION("strings")
{
// create a single-character string for all number types
std::vector<uint8_t> s_i = {'S', 'i', 1, 'a'};
std::vector<uint8_t> const s_U = {'S', 'U', 1, 'a'};
std::vector<uint8_t> const s_I = {'S', 'I', 0, 1, 'a'};
std::vector<uint8_t> const s_l = {'S', 'l', 0, 0, 0, 1, 'a'};
std::vector<uint8_t> const s_L = {'S', 'L', 0, 0, 0, 0, 0, 0, 0, 1, 'a'};
// check if string is parsed correctly to "a"
CHECK(json::from_ubjson(s_i) == "a");
CHECK(json::from_ubjson(s_U) == "a");
CHECK(json::from_ubjson(s_I) == "a");
CHECK(json::from_ubjson(s_l) == "a");
CHECK(json::from_ubjson(s_L) == "a");
// roundtrip: output should be optimized
CHECK(json::to_ubjson(json::from_ubjson(s_i)) == s_i);
CHECK(json::to_ubjson(json::from_ubjson(s_U)) == s_i);
CHECK(json::to_ubjson(json::from_ubjson(s_I)) == s_i);
CHECK(json::to_ubjson(json::from_ubjson(s_l)) == s_i);
CHECK(json::to_ubjson(json::from_ubjson(s_L)) == s_i);
}
SECTION("no-op markers")
{
// A no-op ('N') is valid wherever a value may start; it is consumed
// by get_ignore_noop() before the value is read. It is not valid
// where a string length type specification is expected.
SECTION("accepted where a value may start")
{
// at top level, also repeated
CHECK(json::from_ubjson(std::vector<uint8_t>({'N', 'i', 1})) == json(1));
CHECK(json::from_ubjson(std::vector<uint8_t>({'N', 'N', 'N', 'i', 1})) == json(1));
// inside an array of unknown size, before and after an element
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', 'N', 'i', 1, ']'})) == json({1}));
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', 'i', 1, 'N', ']'})) == json({1}));
// inside an object of unknown size: before a key, between key
// and value, and before the closing '}'
CHECK(json::from_ubjson(std::vector<uint8_t>({'{', 'N', 'U', 1, 'a', 'i', 1, '}'})) == json({{"a", 1}}));
CHECK(json::from_ubjson(std::vector<uint8_t>({'{', 'U', 1, 'a', 'N', 'i', 1, '}'})) == json({{"a", 1}}));
CHECK(json::from_ubjson(std::vector<uint8_t>({'{', 'U', 1, 'a', 'i', 1, 'N', '}'})) == json({{"a", 1}}));
}
SECTION("rejected where a length type specification is expected")
{
json _;
// after the 'S' marker of a string value
std::vector<uint8_t> const v_S = {'S', 'N', 'U', 1, 'a'};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v_S), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing UBJSON string: expected length type specification (U, i, I, l, L); last byte: 0x4E", json::parse_error&);
// as the key length of an object with a known size, where
// no-ops are not permitted in the first place
std::vector<uint8_t> const v_key = {'{', '#', 'i', 1, 'N', 'U', 1, 'a', 'i', 1};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v_key), "[json.exception.parse_error.113] parse error at byte 5: syntax error while parsing UBJSON string: expected length type specification (U, i, I, l, L); last byte: 0x4E", json::parse_error&);
}
}
SECTION("number")
{
SECTION("float")
{
// float32
std::vector<uint8_t> const v_d = {'d', 0x40, 0x49, 0x0f, 0xd0};
CHECK(json::from_ubjson(v_d) == 3.14159f);
// float64
std::vector<uint8_t> const v_D = {'D', 0x40, 0x09, 0x21, 0xf9, 0xf0, 0x1b, 0x86, 0x6e};
CHECK(json::from_ubjson(v_D) == 3.14159);
// float32 is serialized as float64 as the library does not support float32
CHECK(json::to_ubjson(json::from_ubjson(v_d)) == json::to_ubjson(3.14159f));
}
}
SECTION("array")
{
SECTION("optimized version (length only)")
{
// create vector with two elements of the same type
std::vector<uint8_t> const v_TU = {'[', '#', 'U', 2, 'T', 'T'};
std::vector<uint8_t> const v_T = {'[', '#', 'i', 2, 'T', 'T'};
std::vector<uint8_t> const v_F = {'[', '#', 'i', 2, 'F', 'F'};
std::vector<uint8_t> const v_Z = {'[', '#', 'i', 2, 'Z', 'Z'};
std::vector<uint8_t> const v_i = {'[', '#', 'i', 2, 'i', 0x7F, 'i', 0x7F};
std::vector<uint8_t> const v_U = {'[', '#', 'i', 2, 'U', 0xFF, 'U', 0xFF};
std::vector<uint8_t> const v_I = {'[', '#', 'i', 2, 'I', 0x7F, 0xFF, 'I', 0x7F, 0xFF};
std::vector<uint8_t> const v_l = {'[', '#', 'i', 2, 'l', 0x7F, 0xFF, 0xFF, 0xFF, 'l', 0x7F, 0xFF, 0xFF, 0xFF};
std::vector<uint8_t> const v_L = {'[', '#', 'i', 2, 'L', 0x7F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 'L', 0x7F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF};
std::vector<uint8_t> const v_D = {'[', '#', 'i', 2, 'D', 0x40, 0x09, 0x21, 0xfb, 0x4d, 0x12, 0xd8, 0x4a, 'D', 0x40, 0x09, 0x21, 0xfb, 0x4d, 0x12, 0xd8, 0x4a};
std::vector<uint8_t> const v_S = {'[', '#', 'i', 2, 'S', 'i', 1, 'a', 'S', 'i', 1, 'a'};
std::vector<uint8_t> const v_C = {'[', '#', 'i', 2, 'C', 'a', 'C', 'a'};
// check if vector is parsed correctly
CHECK(json::from_ubjson(v_TU) == json({true, true}));
CHECK(json::from_ubjson(v_T) == json({true, true}));
CHECK(json::from_ubjson(v_F) == json({false, false}));
CHECK(json::from_ubjson(v_Z) == json({nullptr, nullptr}));
CHECK(json::from_ubjson(v_i) == json({127, 127}));
CHECK(json::from_ubjson(v_U) == json({255, 255}));
CHECK(json::from_ubjson(v_I) == json({32767, 32767}));
CHECK(json::from_ubjson(v_l) == json({2147483647, 2147483647}));
CHECK(json::from_ubjson(v_L) == json({9223372036854775807, 9223372036854775807}));
CHECK(json::from_ubjson(v_D) == json({3.1415926, 3.1415926}));
CHECK(json::from_ubjson(v_S) == json({"a", "a"}));
CHECK(json::from_ubjson(v_C) == json({"a", "a"}));
// roundtrip: output should be optimized
CHECK(json::to_ubjson(json::from_ubjson(v_T), true) == v_T);
CHECK(json::to_ubjson(json::from_ubjson(v_F), true) == v_F);
CHECK(json::to_ubjson(json::from_ubjson(v_Z), true) == v_Z);
CHECK(json::to_ubjson(json::from_ubjson(v_i), true) == v_i);
CHECK(json::to_ubjson(json::from_ubjson(v_U), true) == v_U);
CHECK(json::to_ubjson(json::from_ubjson(v_I), true) == v_I);
CHECK(json::to_ubjson(json::from_ubjson(v_l), true) == v_l);
CHECK(json::to_ubjson(json::from_ubjson(v_L), true) == v_L);
CHECK(json::to_ubjson(json::from_ubjson(v_D), true) == v_D);
CHECK(json::to_ubjson(json::from_ubjson(v_S), true) == v_S);
CHECK(json::to_ubjson(json::from_ubjson(v_C), true) == v_S); // char is serialized to string
}
SECTION("optimized version (type and length)")
{
// create vector with two elements of the same type
std::vector<uint8_t> const v_N = {'[', '$', 'N', '#', 'i', 2};
std::vector<uint8_t> const v_T = {'[', '$', 'T', '#', 'i', 2};
std::vector<uint8_t> const v_F = {'[', '$', 'F', '#', 'i', 2};
std::vector<uint8_t> const v_Z = {'[', '$', 'Z', '#', 'i', 2};
std::vector<uint8_t> const v_i = {'[', '$', 'i', '#', 'i', 2, 0x7F, 0x7F};
std::vector<uint8_t> const v_U = {'[', '$', 'U', '#', 'i', 2, 0xFF, 0xFF};
std::vector<uint8_t> const v_I = {'[', '$', 'I', '#', 'i', 2, 0x7F, 0xFF, 0x7F, 0xFF};
std::vector<uint8_t> const v_l = {'[', '$', 'l', '#', 'i', 2, 0x7F, 0xFF, 0xFF, 0xFF, 0x7F, 0xFF, 0xFF, 0xFF};
std::vector<uint8_t> const v_L = {'[', '$', 'L', '#', 'i', 2, 0x7F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x7F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF};
std::vector<uint8_t> const v_D = {'[', '$', 'D', '#', 'i', 2, 0x40, 0x09, 0x21, 0xfb, 0x4d, 0x12, 0xd8, 0x4a, 0x40, 0x09, 0x21, 0xfb, 0x4d, 0x12, 0xd8, 0x4a};
std::vector<uint8_t> const v_S = {'[', '$', 'S', '#', 'i', 2, 'i', 1, 'a', 'i', 1, 'a'};
std::vector<uint8_t> const v_C = {'[', '$', 'C', '#', 'i', 2, 'a', 'a'};
// check if vector is parsed correctly
CHECK(json::from_ubjson(v_N) == json::array());
CHECK(json::from_ubjson(v_T) == json({true, true}));
CHECK(json::from_ubjson(v_F) == json({false, false}));
CHECK(json::from_ubjson(v_Z) == json({nullptr, nullptr}));
CHECK(json::from_ubjson(v_i) == json({127, 127}));
CHECK(json::from_ubjson(v_U) == json({255, 255}));
CHECK(json::from_ubjson(v_I) == json({32767, 32767}));
CHECK(json::from_ubjson(v_l) == json({2147483647, 2147483647}));
CHECK(json::from_ubjson(v_L) == json({9223372036854775807, 9223372036854775807}));
CHECK(json::from_ubjson(v_D) == json({3.1415926, 3.1415926}));
CHECK(json::from_ubjson(v_S) == json({"a", "a"}));
CHECK(json::from_ubjson(v_C) == json({"a", "a"}));
// roundtrip: output should be optimized
std::vector<uint8_t> const v_empty = {'[', '#', 'i', 0};
CHECK(json::to_ubjson(json::from_ubjson(v_N), true, true) == v_empty);
CHECK(json::to_ubjson(json::from_ubjson(v_T), true, true) == v_T);
CHECK(json::to_ubjson(json::from_ubjson(v_F), true, true) == v_F);
CHECK(json::to_ubjson(json::from_ubjson(v_Z), true, true) == v_Z);
CHECK(json::to_ubjson(json::from_ubjson(v_i), true, true) == v_i);
CHECK(json::to_ubjson(json::from_ubjson(v_U), true, true) == v_U);
CHECK(json::to_ubjson(json::from_ubjson(v_I), true, true) == v_I);
CHECK(json::to_ubjson(json::from_ubjson(v_l), true, true) == v_l);
CHECK(json::to_ubjson(json::from_ubjson(v_L), true, true) == v_L);
CHECK(json::to_ubjson(json::from_ubjson(v_D), true, true) == v_D);
CHECK(json::to_ubjson(json::from_ubjson(v_S), true, true) == v_S);
CHECK(json::to_ubjson(json::from_ubjson(v_C), true, true) == v_S); // char is serialized to string
}
}
}
SECTION("parse errors")
{
SECTION("empty byte vector")
{
json _;
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(std::vector<uint8_t>()), "[json.exception.parse_error.110] parse error at byte 1: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
}
SECTION("char")
{
SECTION("eof after C byte")
{
std::vector<uint8_t> const v = {'C'};
json _;
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing UBJSON char: unexpected end of input", json::parse_error&);
}
SECTION("byte out of range")
{
std::vector<uint8_t> const v = {'C', 130};
json _;
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing UBJSON char: byte after 'C' must be in range 0x00..0x7F; last byte: 0x82", json::parse_error&);
}
}
SECTION("strings")
{
SECTION("eof after S byte")
{
std::vector<uint8_t> const v = {'S'};
json _;
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
}
SECTION("invalid byte")
{
std::vector<uint8_t> const v = {'S', '1', 'a'};
json _;
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing UBJSON string: expected length type specification (U, i, I, l, L); last byte: 0x31", json::parse_error&);
}
SECTION("negative length")
{
json _;
std::vector<uint8_t> const vi = {'S', 'i', 0xFF};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vi), "[json.exception.parse_error.113] parse error at byte 3: syntax error while parsing UBJSON string: string length must not be negative", json::parse_error&);
CHECK(json::from_ubjson(vi, true, false).is_discarded());
std::vector<uint8_t> const vI = {'S', 'I', 0xFF, 0xFF};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vI), "[json.exception.parse_error.113] parse error at byte 4: syntax error while parsing UBJSON string: string length must not be negative", json::parse_error&);
CHECK(json::from_ubjson(vI, true, false).is_discarded());
std::vector<uint8_t> const vl = {'S', 'l', 0xFF, 0xFF, 0xFF, 0xFF};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vl), "[json.exception.parse_error.113] parse error at byte 6: syntax error while parsing UBJSON string: string length must not be negative", json::parse_error&);
CHECK(json::from_ubjson(vl, true, false).is_discarded());
std::vector<uint8_t> const vL = {'S', 'L', 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vL), "[json.exception.parse_error.113] parse error at byte 10: syntax error while parsing UBJSON string: string length must not be negative", json::parse_error&);
CHECK(json::from_ubjson(vL, true, false).is_discarded());
// a length of zero remains valid and yields an empty string
std::vector<uint8_t> const v0 = {'S', 'i', 0};
CHECK(json::from_ubjson(v0) == json(""));
}
}
SECTION("array")
{
SECTION("optimized array: no size following type")
{
std::vector<uint8_t> const v = {'[', '$', 'i', 2};
json _;
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v), "[json.exception.parse_error.112] parse error at byte 4: syntax error while parsing UBJSON size: expected '#' after type information; last byte: 0x02", json::parse_error&);
}
}
SECTION("strings")
{
std::vector<uint8_t> const vS = {'S'};
json _;
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vS), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
CHECK(json::from_ubjson(vS, true, false).is_discarded());
std::vector<uint8_t> const v = {'S', 'i', '2', 'a'};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v), "[json.exception.parse_error.110] parse error at byte 5: syntax error while parsing UBJSON string: unexpected end of input", json::parse_error&);
CHECK(json::from_ubjson(v, true, false).is_discarded());
std::vector<uint8_t> const vC = {'C'};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vC), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing UBJSON char: unexpected end of input", json::parse_error&);
CHECK(json::from_ubjson(vC, true, false).is_discarded());
}
SECTION("sizes")
{
std::vector<uint8_t> const vU = {'[', '#', 'U'};
json _;
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vU), "[json.exception.parse_error.110] parse error at byte 4: syntax error while parsing UBJSON number: unexpected end of input", json::parse_error&);
CHECK(json::from_ubjson(vU, true, false).is_discarded());
std::vector<uint8_t> const vi = {'[', '#', 'i'};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vi), "[json.exception.parse_error.110] parse error at byte 4: syntax error while parsing UBJSON number: unexpected end of input", json::parse_error&);
CHECK(json::from_ubjson(vi, true, false).is_discarded());
std::vector<uint8_t> const vI = {'[', '#', 'I'};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vI), "[json.exception.parse_error.110] parse error at byte 4: syntax error while parsing UBJSON number: unexpected end of input", json::parse_error&);
CHECK(json::from_ubjson(vI, true, false).is_discarded());
std::vector<uint8_t> const vl = {'[', '#', 'l'};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vl), "[json.exception.parse_error.110] parse error at byte 4: syntax error while parsing UBJSON number: unexpected end of input", json::parse_error&);
CHECK(json::from_ubjson(vl, true, false).is_discarded());
std::vector<uint8_t> const vL = {'[', '#', 'L'};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vL), "[json.exception.parse_error.110] parse error at byte 4: syntax error while parsing UBJSON number: unexpected end of input", json::parse_error&);
CHECK(json::from_ubjson(vL, true, false).is_discarded());
std::vector<uint8_t> const v0 = {'[', '#', 'T', ']'};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v0), "[json.exception.parse_error.113] parse error at byte 3: syntax error while parsing UBJSON size: expected length type specification (U, i, I, l, L) after '#'; last byte: 0x54", json::parse_error&);
CHECK(json::from_ubjson(v0, true, false).is_discarded());
}
SECTION("types")
{
std::vector<uint8_t> const v0 = {'[', '$'};
json _;
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v0), "[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing UBJSON type: unexpected end of input", json::parse_error&);
CHECK(json::from_ubjson(v0, true, false).is_discarded());
std::vector<uint8_t> const vi = {'[', '$', '#'};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vi), "[json.exception.parse_error.112] parse error at byte 3: syntax error while parsing UBJSON type: marker 0x23 is not a permitted optimized array type", json::parse_error&);
CHECK(json::from_ubjson(vi, true, false).is_discarded());
std::vector<uint8_t> const vT = {'[', '$', 'T'};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vT), "[json.exception.parse_error.110] parse error at byte 4: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
CHECK(json::from_ubjson(vT, true, false).is_discarded());
}
SECTION("arrays")
{
std::vector<uint8_t> const vST = {'[', '$', 'i', '#', 'i', 2, 1};
json _;
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vST), "[json.exception.parse_error.110] parse error at byte 8: syntax error while parsing UBJSON number: unexpected end of input", json::parse_error&);
CHECK(json::from_ubjson(vST, true, false).is_discarded());
std::vector<uint8_t> const vS = {'[', '#', 'i', 2, 'i', 1};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vS), "[json.exception.parse_error.110] parse error at byte 7: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
CHECK(json::from_ubjson(vS, true, false).is_discarded());
std::vector<uint8_t> const v = {'[', 'i', 2, 'i', 1};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v), "[json.exception.parse_error.110] parse error at byte 6: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
CHECK(json::from_ubjson(v, true, false).is_discarded());
}
SECTION("objects")
{
std::vector<uint8_t> const vST = {'{', '$', 'i', '#', 'i', 2, 'i', 1, 'a', 1};
json _;
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vST), "[json.exception.parse_error.110] parse error at byte 11: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
CHECK(json::from_ubjson(vST, true, false).is_discarded());
std::vector<uint8_t> const vT = {'{', '$', 'i', 'i', 1, 'a', 1};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vT), "[json.exception.parse_error.112] parse error at byte 4: syntax error while parsing UBJSON size: expected '#' after type information; last byte: 0x69", json::parse_error&);
CHECK(json::from_ubjson(vT, true, false).is_discarded());
std::vector<uint8_t> const vS = {'{', '#', 'i', 2, 'i', 1, 'a', 'i', 1};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vS), "[json.exception.parse_error.110] parse error at byte 10: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
CHECK(json::from_ubjson(vS, true, false).is_discarded());
std::vector<uint8_t> const v = {'{', 'i', 1, 'a', 'i', 1};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v), "[json.exception.parse_error.110] parse error at byte 7: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
CHECK(json::from_ubjson(v, true, false).is_discarded());
std::vector<uint8_t> const v2 = {'{', 'i', 1, 'a', 'i', 1, 'i'};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v2), "[json.exception.parse_error.110] parse error at byte 8: syntax error while parsing UBJSON number: unexpected end of input", json::parse_error&);
CHECK(json::from_ubjson(v2, true, false).is_discarded());
std::vector<uint8_t> const v3 = {'{', 'i', 1, 'a'};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v3), "[json.exception.parse_error.110] parse error at byte 5: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
CHECK(json::from_ubjson(v3, true, false).is_discarded());
std::vector<uint8_t> const vST1 = {'{', '$', 'd', '#', 'i', 2, 'i', 1, 'a'};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vST1), "[json.exception.parse_error.110] parse error at byte 10: syntax error while parsing UBJSON number: unexpected end of input", json::parse_error&);
CHECK(json::from_ubjson(vST1, true, false).is_discarded());
std::vector<uint8_t> const vST2 = {'{', '#', 'i', 2, 'i', 1, 'a'};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vST2), "[json.exception.parse_error.110] parse error at byte 8: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
CHECK(json::from_ubjson(vST2, true, false).is_discarded());
}
}
SECTION("writing optimized values")
{
SECTION("integer")
{
SECTION("array of i")
{
json const j = {1, -1};
std::vector<uint8_t> expected = {'[', '$', 'i', '#', 'i', 2, 1, 0xff};
CHECK(json::to_ubjson(j, true, true) == expected);
}
SECTION("array of U")
{
json const j = {200, 201};
std::vector<uint8_t> expected = {'[', '$', 'U', '#', 'i', 2, 0xC8, 0xC9};
CHECK(json::to_ubjson(j, true, true) == expected);
}
SECTION("array of I")
{
json const j = {30000, -30000};
std::vector<uint8_t> expected = {'[', '$', 'I', '#', 'i', 2, 0x75, 0x30, 0x8a, 0xd0};
CHECK(json::to_ubjson(j, true, true) == expected);
}
SECTION("array of l")
{
json const j = {70000, -70000};
std::vector<uint8_t> expected = {'[', '$', 'l', '#', 'i', 2, 0x00, 0x01, 0x11, 0x70, 0xFF, 0xFE, 0xEE, 0x90};
CHECK(json::to_ubjson(j, true, true) == expected);
}
SECTION("array of L")
{
json const j = {5000000000, -5000000000};
std::vector<uint8_t> expected = {'[', '$', 'L', '#', 'i', 2, 0x00, 0x00, 0x00, 0x01, 0x2A, 0x05, 0xF2, 0x00, 0xFF, 0xFF, 0xFF, 0xFE, 0xD5, 0xFA, 0x0E, 0x00};
CHECK(json::to_ubjson(j, true, true) == expected);
}
}
SECTION("unsigned integer")
{
SECTION("array of i")
{
json const j = {1u, 2u};
std::vector<uint8_t> expected = {'[', '$', 'i', '#', 'i', 2, 1, 2};
std::vector<uint8_t> expected_size = {'[', '#', 'i', 2, 'i', 1, 'i', 2};
CHECK(json::to_ubjson(j, true, true) == expected);
CHECK(json::to_ubjson(j, true) == expected_size);
}
SECTION("array of U")
{
json const j = {200u, 201u};
std::vector<uint8_t> expected = {'[', '$', 'U', '#', 'i', 2, 0xC8, 0xC9};
std::vector<uint8_t> expected_size = {'[', '#', 'i', 2, 'U', 0xC8, 'U', 0xC9};
CHECK(json::to_ubjson(j, true, true) == expected);
CHECK(json::to_ubjson(j, true) == expected_size);
}
SECTION("array of I")
{
json const j = {30000u, 30001u};
std::vector<uint8_t> expected = {'[', '$', 'I', '#', 'i', 2, 0x75, 0x30, 0x75, 0x31};
std::vector<uint8_t> expected_size = {'[', '#', 'i', 2, 'I', 0x75, 0x30, 'I', 0x75, 0x31};
CHECK(json::to_ubjson(j, true, true) == expected);
CHECK(json::to_ubjson(j, true) == expected_size);
}
SECTION("array of l")
{
json const j = {70000u, 70001u};
std::vector<uint8_t> expected = {'[', '$', 'l', '#', 'i', 2, 0x00, 0x01, 0x11, 0x70, 0x00, 0x01, 0x11, 0x71};
std::vector<uint8_t> expected_size = {'[', '#', 'i', 2, 'l', 0x00, 0x01, 0x11, 0x70, 'l', 0x00, 0x01, 0x11, 0x71};
CHECK(json::to_ubjson(j, true, true) == expected);
CHECK(json::to_ubjson(j, true) == expected_size);
}
SECTION("array of L")
{
json const j = {5000000000u, 5000000001u};
std::vector<uint8_t> expected = {'[', '$', 'L', '#', 'i', 2, 0x00, 0x00, 0x00, 0x01, 0x2A, 0x05, 0xF2, 0x00, 0x00, 0x00, 0x00, 0x01, 0x2A, 0x05, 0xF2, 0x01};
std::vector<uint8_t> expected_size = {'[', '#', 'i', 2, 'L', 0x00, 0x00, 0x00, 0x01, 0x2A, 0x05, 0xF2, 0x00, 'L', 0x00, 0x00, 0x00, 0x01, 0x2A, 0x05, 0xF2, 0x01};
CHECK(json::to_ubjson(j, true, true) == expected);
CHECK(json::to_ubjson(j, true) == expected_size);
}
}
SECTION("discarded")
{
// a discarded value cannot be serialized to UBJSON, even as part
// of an optimized array of a single (here: valueless) type
json const j = {json::value_t::discarded, json::value_t::discarded};
#if JSON_DIAGNOSTICS
CHECK_THROWS_WITH_AS(json::to_ubjson(j, true, true), "[json.exception.type_error.321] (/0) cannot serialize discarded value to UBJSON", json::type_error&);
#else
CHECK_THROWS_WITH_AS(json::to_ubjson(j, true, true), "[json.exception.type_error.321] cannot serialize discarded value to UBJSON", json::type_error&);
#endif
}
}
}
TEST_CASE("UBJSON nesting does not consume the call stack")
{
// Containers used to be read by calling back into the value reader once
// per element, so the native call stack grew with the nesting depth of the
// input. '[' alone opens a container, so a payload of repeated '[' crashed
// the process (#5104), as did the optimized forms, which reach the same
// path through a type or size annotation. The containers are kept on a
// heap stack now.
//
// Deeply nested values must not be compared, copied or dumped here: those
// operations are still recursive and would reintroduce the crash.
json _;
SECTION("containers that end at a marker")
{
const std::vector<uint8_t> input(500000, '[');
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(input), "[json.exception.parse_error.110] parse error at byte 500001: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
CHECK(json::from_ubjson(input, true, false).is_discarded());
}
SECTION("containers with a size")
{
std::vector<uint8_t> input;
for (std::size_t i = 0; i < 100000; ++i)
{
input.push_back('[');
input.push_back('#');
input.push_back('i');
input.push_back(1);
}
CHECK_THROWS_AS(_ = json::from_ubjson(input), json::parse_error&);
CHECK(json::from_ubjson(input, true, false).is_discarded());
}
SECTION("containers with a type and a size")
{
// '[' is a permitted optimized type in UBJSON, so each element of such
// a container is itself a container, read without a marker of its own
std::vector<uint8_t> input;
for (std::size_t i = 0; i < 100000; ++i)
{
const std::vector<uint8_t> level = {'[', '$', '[', '#', 'i', 1};
input.insert(input.end(), level.begin(), level.end());
}
CHECK_THROWS_AS(_ = json::from_ubjson(input), json::parse_error&);
CHECK(json::from_ubjson(input, true, false).is_discarded());
}
SECTION("a well-formed deep value is read through the SAX interface")
{
std::vector<uint8_t> input(100000, '[');
input.insert(input.end(), 100000, ']');
SaxCountdown accept_all(1000000);
CHECK(json::sax_parse(input, &accept_all, json::input_format_t::ubjson));
}
SECTION("a well-formed deep value is read into a value")
{
const std::size_t depth = 10000;
std::vector<uint8_t> input(depth, '[');
input.insert(input.end(), depth, ']');
json j = json::from_ubjson(input);
std::size_t measured = 0;
const json* p = &j;
while (p->is_array() && !p->empty())
{
p = &p->front();
++measured;
}
// the innermost array is empty, so the descent stops one level short
CHECK(measured == depth - 1);
}
SECTION("containers are still read the same way")
{
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', ']'})) == json::array());
CHECK(json::from_ubjson(std::vector<uint8_t>({'{', '}'})) == json::object());
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '#', 'i', 0})) == json::array());
CHECK(json::from_ubjson(std::vector<uint8_t>({'{', '#', 'i', 0})) == json::object());
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'i', '#', 'i', 2, 1, 2})) == json({1, 2}));
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '#', 'i', 2, 'i', 1, 'i', 2})) == json({1, 2}));
CHECK(json::from_ubjson(std::vector<uint8_t>({'{', '$', 'i', '#', 'i', 1, 'i', 1, 'a', 1})) == json({{"a", 1}}));
// a no-op is not a value, so a container of them holds none
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'N', '#', 'i', 2})) == json::array());
// sized and unsized forms nested inside one another
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '[', '#', 'i', 2, 'i', 1, 'i', 2, ']'})) == json({{1, 2}}));
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '#', 'i', 1, '[', 'i', 1, ']'})) == json({{1}}));
// an optimized container of containers
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', '[', '#', 'i', 2, 'i', 1, ']', 'i', 2, ']'})) == json({{1}, {2}}));
}
SECTION("BJData containers are still read the same way")
{
// the ND-array wrapper and the binary shortcut are complete values,
// not containers the reader descends into
CHECK(json::from_bjdata(std::vector<uint8_t>({'[', '$', 'U', '#', '[', '$', 'i', '#', 'i', 2, 2, 3, 1, 2, 3, 4, 5, 6})) ==
json({{"_ArrayType_", "uint8"}, {"_ArraySize_", {2, 3}}, {"_ArrayData_", {1, 2, 3, 4, 5, 6}}}));
CHECK(json::from_bjdata(std::vector<uint8_t>({'[', '$', 'i', '#', 'i', 2, 1, 2})) == json({1, 2}));
CHECK(json::from_bjdata(std::vector<uint8_t>({'[', '[', 'i', 1, ']', ']'})) == json({{1}}));
}
}
TEST_CASE("UBJSON input that cannot be read is discarded by every overload")
{
std::vector<std::uint8_t> input = json::to_ubjson(json({{"a", {1, 2}}}));
input.pop_back();
json _;
CHECK_THROWS_AS(_ = json::from_ubjson(input.begin(), input.end()), json::parse_error&);
CHECK(json::from_ubjson(input, true, false).is_discarded());
CHECK(json::from_ubjson(input.begin(), input.end(), true, false).is_discarded());
#ifndef JSON_TEST_DEPRECATED_FUNCTIONS_DELETED
CHECK(json::from_ubjson(input.data(), input.size(), true, false).is_discarded());
CHECK(json::from_ubjson({input.data(), input.size()}, true, false).is_discarded());
#endif
}
TEST_CASE("UBJSON SAX parsing stops at every event")
{
// Containers are opened and closed by the loop that reads them; a SAX
// handler that rejects any event - including the end of a nested
// container - must stop the parse right there.
const auto count_events = [](const std::vector<std::uint8_t>& input)
{
int events = 0;
while (true)
{
SaxCountdown scp(events);
if (json::sax_parse(input, &scp, json::input_format_t::ubjson))
{
return events;
}
++events;
REQUIRE(events < 1000);
}
};
// 20 events: every container kind closes inside another one
const json j = json::parse(R"({"a": [1, {"b": []}], "c": {"d": [[2]]}})");
CHECK(count_events(json::to_ubjson(j)) == 20);
CHECK(count_events(json::to_ubjson(j, true)) == 20);
CHECK(count_events(json::to_ubjson(j, true, true)) == 20);
}
TEST_CASE("UBJSON optimized arrays of a valueless type are bounded")
{
// An element of type 'Z', 'T' or 'F' is encoded by its marker alone, so an
// optimized array of one of those has no payload and the declared count is
// the only thing deciding how much is allocated. Ten bytes used to produce
// billions of values (#2793); every other type costs at least one byte per
// element and is bounded by the end of the input.
json _;
SECTION("an excessive count is rejected")
{
// 'l' is a big-endian int32: 0x7FFFFFFF elements, about 34 GB of value;
// OSS-Fuzz reported this shape as a parse_ubjson_fuzzer timeout
// (testcase 6347769435193344, no issue filed)
for (const auto marker :
{'Z', 'T', 'F'
})
{
const std::vector<uint8_t> input = {'[', '$', static_cast<uint8_t>(marker), '#', 'l', 0x7F, 0xFF, 0xFF, 0xFF};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(input), "[json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive array size", json::out_of_range&);
CHECK(json::from_ubjson(input, true, false).is_discarded());
}
}
SECTION("ordinary counts are unaffected")
{
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'Z', '#', 'i', 3})) == json({nullptr, nullptr, nullptr}));
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'T', '#', 'i', 2})) == json({true, true}));
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'F', '#', 'i', 2})) == json({false, false}));
// 'N' is a no-op rather than a value, and still yields an empty array
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'N', '#', 'i', 2})) == json::array());
}
SECTION("a type with a payload is unaffected")
{
// A count past the limit is not rejected for 'U', which costs a byte
// per element and is bounded by the end of the input instead. The
// count is kept just past the limit rather than made huge, because a
// count that also exceeds the array's max_size() is reported as
// out_of_range before the input runs out, and max_size() depends on
// the width of std::size_t.
const std::vector<uint8_t> input = {'[', '$', 'U', '#', 'l', 0x00, 0x10, 0x00, 0x01};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(input), "[json.exception.parse_error.110] parse error at byte 10: syntax error while parsing UBJSON number: unexpected end of input", json::parse_error&);
CHECK(json::from_ubjson(input, true, false).is_discarded());
}
SECTION("the writer stays within what the reader accepts")
{
// below the limit the optimized form is used and is tiny; above it the
// writer falls back so that the result can still be read back
json const at_limit(1048576, nullptr);
const auto v_at_limit = json::to_ubjson(at_limit, true, true);
CHECK(v_at_limit.size() == 9);
CHECK(v_at_limit.at(1) == '$');
CHECK(json::from_ubjson(v_at_limit) == at_limit);
json const above_limit(1048577, nullptr);
const auto v_above_limit = json::to_ubjson(above_limit, true, true);
CHECK(v_above_limit.at(1) != '$');
CHECK(json::from_ubjson(v_above_limit) == above_limit);
}
}
TEST_CASE("issue #5405 - array reserve for definite-length UBJSON arrays")
{
#if !defined(JSON_NOEXCEPTION)
// this SECTION relies on catching a thrown exception to distinguish
// which of two acceptable, bounded rejections a hostile header took;
// under JSON_NOEXCEPTION, JSON_THROW never produces a catchable C++
// exception (it aborts instead), so this cannot be tested that way here
SECTION("a huge claimed length with no element data must not over-allocate")
{
// optimized form [$type#count: type 'i' (int8), count as a four-byte
// 'l' (int32) of 0x7FFFFFFF (2147483647), but no element data at all.
// max_size() for a std::vector is far larger than this count, so it
// does not reject the header outright; the (capped) reservation must
// not attempt to allocate space for billions of elements before the
// missing data is detected.
json _;
const std::vector<uint8_t> input = {'[', '$', 'i', '#', 'l', 0x7F, 0xFF, 0xFF, 0xFF};
// On a platform where std::vector<json>::max_size() is smaller than
// the claimed count (e.g. 32-bit, where max_size() is bounded by a
// 32-bit SIZE_MAX divided by sizeof(json)), the SAX consumer's own
// check rejects the header outright (out_of_range.408, with the
// claimed count in the message) instead of accepting it and only
// finding it short of data once the (capped) reservation looks for
// element bytes that were never provided (parse_error.110). Either
// is an acceptable, bounded rejection of the hostile header -- the
// property under test is that no path attempts to allocate space
// for billions of elements.
bool threw = false;
try
{
_ = json::from_ubjson(input);
}
catch (const json::parse_error& e)
{
threw = true;
CHECK(e.id == 110);
CHECK(std::string(e.what()) == "[json.exception.parse_error.110] parse error at byte 10: syntax error while parsing UBJSON number: unexpected end of input");
}
catch (const json::out_of_range& e)
{
threw = true;
CHECK(e.id == 408);
CHECK(std::string(e.what()).find("excessive array size") != std::string::npos);
}
CHECK(threw);
// json_sax_dom_parser::start_array()'s max_size() check (unlike the
// scanner's own parse_error path) throws unconditionally via
// JSON_THROW rather than going through sax->parse_error(), so it is
// not gated by allow_exceptions=false on a platform where this
// header hits that check (e.g. 32-bit, see above) -- allow either
// a discarded result or the same out_of_range it throws with
// exceptions enabled.
try
{
CHECK(json::from_ubjson(input, true, false).is_discarded());
}
catch (const json::out_of_range& e)
{
CHECK(e.id == 408);
}
}
#endif
SECTION("arrays of various sizes decode to the same value as before the reserve optimization")
{
for (const auto size :
{
std::size_t{0}, std::size_t{1}, std::size_t{5}, // small
std::size_t{16384}, // exactly at the reserve cap
std::size_t{20000} // above the reserve cap
})
{
CAPTURE(size)
json j = json::array();
for (std::size_t i = 0; i < size; ++i)
{
j.push_back(static_cast<int>(i % 1000));
}
// exercise both the plain and the optimized [$type#count encoding
const auto packed_plain = json::to_ubjson(j);
CHECK(json::from_ubjson(packed_plain) == j);
const auto packed_optimized = json::to_ubjson(j, true, true);
CHECK(json::from_ubjson(packed_optimized) == j);
}
}
SECTION("a user-defined SAX consumer is unaffected by the internal DOM reserve optimization")
{
// the reserve() call is local to json_sax_dom_parser / json_sax_dom_callback_parser;
// a custom SAX consumer that does not touch a DOM array sees identical events
json j = json::array();
for (int i = 0; i < 100; ++i)
{
j.push_back(i);
}
const auto packed = json::to_ubjson(j, true, true);
SaxCountdown scp(1000000); // large enough to never trigger an abort
CHECK(json::sax_parse(packed, &scp, json::input_format_t::ubjson));
}
}
TEST_CASE("Universal Binary JSON Specification Examples 1")
{
SECTION("Null Value")
{
json const j = {{"passcode", nullptr}};
std::vector<uint8_t> const v = {'{', 'i', 8, 'p', 'a', 's', 's', 'c', 'o', 'd', 'e', 'Z', '}'};
CHECK(json::to_ubjson(j) == v);
CHECK(json::from_ubjson(v) == j);
}
SECTION("No-Op Value")
{
json const j = {"foo", "bar", "baz"};
std::vector<uint8_t> const v = {'[', 'S', 'i', 3, 'f', 'o', 'o',
'S', 'i', 3, 'b', 'a', 'r',
'S', 'i', 3, 'b', 'a', 'z', ']'
};
std::vector<uint8_t> const v2 = {'[', 'S', 'i', 3, 'f', 'o', 'o', 'N',
'S', 'i', 3, 'b', 'a', 'r', 'N', 'N', 'N',
'S', 'i', 3, 'b', 'a', 'z', 'N', 'N', ']'
};
CHECK(json::to_ubjson(j) == v);
CHECK(json::from_ubjson(v) == j);
CHECK(json::from_ubjson(v2) == j);
}
SECTION("Boolean Types")
{
json const j = {{"authorized", true}, {"verified", false}};
std::vector<uint8_t> const v = {'{', 'i', 10, 'a', 'u', 't', 'h', 'o', 'r', 'i', 'z', 'e', 'd', 'T',
'i', 8, 'v', 'e', 'r', 'i', 'f', 'i', 'e', 'd', 'F', '}'
};
CHECK(json::to_ubjson(j) == v);
CHECK(json::from_ubjson(v) == j);
}
SECTION("Numeric Types")
{
json const j =
{
{"int8", 16},
{"uint8", 255},
{"int16", 32767},
{"int32", 2147483647},
{"int64", 9223372036854775807},
{"float64", 113243.7863123}
};
std::vector<uint8_t> const v = {'{',
'i', 7, 'f', 'l', 'o', 'a', 't', '6', '4', 'D', 0x40, 0xfb, 0xa5, 0xbc, 0x94, 0xbc, 0x34, 0xcf,
'i', 5, 'i', 'n', 't', '1', '6', 'I', 0x7f, 0xff,
'i', 5, 'i', 'n', 't', '3', '2', 'l', 0x7f, 0xff, 0xff, 0xff,
'i', 5, 'i', 'n', 't', '6', '4', 'L', 0x7f, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
'i', 4, 'i', 'n', 't', '8', 'i', 16,
'i', 5, 'u', 'i', 'n', 't', '8', 'U', 0xff,
'}'
};
CHECK(json::to_ubjson(j) == v);
CHECK(json::from_ubjson(v) == j);
}
SECTION("Char Type")
{
json const j = {{"rolecode", "a"}, {"delim", ";"}};
std::vector<uint8_t> const v = {'{', 'i', 5, 'd', 'e', 'l', 'i', 'm', 'C', ';', 'i', 8, 'r', 'o', 'l', 'e', 'c', 'o', 'd', 'e', 'C', 'a', '}'};
//CHECK(json::to_ubjson(j) == v);
CHECK(json::from_ubjson(v) == j);
}
SECTION("String Type")
{
SECTION("English")
{
json const j = "hello";
std::vector<uint8_t> const v = {'S', 'i', 5, 'h', 'e', 'l', 'l', 'o'};
CHECK(json::to_ubjson(j) == v);
CHECK(json::from_ubjson(v) == j);
}
SECTION("Russian")
{
json const j = "привет";
std::vector<uint8_t> const v = {'S', 'i', 12, 0xD0, 0xBF, 0xD1, 0x80, 0xD0, 0xB8, 0xD0, 0xB2, 0xD0, 0xB5, 0xD1, 0x82};
CHECK(json::to_ubjson(j) == v);
CHECK(json::from_ubjson(v) == j);
}
SECTION("Russian")
{
json const j = "مرحبا";
std::vector<uint8_t> const v = {'S', 'i', 10, 0xD9, 0x85, 0xD8, 0xB1, 0xD8, 0xAD, 0xD8, 0xA8, 0xD8, 0xA7};
CHECK(json::to_ubjson(j) == v);
CHECK(json::from_ubjson(v) == j);
}
SECTION("ill-formed UTF-8 (see #5529, #5651)")
{
// none of the binary format specs requires a decoder to reject
// ill-formed UTF-8 in a text string, so a value whose bytes are
// not valid UTF-8 (0xC0 0xAE is an overlong encoding of '.')
// round-trips byte for byte as a string value; to_ubjson() writes
// the bytes unchanged, as before 3.13.0, unless
// JSON_STRICT_BINARY_UTF8 is enabled (see
// unit-binary_utf8_strict.cpp)
const std::vector<uint8_t> v = {'S', 'i', 2, 0xc0, 0xae};
json j;
CHECK_NOTHROW(j = json::from_ubjson(v));
REQUIRE(j.is_string());
CHECK(j.get_ref<const json::string_t&>() == std::string("\xc0\xae"));
CHECK_THROWS_AS(utils::ignore_return_value(j.dump()), json::type_error&);
CHECK(json::from_ubjson(json::to_ubjson(j)) == j);
// the same bytes as an object key round-trip as well
const std::vector<uint8_t> v_key = {'{', 'i', 2, 0xc0, 0xae, 'i', 1, '}'};
json j_key;
CHECK_NOTHROW(j_key = json::from_ubjson(v_key));
REQUIRE(j_key.is_object());
CHECK(j_key.contains(std::string("\xc0\xae")));
CHECK(json::from_ubjson(json::to_ubjson(j_key)) == j_key);
CHECK(json::from_ubjson(json::to_ubjson(json("\xFF"))) == json("\xFF"));
// a truncated multi-byte sequence
CHECK(json::from_ubjson(json::to_ubjson(json("\xC3"))) == json("\xC3"));
// an encoded surrogate half (U+D800)
CHECK(json::from_ubjson(json::to_ubjson(json("\xED\xA0\x80"))) == json("\xED\xA0\x80"));
// an overlong encoding of '.'
CHECK(json::from_ubjson(json::to_ubjson(json("\xC0\xAF"))) == json("\xC0\xAF"));
// an object key with ill-formed UTF-8 is kept the same way
CHECK(json::from_ubjson(json::to_ubjson(json{{"\xFF", 1}})) == json{{"\xFF", 1}});
}
}
SECTION("Array Type")
{
SECTION("size=false type=false")
{
// note the float has been replaced by a double
json const j = {nullptr, true, false, 4782345193, 153.132, "ham"};
std::vector<uint8_t> const v = {'[', 'Z', 'T', 'F', 'L', 0x00, 0x00, 0x00, 0x01, 0x1D, 0x0C, 0xCB, 0xE9, 'D', 0x40, 0x63, 0x24, 0x39, 0x58, 0x10, 0x62, 0x4e, 'S', 'i', 3, 'h', 'a', 'm', ']'};
CHECK(json::to_ubjson(j) == v);
CHECK(json::from_ubjson(v) == j);
}
SECTION("size=true type=false")
{
// note the float has been replaced by a double
json const j = {nullptr, true, false, 4782345193, 153.132, "ham"};
std::vector<uint8_t> const v = {'[', '#', 'i', 6, 'Z', 'T', 'F', 'L', 0x00, 0x00, 0x00, 0x01, 0x1D, 0x0C, 0xCB, 0xE9, 'D', 0x40, 0x63, 0x24, 0x39, 0x58, 0x10, 0x62, 0x4e, 'S', 'i', 3, 'h', 'a', 'm'};
CHECK(json::to_ubjson(j, true) == v);
CHECK(json::from_ubjson(v) == j);
}
SECTION("size=true type=true")
{
// note the float has been replaced by a double
json const j = {nullptr, true, false, 4782345193, 153.132, "ham"};
std::vector<uint8_t> const v = {'[', '#', 'i', 6, 'Z', 'T', 'F', 'L', 0x00, 0x00, 0x00, 0x01, 0x1D, 0x0C, 0xCB, 0xE9, 'D', 0x40, 0x63, 0x24, 0x39, 0x58, 0x10, 0x62, 0x4e, 'S', 'i', 3, 'h', 'a', 'm'};
CHECK(json::to_ubjson(j, true, true) == v);
CHECK(json::from_ubjson(v) == j);
}
}
SECTION("Object Type")
{
SECTION("size=false type=false")
{
json const j =
{
{
"post", {
{"id", 1137},
{"author", "rkalla"},
{"timestamp", 1364482090592},
{"body", "I totally agree!"}
}
}
};
std::vector<uint8_t> const v = {'{', 'i', 4, 'p', 'o', 's', 't', '{',
'i', 6, 'a', 'u', 't', 'h', 'o', 'r', 'S', 'i', 6, 'r', 'k', 'a', 'l', 'l', 'a',
'i', 4, 'b', 'o', 'd', 'y', 'S', 'i', 16, 'I', ' ', 't', 'o', 't', 'a', 'l', 'l', 'y', ' ', 'a', 'g', 'r', 'e', 'e', '!',
'i', 2, 'i', 'd', 'I', 0x04, 0x71,
'i', 9, 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', 'L', 0x00, 0x00, 0x01, 0x3D, 0xB1, 0x78, 0x66, 0x60,
'}', '}'
};
CHECK(json::to_ubjson(j) == v);
CHECK(json::from_ubjson(v) == j);
}
SECTION("size=true type=false")
{
json const j =
{
{
"post", {
{"id", 1137},
{"author", "rkalla"},
{"timestamp", 1364482090592},
{"body", "I totally agree!"}
}
}
};
std::vector<uint8_t> const v = {'{', '#', 'i', 1, 'i', 4, 'p', 'o', 's', 't', '{', '#', 'i', 4,
'i', 6, 'a', 'u', 't', 'h', 'o', 'r', 'S', 'i', 6, 'r', 'k', 'a', 'l', 'l', 'a',
'i', 4, 'b', 'o', 'd', 'y', 'S', 'i', 16, 'I', ' ', 't', 'o', 't', 'a', 'l', 'l', 'y', ' ', 'a', 'g', 'r', 'e', 'e', '!',
'i', 2, 'i', 'd', 'I', 0x04, 0x71,
'i', 9, 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', 'L', 0x00, 0x00, 0x01, 0x3D, 0xB1, 0x78, 0x66, 0x60
};
CHECK(json::to_ubjson(j, true) == v);
CHECK(json::from_ubjson(v) == j);
}
SECTION("size=true type=true")
{
json const j =
{
{
"post", {
{"id", 1137},
{"author", "rkalla"},
{"timestamp", 1364482090592},
{"body", "I totally agree!"}
}
}
};
std::vector<uint8_t> const v = {'{', '$', '{', '#', 'i', 1, 'i', 4, 'p', 'o', 's', 't', '#', 'i', 4,
'i', 6, 'a', 'u', 't', 'h', 'o', 'r', 'S', 'i', 6, 'r', 'k', 'a', 'l', 'l', 'a',
'i', 4, 'b', 'o', 'd', 'y', 'S', 'i', 16, 'I', ' ', 't', 'o', 't', 'a', 'l', 'l', 'y', ' ', 'a', 'g', 'r', 'e', 'e', '!',
'i', 2, 'i', 'd', 'I', 0x04, 0x71,
'i', 9, 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', 'L', 0x00, 0x00, 0x01, 0x3D, 0xB1, 0x78, 0x66, 0x60
};
CHECK(json::to_ubjson(j, true, true) == v);
CHECK(json::from_ubjson(v) == j);
}
}
SECTION("Optimized Format")
{
SECTION("Array Example")
{
SECTION("No Optimization")
{
// note the floats have been replaced by doubles
json const j = {29.97, 31.13, 67.0, 2.113, 23.888};
std::vector<uint8_t> const v = {'[',
'D', 0x40, 0x3d, 0xf8, 0x51, 0xeb, 0x85, 0x1e, 0xb8,
'D', 0x40, 0x3f, 0x21, 0x47, 0xae, 0x14, 0x7a, 0xe1,
'D', 0x40, 0x50, 0xc0, 0x00, 0x00, 0x00, 0x00, 0x00,
'D', 0x40, 0x00, 0xe7, 0x6c, 0x8b, 0x43, 0x95, 0x81,
'D', 0x40, 0x37, 0xe3, 0x53, 0xf7, 0xce, 0xd9, 0x17,
']'
};
CHECK(json::to_ubjson(j) == v);
CHECK(json::from_ubjson(v) == j);
}
SECTION("Optimized with count")
{
// note the floats have been replaced by doubles
json const j = {29.97, 31.13, 67.0, 2.113, 23.888};
std::vector<uint8_t> const v = {'[', '#', 'i', 5,
'D', 0x40, 0x3d, 0xf8, 0x51, 0xeb, 0x85, 0x1e, 0xb8,
'D', 0x40, 0x3f, 0x21, 0x47, 0xae, 0x14, 0x7a, 0xe1,
'D', 0x40, 0x50, 0xc0, 0x00, 0x00, 0x00, 0x00, 0x00,
'D', 0x40, 0x00, 0xe7, 0x6c, 0x8b, 0x43, 0x95, 0x81,
'D', 0x40, 0x37, 0xe3, 0x53, 0xf7, 0xce, 0xd9, 0x17
};
CHECK(json::to_ubjson(j, true) == v);
CHECK(json::from_ubjson(v) == j);
}
SECTION("Optimized with type & count")
{
// note the floats have been replaced by doubles
json const j = {29.97, 31.13, 67.0, 2.113, 23.888};
std::vector<uint8_t> const v = {'[', '$', 'D', '#', 'i', 5,
0x40, 0x3d, 0xf8, 0x51, 0xeb, 0x85, 0x1e, 0xb8,
0x40, 0x3f, 0x21, 0x47, 0xae, 0x14, 0x7a, 0xe1,
0x40, 0x50, 0xc0, 0x00, 0x00, 0x00, 0x00, 0x00,
0x40, 0x00, 0xe7, 0x6c, 0x8b, 0x43, 0x95, 0x81,
0x40, 0x37, 0xe3, 0x53, 0xf7, 0xce, 0xd9, 0x17
};
CHECK(json::to_ubjson(j, true, true) == v);
CHECK(json::from_ubjson(v) == j);
}
}
SECTION("Object Example")
{
SECTION("No Optimization")
{
// note the floats have been replaced by doubles
json const j = { {"lat", 29.976}, {"long", 31.131}, {"alt", 67.0} };
std::vector<uint8_t> const v = {'{',
'i', 3, 'a', 'l', 't', 'D', 0x40, 0x50, 0xc0, 0x00, 0x00, 0x00, 0x00, 0x00,
'i', 3, 'l', 'a', 't', 'D', 0x40, 0x3d, 0xf9, 0xdb, 0x22, 0xd0, 0xe5, 0x60,
'i', 4, 'l', 'o', 'n', 'g', 'D', 0x40, 0x3f, 0x21, 0x89, 0x37, 0x4b, 0xc6, 0xa8,
'}'
};
CHECK(json::to_ubjson(j) == v);
CHECK(json::from_ubjson(v) == j);
}
SECTION("Optimized with count")
{
// note the floats have been replaced by doubles
json const j = { {"lat", 29.976}, {"long", 31.131}, {"alt", 67.0} };
std::vector<uint8_t> const v = {'{', '#', 'i', 3,
'i', 3, 'a', 'l', 't', 'D', 0x40, 0x50, 0xc0, 0x00, 0x00, 0x00, 0x00, 0x00,
'i', 3, 'l', 'a', 't', 'D', 0x40, 0x3d, 0xf9, 0xdb, 0x22, 0xd0, 0xe5, 0x60,
'i', 4, 'l', 'o', 'n', 'g', 'D', 0x40, 0x3f, 0x21, 0x89, 0x37, 0x4b, 0xc6, 0xa8
};
CHECK(json::to_ubjson(j, true) == v);
CHECK(json::from_ubjson(v) == j);
}
SECTION("Optimized with type & count")
{
// note the floats have been replaced by doubles
json const j = { {"lat", 29.976}, {"long", 31.131}, {"alt", 67.0} };
std::vector<uint8_t> const v = {'{', '$', 'D', '#', 'i', 3,
'i', 3, 'a', 'l', 't', 0x40, 0x50, 0xc0, 0x00, 0x00, 0x00, 0x00, 0x00,
'i', 3, 'l', 'a', 't', 0x40, 0x3d, 0xf9, 0xdb, 0x22, 0xd0, 0xe5, 0x60,
'i', 4, 'l', 'o', 'n', 'g', 0x40, 0x3f, 0x21, 0x89, 0x37, 0x4b, 0xc6, 0xa8
};
CHECK(json::to_ubjson(j, true, true) == v);
CHECK(json::from_ubjson(v) == j);
}
}
SECTION("Special Cases (Null, No-Op and Boolean)")
{
SECTION("Array")
{
std::vector<uint8_t> const v = {'[', '$', 'N', '#', 'I', 0x02, 0x00};
CHECK(json::from_ubjson(v) == json::array());
}
SECTION("Object")
{
std::vector<uint8_t> const v = {'{', '$', 'Z', '#', 'i', 3, 'i', 4, 'n', 'a', 'm', 'e', 'i', 8, 'p', 'a', 's', 's', 'w', 'o', 'r', 'd', 'i', 5, 'e', 'm', 'a', 'i', 'l'};
CHECK(json::from_ubjson(v) == json({ {"name", nullptr}, {"password", nullptr}, {"email", nullptr} }));
}
}
}
}
TEST_CASE("Parse UBJSON directly from a file using iterator and sentinel")
{
std::string const filename = TEST_DATA_DIRECTORY "/json_testsuite/sample.json.ubjson";
std::ifstream file(filename, std::ios::binary);
const std::istreambuf_iterator<char> first(file);
const json parsed = json::from_ubjson(first, utils::istreambuf_sentinel{});
CHECK((parsed.is_object() || parsed.is_array()));
}
#if !defined(JSON_NOEXCEPTION)
TEST_CASE("all UBJSON first bytes")
{
// these bytes will fail immediately with exception parse_error.112
std::set<uint8_t> supported =
{
'T', 'F', 'Z', 'U', 'i', 'I', 'l', 'L', 'd', 'D', 'C', 'S', '[', '{', 'N', 'H'
};
for (auto i = 0; i < 256; ++i)
{
const auto byte = static_cast<uint8_t>(i);
CAPTURE(byte)
try
{
auto res = json::from_ubjson(std::vector<uint8_t>(1, byte));
}
catch (const json::parse_error& e)
{
// check that parse_error.112 is only thrown if the
// first byte is not in the supported set
INFO_WITH_TEMP(e.what());
if (supported.find(byte) == supported.end())
{
CHECK(e.id == 112);
}
else
{
CHECK(e.id != 112);
}
}
}
}
#endif
TEST_CASE("UBJSON use_type requires use_size")
{
SECTION("non-empty array throws other_error.502")
{
const json j = {1, 2, 3};
CHECK_THROWS_WITH_AS(json::to_ubjson(j, false, true),
"[json.exception.other_error.502] use_type requires use_size = true",
json::other_error&);
}
SECTION("non-empty object throws other_error.502")
{
const json j = {{"a", 1}, {"b", 2}};
CHECK_THROWS_WITH_AS(json::to_ubjson(j, false, true),
"[json.exception.other_error.502] use_type requires use_size = true",
json::other_error&);
}
SECTION("scalars do not throw with use_type=true, use_count=false")
{
CHECK_NOTHROW(json::to_ubjson(42, false, true));
CHECK_NOTHROW(json::to_ubjson(3.14, false, true));
CHECK_NOTHROW(json::to_ubjson("hello", false, true));
CHECK_NOTHROW(json::to_ubjson(true, false, true));
CHECK_NOTHROW(json::to_ubjson(nullptr, false, true));
}
SECTION("empty containers do not throw with use_type=true, use_count=false")
{
CHECK_NOTHROW(json::to_ubjson(json::array(), false, true));
CHECK_NOTHROW(json::to_ubjson(json::object(), false, true));
}
SECTION("valid combinations on non-empty containers")
{
const json j = {1, 2, 3};
CHECK_NOTHROW(json::to_ubjson(j, false, false));
CHECK_NOTHROW(json::to_ubjson(j, true, false));
CHECK_NOTHROW(json::to_ubjson(j, true, true));
}
}
TEST_CASE("UBJSON round-trip invariants")
{
// This checks what the parse_ubjson_fuzzer driver checks (see
// tests/src/fuzzer-parse_ubjson.cpp), so that a regression shows up in CI
// rather than as an OSS-Fuzz report: every value from_ubjson() returns
// (j1) can be serialized with any combination of options, the result can
// be parsed back (j2), and serializing j2 again with the same options
// reproduces the exact bytes. Beyond the driver, this also checks that j2
// equals j1. Values are compared with dump() rather than operator==,
// because a NaN never compares equal to itself.
struct options
{
bool use_size;
bool use_type;
};
const std::vector<options> all_options =
{
{false, false},
{true, false},
{true, true},
};
for (const auto& j0 : utils::round_trip_corpus::values())
{
// turn the corpus value into a value as from_ubjson() returns it; this
// has no binary values, as UBJSON writes them as arrays of integers
for (const auto& initial : all_options)
{
const json j1 = json::from_ubjson(json::to_ubjson(j0, initial.use_size, initial.use_type));
for (const auto& o : all_options)
{
INFO("j1 = " << j1.dump() << ", use_size = " << o.use_size << ", use_type = " << o.use_type);
const std::vector<std::uint8_t> vec = json::to_ubjson(j1, o.use_size, o.use_type);
json j2;
// anything the library writes must be parsable by the library
REQUIRE_NOTHROW(j2 = json::from_ubjson(vec));
CHECK(j2.dump() == j1.dump());
CHECK(json::to_ubjson(j2, o.use_size, o.use_type) == vec);
}
}
}
}
TEST_CASE("UBJSON roundtrips" * doctest::skip())
{
SECTION("input from self-generated UBJSON files")
{
for (std::string filename :
{
TEST_DATA_DIRECTORY "/json_nlohmann_tests/all_unicode.json",
TEST_DATA_DIRECTORY "/json.org/1.json",
TEST_DATA_DIRECTORY "/json.org/2.json",
TEST_DATA_DIRECTORY "/json.org/3.json",
TEST_DATA_DIRECTORY "/json.org/4.json",
TEST_DATA_DIRECTORY "/json.org/5.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip01.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip02.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip03.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip04.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip05.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip06.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip07.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip08.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip09.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip10.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip11.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip12.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip13.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip14.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip15.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip16.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip17.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip18.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip19.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip20.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip21.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip22.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip23.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip24.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip25.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip26.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip27.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip28.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip29.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip30.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip31.json",
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip32.json",
TEST_DATA_DIRECTORY "/json_testsuite/sample.json",
TEST_DATA_DIRECTORY "/json_tests/pass1.json",
TEST_DATA_DIRECTORY "/json_tests/pass2.json",
TEST_DATA_DIRECTORY "/json_tests/pass3.json"
})
{
CAPTURE(filename)
std::ifstream f_json(filename);
json const j1 = json::parse(f_json);
auto const packed = utils::read_binary_file(filename + ".ubjson");
{
INFO_WITH_TEMP(filename + ": std::vector<uint8_t>");
json j2;
CHECK_NOTHROW(j2 = json::from_ubjson(packed));
CHECK(j1 == j2);
}
{
INFO_WITH_TEMP(filename + ": std::ifstream");
std::ifstream f_ubjson(filename + ".ubjson", std::ios::binary);
json j2;
CHECK_NOTHROW(j2 = json::from_ubjson(f_ubjson));
CHECK(j1 == j2);
}
#ifndef JSON_TEST_DEPRECATED_FUNCTIONS_DELETED
{
INFO_WITH_TEMP(filename + ": uint8_t* and size");
json j2;
CHECK_NOTHROW(j2 = json::from_ubjson({packed.data(), packed.size()}));
CHECK(j1 == j2);
}
#endif
{
INFO_WITH_TEMP(filename + ": output to output adapters");
{
INFO_WITH_TEMP(filename + ": output adapters: std::vector<uint8_t>");
std::vector<uint8_t> vec;
json::to_ubjson(j1, vec);
CHECK(vec == packed);
}
}
}
}
}
TEST_CASE("UBJSON optimized array of unsigned integers beyond int64")
{
// UBJSON has no unsigned 64-bit type, so such values are written as
// high-precision numbers - also as the type of an optimized container
const json j = {18446744073709551615ULL, 9223372036854775808ULL};
const std::vector<std::uint8_t> expected =
{
'[', '$', 'H', '#', 'i', 2,
'i', 20, '1', '8', '4', '4', '6', '7', '4', '4', '0', '7', '3', '7', '0', '9', '5', '5', '1', '6', '1', '5',
'i', 19, '9', '2', '2', '3', '3', '7', '2', '0', '3', '6', '8', '5', '4', '7', '7', '5', '8', '0', '8'
};
CHECK(json::to_ubjson(j, true, true) == expected);
CHECK(json::from_ubjson(expected) == j);
}
namespace
{
// the bytes that follow the marker of an integer: the value in the width of
// the marker (big endian for UBJSON, little endian for BJData), or, for a
// high-precision number, the length and the decimal digits
std::vector<std::uint8_t> integer_payload(const char marker, const json& value, const bool little_endian)
{
std::size_t width = 0;
switch (marker)
{
case 'i':
case 'U':
width = 1;
break;
case 'I':
case 'u':
width = 2;
break;
case 'l':
case 'm':
width = 4;
break;
case 'L':
case 'M':
width = 8;
break;
default:
{
const std::string digits = value.dump();
std::vector<std::uint8_t> result = {'i', static_cast<std::uint8_t>(digits.size())};
for (const char c : digits)
{
result.push_back(static_cast<std::uint8_t>(c));
}
return result;
}
}
const std::uint64_t bits = value.is_number_unsigned()
? value.get<std::uint64_t>()
: static_cast<std::uint64_t>(value.get<std::int64_t>());
std::vector<std::uint8_t> result(width);
for (std::size_t i = 0; i < width; ++i)
{
result[little_endian ? i : width - 1 - i] = static_cast<std::uint8_t>(bits >> (8 * i));
}
return result;
}
json i64(const std::int64_t v)
{
return v;
}
json u64(const std::uint64_t v)
{
return v;
}
} // namespace
TEST_CASE("UBJSON and BJData integer markers at every range edge")
{
// An optimized container announces the marker of its values after `$` and
// then writes every value without a marker, so the marker the writer
// announces and the width it writes must match for every value. This
// checks both for the values around each edge of the integer types, as
// scalars and as the values of optimized arrays and objects.
struct integer_case
{
json value;
char ubjson; // expected UBJSON marker
char bjdata; // expected BJData marker
};
const std::int64_t int64_min = (std::numeric_limits<std::int64_t>::min)();
const std::int64_t int64_max = (std::numeric_limits<std::int64_t>::max)();
const std::uint64_t uint64_max = (std::numeric_limits<std::uint64_t>::max)();
const std::vector<integer_case> cases =
{
// int8
{i64(-129), 'I', 'I'},
{i64(-128), 'i', 'i'},
{i64(-127), 'i', 'i'},
{i64(-1), 'i', 'i'},
{i64(0), 'i', 'i'},
{u64(0), 'i', 'i'},
{i64(126), 'i', 'i'},
{i64(127), 'i', 'i'},
{u64(127), 'i', 'i'},
{i64(128), 'U', 'U'},
{u64(128), 'U', 'U'},
// uint8
{i64(254), 'U', 'U'},
{i64(255), 'U', 'U'},
{u64(255), 'U', 'U'},
{i64(256), 'I', 'I'},
{u64(256), 'I', 'I'},
// int16
{i64(-32769), 'l', 'l'},
{i64(-32768), 'I', 'I'},
{i64(-32767), 'I', 'I'},
{i64(32766), 'I', 'I'},
{i64(32767), 'I', 'I'},
{u64(32767), 'I', 'I'},
{i64(32768), 'l', 'u'},
{u64(32768), 'l', 'u'},
// uint16 (BJData only)
{i64(65534), 'l', 'u'},
{i64(65535), 'l', 'u'},
{u64(65535), 'l', 'u'},
{i64(65536), 'l', 'l'},
{u64(65536), 'l', 'l'},
// int32
{i64(-2147483649LL), 'L', 'L'},
{i64(-2147483648LL), 'l', 'l'},
{i64(-2147483647LL), 'l', 'l'},
{i64(2147483646LL), 'l', 'l'},
{i64(2147483647LL), 'l', 'l'},
{u64(2147483647ULL), 'l', 'l'},
{i64(2147483648LL), 'L', 'm'},
{u64(2147483648ULL), 'L', 'm'},
// uint32 (BJData only)
{i64(4294967294LL), 'L', 'm'},
{i64(4294967295LL), 'L', 'm'},
{u64(4294967295ULL), 'L', 'm'},
{i64(4294967296LL), 'L', 'L'},
{u64(4294967296ULL), 'L', 'L'},
// int64
{i64(int64_min), 'L', 'L'},
{i64(int64_min + 1), 'L', 'L'},
{i64(int64_max - 1), 'L', 'L'},
{i64(int64_max), 'L', 'L'},
{u64(static_cast<std::uint64_t>(int64_max)), 'L', 'L'},
// uint64 (BJData only; UBJSON writes a high-precision number)
{u64(static_cast<std::uint64_t>(int64_max) + 1), 'H', 'M'},
{u64(uint64_max - 1), 'H', 'M'},
{u64(uint64_max), 'H', 'M'},
};
for (const auto& c : cases)
{
for (const bool bjdata :
{
false, true
})
{
const char marker = bjdata ? c.bjdata : c.ubjson;
const std::vector<std::uint8_t> payload = integer_payload(marker, c.value, bjdata);
const auto to_binary = [bjdata](const json & j, const bool use_size, const bool use_type)
{
return bjdata ? json::to_bjdata(j, use_size, use_type) : json::to_ubjson(j, use_size, use_type);
};
const auto from_binary = [bjdata](const std::vector<std::uint8_t>& v)
{
return bjdata ? json::from_bjdata(v) : json::from_ubjson(v);
};
INFO("value = " << c.value.dump() << (c.value.is_number_unsigned() ? " (unsigned)" : "") << ", format = " << (bjdata ? "BJData" : "UBJSON"));
// scalar
std::vector<std::uint8_t> expected = {static_cast<std::uint8_t>(marker)};
expected.insert(expected.end(), payload.begin(), payload.end());
for (const bool use_size :
{
false, true
})
{
CHECK(to_binary(c.value, use_size, false) == expected);
}
CHECK(from_binary(expected) == c.value);
const json arr = {c.value, c.value, c.value};
// array without count or type: every value has its marker
expected = {'['};
for (int i = 0; i < 3; ++i)
{
expected.push_back(static_cast<std::uint8_t>(marker));
expected.insert(expected.end(), payload.begin(), payload.end());
}
expected.push_back(']');
CHECK(to_binary(arr, false, false) == expected);
CHECK(from_binary(expected) == arr);
// array with count: every value has its marker
expected = {'[', '#', 'i', 3};
for (int i = 0; i < 3; ++i)
{
expected.push_back(static_cast<std::uint8_t>(marker));
expected.insert(expected.end(), payload.begin(), payload.end());
}
CHECK(to_binary(arr, true, false) == expected);
CHECK(from_binary(expected) == arr);
// array with type and count: the marker once, then the payloads
expected = {'[', '$', static_cast<std::uint8_t>(marker), '#', 'i', 3};
for (int i = 0; i < 3; ++i)
{
expected.insert(expected.end(), payload.begin(), payload.end());
}
CHECK(to_binary(arr, true, true) == expected);
CHECK(from_binary(expected) == arr);
// object with type and count: the marker once, then key and payload
const json obj = {{"a", c.value}, {"b", c.value}};
expected = {'{', '$', static_cast<std::uint8_t>(marker), '#', 'i', 2};
for (const char key :
{'a', 'b'
})
{
expected.push_back('i');
expected.push_back(1);
expected.push_back(static_cast<std::uint8_t>(key));
expected.insert(expected.end(), payload.begin(), payload.end());
}
CHECK(to_binary(obj, true, true) == expected);
CHECK(from_binary(expected) == obj);
}
}
}
TEST_CASE("UBJSON large strings (chunked reader)")
{
// get_ubjson_string() reads through get_string(), which reads in bounded
// chunks (binary_reader.hpp, chunk_size == 4096); make sure roundtripping
// is correct for lengths around and beyond that chunk size, for both
// vector (iterator) and pointer inputs.
//
// A binary value is not included here: plain UBJSON (unlike BJData, see
// the "BJData large strings and binaries" test) has no reader-side binary
// type, so even the optimized uint8_t-array encoding of a binary value is
// read back element-by-element as a JSON array of numbers rather than
// through get_binary() - it never reaches the chunked path this test is
// about (see the "roundtrip only works to an array of numbers" case
// above).
for (const std::size_t len :
{
std::size_t{0}, std::size_t{1}, std::size_t{4095}, std::size_t{4096},
std::size_t{4097}, std::size_t{8192}, std::size_t{100000}
})
{
CAPTURE(len)
const json j_string = std::string(len, 'x');
const std::vector<std::uint8_t> v_string = json::to_ubjson(j_string);
CHECK(json::from_ubjson(v_string) == j_string);
// pointer input exercises the std::memcpy fast path
CHECK(json::from_ubjson(reinterpret_cast<const char*>(v_string.data()),
reinterpret_cast<const char*>(v_string.data()) + v_string.size()) == j_string);
// a truncated payload must still be reported as an error
if (len > 16)
{
std::vector<std::uint8_t> truncated = v_string;
truncated.resize(truncated.size() - 8);
json _;
CHECK_THROWS_AS(_ = json::from_ubjson(truncated), json::parse_error);
}
}
}
TEST_CASE("UBJSON optimized container type marker is validated for count 0")
{
// the '$' type marker must be a legal marker even if no element follows
SECTION("invalid marker")
{
for (const char marker :
{'x', 'E', 'B', 'u', 'm', 'M', 'h'
})
{
for (const char open :
{'[', '{'
})
{
for (const int count :
{
0, 1
})
{
CAPTURE(marker)
CAPTURE(open)
CAPTURE(count)
const std::vector<std::uint8_t> v = {static_cast<std::uint8_t>(open), '$', static_cast<std::uint8_t>(marker), '#', 'i', static_cast<std::uint8_t>(count)};
json _;
CHECK_THROWS_AS(_ = json::from_ubjson(v), json::parse_error&);
CHECK(json::from_ubjson(v, true, false).is_discarded());
}
}
}
}
SECTION("error message")
{
const std::vector<std::uint8_t> v = {'[', '$', 'x', '#', 'i', 0};
json _;
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v), "[json.exception.parse_error.112] parse error at byte 3: syntax error while parsing UBJSON type: marker 0x78 is not a permitted optimized array type", json::parse_error&);
}
SECTION("valid marker")
{
for (const char marker :
{'Z', 'N', 'T', 'F', 'i', 'U', 'I', 'l', 'L', 'd', 'D', 'H', 'C', 'S', '[', '{'
})
{
CAPTURE(marker)
const std::vector<std::uint8_t> va = {'[', '$', static_cast<std::uint8_t>(marker), '#', 'i', 0};
const std::vector<std::uint8_t> vo = {'{', '$', static_cast<std::uint8_t>(marker), '#', 'i', 0};
CHECK(json::from_ubjson(va) == json::array());
CHECK(json::from_ubjson(vo) == json::object());
}
}
}
Reference in new issue View Git Blame Copy Permalink