Compare commits

..
Author SHA1 Message Date
Niels Lohmann b0a4218e38 Report the same error position for bulk number runs as for byte-wise input
For contiguous input, the CBOR and MessagePack readers read the
scalars of an array in one loop, and the UBJSON/BJData readers the
numbers of an optimized array. These loops passed a value on before
they advanced the position and the last byte. With a number_float_t
narrower than the value (for example float), passing the value on
reports out_of_range.406, and SAX parse_error() then got a different
position and last token than for the same bytes read through a
non-contiguous adapter.

Advance before passing the value on in the CBOR and MessagePack loops.
The last byte is the head byte, as get_number() does not change it;
the MessagePack loop used the last byte of the value. In the typed
UBJSON/BJData loop, update the position per value only where passing
it on can report an error, which is decided at compile time and never
the case for the default number types, and do not change the last byte,
as reading the values one by one does not.

Instruction counts of SAX and DOM parsing of arrays of one million
numbers: CBOR -1 to -4 %, MessagePack -2 to -10 %, UBJSON/BJData typed
arrays -0.8 to +0.6 % (clang, -O2).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-11-16 07:41:29 +01:00
7 changed files with 303 additions and 274 deletions

No files matched your search

@@ -79,12 +79,7 @@ basic_json(basic_json&& other) noexcept;
(e.g., `json j = std::move(vec);` or `json j(std::vector<std::string>{...});`), the elements are moved into
the JSON value instead of copied. Afterwards, the elements of the source container are in a valid but
unspecified state (**moved-from**). Keys of associative containers are `const` and therefore still copied.
Lvalues and `const` rvalues are copied, and the source is left unchanged. Only containers that own their
elements are moved from, that is, containers whose elements are `const` when accessed through a `const`
container. Non-owning ranges such as spans or iterator ranges (e.g., `gsl::span`, `std::span`) are copied, so
converting an rvalue span leaves the referenced elements unchanged.
A user-defined view whose elements are `const` through a `const` view, but which does not own them, is
treated as owning; pass such a view as an lvalue to copy the elements.
Lvalues and `const` rvalues are copied, and the source is left unchanged.
- **numbers**: [`number_integer_t`](number_integer_t.md), [`number_unsigned_t`](number_unsigned_t.md),
[`number_float_t`](number_float_t.md), and all convertible number types such as `int`, `size_t`, `int64_t`, `float`
or `double` can be used.
+1 -3
View File
@@ -7,9 +7,7 @@ These changes are part of the `develop` branch, but not of a release yet.
2.3 ms). Two consequences: the source container is left in a valid but unspecified (moved-from) state, and if an
exception is thrown during the conversion, the source may be partially moved-from, so the strong exception guarantee
no longer holds for the source of an rvalue conversion (the JSON value itself is unaffected). Lvalues and `const`
objects are still copied, and so are non-owning ranges such as spans (a range whose elements stay mutable through a
`const` range): converting an rvalue span leaves the referenced elements unchanged.
See [`basic_json`](../../api/basic_json/basic_json.md).
objects are still copied. See [`basic_json`](../../api/basic_json/basic_json.md).
- In C++20 (library support for `char8_t`, `__cpp_lib_char8_t`), a `#!cpp std::u8string` is a JSON **string**:
`json j = std::u8string(u8"abc");` stores `"abc"`, and `#!cpp j.get<std::u8string>()` reads it back. Version 3.12.0
stored a `#!cpp std::u8string` as an array of code units (`[97,98,99]`); on the 3.13.0 development branch it was
+17 -35
View File
@@ -48,47 +48,30 @@ namespace detail
* leave j with the type of the new value, but the pointer to the destroyed old one.
*/
// reference type of the elements of a const T
template<typename T>
using const_range_reference_t = decltype(*std::declval<iterator_t<const T>&>());
// a container that owns its elements propagates constness to them (deep
// constness): the elements of a const std::vector<std::string> are const. A
// non-owning range (a span, an iterator range) has shallow constness: a const
// span still gives mutable access to the elements it refers to. Moving from an
// rvalue of such a range would empty the storage it refers to, so the elements
// of an rvalue range are only moved if it has deep constness.
template<typename T>
using owns_its_elements = std::is_const<typename std::remove_reference<detected_t<const_range_reference_t, T>>::type>;
// a move iterator over the elements of an rvalue container, unless they are
// const (e.g., std::set): libstdc++ before GCC 5 defines the reference of a
// std::move_iterator as value_type&&, which cannot bind to a const element
template<typename Iterator>
using is_mutable_iterator = std::integral_constant < bool,
!std::is_const<typename std::remove_reference<decltype(*std::declval<Iterator&>())>::type>::value >;
// a move iterator over the elements of an rvalue container that owns them,
// unless they are const (e.g., std::set): libstdc++ before GCC 5 defines the
// reference of a std::move_iterator as value_type&&, which cannot bind to a
// const element
template<typename Iterator>
std::move_iterator<Iterator> make_move_iterator_if_impl(const Iterator& it, std::true_type /*move*/)
std::move_iterator<Iterator> make_move_iterator_if_mutable_impl(const Iterator& it, std::true_type /*mutable*/)
{
return std::make_move_iterator(it);
}
template<typename Iterator>
Iterator make_move_iterator_if_impl(const Iterator& it, std::false_type /*move*/)
Iterator make_move_iterator_if_mutable_impl(const Iterator& it, std::false_type /*mutable*/)
{
return it;
}
template<typename Container, typename Iterator>
using moves_elements = conjunction<owns_its_elements<Container>, is_mutable_iterator<Iterator>>;
template<typename Iterator>
using is_mutable_iterator = std::integral_constant < bool,
!std::is_const<typename std::remove_reference<decltype(*std::declval<Iterator&>())>::type>::value >;
template<typename Container, typename Iterator>
auto make_move_iterator_if_owned(const Iterator& it)
-> decltype(make_move_iterator_if_impl(it, moves_elements<Container, Iterator> {}))
template<typename Iterator>
auto make_move_iterator_if_mutable(const Iterator& it)
-> decltype(make_move_iterator_if_mutable_impl(it, is_mutable_iterator<Iterator> {}))
{
return make_move_iterator_if_impl(it, moves_elements<Container, Iterator> {});
return make_move_iterator_if_mutable_impl(it, is_mutable_iterator<Iterator> {});
}
template<value_t> struct external_constructor;
@@ -210,14 +193,14 @@ struct external_constructor<value_t::array>
j.replace_value(value_t::array, value);
}
/// an rvalue container: move its elements if it owns them
/// an rvalue container: move its elements
template<typename BasicJsonType, typename CompatibleArrayType>
static void construct_moving(BasicJsonType& j, CompatibleArrayType& arr)
{
using std::begin;
using std::end;
const typename BasicJsonType::json_value value(j.template create<typename BasicJsonType::array_t>(make_move_iterator_if_owned<CompatibleArrayType>(begin(arr)), make_move_iterator_if_owned<CompatibleArrayType>(end(arr))));
const typename BasicJsonType::json_value value(j.template create<typename BasicJsonType::array_t>(make_move_iterator_if_mutable(begin(arr)), make_move_iterator_if_mutable(end(arr))));
j.replace_value(value_t::array, value);
}
@@ -271,14 +254,14 @@ struct external_constructor<value_t::object>
j.replace_value(value_t::object, value);
}
/// an rvalue container: move its values if it owns them
/// an rvalue container: move its values
template<typename BasicJsonType, typename CompatibleObjectType>
static void construct_moving(BasicJsonType& j, CompatibleObjectType& obj)
{
using std::begin;
using std::end;
const typename BasicJsonType::json_value value(j.template create<typename BasicJsonType::object_t>(make_move_iterator_if_owned<CompatibleObjectType>(begin(obj)), make_move_iterator_if_owned<CompatibleObjectType>(end(obj))));
const typename BasicJsonType::json_value value(j.template create<typename BasicJsonType::object_t>(make_move_iterator_if_mutable(begin(obj)), make_move_iterator_if_mutable(end(obj))));
j.replace_value(value_t::object, value);
}
@@ -473,8 +456,7 @@ using is_rvalue_array_to_json = conjunction<is_movable_container_rvalue<T>,
is_compatible_string_type<BasicJsonType, T>,
is_compatible_binary_type<BasicJsonType, T>>>>;
// rvalue containers: their elements are moved instead of copied if the
// container owns them (see owns_its_elements)
// rvalue containers: their elements are moved instead of copied
template<typename BasicJsonType, typename CompatibleArrayType,
enable_if_t<is_rvalue_array_to_json<BasicJsonType, CompatibleArrayType>::value, int> = 0>
inline void to_json(BasicJsonType& j, CompatibleArrayType && arr) // NOLINT(cppcoreguidelines-missing-std-forward)
+72 -43
View File
@@ -1927,6 +1927,21 @@ class binary_reader
return result;
}
/*!
@brief move past a value of @a size bytes, starting with @a head, that a
bulk run reads, as get() and get_number() would
Called before the value is passed on, so that an error reported while
passing it on (see @ref emit_float) has the position and the last byte
that reading the value byte by byte gives.
*/
void bulk_advance(const std::size_t size, const unsigned char head)
{
ia.bulk_skip(size);
chars_read += size;
current = head;
}
/*!
@brief read the integers, floats, booleans, and nulls that follow in a
definite-length array, without the per-element bookkeeping of
@@ -1952,71 +1967,70 @@ class binary_reader
{
const auto* const data = reinterpret_cast<const unsigned char*>(ia.bulk_data());
const unsigned char head = data[0];
std::size_t size = 0;
bool accepted = false;
switch (head)
{
case 0x18:
size = 2;
bulk_advance(2, head);
accepted = emit_unsigned(data[1]);
break;
case 0x19:
size = 3;
bulk_advance(3, head);
accepted = emit_unsigned(load_big_endian<std::uint16_t>(data + 1));
break;
case 0x1A:
size = 5;
bulk_advance(5, head);
accepted = emit_unsigned(load_big_endian<std::uint32_t>(data + 1));
break;
case 0x1B:
size = 9;
bulk_advance(9, head);
accepted = emit_unsigned(load_big_endian<std::uint64_t>(data + 1));
break;
case 0x38:
size = 2;
bulk_advance(2, head);
accepted = emit_cbor_negative_integer(data[1]);
break;
case 0x39:
size = 3;
bulk_advance(3, head);
accepted = emit_cbor_negative_integer(load_big_endian<std::uint16_t>(data + 1));
break;
case 0x3A:
size = 5;
bulk_advance(5, head);
accepted = emit_cbor_negative_integer(load_big_endian<std::uint32_t>(data + 1));
break;
case 0x3B:
size = 9;
bulk_advance(9, head);
accepted = emit_cbor_negative_integer(load_big_endian<std::uint64_t>(data + 1));
break;
case 0xF4:
size = 1;
bulk_advance(1, head);
accepted = sax->boolean(false);
break;
case 0xF5:
size = 1;
bulk_advance(1, head);
accepted = sax->boolean(true);
break;
case 0xF6:
size = 1;
bulk_advance(1, head);
accepted = sax->null();
break;
case 0xFA:
size = 5;
bulk_advance(5, head);
accepted = emit_float(load_big_endian<float>(data + 1));
break;
case 0xFB:
size = 9;
bulk_advance(9, head);
accepted = emit_float(load_big_endian<double>(data + 1));
break;
default:
if (head <= 0x17)
{
size = 1;
bulk_advance(1, head);
accepted = sax->number_unsigned(static_cast<number_unsigned_t>(head));
}
else if (head >= 0x20 && head <= 0x37)
{
size = 1;
bulk_advance(1, head);
accepted = sax->number_integer(static_cast<std::int8_t>(0x20 - 1 - head));
}
else
@@ -2026,10 +2040,6 @@ class binary_reader
}
break;
}
// as if read by get() and get_number(): current is the head byte
ia.bulk_skip(size);
chars_read += size;
current = head;
--remaining;
if (JSON_HEDLEY_UNLIKELY(!accepted))
{
@@ -2867,73 +2877,72 @@ class binary_reader
{
const auto* const data = reinterpret_cast<const unsigned char*>(ia.bulk_data());
const unsigned char head = data[0];
std::size_t size = 0;
bool accepted = false;
switch (head)
{
case 0xC0: // nil
size = 1;
bulk_advance(1, head);
accepted = sax->null();
break;
case 0xC2: // false
size = 1;
bulk_advance(1, head);
accepted = sax->boolean(false);
break;
case 0xC3: // true
size = 1;
bulk_advance(1, head);
accepted = sax->boolean(true);
break;
case 0xCA: // float 32
size = 5;
bulk_advance(5, head);
accepted = emit_float(load_big_endian<float>(data + 1));
break;
case 0xCB: // float 64
size = 9;
bulk_advance(9, head);
accepted = emit_float(load_big_endian<double>(data + 1));
break;
case 0xCC: // uint 8
size = 2;
bulk_advance(2, head);
accepted = emit_unsigned(data[1]);
break;
case 0xCD: // uint 16
size = 3;
bulk_advance(3, head);
accepted = emit_unsigned(load_big_endian<std::uint16_t>(data + 1));
break;
case 0xCE: // uint 32
size = 5;
bulk_advance(5, head);
accepted = emit_unsigned(load_big_endian<std::uint32_t>(data + 1));
break;
case 0xCF: // uint 64
size = 9;
bulk_advance(9, head);
accepted = emit_unsigned(load_big_endian<std::uint64_t>(data + 1));
break;
case 0xD0: // int 8
size = 2;
bulk_advance(2, head);
accepted = emit_signed(static_cast<std::int8_t>(data[1]));
break;
case 0xD1: // int 16
size = 3;
bulk_advance(3, head);
accepted = emit_signed(load_big_endian<std::int16_t>(data + 1));
break;
case 0xD2: // int 32
size = 5;
bulk_advance(5, head);
accepted = emit_signed(load_big_endian<std::int32_t>(data + 1));
break;
case 0xD3: // int 64
size = 9;
bulk_advance(9, head);
accepted = emit_signed(load_big_endian<std::int64_t>(data + 1));
break;
default:
if (head <= 0x7F)
{
// positive fixint
size = 1;
bulk_advance(1, head);
accepted = sax->number_unsigned(static_cast<number_unsigned_t>(head));
}
else if (head >= 0xE0)
{
// negative fixint
size = 1;
bulk_advance(1, head);
accepted = sax->number_integer(static_cast<std::int8_t>(head));
}
else
@@ -2943,10 +2952,6 @@ class binary_reader
}
break;
}
// as if read by get() and get_number(): current is the last byte
ia.bulk_skip(size);
chars_read += size;
current = data[size - 1];
--remaining;
if (JSON_HEDLEY_UNLIKELY(!accepted))
{
@@ -3850,6 +3855,7 @@ class binary_reader
const auto* const data = reinterpret_cast<const unsigned char*>(ia.bulk_data());
// UBJSON stores numbers big-endian, BJData little-endian
const bool swap = is_little_endian != (input_format == input_format_t::bjdata);
const std::size_t start = chars_read;
std::size_t read = 0;
bool accepted = true;
while (read < count && accepted)
@@ -3861,16 +3867,39 @@ class binary_reader
byte_swap(number);
}
++read;
// where passing the value on can report an error (see
// emit_float), the position is that after the value, as if it
// was read by get_number()
set_chars_read(start + (read * size), std::integral_constant<bool, may_overflow_number_float<NumberType>()> {});
accepted = emit_ubjson_number(number);
}
// as if read by get_number(): current is the last byte read
// get_number() does not change current
ia.bulk_skip(read * size);
chars_read += read * size;
current = data[(read * size) - 1];
chars_read = start + (read * size);
remaining -= read;
return accepted;
}
/*!
@brief whether a value of type @a NumberType can be finite and still
overflow number_float_t, which @ref emit_float reports as an error
False for the default number types, so that the loop in @ref
parse_ubjson_typed_run need not track the position per value.
*/
template<typename NumberType>
static constexpr bool may_overflow_number_float() noexcept
{
return static_cast<long double>((std::numeric_limits<number_float_t>::max)()) < static_cast<long double>((std::numeric_limits<NumberType>::max)());
}
void set_chars_read(const std::size_t position, std::true_type /*track*/) noexcept
{
chars_read = position;
}
void set_chars_read(const std::size_t /*position*/, std::false_type /*track*/) const noexcept {}
template<typename NumberType, enable_if_t<std::is_floating_point<NumberType>::value, int> = 0>
bool emit_ubjson_number(const NumberType number)
{
+89 -78
View File
@@ -7810,47 +7810,30 @@ namespace detail
* leave j with the type of the new value, but the pointer to the destroyed old one.
*/
// reference type of the elements of a const T
template<typename T>
using const_range_reference_t = decltype(*std::declval<iterator_t<const T>&>());
// a container that owns its elements propagates constness to them (deep
// constness): the elements of a const std::vector<std::string> are const. A
// non-owning range (a span, an iterator range) has shallow constness: a const
// span still gives mutable access to the elements it refers to. Moving from an
// rvalue of such a range would empty the storage it refers to, so the elements
// of an rvalue range are only moved if it has deep constness.
template<typename T>
using owns_its_elements = std::is_const<typename std::remove_reference<detected_t<const_range_reference_t, T>>::type>;
// a move iterator over the elements of an rvalue container, unless they are
// const (e.g., std::set): libstdc++ before GCC 5 defines the reference of a
// std::move_iterator as value_type&&, which cannot bind to a const element
template<typename Iterator>
using is_mutable_iterator = std::integral_constant < bool,
!std::is_const<typename std::remove_reference<decltype(*std::declval<Iterator&>())>::type>::value >;
// a move iterator over the elements of an rvalue container that owns them,
// unless they are const (e.g., std::set): libstdc++ before GCC 5 defines the
// reference of a std::move_iterator as value_type&&, which cannot bind to a
// const element
template<typename Iterator>
std::move_iterator<Iterator> make_move_iterator_if_impl(const Iterator& it, std::true_type /*move*/)
std::move_iterator<Iterator> make_move_iterator_if_mutable_impl(const Iterator& it, std::true_type /*mutable*/)
{
return std::make_move_iterator(it);
}
template<typename Iterator>
Iterator make_move_iterator_if_impl(const Iterator& it, std::false_type /*move*/)
Iterator make_move_iterator_if_mutable_impl(const Iterator& it, std::false_type /*mutable*/)
{
return it;
}
template<typename Container, typename Iterator>
using moves_elements = conjunction<owns_its_elements<Container>, is_mutable_iterator<Iterator>>;
template<typename Iterator>
using is_mutable_iterator = std::integral_constant < bool,
!std::is_const<typename std::remove_reference<decltype(*std::declval<Iterator&>())>::type>::value >;
template<typename Container, typename Iterator>
auto make_move_iterator_if_owned(const Iterator& it)
-> decltype(make_move_iterator_if_impl(it, moves_elements<Container, Iterator> {}))
template<typename Iterator>
auto make_move_iterator_if_mutable(const Iterator& it)
-> decltype(make_move_iterator_if_mutable_impl(it, is_mutable_iterator<Iterator> {}))
{
return make_move_iterator_if_impl(it, moves_elements<Container, Iterator> {});
return make_move_iterator_if_mutable_impl(it, is_mutable_iterator<Iterator> {});
}
template<value_t> struct external_constructor;
@@ -7972,14 +7955,14 @@ struct external_constructor<value_t::array>
j.replace_value(value_t::array, value);
}
/// an rvalue container: move its elements if it owns them
/// an rvalue container: move its elements
template<typename BasicJsonType, typename CompatibleArrayType>
static void construct_moving(BasicJsonType& j, CompatibleArrayType& arr)
{
using std::begin;
using std::end;
const typename BasicJsonType::json_value value(j.template create<typename BasicJsonType::array_t>(make_move_iterator_if_owned<CompatibleArrayType>(begin(arr)), make_move_iterator_if_owned<CompatibleArrayType>(end(arr))));
const typename BasicJsonType::json_value value(j.template create<typename BasicJsonType::array_t>(make_move_iterator_if_mutable(begin(arr)), make_move_iterator_if_mutable(end(arr))));
j.replace_value(value_t::array, value);
}
@@ -8033,14 +8016,14 @@ struct external_constructor<value_t::object>
j.replace_value(value_t::object, value);
}
/// an rvalue container: move its values if it owns them
/// an rvalue container: move its values
template<typename BasicJsonType, typename CompatibleObjectType>
static void construct_moving(BasicJsonType& j, CompatibleObjectType& obj)
{
using std::begin;
using std::end;
const typename BasicJsonType::json_value value(j.template create<typename BasicJsonType::object_t>(make_move_iterator_if_owned<CompatibleObjectType>(begin(obj)), make_move_iterator_if_owned<CompatibleObjectType>(end(obj))));
const typename BasicJsonType::json_value value(j.template create<typename BasicJsonType::object_t>(make_move_iterator_if_mutable(begin(obj)), make_move_iterator_if_mutable(end(obj))));
j.replace_value(value_t::object, value);
}
@@ -8235,8 +8218,7 @@ using is_rvalue_array_to_json = conjunction<is_movable_container_rvalue<T>,
is_compatible_string_type<BasicJsonType, T>,
is_compatible_binary_type<BasicJsonType, T>>>>;
// rvalue containers: their elements are moved instead of copied if the
// container owns them (see owns_its_elements)
// rvalue containers: their elements are moved instead of copied
template<typename BasicJsonType, typename CompatibleArrayType,
enable_if_t<is_rvalue_array_to_json<BasicJsonType, CompatibleArrayType>::value, int> = 0>
inline void to_json(BasicJsonType& j, CompatibleArrayType && arr) // NOLINT(cppcoreguidelines-missing-std-forward)
@@ -20623,6 +20605,21 @@ class binary_reader
return result;
}
/*!
@brief move past a value of @a size bytes, starting with @a head, that a
bulk run reads, as get() and get_number() would
Called before the value is passed on, so that an error reported while
passing it on (see @ref emit_float) has the position and the last byte
that reading the value byte by byte gives.
*/
void bulk_advance(const std::size_t size, const unsigned char head)
{
ia.bulk_skip(size);
chars_read += size;
current = head;
}
/*!
@brief read the integers, floats, booleans, and nulls that follow in a
definite-length array, without the per-element bookkeeping of
@@ -20648,71 +20645,70 @@ class binary_reader
{
const auto* const data = reinterpret_cast<const unsigned char*>(ia.bulk_data());
const unsigned char head = data[0];
std::size_t size = 0;
bool accepted = false;
switch (head)
{
case 0x18:
size = 2;
bulk_advance(2, head);
accepted = emit_unsigned(data[1]);
break;
case 0x19:
size = 3;
bulk_advance(3, head);
accepted = emit_unsigned(load_big_endian<std::uint16_t>(data + 1));
break;
case 0x1A:
size = 5;
bulk_advance(5, head);
accepted = emit_unsigned(load_big_endian<std::uint32_t>(data + 1));
break;
case 0x1B:
size = 9;
bulk_advance(9, head);
accepted = emit_unsigned(load_big_endian<std::uint64_t>(data + 1));
break;
case 0x38:
size = 2;
bulk_advance(2, head);
accepted = emit_cbor_negative_integer(data[1]);
break;
case 0x39:
size = 3;
bulk_advance(3, head);
accepted = emit_cbor_negative_integer(load_big_endian<std::uint16_t>(data + 1));
break;
case 0x3A:
size = 5;
bulk_advance(5, head);
accepted = emit_cbor_negative_integer(load_big_endian<std::uint32_t>(data + 1));
break;
case 0x3B:
size = 9;
bulk_advance(9, head);
accepted = emit_cbor_negative_integer(load_big_endian<std::uint64_t>(data + 1));
break;
case 0xF4:
size = 1;
bulk_advance(1, head);
accepted = sax->boolean(false);
break;
case 0xF5:
size = 1;
bulk_advance(1, head);
accepted = sax->boolean(true);
break;
case 0xF6:
size = 1;
bulk_advance(1, head);
accepted = sax->null();
break;
case 0xFA:
size = 5;
bulk_advance(5, head);
accepted = emit_float(load_big_endian<float>(data + 1));
break;
case 0xFB:
size = 9;
bulk_advance(9, head);
accepted = emit_float(load_big_endian<double>(data + 1));
break;
default:
if (head <= 0x17)
{
size = 1;
bulk_advance(1, head);
accepted = sax->number_unsigned(static_cast<number_unsigned_t>(head));
}
else if (head >= 0x20 && head <= 0x37)
{
size = 1;
bulk_advance(1, head);
accepted = sax->number_integer(static_cast<std::int8_t>(0x20 - 1 - head));
}
else
@@ -20722,10 +20718,6 @@ class binary_reader
}
break;
}
// as if read by get() and get_number(): current is the head byte
ia.bulk_skip(size);
chars_read += size;
current = head;
--remaining;
if (JSON_HEDLEY_UNLIKELY(!accepted))
{
@@ -21563,73 +21555,72 @@ class binary_reader
{
const auto* const data = reinterpret_cast<const unsigned char*>(ia.bulk_data());
const unsigned char head = data[0];
std::size_t size = 0;
bool accepted = false;
switch (head)
{
case 0xC0: // nil
size = 1;
bulk_advance(1, head);
accepted = sax->null();
break;
case 0xC2: // false
size = 1;
bulk_advance(1, head);
accepted = sax->boolean(false);
break;
case 0xC3: // true
size = 1;
bulk_advance(1, head);
accepted = sax->boolean(true);
break;
case 0xCA: // float 32
size = 5;
bulk_advance(5, head);
accepted = emit_float(load_big_endian<float>(data + 1));
break;
case 0xCB: // float 64
size = 9;
bulk_advance(9, head);
accepted = emit_float(load_big_endian<double>(data + 1));
break;
case 0xCC: // uint 8
size = 2;
bulk_advance(2, head);
accepted = emit_unsigned(data[1]);
break;
case 0xCD: // uint 16
size = 3;
bulk_advance(3, head);
accepted = emit_unsigned(load_big_endian<std::uint16_t>(data + 1));
break;
case 0xCE: // uint 32
size = 5;
bulk_advance(5, head);
accepted = emit_unsigned(load_big_endian<std::uint32_t>(data + 1));
break;
case 0xCF: // uint 64
size = 9;
bulk_advance(9, head);
accepted = emit_unsigned(load_big_endian<std::uint64_t>(data + 1));
break;
case 0xD0: // int 8
size = 2;
bulk_advance(2, head);
accepted = emit_signed(static_cast<std::int8_t>(data[1]));
break;
case 0xD1: // int 16
size = 3;
bulk_advance(3, head);
accepted = emit_signed(load_big_endian<std::int16_t>(data + 1));
break;
case 0xD2: // int 32
size = 5;
bulk_advance(5, head);
accepted = emit_signed(load_big_endian<std::int32_t>(data + 1));
break;
case 0xD3: // int 64
size = 9;
bulk_advance(9, head);
accepted = emit_signed(load_big_endian<std::int64_t>(data + 1));
break;
default:
if (head <= 0x7F)
{
// positive fixint
size = 1;
bulk_advance(1, head);
accepted = sax->number_unsigned(static_cast<number_unsigned_t>(head));
}
else if (head >= 0xE0)
{
// negative fixint
size = 1;
bulk_advance(1, head);
accepted = sax->number_integer(static_cast<std::int8_t>(head));
}
else
@@ -21639,10 +21630,6 @@ class binary_reader
}
break;
}
// as if read by get() and get_number(): current is the last byte
ia.bulk_skip(size);
chars_read += size;
current = data[size - 1];
--remaining;
if (JSON_HEDLEY_UNLIKELY(!accepted))
{
@@ -22546,6 +22533,7 @@ class binary_reader
const auto* const data = reinterpret_cast<const unsigned char*>(ia.bulk_data());
// UBJSON stores numbers big-endian, BJData little-endian
const bool swap = is_little_endian != (input_format == input_format_t::bjdata);
const std::size_t start = chars_read;
std::size_t read = 0;
bool accepted = true;
while (read < count && accepted)
@@ -22557,16 +22545,39 @@ class binary_reader
byte_swap(number);
}
++read;
// where passing the value on can report an error (see
// emit_float), the position is that after the value, as if it
// was read by get_number()
set_chars_read(start + (read * size), std::integral_constant<bool, may_overflow_number_float<NumberType>()> {});
accepted = emit_ubjson_number(number);
}
// as if read by get_number(): current is the last byte read
// get_number() does not change current
ia.bulk_skip(read * size);
chars_read += read * size;
current = data[(read * size) - 1];
chars_read = start + (read * size);
remaining -= read;
return accepted;
}
/*!
@brief whether a value of type @a NumberType can be finite and still
overflow number_float_t, which @ref emit_float reports as an error
False for the default number types, so that the loop in @ref
parse_ubjson_typed_run need not track the position per value.
*/
template<typename NumberType>
static constexpr bool may_overflow_number_float() noexcept
{
return static_cast<long double>((std::numeric_limits<number_float_t>::max)()) < static_cast<long double>((std::numeric_limits<NumberType>::max)());
}
void set_chars_read(const std::size_t position, std::true_type /*track*/) noexcept
{
chars_read = position;
}
void set_chars_read(const std::size_t /*position*/, std::false_type /*track*/) const noexcept {}
template<typename NumberType, enable_if_t<std::is_floating_point<NumberType>::value, int> = 0>
bool emit_ubjson_number(const NumberType number)
{
+123
View File
@@ -14,6 +14,7 @@ using nlohmann::json;
#include <cmath>
#include <fstream>
#include <limits>
#include <list>
#include <map>
#include <string>
#include <vector>
@@ -367,3 +368,125 @@ TEST_CASE("Binary formats with narrow number types")
CHECK(format.decode(format.encode(json(1e300)), false).is_discarded());
}
}
namespace
{
// records the SAX events of narrow_json, including the arguments of parse_error
struct narrow_event_recorder
{
using number_integer_t = narrow_json::number_integer_t;
using number_unsigned_t = narrow_json::number_unsigned_t;
using number_float_t = narrow_json::number_float_t;
using string_t = narrow_json::string_t;
using binary_t = narrow_json::binary_t;
bool null()
{
events.emplace_back("null");
return true;
}
bool boolean(bool val)
{
events.emplace_back(val ? "true" : "false");
return true;
}
bool number_integer(number_integer_t val)
{
events.push_back(std::to_string(val));
return true;
}
bool number_unsigned(number_unsigned_t val)
{
events.push_back(std::to_string(val));
return true;
}
bool number_float(number_float_t val, const string_t& /*unused*/)
{
events.push_back(std::to_string(val));
return true;
}
bool string(string_t& val)
{
events.push_back(val);
return true;
}
bool binary(binary_t& /*unused*/)
{
events.emplace_back("binary");
return true;
}
bool start_object(std::size_t /*unused*/)
{
events.emplace_back("{");
return true;
}
bool key(string_t& val)
{
events.push_back(val);
return true;
}
bool end_object()
{
events.emplace_back("}");
return true;
}
bool start_array(std::size_t /*unused*/)
{
events.emplace_back("[");
return true;
}
bool end_array()
{
events.emplace_back("]");
return true;
}
bool parse_error(std::size_t position, const std::string& last_token, const nlohmann::detail::exception& ex)
{
events.push_back("parse_error(" + std::to_string(position) + ", " + last_token + ", " + ex.what() + ")");
return false;
}
std::vector<std::string> events {}; // NOLINT(readability-redundant-member-init)
};
} // namespace
TEST_CASE("Binary formats with narrow number types from contiguous and non-contiguous input")
{
// contiguous input reads runs of numbers in one loop; a number that
// overflows number_float_t must be reported with the same position and
// last byte as when the input is read byte by byte
const json values = {1, 70000, -70000, 1.5, 2.5, 1e300, 3.5, 4.5, 5.5, 6.5, 7.5, 8.5};
const json doubles = {1.5, 2.5, 3.5, 1e300, 4.5, 5.5, 6.5, 7.5, 8.5};
struct binary_input
{
const char* name;
json::input_format_t format;
bytes data;
};
const std::vector<binary_input> inputs =
{
{"CBOR", json::input_format_t::cbor, json::to_cbor(values)},
{"CBOR doubles", json::input_format_t::cbor, json::to_cbor(doubles)},
{"MessagePack", json::input_format_t::msgpack, json::to_msgpack(values)},
{"MessagePack doubles", json::input_format_t::msgpack, json::to_msgpack(doubles)},
{"UBJSON", json::input_format_t::ubjson, json::to_ubjson(values, true)},
{"UBJSON typed doubles", json::input_format_t::ubjson, json::to_ubjson(doubles, true, true)},
{"BJData", json::input_format_t::bjdata, json::to_bjdata(values, true)},
{"BJData typed doubles", json::input_format_t::bjdata, json::to_bjdata(doubles, true, true)},
};
for (const auto& input : inputs)
{
const std::string name = input.name;
CAPTURE(name)
const std::list<std::uint8_t> bytewise(input.data.begin(), input.data.end());
narrow_event_recorder from_contiguous;
narrow_event_recorder from_bytewise;
CHECK_FALSE(narrow_json::sax_parse(input.data, &from_contiguous, input.format));
CHECK_FALSE(narrow_json::sax_parse(bytewise.begin(), bytewise.end(), &from_bytewise, input.format));
CHECK(from_contiguous.events == from_bytewise.events);
REQUIRE(!from_contiguous.events.empty());
CHECK(from_contiguous.events.back().find("out_of_range.406") != std::string::npos);
}
}
-109
View File
@@ -67,67 +67,6 @@ void to_json(json& j, tracked&& t)
t.moved_from = true;
}
// non-owning range over contiguous storage (like gsl::span or a pre-C++20 span
// backport): its constness is shallow, so a const span gives mutable elements
template<typename T>
struct span_like
{
using value_type = T;
using iterator = T*;
using const_iterator = T*;
T* first;
T* last;
T* begin() const
{
return first;
}
T* end() const
{
return last;
}
};
// non-owning view of key/value pairs with key_type and mapped_type, so it is
// converted like an object
template<typename K, typename V>
struct map_span_like
{
using key_type = K;
using mapped_type = V;
using value_type = std::pair<const K, V>;
using iterator = value_type*;
using const_iterator = value_type*;
value_type* first;
value_type* last;
value_type* begin() const
{
return first;
}
value_type* end() const
{
return last;
}
};
// owning container that can only be iterated when it is not const
struct mutable_only_range
{
std::vector<std::string> values;
std::vector<std::string>::iterator begin()
{
return values.begin();
}
std::vector<std::string>::iterator end()
{
return values.end();
}
};
// user type with its own rvalue-aware serializer
struct udt_container
{
@@ -205,34 +144,6 @@ TEST_CASE("to_json from rvalue containers")
CHECK(tracked_moves == 2);
}
SECTION("tracked elements: standard containers move")
{
tracked_copies = 0;
tracked_moves = 0;
std::list<tracked> l;
l.emplace_back("l");
CHECK(json(std::move(l)) == json::array({"l"}));
std::deque<tracked> d;
d.emplace_back("d");
CHECK(json(std::move(d)) == json::array({"d"}));
std::forward_list<tracked> fl;
fl.emplace_front("f");
CHECK(json(std::move(fl)) == json::array({"f"}));
std::array<tracked, 1> a{{tracked("a")}};
CHECK(json(std::move(a)) == json::array({"a"}));
std::unordered_map<std::string, tracked> um;
um.emplace("k", tracked("u"));
CHECK(json(std::move(um)) == json({{"k", "u"}}));
CHECK(tracked_moves == 5);
CHECK(tracked_copies == 0);
}
SECTION("const rvalue is copied")
{
tracked_copies = 0;
@@ -310,26 +221,6 @@ TEST_CASE("to_json from rvalue containers")
CHECK(json(std::move(obj)) == json({{"a", 1}}));
}
SECTION("non-owning ranges are copied")
{
std::vector<std::string> v{long_a, long_b};
const json j = span_like<std::string> {v.data(), v.data() + v.size()};
CHECK(j == json::array({long_a, long_b}));
CHECK(v[0] == long_a);
CHECK(v[1] == long_b);
std::vector<std::pair<const std::string, std::string>> m{{"a", long_a}};
const json jm = map_span_like<std::string, std::string> {m.data(), m.data() + m.size()};
CHECK(jm == json({{"a", long_a}}));
CHECK(m[0].second == long_a);
// without a const begin(), it cannot be told whether the range owns
// its elements, so they are copied
mutable_only_range r{{long_a}};
CHECK(json(std::move(r)) == json::array({long_a}));
CHECK(r.values[0] == long_a); // NOLINT(bugprone-use-after-move,hicpp-invalid-access-moved)
}
SECTION("user type with its own to_json overloads")
{
udt_container c{{1, 2}};