Compare commits

..
Author SHA1 Message Date
Niels Lohmann c194c7703d Assert that update() and merge_patch() are not called with *this
Passing *this itself is cheap to detect; a value contained in *this is not. Document the assertion. Addresses review comment by @gregmarr.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-30 07:33:15 +02:00
Niels Lohmann 71f3554243 Document that update() and merge_patch() must not alias *this
update() and merge_patch() read their argument while they modify
*this. When the argument is *this or a value nested inside *this
(for example a subobject returned by operator[]), the modification
destroys or relocates the value while it is still being iterated,
so the functions read freed memory or dereference invalidated
iterators (heap-use-after-free, or an uncaught invalid_iterator.214
for update()). This reproduces with plain std::map-backed json and,
for update() on ordered_json, also via reallocation of the
underlying vector.

A fix would require copying the argument whenever it may alias
*this, which cannot be checked in constant time without parent
pointers (only available under JSON_DIAGNOSTICS), and would cost an
unconditional deep copy per call otherwise. The maintainer decided
to document the restriction instead of changing the library.

Add a "Notes" section with a "!!! danger" admonition to
update.md and merge_patch.md explaining that the argument must not
be *this or refer into *this, and showing the workaround of passing
a copy, e.g. j.update(json(j["a"])) and j.merge_patch(json(j)).

Fixes #5641.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-29 23:14:33 +02:00
14 changed files with 227 additions and 443 deletions
@@ -37,6 +37,25 @@ Thereby, `Target` is the current object; that is, the patch is applied to the cu
Linear in the lengths of `apply_patch`.
## Notes
!!! danger "Undefined behavior and runtime assertions"
`merge_patch()` reads `apply_patch` while it modifies `#!cpp *this`. `apply_patch` must not be `#!cpp *this`
itself and must not refer to a value contained in `#!cpp *this` (for example, a subobject returned by
`#!cpp (*this)[key]`). Calling `merge_patch()` with such an argument reads the argument after it has been
invalidated by the modification, which is undefined behavior. If the patch may alias `#!cpp *this`, pass a copy
instead:
```cpp
j.merge_patch(json(j)); // instead of j.merge_patch(j)
```
Passing `#!cpp *this` itself is **guarded by a [runtime assertion](../../features/assertions.md)**; a value
contained in `#!cpp *this` is not detected.
See [GitHub issue #5641](https://github.com/nlohmann/json/issues/5641) for more information.
## Examples
??? example
@@ -61,3 +80,5 @@ Linear in the lengths of `apply_patch`.
## Version history
- Added in version 3.0.0.
- Documented that `apply_patch` must not be `#!cpp *this` or refer to a value contained in `#!cpp *this`, in version
3.13.0.
@@ -55,10 +55,6 @@ This implementation does exactly follow this approach, as it uses double precisi
smaller than `-1.79769313486232e+308` and values greater than `1.79769313486232e+308` will be stored as NaN internally
and be serialized to `null`.
During deserialization (from JSON text or any of the binary formats), a finite number that does not fit into
`number_float_t` is rejected with [`out_of_range.406`](../../home/exceptions.md#jsonexceptionout_of_range406), for
example a double-precision number in a binary format when `number_float_t` is `#!cpp float`.
#### Storage
Floating-point number values are stored directly inside a `basic_json` type.
@@ -47,9 +47,8 @@ With the default values for `NumberIntegerType` (`std::int64_t`), the default va
When the default type is used, the maximal integer number that can be stored is `9223372036854775807` (INT64_MAX) and
the minimal integer number that can be stored is `-9223372036854775808` (INT64_MIN). Integer numbers that are out of
range will yield over/underflow when used in a constructor. During deserialization (from JSON text or any of the binary
formats), too large or small integer numbers will automatically be stored as [`number_unsigned_t`](number_unsigned_t.md)
or [`number_float_t`](number_float_t.md).
range will yield over/underflow when used in a constructor. During deserialization, too large or small integer numbers
will automatically be stored as [`number_unsigned_t`](number_unsigned_t.md) or [`number_float_t`](number_float_t.md).
[RFC 8259](https://tools.ietf.org/html/rfc8259) further states:
> Note that when such software is used, numbers that are integers and are in the range $[-2^{53}+1, 2^{53}-1]$ are
@@ -48,9 +48,8 @@ With the default values for `NumberUnsignedType` (`std::uint64_t`), the default
When the default type is used, the maximal integer number that can be stored is `18446744073709551615` (UINT64_MAX) and
the minimal integer number that can be stored is `0`. Integer numbers that are out of range will yield over/underflow
when used in a constructor. During deserialization (from JSON text or any of the binary formats), too large or small
integer numbers will automatically be stored as [`number_integer_t`](number_integer_t.md) or
[`number_float_t`](number_float_t.md).
when used in a constructor. During deserialization, too large or small integer numbers will automatically be stored
as [`number_integer_t`](number_integer_t.md) or [`number_float_t`](number_float_t.md).
[RFC 8259](https://tools.ietf.org/html/rfc8259) further states:
> Note that when such software is used, numbers that are integers and are in the range $[-2^{53}+1, 2^{53}-1]$ are
+21
View File
@@ -59,6 +59,25 @@ Basic guarantee: if an exception is thrown during the operation, the JSON value
1. O(N*log(size() + N)), where N is the number of elements to insert.
2. O(N*log(size() + N)), where N is the number of elements to insert.
## Notes
!!! danger "Undefined behavior and runtime assertions"
Both overloads read the argument while they modify `#!cpp *this`. The argument `j` (or, for overload (2), the
range `[first, last)`) must not be `#!cpp *this` itself and must not refer to a value contained in
`#!cpp *this` (for example, a subobject returned by `#!cpp (*this)[key]`). Calling `update()` with such an
argument reads the argument after it has been invalidated by the modification, which is undefined behavior. If
the argument may alias `#!cpp *this`, pass a copy instead:
```cpp
j.update(json(j["defaults"])); // instead of j.update(j["defaults"])
```
Passing `#!cpp *this` itself is **guarded by a [runtime assertion](../../features/assertions.md)**; a value
contained in `#!cpp *this` is not detected.
See [GitHub issue #5641](https://github.com/nlohmann/json/issues/5641) for more information.
## Examples
??? example
@@ -155,3 +174,5 @@ Basic guarantee: if an exception is thrown during the operation, the JSON value
- Added in version 3.0.0.
- Added `merge_objects` parameter in 3.10.5.
- Documented that the argument must not be `#!cpp *this` or refer to a value contained in `#!cpp *this`, in version
3.13.0.
+30
View File
@@ -103,6 +103,36 @@ behavior and yields a runtime assertion.
Assertion failed: (m_object != nullptr), function operator++, file iter_impl.hpp, line 368.
```
### Updating or merge-patching a value with itself
Functions [`update`](../api/basic_json/update.md) and [`merge_patch`](../api/basic_json/merge_patch.md) read their
argument while they modify the value they are called on. Passing that value itself as the argument is undefined
behavior and yields a runtime assertion. Pass a copy instead, for example `#!cpp j.merge_patch(json(j))`. An argument
that refers to a value contained in the value (e.g., `#!cpp j.update(j["defaults"])`) is undefined behavior as well, but
is not detected.
??? example "Example 4: Merge-patching a value with itself"
The following code will trigger an assertion at runtime:
```cpp
#include <nlohmann/json.hpp>
using json = nlohmann::json;
int main()
{
json j = {{"key", "value"}};
j.merge_patch(j);
}
```
Output:
```
Assertion failed: (&apply_patch != this), function merge_patch, file json.hpp, line 6310.
```
## Changes
### Reading from a null `FILE` or `char` pointer
@@ -168,9 +168,9 @@ The library maps CBOR types to JSON value types as follows:
!!! warning "Negative integer overflow"
CBOR negative integers (major type 1) are decoded as `-1 - n`. If the encoded magnitude `n` is too large for the
result to fit into `number_integer_t` (`std::int64_t` by default), the result is stored as `number_float_t`, like
a too small integer in JSON text. For example, `-18446744073709551616` (`0x3B` followed by eight `0xFF` bytes) is
stored as `-1.8446744073709552e+19`.
result to fit into `number_integer_t` (`std::int64_t` by default), parsing fails with a
[`parse_error.112`](../../home/exceptions.md#jsonexceptionparse_error112) exception rather than overflowing
silently.
!!! warning "Object keys"
+5 -7
View File
@@ -331,6 +331,9 @@ An unexpected byte was read in a [binary format](../features/binary_formats/inde
[json.exception.parse_error.112] parse error at byte 15: syntax error while parsing BSON binary: byte array length cannot be negative, is -1
```
```
[json.exception.parse_error.112] parse error at byte 9: syntax error while parsing CBOR value: negative integer overflow
```
```
[json.exception.parse_error.112] parse error at byte 5: syntax error while parsing BSON document: document size 6 does not match the number of bytes read (5)
```
@@ -851,18 +854,13 @@ The JSON Patch operations 'remove' and 'add' cannot be applied to the root eleme
### json.exception.out_of_range.406
A parsed number could not be stored without changing it to NaN or INF. For the binary formats, this happens when a
finite floating-point number does not fit into [`number_float_t`](../api/basic_json/number_float_t.md), for example a
double-precision number when `number_float_t` is `#!cpp float`.
A parsed number could not be stored as without changing it to NaN or INF.
!!! failure "Example messages"
!!! failure "Example message"
```
number overflow parsing '10E1000'
```
```
[json.exception.out_of_range.406] syntax error while parsing CBOR value: number overflow
```
### json.exception.out_of_range.407
+45 -133
View File
@@ -559,7 +559,7 @@ class binary_reader
case 0x01: // double
{
double number{};
return get_number<double, true>(input_format_t::bson, number) && emit_float(input_format_t::bson, number);
return get_number<double, true>(input_format_t::bson, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 0x02: // string
@@ -600,19 +600,19 @@ class binary_reader
case 0x10: // int32
{
std::int32_t value{};
return get_number<std::int32_t, true>(input_format_t::bson, value) && emit_signed(input_format_t::bson, value);
return get_number<std::int32_t, true>(input_format_t::bson, value) && sax->number_integer(value);
}
case 0x12: // int64
{
std::int64_t value{};
return get_number<std::int64_t, true>(input_format_t::bson, value) && emit_signed(input_format_t::bson, value);
return get_number<std::int64_t, true>(input_format_t::bson, value) && sax->number_integer(value);
}
case 0x11: // uint64
{
std::uint64_t value{};
return get_number<std::uint64_t, true>(input_format_t::bson, value) && emit_unsigned(input_format_t::bson, value);
return get_number<std::uint64_t, true>(input_format_t::bson, value) && sax->number_unsigned(value);
}
default: // anything else is not supported (yet)
@@ -638,19 +638,14 @@ class binary_reader
{
return false;
}
// the value is -1 - number, which fits into number_integer_t
// whenever number does
if (JSON_HEDLEY_LIKELY(value_in_range_of<number_integer_t>(number)))
const auto max_val = static_cast<NumberType>((std::numeric_limits<number_integer_t>::max)());
if (number > max_val)
{
return sax->number_integer(static_cast<number_integer_t>(-1) - static_cast<number_integer_t>(number));
return sax->parse_error(chars_read, get_token_string(),
parse_error::create(112, chars_read,
exception_message(input_format_t::cbor, "negative integer overflow", "value"), nullptr));
}
// like the lexer does for JSON text, store a value too small for
// number_integer_t as number_float_t; compute it as long double so
// that emit_float sees a finite value and can detect an overflow of
// number_float_t
return emit_float(input_format_t::cbor, static_cast<long double>(-1) - static_cast<long double>(number));
return sax->number_integer(static_cast<number_integer_t>(-1) - static_cast<number_integer_t>(number));
}
/*!
@@ -707,25 +702,25 @@ class binary_reader
case 0x18: // Unsigned integer (one-byte uint8_t follows)
{
std::uint8_t number{};
return get_number(input_format_t::cbor, number) && emit_unsigned(input_format_t::cbor, number);
return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
}
case 0x19: // Unsigned integer (two-byte uint16_t follows)
{
std::uint16_t number{};
return get_number(input_format_t::cbor, number) && emit_unsigned(input_format_t::cbor, number);
return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
}
case 0x1A: // Unsigned integer (four-byte uint32_t follows)
{
std::uint32_t number{};
return get_number(input_format_t::cbor, number) && emit_unsigned(input_format_t::cbor, number);
return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
}
case 0x1B: // Unsigned integer (eight-byte uint64_t follows)
{
std::uint64_t number{};
return get_number(input_format_t::cbor, number) && emit_unsigned(input_format_t::cbor, number);
return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
}
// Negative integer -1-0x00..-1-0x17 (-1..-24)
@@ -1170,13 +1165,13 @@ class binary_reader
case 0xFA: // Single-Precision Float (four-byte IEEE 754)
{
float number{};
return get_number(input_format_t::cbor, number) && emit_float(input_format_t::cbor, number);
return get_number(input_format_t::cbor, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 0xFB: // Double-Precision Float (eight-byte IEEE 754)
{
double number{};
return get_number(input_format_t::cbor, number) && emit_float(input_format_t::cbor, number);
return get_number(input_format_t::cbor, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
default: // anything else (0xFF is handled inside the other types)
@@ -1940,61 +1935,61 @@ class binary_reader
case 0xCA: // float 32
{
float number{};
return get_number(input_format_t::msgpack, number) && emit_float(input_format_t::msgpack, number);
return get_number(input_format_t::msgpack, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 0xCB: // float 64
{
double number{};
return get_number(input_format_t::msgpack, number) && emit_float(input_format_t::msgpack, number);
return get_number(input_format_t::msgpack, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 0xCC: // uint 8
{
std::uint8_t number{};
return get_number(input_format_t::msgpack, number) && emit_unsigned(input_format_t::msgpack, number);
return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
}
case 0xCD: // uint 16
{
std::uint16_t number{};
return get_number(input_format_t::msgpack, number) && emit_unsigned(input_format_t::msgpack, number);
return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
}
case 0xCE: // uint 32
{
std::uint32_t number{};
return get_number(input_format_t::msgpack, number) && emit_unsigned(input_format_t::msgpack, number);
return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
}
case 0xCF: // uint 64
{
std::uint64_t number{};
return get_number(input_format_t::msgpack, number) && emit_unsigned(input_format_t::msgpack, number);
return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
}
case 0xD0: // int 8
{
std::int8_t number{};
return get_number(input_format_t::msgpack, number) && emit_signed(input_format_t::msgpack, number);
return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
}
case 0xD1: // int 16
{
std::int16_t number{};
return get_number(input_format_t::msgpack, number) && emit_signed(input_format_t::msgpack, number);
return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
}
case 0xD2: // int 32
{
std::int32_t number{};
return get_number(input_format_t::msgpack, number) && emit_signed(input_format_t::msgpack, number);
return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
}
case 0xD3: // int 64
{
std::int64_t number{};
return get_number(input_format_t::msgpack, number) && emit_signed(input_format_t::msgpack, number);
return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
}
case 0xDC: // array 16
@@ -2927,7 +2922,7 @@ class binary_reader
{
return sax->parse_error(chars_read, get_token_string(), out_of_range::create(408, exception_message(input_format, "excessive ndarray size caused overflow", "size"), nullptr));
}
if (JSON_HEDLEY_UNLIKELY(!emit_unsigned(input_format, i)))
if (JSON_HEDLEY_UNLIKELY(!sax->number_unsigned(static_cast<number_unsigned_t>(i))))
{
return false;
}
@@ -3059,37 +3054,37 @@ class binary_reader
break;
}
std::uint8_t number{};
return get_number(input_format, number) && emit_unsigned(input_format, number);
return get_number(input_format, number) && sax->number_unsigned(number);
}
case 'U':
{
std::uint8_t number{};
return get_number(input_format, number) && emit_unsigned(input_format, number);
return get_number(input_format, number) && sax->number_unsigned(number);
}
case 'i':
{
std::int8_t number{};
return get_number(input_format, number) && emit_signed(input_format, number);
return get_number(input_format, number) && sax->number_integer(number);
}
case 'I':
{
std::int16_t number{};
return get_number(input_format, number) && emit_signed(input_format, number);
return get_number(input_format, number) && sax->number_integer(number);
}
case 'l':
{
std::int32_t number{};
return get_number(input_format, number) && emit_signed(input_format, number);
return get_number(input_format, number) && sax->number_integer(number);
}
case 'L':
{
std::int64_t number{};
return get_number(input_format, number) && emit_signed(input_format, number);
return get_number(input_format, number) && sax->number_integer(number);
}
case 'u':
@@ -3099,7 +3094,7 @@ class binary_reader
break;
}
std::uint16_t number{};
return get_number(input_format, number) && emit_unsigned(input_format, number);
return get_number(input_format, number) && sax->number_unsigned(number);
}
case 'm':
@@ -3109,7 +3104,7 @@ class binary_reader
break;
}
std::uint32_t number{};
return get_number(input_format, number) && emit_unsigned(input_format, number);
return get_number(input_format, number) && sax->number_unsigned(number);
}
case 'M':
@@ -3119,7 +3114,7 @@ class binary_reader
break;
}
std::uint64_t number{};
return get_number(input_format, number) && emit_unsigned(input_format, number);
return get_number(input_format, number) && sax->number_unsigned(number);
}
case 'h':
@@ -3177,13 +3172,13 @@ class binary_reader
case 'd':
{
float number{};
return get_number(input_format, number) && emit_float(input_format, number);
return get_number(input_format, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 'D':
{
double number{};
return get_number(input_format, number) && emit_float(input_format, number);
return get_number(input_format, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 'H':
@@ -3650,13 +3645,13 @@ class binary_reader
case 0x8E: // binary32
{
float number{};
return get_number(input_format_t::bon8, number) && emit_float(input_format_t::bon8, number);
return get_number(input_format_t::bon8, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 0x8F: // binary64
{
double number{};
return get_number(input_format_t::bon8, number) && emit_float(input_format_t::bon8, number);
return get_number(input_format_t::bon8, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 0xF8:
@@ -3722,9 +3717,7 @@ class binary_reader
@brief pass an integer to the SAX parser
Non-negative integers are passed as unsigned, negative integers as signed
numbers, like the other binary formats do. A value that does not fit the
number type is passed as described for @ref emit_unsigned and
@ref emit_signed.
numbers, like the other binary formats do.
@param[in] number the integer
@return whether the SAX parser accepted the value
@@ -3733,9 +3726,9 @@ class binary_reader
{
if (number >= 0)
{
return emit_unsigned(input_format_t::bon8, static_cast<std::uint64_t>(number));
return sax->number_unsigned(static_cast<number_unsigned_t>(number));
}
return emit_signed(input_format_t::bon8, number);
return sax->number_integer(static_cast<number_integer_t>(number));
}
/*!
@@ -3792,7 +3785,8 @@ class binary_reader
value = (value << 8) | static_cast<std::int64_t>(current);
}
return emit_bon8_integer(negative ? -(value + offset) : value + offset);
return negative ? sax->number_integer(static_cast<number_integer_t>(-(value + offset)))
: sax->number_unsigned(static_cast<number_unsigned_t>(value + offset));
}
/*!
@@ -4089,88 +4083,6 @@ class binary_reader
return true;
}
/*!
@brief pass a signed integer read from the input to the SAX parser
Like the lexer does for JSON text, a value that does not fit into
number_integer_t is passed as number_unsigned_t if it is non-negative and
fits there, and as number_float_t otherwise. With the default number
types, every integer the binary formats can encode fits, so this only
matters for narrower custom number types.
@tparam NumberType a signed integer type
@param[in] format the current format (for diagnostics)
@param[in] number the integer
@return whether the SAX parser accepted the value
@throw out_of_range.406 if @a number overflows number_float_t (see
@ref emit_float)
*/
template<typename NumberType>
bool emit_signed(const input_format_t format, const NumberType number)
{
if (JSON_HEDLEY_LIKELY(value_in_range_of<number_integer_t>(number)))
{
return sax->number_integer(static_cast<number_integer_t>(number));
}
if (value_in_range_of<number_unsigned_t>(number))
{
return sax->number_unsigned(static_cast<number_unsigned_t>(number));
}
return emit_float(format, number);
}
/*!
@brief pass an unsigned integer read from the input to the SAX parser
Like the lexer does for JSON text, a value that does not fit into
number_unsigned_t is passed as number_float_t.
@tparam NumberType an unsigned integer type
@param[in] format the current format (for diagnostics)
@param[in] number the integer
@return whether the SAX parser accepted the value
@throw out_of_range.406 if @a number overflows number_float_t (see
@ref emit_float)
*/
template<typename NumberType>
bool emit_unsigned(const input_format_t format, const NumberType number)
{
if (JSON_HEDLEY_LIKELY(value_in_range_of<number_unsigned_t>(number)))
{
return sax->number_unsigned(static_cast<number_unsigned_t>(number));
}
return emit_float(format, number);
}
/*!
@brief pass a floating-point number read from the input to the SAX parser
Like the lexer does for JSON text, a finite value that overflows
number_float_t is rejected instead of silently becoming infinity. Infinity
and NaN in the input are passed on unchanged. Integers only overflow if
number_float_t cannot represent 2^64, e.g., a half-precision type.
@tparam NumberType a floating-point or integer type
@param[in] format the current format (for diagnostics)
@param[in] number the number
@return whether the SAX parser accepted the value
@throw out_of_range.406 if a finite @a number overflows number_float_t
*/
template<typename NumberType>
bool emit_float(const input_format_t format, const NumberType number)
{
const auto result = static_cast<number_float_t>(number);
if (JSON_HEDLEY_UNLIKELY(std::isfinite(number) && !std::isfinite(result)))
{
return sax->parse_error(chars_read, get_token_string(),
out_of_range::create(406, exception_message(format, "number overflow", "value"), nullptr));
}
return sax->number_float(result, "");
}
/*!
@brief create a string by reading characters from the input
+6
View File
@@ -4218,6 +4218,9 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
JSON_THROW(type_error::create(312, detail::concat("cannot use update() with ", first.m_object->type_name()), first.m_object));
}
// the range must not be *this; a range inside *this is not detected
JSON_ASSERT(first.m_object != this);
update_members(first, last, merge_objects, 0);
}
@@ -6303,6 +6306,9 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
/// @sa https://json.nlohmann.me/api/basic_json/merge_patch/
void merge_patch(const basic_json& apply_patch)
{
// the patch must not be *this; a patch inside *this is not detected
JSON_ASSERT(&apply_patch != this);
apply_merge_patch(apply_patch, 0);
}
+51 -133
View File
@@ -13326,7 +13326,7 @@ class binary_reader
case 0x01: // double
{
double number{};
return get_number<double, true>(input_format_t::bson, number) && emit_float(input_format_t::bson, number);
return get_number<double, true>(input_format_t::bson, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 0x02: // string
@@ -13367,19 +13367,19 @@ class binary_reader
case 0x10: // int32
{
std::int32_t value{};
return get_number<std::int32_t, true>(input_format_t::bson, value) && emit_signed(input_format_t::bson, value);
return get_number<std::int32_t, true>(input_format_t::bson, value) && sax->number_integer(value);
}
case 0x12: // int64
{
std::int64_t value{};
return get_number<std::int64_t, true>(input_format_t::bson, value) && emit_signed(input_format_t::bson, value);
return get_number<std::int64_t, true>(input_format_t::bson, value) && sax->number_integer(value);
}
case 0x11: // uint64
{
std::uint64_t value{};
return get_number<std::uint64_t, true>(input_format_t::bson, value) && emit_unsigned(input_format_t::bson, value);
return get_number<std::uint64_t, true>(input_format_t::bson, value) && sax->number_unsigned(value);
}
default: // anything else is not supported (yet)
@@ -13405,19 +13405,14 @@ class binary_reader
{
return false;
}
// the value is -1 - number, which fits into number_integer_t
// whenever number does
if (JSON_HEDLEY_LIKELY(value_in_range_of<number_integer_t>(number)))
const auto max_val = static_cast<NumberType>((std::numeric_limits<number_integer_t>::max)());
if (number > max_val)
{
return sax->number_integer(static_cast<number_integer_t>(-1) - static_cast<number_integer_t>(number));
return sax->parse_error(chars_read, get_token_string(),
parse_error::create(112, chars_read,
exception_message(input_format_t::cbor, "negative integer overflow", "value"), nullptr));
}
// like the lexer does for JSON text, store a value too small for
// number_integer_t as number_float_t; compute it as long double so
// that emit_float sees a finite value and can detect an overflow of
// number_float_t
return emit_float(input_format_t::cbor, static_cast<long double>(-1) - static_cast<long double>(number));
return sax->number_integer(static_cast<number_integer_t>(-1) - static_cast<number_integer_t>(number));
}
/*!
@@ -13474,25 +13469,25 @@ class binary_reader
case 0x18: // Unsigned integer (one-byte uint8_t follows)
{
std::uint8_t number{};
return get_number(input_format_t::cbor, number) && emit_unsigned(input_format_t::cbor, number);
return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
}
case 0x19: // Unsigned integer (two-byte uint16_t follows)
{
std::uint16_t number{};
return get_number(input_format_t::cbor, number) && emit_unsigned(input_format_t::cbor, number);
return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
}
case 0x1A: // Unsigned integer (four-byte uint32_t follows)
{
std::uint32_t number{};
return get_number(input_format_t::cbor, number) && emit_unsigned(input_format_t::cbor, number);
return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
}
case 0x1B: // Unsigned integer (eight-byte uint64_t follows)
{
std::uint64_t number{};
return get_number(input_format_t::cbor, number) && emit_unsigned(input_format_t::cbor, number);
return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
}
// Negative integer -1-0x00..-1-0x17 (-1..-24)
@@ -13937,13 +13932,13 @@ class binary_reader
case 0xFA: // Single-Precision Float (four-byte IEEE 754)
{
float number{};
return get_number(input_format_t::cbor, number) && emit_float(input_format_t::cbor, number);
return get_number(input_format_t::cbor, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 0xFB: // Double-Precision Float (eight-byte IEEE 754)
{
double number{};
return get_number(input_format_t::cbor, number) && emit_float(input_format_t::cbor, number);
return get_number(input_format_t::cbor, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
default: // anything else (0xFF is handled inside the other types)
@@ -14707,61 +14702,61 @@ class binary_reader
case 0xCA: // float 32
{
float number{};
return get_number(input_format_t::msgpack, number) && emit_float(input_format_t::msgpack, number);
return get_number(input_format_t::msgpack, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 0xCB: // float 64
{
double number{};
return get_number(input_format_t::msgpack, number) && emit_float(input_format_t::msgpack, number);
return get_number(input_format_t::msgpack, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 0xCC: // uint 8
{
std::uint8_t number{};
return get_number(input_format_t::msgpack, number) && emit_unsigned(input_format_t::msgpack, number);
return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
}
case 0xCD: // uint 16
{
std::uint16_t number{};
return get_number(input_format_t::msgpack, number) && emit_unsigned(input_format_t::msgpack, number);
return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
}
case 0xCE: // uint 32
{
std::uint32_t number{};
return get_number(input_format_t::msgpack, number) && emit_unsigned(input_format_t::msgpack, number);
return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
}
case 0xCF: // uint 64
{
std::uint64_t number{};
return get_number(input_format_t::msgpack, number) && emit_unsigned(input_format_t::msgpack, number);
return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
}
case 0xD0: // int 8
{
std::int8_t number{};
return get_number(input_format_t::msgpack, number) && emit_signed(input_format_t::msgpack, number);
return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
}
case 0xD1: // int 16
{
std::int16_t number{};
return get_number(input_format_t::msgpack, number) && emit_signed(input_format_t::msgpack, number);
return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
}
case 0xD2: // int 32
{
std::int32_t number{};
return get_number(input_format_t::msgpack, number) && emit_signed(input_format_t::msgpack, number);
return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
}
case 0xD3: // int 64
{
std::int64_t number{};
return get_number(input_format_t::msgpack, number) && emit_signed(input_format_t::msgpack, number);
return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
}
case 0xDC: // array 16
@@ -15694,7 +15689,7 @@ class binary_reader
{
return sax->parse_error(chars_read, get_token_string(), out_of_range::create(408, exception_message(input_format, "excessive ndarray size caused overflow", "size"), nullptr));
}
if (JSON_HEDLEY_UNLIKELY(!emit_unsigned(input_format, i)))
if (JSON_HEDLEY_UNLIKELY(!sax->number_unsigned(static_cast<number_unsigned_t>(i))))
{
return false;
}
@@ -15826,37 +15821,37 @@ class binary_reader
break;
}
std::uint8_t number{};
return get_number(input_format, number) && emit_unsigned(input_format, number);
return get_number(input_format, number) && sax->number_unsigned(number);
}
case 'U':
{
std::uint8_t number{};
return get_number(input_format, number) && emit_unsigned(input_format, number);
return get_number(input_format, number) && sax->number_unsigned(number);
}
case 'i':
{
std::int8_t number{};
return get_number(input_format, number) && emit_signed(input_format, number);
return get_number(input_format, number) && sax->number_integer(number);
}
case 'I':
{
std::int16_t number{};
return get_number(input_format, number) && emit_signed(input_format, number);
return get_number(input_format, number) && sax->number_integer(number);
}
case 'l':
{
std::int32_t number{};
return get_number(input_format, number) && emit_signed(input_format, number);
return get_number(input_format, number) && sax->number_integer(number);
}
case 'L':
{
std::int64_t number{};
return get_number(input_format, number) && emit_signed(input_format, number);
return get_number(input_format, number) && sax->number_integer(number);
}
case 'u':
@@ -15866,7 +15861,7 @@ class binary_reader
break;
}
std::uint16_t number{};
return get_number(input_format, number) && emit_unsigned(input_format, number);
return get_number(input_format, number) && sax->number_unsigned(number);
}
case 'm':
@@ -15876,7 +15871,7 @@ class binary_reader
break;
}
std::uint32_t number{};
return get_number(input_format, number) && emit_unsigned(input_format, number);
return get_number(input_format, number) && sax->number_unsigned(number);
}
case 'M':
@@ -15886,7 +15881,7 @@ class binary_reader
break;
}
std::uint64_t number{};
return get_number(input_format, number) && emit_unsigned(input_format, number);
return get_number(input_format, number) && sax->number_unsigned(number);
}
case 'h':
@@ -15944,13 +15939,13 @@ class binary_reader
case 'd':
{
float number{};
return get_number(input_format, number) && emit_float(input_format, number);
return get_number(input_format, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 'D':
{
double number{};
return get_number(input_format, number) && emit_float(input_format, number);
return get_number(input_format, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 'H':
@@ -16417,13 +16412,13 @@ class binary_reader
case 0x8E: // binary32
{
float number{};
return get_number(input_format_t::bon8, number) && emit_float(input_format_t::bon8, number);
return get_number(input_format_t::bon8, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 0x8F: // binary64
{
double number{};
return get_number(input_format_t::bon8, number) && emit_float(input_format_t::bon8, number);
return get_number(input_format_t::bon8, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 0xF8:
@@ -16489,9 +16484,7 @@ class binary_reader
@brief pass an integer to the SAX parser
Non-negative integers are passed as unsigned, negative integers as signed
numbers, like the other binary formats do. A value that does not fit the
number type is passed as described for @ref emit_unsigned and
@ref emit_signed.
numbers, like the other binary formats do.
@param[in] number the integer
@return whether the SAX parser accepted the value
@@ -16500,9 +16493,9 @@ class binary_reader
{
if (number >= 0)
{
return emit_unsigned(input_format_t::bon8, static_cast<std::uint64_t>(number));
return sax->number_unsigned(static_cast<number_unsigned_t>(number));
}
return emit_signed(input_format_t::bon8, number);
return sax->number_integer(static_cast<number_integer_t>(number));
}
/*!
@@ -16559,7 +16552,8 @@ class binary_reader
value = (value << 8) | static_cast<std::int64_t>(current);
}
return emit_bon8_integer(negative ? -(value + offset) : value + offset);
return negative ? sax->number_integer(static_cast<number_integer_t>(-(value + offset)))
: sax->number_unsigned(static_cast<number_unsigned_t>(value + offset));
}
/*!
@@ -16856,88 +16850,6 @@ class binary_reader
return true;
}
/*!
@brief pass a signed integer read from the input to the SAX parser
Like the lexer does for JSON text, a value that does not fit into
number_integer_t is passed as number_unsigned_t if it is non-negative and
fits there, and as number_float_t otherwise. With the default number
types, every integer the binary formats can encode fits, so this only
matters for narrower custom number types.
@tparam NumberType a signed integer type
@param[in] format the current format (for diagnostics)
@param[in] number the integer
@return whether the SAX parser accepted the value
@throw out_of_range.406 if @a number overflows number_float_t (see
@ref emit_float)
*/
template<typename NumberType>
bool emit_signed(const input_format_t format, const NumberType number)
{
if (JSON_HEDLEY_LIKELY(value_in_range_of<number_integer_t>(number)))
{
return sax->number_integer(static_cast<number_integer_t>(number));
}
if (value_in_range_of<number_unsigned_t>(number))
{
return sax->number_unsigned(static_cast<number_unsigned_t>(number));
}
return emit_float(format, number);
}
/*!
@brief pass an unsigned integer read from the input to the SAX parser
Like the lexer does for JSON text, a value that does not fit into
number_unsigned_t is passed as number_float_t.
@tparam NumberType an unsigned integer type
@param[in] format the current format (for diagnostics)
@param[in] number the integer
@return whether the SAX parser accepted the value
@throw out_of_range.406 if @a number overflows number_float_t (see
@ref emit_float)
*/
template<typename NumberType>
bool emit_unsigned(const input_format_t format, const NumberType number)
{
if (JSON_HEDLEY_LIKELY(value_in_range_of<number_unsigned_t>(number)))
{
return sax->number_unsigned(static_cast<number_unsigned_t>(number));
}
return emit_float(format, number);
}
/*!
@brief pass a floating-point number read from the input to the SAX parser
Like the lexer does for JSON text, a finite value that overflows
number_float_t is rejected instead of silently becoming infinity. Infinity
and NaN in the input are passed on unchanged. Integers only overflow if
number_float_t cannot represent 2^64, e.g., a half-precision type.
@tparam NumberType a floating-point or integer type
@param[in] format the current format (for diagnostics)
@param[in] number the number
@return whether the SAX parser accepted the value
@throw out_of_range.406 if a finite @a number overflows number_float_t
*/
template<typename NumberType>
bool emit_float(const input_format_t format, const NumberType number)
{
const auto result = static_cast<number_float_t>(number);
if (JSON_HEDLEY_UNLIKELY(std::isfinite(number) && !std::isfinite(result)))
{
return sax->parse_error(chars_read, get_token_string(),
out_of_range::create(406, exception_message(format, "number overflow", "value"), nullptr));
}
return sax->number_float(result, "");
}
/*!
@brief create a string by reading characters from the input
@@ -30387,6 +30299,9 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
JSON_THROW(type_error::create(312, detail::concat("cannot use update() with ", first.m_object->type_name()), first.m_object));
}
// the range must not be *this; a range inside *this is not detected
JSON_ASSERT(first.m_object != this);
update_members(first, last, merge_objects, 0);
}
@@ -32472,6 +32387,9 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
/// @sa https://json.nlohmann.me/api/basic_json/merge_patch/
void merge_patch(const basic_json& apply_patch)
{
// the patch must not be *this; a patch inside *this is not detected
JSON_ASSERT(&apply_patch != this);
apply_merge_patch(apply_patch, 0);
}
+27
View File
@@ -41,6 +41,33 @@ TEST_CASE("JSON_ASSERT(x)")
// check that assertion actually happened
CHECK(assert_counter == 1);
}
SECTION("update() and merge_patch() with *this")
{
// update() and merge_patch() must not be called with *this (#5641);
// the values are chosen so the calls still happen to work without
// aborting, and only the assertion counter is checked
json j = {{"a", 1}};
assert_counter = 0;
j.update(j);
CHECK(assert_counter == 1);
assert_counter = 0;
j.update(j.cbegin(), j.cend());
CHECK(assert_counter == 1);
assert_counter = 0;
j.merge_patch(j);
CHECK(assert_counter == 1);
// a copy is fine
assert_counter = 0;
j.update(json(j));
j.merge_patch(json(j));
CHECK(assert_counter == 0);
CHECK(j == json({{"a", 1}}));
}
}
#endif
-141
View File
@@ -11,12 +11,7 @@
#include <nlohmann/json.hpp>
using nlohmann::json;
#include <cmath>
#include <fstream>
#include <limits>
#include <map>
#include <string>
#include <vector>
#include "make_test_data_available.hpp"
TEST_CASE("Binary Formats" * doctest::skip())
@@ -229,139 +224,3 @@ TEST_CASE("Binary Formats" * doctest::skip())
CHECK((100.0 * double(ubjson_3_size) / double(json_size)) == Approx(89.450));
}
}
namespace
{
// the binary formats as function pointers for "Binary formats with narrow number types";
// named functions rather than lambdas, because clang 3.5 cannot convert a lambda
// to a function pointer in the braced initializer of the format table
using narrow_json = nlohmann::basic_json<std::map, std::vector, std::string, bool, std::int32_t, std::uint32_t, float>;
using bytes = std::vector<std::uint8_t>;
bytes encode_cbor(const json& j)
{
return json::to_cbor(j);
}
narrow_json decode_cbor(const bytes& v, bool allow_exceptions)
{
return narrow_json::from_cbor(v, true, allow_exceptions);
}
bytes encode_msgpack(const json& j)
{
return json::to_msgpack(j);
}
narrow_json decode_msgpack(const bytes& v, bool allow_exceptions)
{
return narrow_json::from_msgpack(v, true, allow_exceptions);
}
bytes encode_ubjson(const json& j)
{
return json::to_ubjson(j);
}
narrow_json decode_ubjson(const bytes& v, bool allow_exceptions)
{
return narrow_json::from_ubjson(v, true, allow_exceptions);
}
bytes encode_bjdata(const json& j)
{
return json::to_bjdata(j);
}
narrow_json decode_bjdata(const bytes& v, bool allow_exceptions)
{
return narrow_json::from_bjdata(v, true, allow_exceptions);
}
// BSON can only store numbers as object members
bytes encode_bson(const json& j)
{
return json::to_bson(json{{"a", j}});
}
narrow_json decode_bson(const bytes& v, bool allow_exceptions)
{
const auto result = narrow_json::from_bson(v, true, allow_exceptions);
return result.is_discarded() ? result : result.at("a");
}
bytes encode_bon8(const json& j)
{
return json::to_bon8(j);
}
narrow_json decode_bon8(const bytes& v, bool allow_exceptions)
{
return narrow_json::from_bon8(v, true, allow_exceptions);
}
} // namespace
TEST_CASE("Binary formats with narrow number types")
{
// Numbers that do not fit the number types are handled like the lexer
// handles them in JSON text: an integer that fits neither integer type is
// stored as a floating-point number, and a finite floating-point number
// that overflows number_float_t is rejected with out_of_range.406.
struct binary_format
{
const char* name;
bytes (*encode)(const json&);
narrow_json (*decode)(const bytes&, bool);
};
const std::vector<binary_format> formats =
{
{"CBOR", encode_cbor, decode_cbor},
{"MessagePack", encode_msgpack, decode_msgpack},
{"UBJSON", encode_ubjson, decode_ubjson},
{"BJData", encode_bjdata, decode_bjdata},
{"BSON", encode_bson, decode_bson},
{"BON8", encode_bon8, decode_bon8},
};
for (const auto& format : formats)
{
const std::string name = format.name;
INFO("format := ", name);
const auto roundtrip = [&format](const json & j)
{
return format.decode(format.encode(j), true);
};
// integers that fit keep their type
CHECK(roundtrip(json(-5)).is_number_integer());
CHECK(roundtrip(json(-5)).get<std::int32_t>() == -5);
CHECK(roundtrip(json(3000000000u)).is_number_unsigned());
CHECK(roundtrip(json(3000000000u)).get<std::uint32_t>() == 3000000000u);
// integers that fit neither integer type are stored as float
CHECK(roundtrip(json(5000000000u)).is_number_float());
CHECK(roundtrip(json(5000000000u)).get<float>() == 5000000000.0f);
if (name != "BON8") // BON8 cannot encode integers above INT64_MAX
{
CHECK(roundtrip(json(10000000000000000000u)).is_number_float());
CHECK(roundtrip(json(10000000000000000000u)).get<float>() == 10000000000000000000.0f);
}
CHECK(roundtrip(json(-3000000000LL)).is_number_float());
CHECK(roundtrip(json(-3000000000LL)).get<float>() == -3000000000.0f);
CHECK(roundtrip(json(-5000000000LL)).is_number_float());
CHECK(roundtrip(json(-5000000000LL)).get<float>() == -5000000000.0f);
// floating-point numbers that fit
CHECK(roundtrip(json(1.5)).get<float>() == 1.5f);
const auto just_above_max = std::nextafter(static_cast<double>((std::numeric_limits<float>::max)()),
std::numeric_limits<double>::infinity());
CHECK(roundtrip(json(just_above_max)).get<float>() == (std::numeric_limits<float>::max)());
// infinity and NaN are passed on
CHECK(std::isinf(roundtrip(json(std::numeric_limits<double>::infinity())).get<float>()));
CHECK(std::isnan(roundtrip(json(std::numeric_limits<double>::quiet_NaN())).get<float>()));
// finite floating-point numbers that overflow number_float_t are rejected
const std::string message = "[json.exception.out_of_range.406] syntax error while parsing " + name
+ " value: number overflow";
CHECK_THROWS_WITH_AS(roundtrip(json(1e300)), message.c_str(), narrow_json::out_of_range&);
CHECK_THROWS_WITH_AS(roundtrip(json(-1e300)), message.c_str(), narrow_json::out_of_range&);
CHECK(format.decode(format.encode(json(1e300)), false).is_discarded());
}
}
+14 -16
View File
@@ -3187,8 +3187,7 @@ TEST_CASE("Tagged values")
// CBOR encodes negative integers as: result = -1 - n
// For type 0x3B, n is an 8-byte uint64_t. Valid range for n with
// the default int64_t is [0, INT64_MAX], producing results in [INT64_MIN, -1].
// When n > INT64_MAX, the result exceeds int64_t range and is stored
// as a floating-point number, as the lexer does for JSON text.
// When n > INT64_MAX, the result exceeds int64_t range and is rejected.
SECTION("n = 0 is valid (result = -1)")
{
@@ -3209,34 +3208,33 @@ TEST_CASE("Tagged values")
CHECK(result.get<int64_t>() == (std::numeric_limits<int64_t>::min)());
}
SECTION("n = INT64_MAX + 1 is stored as float")
SECTION("n = INT64_MAX + 1 is rejected (overflow)")
{
// n = INT64_MAX + 1 (0x8000000000000000)
// result = -1 - n = -9223372036854775809, which exceeds int64_t range;
// the nearest double is -9223372036854775808.0
// result = -1 - n = -9223372036854775809, which exceeds int64_t range
const std::vector<uint8_t> input = {0x3B, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
const auto result = json::from_cbor(input);
CHECK(result.is_number_float());
CHECK(result.get<double>() == -9223372036854775808.0);
CHECK(result == json::parse("-9223372036854775809"));
json _;
CHECK_THROWS_WITH_AS(_ = json::from_cbor(input),
"[json.exception.parse_error.112] parse error at byte 9: syntax error while parsing CBOR value: negative integer overflow",
json::parse_error);
}
SECTION("n = UINT64_MAX is stored as float")
SECTION("n = UINT64_MAX is rejected (overflow)")
{
// n = UINT64_MAX (0xFFFFFFFFFFFFFFFF)
// result = -1 - n = -18446744073709551616, which exceeds int64_t range
const std::vector<uint8_t> input = {0x3B, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF};
const auto result = json::from_cbor(input);
CHECK(result.is_number_float());
CHECK(result.get<double>() == -18446744073709551616.0);
CHECK(result == json::parse("-18446744073709551616"));
json _;
CHECK_THROWS_WITH_AS(_ = json::from_cbor(input),
"[json.exception.parse_error.112] parse error at byte 9: syntax error while parsing CBOR value: negative integer overflow",
json::parse_error);
}
SECTION("overflow with allow_exceptions=false is not an error")
SECTION("overflow with allow_exceptions=false returns discarded")
{
const std::vector<uint8_t> input = {0x3B, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
const auto result = json::from_cbor(input, true, false);
CHECK(result.is_number_float());
CHECK(result.is_discarded());
}
}