Compare commits

..
Author SHA1 Message Date
Niels Lohmann 7fdf97274b Merge branch 'develop' into claude/non-string-object-keys-2d35fc
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-27 20:58:05 +02:00
Niels Lohmann 6508d1d0e1 Point the MessagePack key note to the spec's profile section
The note linked to "Serialization: type to format conversion", which says nothing about key types. Restricting map keys to strings is only mentioned in the "Profile" section (under "Future discussion") as an example of a JSON-compatible profile, so link there and describe it as such instead of as a permission.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-27 18:20:36 +02:00
Niels Lohmann df9a000e3a Name the key type when rejecting non-string CBOR/MessagePack map keys
CBOR and MessagePack allow map keys of any type, but JSON object keys
are always strings, so such maps are rejected. The error so far was the
one for a malformed string (e.g. "expected length specification
(0xA0-0xBF, 0xD9-0xDB); last byte: 0xC0" for a nil key), which does not
tell the user what went wrong. Report the type of the key instead:

  syntax error while parsing MessagePack object key: only string keys
  are supported, but found nil; last byte: 0xC0

The exception id (parse_error.113) and type are unchanged. Malformed
string keys and a missing key keep their previous messages. Document
the restriction on the CBOR and MessagePack pages.

Refs #2766, #3381

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-27 16:40:40 +02:00
14 changed files with 600 additions and 413 deletions
+2 -2
View File
@@ -80,8 +80,8 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
the end of the file was not reached when `strict` was set to true
- Throws [parse_error.112](../../home/exceptions.md#jsonexceptionparse_error112) if unsupported features from CBOR were
used in the given input or if the input is not valid CBOR
- Throws [parse_error.113](../../home/exceptions.md#jsonexceptionparse_error113) if a string was expected as a map key,
but not found
- Throws [parse_error.113](../../home/exceptions.md#jsonexceptionparse_error113) if a map key is not a string (keys of other
types are not supported, as JSON object keys are always strings) or a string is malformed
## Complexity
@@ -73,8 +73,8 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
the end of the file was not reached when `strict` was set to true
- Throws [parse_error.112](../../home/exceptions.md#jsonexceptionparse_error112) if unsupported features from
MessagePack were used in the given input or if the input is not valid MessagePack
- Throws [parse_error.113](../../home/exceptions.md#jsonexceptionparse_error113) if a string was expected as a map key,
but not found
- Throws [parse_error.113](../../home/exceptions.md#jsonexceptionparse_error113) if a map key is not a string (keys of other
types are not supported, as JSON object keys are always strings) or a string is malformed
## Complexity
@@ -55,10 +55,6 @@ This implementation does exactly follow this approach, as it uses double precisi
smaller than `-1.79769313486232e+308` and values greater than `1.79769313486232e+308` will be stored as NaN internally
and be serialized to `null`.
During deserialization (from JSON text or any of the binary formats), a finite number that does not fit into
`number_float_t` is rejected with [`out_of_range.406`](../../home/exceptions.md#jsonexceptionout_of_range406), for
example a double-precision number in a binary format when `number_float_t` is `#!cpp float`.
#### Storage
Floating-point number values are stored directly inside a `basic_json` type.
@@ -47,9 +47,8 @@ With the default values for `NumberIntegerType` (`std::int64_t`), the default va
When the default type is used, the maximal integer number that can be stored is `9223372036854775807` (INT64_MAX) and
the minimal integer number that can be stored is `-9223372036854775808` (INT64_MIN). Integer numbers that are out of
range will yield over/underflow when used in a constructor. During deserialization (from JSON text or any of the binary
formats), too large or small integer numbers will automatically be stored as [`number_unsigned_t`](number_unsigned_t.md)
or [`number_float_t`](number_float_t.md).
range will yield over/underflow when used in a constructor. During deserialization, too large or small integer numbers
will automatically be stored as [`number_unsigned_t`](number_unsigned_t.md) or [`number_float_t`](number_float_t.md).
[RFC 8259](https://tools.ietf.org/html/rfc8259) further states:
> Note that when such software is used, numbers that are integers and are in the range $[-2^{53}+1, 2^{53}-1]$ are
@@ -48,9 +48,8 @@ With the default values for `NumberUnsignedType` (`std::uint64_t`), the default
When the default type is used, the maximal integer number that can be stored is `18446744073709551615` (UINT64_MAX) and
the minimal integer number that can be stored is `0`. Integer numbers that are out of range will yield over/underflow
when used in a constructor. During deserialization (from JSON text or any of the binary formats), too large or small
integer numbers will automatically be stored as [`number_integer_t`](number_integer_t.md) or
[`number_float_t`](number_float_t.md).
when used in a constructor. During deserialization, too large or small integer numbers will automatically be stored
as [`number_integer_t`](number_integer_t.md) or [`number_float_t`](number_float_t.md).
[RFC 8259](https://tools.ietf.org/html/rfc8259) further states:
> Note that when such software is used, numbers that are integers and are in the range $[-2^{53}+1, 2^{53}-1]$ are
@@ -168,13 +168,26 @@ The library maps CBOR types to JSON value types as follows:
!!! warning "Negative integer overflow"
CBOR negative integers (major type 1) are decoded as `-1 - n`. If the encoded magnitude `n` is too large for the
result to fit into `number_integer_t` (`std::int64_t` by default), the result is stored as `number_float_t`, like
a too small integer in JSON text. For example, `-18446744073709551616` (`0x3B` followed by eight `0xFF` bytes) is
stored as `-1.8446744073709552e+19`.
result to fit into `number_integer_t` (`std::int64_t` by default), parsing fails with a
[`parse_error.112`](../../home/exceptions.md#jsonexceptionparse_error112) exception rather than overflowing
silently.
!!! warning "Object keys"
CBOR allows map keys of any type, whereas JSON only allows strings as keys in object values. Therefore, CBOR maps with keys other than UTF-8 strings are rejected.
CBOR allows map keys of any type, whereas JSON only allows strings as keys in object values. Therefore, CBOR maps
with keys other than text strings (major type 3) are rejected with a
[`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113) exception (or, with `allow_exceptions` set
to `false`, a discarded value) naming the type of the key that was found, for instance:
```
[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR object key: only string keys are supported, but found an unsigned integer; last byte: 0x01
```
This applies to the [SAX interface](../parsing/sax_interface.md) as well, as the key is read before it is passed
on. This is a deliberate restriction of the library's JSON value model, not an oversight: formats built on CBOR
maps with integer keys, such as COSE ([RFC 9052](https://www.rfc-editor.org/rfc/rfc9052.html)) or CWT
([RFC 8392](https://www.rfc-editor.org/rfc/rfc8392.html)), cannot be read with this library and need a
general-purpose CBOR library instead.
!!! warning "UTF-8 validation of text strings"
@@ -138,6 +138,21 @@ The library maps MessagePack types to JSON value types as follows:
Any MessagePack output created by `to_msgpack` can be successfully parsed by `from_msgpack`.
!!! warning "Object keys"
MessagePack allows map keys of any type, whereas JSON only allows strings as keys in object values. Like the
JSON-compatible [profile](https://github.com/msgpack/msgpack/blob/master/spec.md#profile) sketched in the
MessagePack specification, this library restricts map keys to `str` values. Maps with keys of any other type are
rejected with a [`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113) exception (or, with
`allow_exceptions` set to `false`, a discarded value) naming the type of the key that was found, for instance:
```
[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing MessagePack object key: only string keys are supported, but found nil; last byte: 0xC0
```
This applies to the [SAX interface](../parsing/sax_interface.md) as well, as the key is read before it is passed
on. Such input needs a general-purpose MessagePack library instead.
!!! warning "UTF-8 validation of string values"
The MessagePack specification requires `str` values (`fixstr`, `str 8`, `str 16`, `str 32`) to be valid UTF-8.
+14 -9
View File
@@ -331,6 +331,9 @@ An unexpected byte was read in a [binary format](../features/binary_formats/inde
[json.exception.parse_error.112] parse error at byte 15: syntax error while parsing BSON binary: byte array length cannot be negative, is -1
```
```
[json.exception.parse_error.112] parse error at byte 9: syntax error while parsing CBOR value: negative integer overflow
```
```
[json.exception.parse_error.112] parse error at byte 5: syntax error while parsing BSON document: document size 6 does not match the number of bytes read (5)
```
@@ -340,13 +343,20 @@ A string could not be read from a [binary format](../features/binary_formats/ind
string was read where one was required (for instance as a map key), the string's length specification is invalid, or
the string's bytes are not valid UTF-8.
CBOR and MessagePack allow map keys of any type, but JSON object keys are always strings. Maps with keys of any other
type (for instance integers or `null`) are therefore not supported; see the notes on
[CBOR](../features/binary_formats/cbor.md) and [MessagePack](../features/binary_formats/messagepack.md).
!!! failure "Example messages"
```
[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR string: expected length specification (0x60-0x7B) or indefinite string type (0x7F); last byte: 0xFF
[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR object key: only string keys are supported, but found an unsigned integer; last byte: 0x01
```
```
[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing MessagePack string: expected length specification (0xA0-0xBF, 0xD9-0xDB); last byte: 0xFF
[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing MessagePack object key: only string keys are supported, but found nil; last byte: 0xC0
```
```
[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR string: expected length specification (0x60-0x7B) or indefinite string type (0x7F); last byte: 0x7C
```
```
[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing UBJSON char: byte after 'C' must be in range 0x00..0x7F; last byte: 0x82
@@ -844,18 +854,13 @@ The JSON Patch operations 'remove' and 'add' cannot be applied to the root eleme
### json.exception.out_of_range.406
A parsed number could not be stored without changing it to NaN or INF. For the binary formats, this happens when a
finite floating-point number does not fit into [`number_float_t`](../api/basic_json/number_float_t.md), for example a
double-precision number when `number_float_t` is `#!cpp float`.
A parsed number could not be stored as without changing it to NaN or INF.
!!! failure "Example messages"
!!! failure "Example message"
```
number overflow parsing '10E1000'
```
```
[json.exception.out_of_range.406] syntax error while parsing CBOR value: number overflow
```
### json.exception.out_of_range.407
+213 -124
View File
@@ -559,7 +559,7 @@ class binary_reader
case 0x01: // double
{
double number{};
return get_number<double, true>(input_format_t::bson, number) && emit_float(input_format_t::bson, number);
return get_number<double, true>(input_format_t::bson, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 0x02: // string
@@ -600,19 +600,19 @@ class binary_reader
case 0x10: // int32
{
std::int32_t value{};
return get_number<std::int32_t, true>(input_format_t::bson, value) && emit_signed(value);
return get_number<std::int32_t, true>(input_format_t::bson, value) && sax->number_integer(value);
}
case 0x12: // int64
{
std::int64_t value{};
return get_number<std::int64_t, true>(input_format_t::bson, value) && emit_signed(value);
return get_number<std::int64_t, true>(input_format_t::bson, value) && sax->number_integer(value);
}
case 0x11: // uint64
{
std::uint64_t value{};
return get_number<std::uint64_t, true>(input_format_t::bson, value) && emit_unsigned(value);
return get_number<std::uint64_t, true>(input_format_t::bson, value) && sax->number_unsigned(value);
}
default: // anything else is not supported (yet)
@@ -638,17 +638,14 @@ class binary_reader
{
return false;
}
// the value is -1 - number, which fits into number_integer_t
// whenever number does
if (JSON_HEDLEY_LIKELY(value_in_range_of<number_integer_t>(number)))
const auto max_val = static_cast<NumberType>((std::numeric_limits<number_integer_t>::max)());
if (number > max_val)
{
return sax->number_integer(static_cast<number_integer_t>(-1) - static_cast<number_integer_t>(number));
return sax->parse_error(chars_read, get_token_string(),
parse_error::create(112, chars_read,
exception_message(input_format_t::cbor, "negative integer overflow", "value"), nullptr));
}
// like the lexer does for JSON text, store a value too small for
// number_integer_t as number_float_t
return sax->number_float(static_cast<number_float_t>(-1) - static_cast<number_float_t>(number), "");
return sax->number_integer(static_cast<number_integer_t>(-1) - static_cast<number_integer_t>(number));
}
/*!
@@ -705,25 +702,25 @@ class binary_reader
case 0x18: // Unsigned integer (one-byte uint8_t follows)
{
std::uint8_t number{};
return get_number(input_format_t::cbor, number) && emit_unsigned(number);
return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
}
case 0x19: // Unsigned integer (two-byte uint16_t follows)
{
std::uint16_t number{};
return get_number(input_format_t::cbor, number) && emit_unsigned(number);
return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
}
case 0x1A: // Unsigned integer (four-byte uint32_t follows)
{
std::uint32_t number{};
return get_number(input_format_t::cbor, number) && emit_unsigned(number);
return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
}
case 0x1B: // Unsigned integer (eight-byte uint64_t follows)
{
std::uint64_t number{};
return get_number(input_format_t::cbor, number) && emit_unsigned(number);
return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
}
// Negative integer -1-0x00..-1-0x17 (-1..-24)
@@ -1168,13 +1165,13 @@ class binary_reader
case 0xFA: // Single-Precision Float (four-byte IEEE 754)
{
float number{};
return get_number(input_format_t::cbor, number) && emit_float(input_format_t::cbor, number);
return get_number(input_format_t::cbor, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 0xFB: // Double-Precision Float (eight-byte IEEE 754)
{
double number{};
return get_number(input_format_t::cbor, number) && emit_float(input_format_t::cbor, number);
return get_number(input_format_t::cbor, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
default: // anything else (0xFF is handled inside the other types)
@@ -1327,6 +1324,80 @@ class binary_reader
}
}
/*!
@brief reads a CBOR object key
RFC 8949 allows any data item as a map key, but only strings have a
counterpart in JSON. A key of any other type is rejected with a message
naming that type, rather than the one @ref get_cbor_string gives for a
malformed string.
@param[out] result created key
@return whether key creation completed
*/
bool get_cbor_object_key(string_t& result)
{
// EOF and major type 3 (text string) are left to get_cbor_string
if (current == char_traits<char_type>::eof() || (static_cast<unsigned int>(current) & 0xE0u) == 0x60u)
{
return get_cbor_string(result);
}
const char* found = nullptr;
switch (static_cast<unsigned int>(current) >> 5u)
{
case 0:
found = "an unsigned integer";
break;
case 1:
found = "a negative integer";
break;
case 2:
found = "a byte string";
break;
case 4:
found = "an array";
break;
case 5:
found = "a map";
break;
case 6:
found = "a tag";
break;
default: // major type 7
switch (current)
{
case 0xF4:
case 0xF5:
found = "a boolean";
break;
case 0xF6:
found = "null";
break;
case 0xF7:
found = "undefined";
break;
case 0xF9:
case 0xFA:
case 0xFB:
found = "a floating-point number";
break;
case 0xFF:
found = "a break stop code";
break;
default:
found = "a simple value";
break;
}
break;
}
auto last_token = get_token_string();
return sax->parse_error(chars_read, last_token, parse_error::create(113, chars_read,
exception_message(input_format_t::cbor, concat("only string keys are supported, but found ", found, "; last byte: 0x", last_token), "object key"), nullptr));
}
/*!
@brief reads a definite-length CBOR byte array
@@ -1571,7 +1642,7 @@ class binary_reader
if (top.is_object)
{
key.clear();
if (JSON_HEDLEY_UNLIKELY(!get_cbor_string(key) || !sax->key(key)))
if (JSON_HEDLEY_UNLIKELY(!get_cbor_object_key(key) || !sax->key(key)))
{
return false;
}
@@ -1864,61 +1935,61 @@ class binary_reader
case 0xCA: // float 32
{
float number{};
return get_number(input_format_t::msgpack, number) && emit_float(input_format_t::msgpack, number);
return get_number(input_format_t::msgpack, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 0xCB: // float 64
{
double number{};
return get_number(input_format_t::msgpack, number) && emit_float(input_format_t::msgpack, number);
return get_number(input_format_t::msgpack, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 0xCC: // uint 8
{
std::uint8_t number{};
return get_number(input_format_t::msgpack, number) && emit_unsigned(number);
return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
}
case 0xCD: // uint 16
{
std::uint16_t number{};
return get_number(input_format_t::msgpack, number) && emit_unsigned(number);
return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
}
case 0xCE: // uint 32
{
std::uint32_t number{};
return get_number(input_format_t::msgpack, number) && emit_unsigned(number);
return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
}
case 0xCF: // uint 64
{
std::uint64_t number{};
return get_number(input_format_t::msgpack, number) && emit_unsigned(number);
return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
}
case 0xD0: // int 8
{
std::int8_t number{};
return get_number(input_format_t::msgpack, number) && emit_signed(number);
return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
}
case 0xD1: // int 16
{
std::int16_t number{};
return get_number(input_format_t::msgpack, number) && emit_signed(number);
return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
}
case 0xD2: // int 32
{
std::int32_t number{};
return get_number(input_format_t::msgpack, number) && emit_signed(number);
return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
}
case 0xD3: // int 64
{
std::int64_t number{};
return get_number(input_format_t::msgpack, number) && emit_signed(number);
return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
}
case 0xDC: // array 16
@@ -2072,6 +2143,98 @@ class binary_reader
}
}
/*!
@brief reads a MessagePack object key
The MessagePack specification allows any type as a map key, but only
strings have a counterpart in JSON. A key of any other type is rejected
with a message naming that type, rather than the one @ref
get_msgpack_string gives for a malformed string.
@param[out] result created key
@return whether key creation completed
*/
bool get_msgpack_object_key(string_t& result)
{
const char* found = nullptr;
switch (current)
{
case 0xC0:
found = "nil";
break;
case 0xC2:
case 0xC3:
found = "a boolean";
break;
case 0xCA:
case 0xCB:
found = "a float";
break;
case 0xC4:
case 0xC5:
case 0xC6:
found = "a bin";
break;
case 0xC7:
case 0xC8:
case 0xC9:
case 0xD4:
case 0xD5:
case 0xD6:
case 0xD7:
case 0xD8:
found = "an ext";
break;
case 0xCC:
case 0xCD:
case 0xCE:
case 0xCF:
case 0xD0:
case 0xD1:
case 0xD2:
case 0xD3:
found = "an integer";
break;
case 0xDC:
case 0xDD:
found = "an array";
break;
case 0xDE:
case 0xDF:
found = "a map";
break;
default:
// fixint, fixmap, and fixarray; strings, EOF, and the unused
// byte 0xC1 are left to get_msgpack_string
if (current == char_traits<char_type>::eof())
{
return get_msgpack_string(result);
}
if (current <= 0x7F || current >= 0xE0)
{
found = "an integer";
}
else if (current <= 0x8F)
{
found = "a map";
}
else if (current <= 0x9F)
{
found = "an array";
}
else
{
return get_msgpack_string(result);
}
break;
}
auto last_token = get_token_string();
return sax->parse_error(chars_read, last_token, parse_error::create(113, chars_read,
exception_message(input_format_t::msgpack, concat("only string keys are supported, but found ", found, "; last byte: 0x", last_token), "object key"), nullptr));
}
/*!
@brief reads a MessagePack byte array
@@ -2234,7 +2397,7 @@ class binary_reader
{
get();
key.clear();
if (JSON_HEDLEY_UNLIKELY(!get_msgpack_string(key) || !sax->key(key)))
if (JSON_HEDLEY_UNLIKELY(!get_msgpack_object_key(key) || !sax->key(key)))
{
return false;
}
@@ -2759,7 +2922,7 @@ class binary_reader
{
return sax->parse_error(chars_read, get_token_string(), out_of_range::create(408, exception_message(input_format, "excessive ndarray size caused overflow", "size"), nullptr));
}
if (JSON_HEDLEY_UNLIKELY(!emit_unsigned(i)))
if (JSON_HEDLEY_UNLIKELY(!sax->number_unsigned(static_cast<number_unsigned_t>(i))))
{
return false;
}
@@ -2891,37 +3054,37 @@ class binary_reader
break;
}
std::uint8_t number{};
return get_number(input_format, number) && emit_unsigned(number);
return get_number(input_format, number) && sax->number_unsigned(number);
}
case 'U':
{
std::uint8_t number{};
return get_number(input_format, number) && emit_unsigned(number);
return get_number(input_format, number) && sax->number_unsigned(number);
}
case 'i':
{
std::int8_t number{};
return get_number(input_format, number) && emit_signed(number);
return get_number(input_format, number) && sax->number_integer(number);
}
case 'I':
{
std::int16_t number{};
return get_number(input_format, number) && emit_signed(number);
return get_number(input_format, number) && sax->number_integer(number);
}
case 'l':
{
std::int32_t number{};
return get_number(input_format, number) && emit_signed(number);
return get_number(input_format, number) && sax->number_integer(number);
}
case 'L':
{
std::int64_t number{};
return get_number(input_format, number) && emit_signed(number);
return get_number(input_format, number) && sax->number_integer(number);
}
case 'u':
@@ -2931,7 +3094,7 @@ class binary_reader
break;
}
std::uint16_t number{};
return get_number(input_format, number) && emit_unsigned(number);
return get_number(input_format, number) && sax->number_unsigned(number);
}
case 'm':
@@ -2941,7 +3104,7 @@ class binary_reader
break;
}
std::uint32_t number{};
return get_number(input_format, number) && emit_unsigned(number);
return get_number(input_format, number) && sax->number_unsigned(number);
}
case 'M':
@@ -2951,7 +3114,7 @@ class binary_reader
break;
}
std::uint64_t number{};
return get_number(input_format, number) && emit_unsigned(number);
return get_number(input_format, number) && sax->number_unsigned(number);
}
case 'h':
@@ -3009,13 +3172,13 @@ class binary_reader
case 'd':
{
float number{};
return get_number(input_format, number) && emit_float(input_format, number);
return get_number(input_format, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 'D':
{
double number{};
return get_number(input_format, number) && emit_float(input_format, number);
return get_number(input_format, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 'H':
@@ -3482,13 +3645,13 @@ class binary_reader
case 0x8E: // binary32
{
float number{};
return get_number(input_format_t::bon8, number) && emit_float(input_format_t::bon8, number);
return get_number(input_format_t::bon8, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 0x8F: // binary64
{
double number{};
return get_number(input_format_t::bon8, number) && emit_float(input_format_t::bon8, number);
return get_number(input_format_t::bon8, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 0xF8:
@@ -3554,9 +3717,7 @@ class binary_reader
@brief pass an integer to the SAX parser
Non-negative integers are passed as unsigned, negative integers as signed
numbers, like the other binary formats do. A value that does not fit the
number type is passed as described for @ref emit_unsigned and
@ref emit_signed.
numbers, like the other binary formats do.
@param[in] number the integer
@return whether the SAX parser accepted the value
@@ -3565,9 +3726,9 @@ class binary_reader
{
if (number >= 0)
{
return emit_unsigned(static_cast<std::uint64_t>(number));
return sax->number_unsigned(static_cast<number_unsigned_t>(number));
}
return emit_signed(number);
return sax->number_integer(static_cast<number_integer_t>(number));
}
/*!
@@ -3624,7 +3785,8 @@ class binary_reader
value = (value << 8) | static_cast<std::int64_t>(current);
}
return emit_bon8_integer(negative ? -(value + offset) : value + offset);
return negative ? sax->number_integer(static_cast<number_integer_t>(-(value + offset)))
: sax->number_unsigned(static_cast<number_unsigned_t>(value + offset));
}
/*!
@@ -3921,79 +4083,6 @@ class binary_reader
return true;
}
/*!
@brief pass a signed integer read from the input to the SAX parser
Like the lexer does for JSON text, a value that does not fit into
number_integer_t is passed as number_unsigned_t if it is non-negative and
fits there, and as number_float_t otherwise. With the default number
types, every integer the binary formats can encode fits, so this only
matters for narrower custom number types.
@tparam NumberType a signed integer type
@param[in] number the integer
@return whether the SAX parser accepted the value
*/
template<typename NumberType>
bool emit_signed(const NumberType number)
{
if (JSON_HEDLEY_LIKELY(value_in_range_of<number_integer_t>(number)))
{
return sax->number_integer(static_cast<number_integer_t>(number));
}
if (value_in_range_of<number_unsigned_t>(number))
{
return sax->number_unsigned(static_cast<number_unsigned_t>(number));
}
return sax->number_float(static_cast<number_float_t>(number), "");
}
/*!
@brief pass an unsigned integer read from the input to the SAX parser
Like the lexer does for JSON text, a value that does not fit into
number_unsigned_t is passed as number_float_t.
@tparam NumberType an unsigned integer type
@param[in] number the integer
@return whether the SAX parser accepted the value
*/
template<typename NumberType>
bool emit_unsigned(const NumberType number)
{
if (JSON_HEDLEY_LIKELY(value_in_range_of<number_unsigned_t>(number)))
{
return sax->number_unsigned(static_cast<number_unsigned_t>(number));
}
return sax->number_float(static_cast<number_float_t>(number), "");
}
/*!
@brief pass a floating-point number read from the input to the SAX parser
Like the lexer does for JSON text, a finite value that overflows
number_float_t is rejected instead of silently becoming infinity. Infinity
and NaN in the input are passed on unchanged.
@tparam NumberType a floating-point type
@param[in] format the current format (for diagnostics)
@param[in] number the number
@return whether the SAX parser accepted the value
@throw out_of_range.406 if a finite @a number overflows number_float_t
*/
template<typename NumberType>
bool emit_float(const input_format_t format, const NumberType number)
{
const auto result = static_cast<number_float_t>(number);
if (JSON_HEDLEY_UNLIKELY(std::isfinite(number) && !std::isfinite(result)))
{
return sax->parse_error(chars_read, get_token_string(),
out_of_range::create(406, exception_message(format, "number overflow", "value"), nullptr));
}
return sax->number_float(result, "");
}
/*!
@brief create a string by reading characters from the input
+213 -124
View File
@@ -13294,7 +13294,7 @@ class binary_reader
case 0x01: // double
{
double number{};
return get_number<double, true>(input_format_t::bson, number) && emit_float(input_format_t::bson, number);
return get_number<double, true>(input_format_t::bson, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 0x02: // string
@@ -13335,19 +13335,19 @@ class binary_reader
case 0x10: // int32
{
std::int32_t value{};
return get_number<std::int32_t, true>(input_format_t::bson, value) && emit_signed(value);
return get_number<std::int32_t, true>(input_format_t::bson, value) && sax->number_integer(value);
}
case 0x12: // int64
{
std::int64_t value{};
return get_number<std::int64_t, true>(input_format_t::bson, value) && emit_signed(value);
return get_number<std::int64_t, true>(input_format_t::bson, value) && sax->number_integer(value);
}
case 0x11: // uint64
{
std::uint64_t value{};
return get_number<std::uint64_t, true>(input_format_t::bson, value) && emit_unsigned(value);
return get_number<std::uint64_t, true>(input_format_t::bson, value) && sax->number_unsigned(value);
}
default: // anything else is not supported (yet)
@@ -13373,17 +13373,14 @@ class binary_reader
{
return false;
}
// the value is -1 - number, which fits into number_integer_t
// whenever number does
if (JSON_HEDLEY_LIKELY(value_in_range_of<number_integer_t>(number)))
const auto max_val = static_cast<NumberType>((std::numeric_limits<number_integer_t>::max)());
if (number > max_val)
{
return sax->number_integer(static_cast<number_integer_t>(-1) - static_cast<number_integer_t>(number));
return sax->parse_error(chars_read, get_token_string(),
parse_error::create(112, chars_read,
exception_message(input_format_t::cbor, "negative integer overflow", "value"), nullptr));
}
// like the lexer does for JSON text, store a value too small for
// number_integer_t as number_float_t
return sax->number_float(static_cast<number_float_t>(-1) - static_cast<number_float_t>(number), "");
return sax->number_integer(static_cast<number_integer_t>(-1) - static_cast<number_integer_t>(number));
}
/*!
@@ -13440,25 +13437,25 @@ class binary_reader
case 0x18: // Unsigned integer (one-byte uint8_t follows)
{
std::uint8_t number{};
return get_number(input_format_t::cbor, number) && emit_unsigned(number);
return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
}
case 0x19: // Unsigned integer (two-byte uint16_t follows)
{
std::uint16_t number{};
return get_number(input_format_t::cbor, number) && emit_unsigned(number);
return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
}
case 0x1A: // Unsigned integer (four-byte uint32_t follows)
{
std::uint32_t number{};
return get_number(input_format_t::cbor, number) && emit_unsigned(number);
return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
}
case 0x1B: // Unsigned integer (eight-byte uint64_t follows)
{
std::uint64_t number{};
return get_number(input_format_t::cbor, number) && emit_unsigned(number);
return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
}
// Negative integer -1-0x00..-1-0x17 (-1..-24)
@@ -13903,13 +13900,13 @@ class binary_reader
case 0xFA: // Single-Precision Float (four-byte IEEE 754)
{
float number{};
return get_number(input_format_t::cbor, number) && emit_float(input_format_t::cbor, number);
return get_number(input_format_t::cbor, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 0xFB: // Double-Precision Float (eight-byte IEEE 754)
{
double number{};
return get_number(input_format_t::cbor, number) && emit_float(input_format_t::cbor, number);
return get_number(input_format_t::cbor, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
default: // anything else (0xFF is handled inside the other types)
@@ -14062,6 +14059,80 @@ class binary_reader
}
}
/*!
@brief reads a CBOR object key
RFC 8949 allows any data item as a map key, but only strings have a
counterpart in JSON. A key of any other type is rejected with a message
naming that type, rather than the one @ref get_cbor_string gives for a
malformed string.
@param[out] result created key
@return whether key creation completed
*/
bool get_cbor_object_key(string_t& result)
{
// EOF and major type 3 (text string) are left to get_cbor_string
if (current == char_traits<char_type>::eof() || (static_cast<unsigned int>(current) & 0xE0u) == 0x60u)
{
return get_cbor_string(result);
}
const char* found = nullptr;
switch (static_cast<unsigned int>(current) >> 5u)
{
case 0:
found = "an unsigned integer";
break;
case 1:
found = "a negative integer";
break;
case 2:
found = "a byte string";
break;
case 4:
found = "an array";
break;
case 5:
found = "a map";
break;
case 6:
found = "a tag";
break;
default: // major type 7
switch (current)
{
case 0xF4:
case 0xF5:
found = "a boolean";
break;
case 0xF6:
found = "null";
break;
case 0xF7:
found = "undefined";
break;
case 0xF9:
case 0xFA:
case 0xFB:
found = "a floating-point number";
break;
case 0xFF:
found = "a break stop code";
break;
default:
found = "a simple value";
break;
}
break;
}
auto last_token = get_token_string();
return sax->parse_error(chars_read, last_token, parse_error::create(113, chars_read,
exception_message(input_format_t::cbor, concat("only string keys are supported, but found ", found, "; last byte: 0x", last_token), "object key"), nullptr));
}
/*!
@brief reads a definite-length CBOR byte array
@@ -14306,7 +14377,7 @@ class binary_reader
if (top.is_object)
{
key.clear();
if (JSON_HEDLEY_UNLIKELY(!get_cbor_string(key) || !sax->key(key)))
if (JSON_HEDLEY_UNLIKELY(!get_cbor_object_key(key) || !sax->key(key)))
{
return false;
}
@@ -14599,61 +14670,61 @@ class binary_reader
case 0xCA: // float 32
{
float number{};
return get_number(input_format_t::msgpack, number) && emit_float(input_format_t::msgpack, number);
return get_number(input_format_t::msgpack, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 0xCB: // float 64
{
double number{};
return get_number(input_format_t::msgpack, number) && emit_float(input_format_t::msgpack, number);
return get_number(input_format_t::msgpack, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 0xCC: // uint 8
{
std::uint8_t number{};
return get_number(input_format_t::msgpack, number) && emit_unsigned(number);
return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
}
case 0xCD: // uint 16
{
std::uint16_t number{};
return get_number(input_format_t::msgpack, number) && emit_unsigned(number);
return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
}
case 0xCE: // uint 32
{
std::uint32_t number{};
return get_number(input_format_t::msgpack, number) && emit_unsigned(number);
return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
}
case 0xCF: // uint 64
{
std::uint64_t number{};
return get_number(input_format_t::msgpack, number) && emit_unsigned(number);
return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
}
case 0xD0: // int 8
{
std::int8_t number{};
return get_number(input_format_t::msgpack, number) && emit_signed(number);
return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
}
case 0xD1: // int 16
{
std::int16_t number{};
return get_number(input_format_t::msgpack, number) && emit_signed(number);
return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
}
case 0xD2: // int 32
{
std::int32_t number{};
return get_number(input_format_t::msgpack, number) && emit_signed(number);
return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
}
case 0xD3: // int 64
{
std::int64_t number{};
return get_number(input_format_t::msgpack, number) && emit_signed(number);
return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
}
case 0xDC: // array 16
@@ -14807,6 +14878,98 @@ class binary_reader
}
}
/*!
@brief reads a MessagePack object key
The MessagePack specification allows any type as a map key, but only
strings have a counterpart in JSON. A key of any other type is rejected
with a message naming that type, rather than the one @ref
get_msgpack_string gives for a malformed string.
@param[out] result created key
@return whether key creation completed
*/
bool get_msgpack_object_key(string_t& result)
{
const char* found = nullptr;
switch (current)
{
case 0xC0:
found = "nil";
break;
case 0xC2:
case 0xC3:
found = "a boolean";
break;
case 0xCA:
case 0xCB:
found = "a float";
break;
case 0xC4:
case 0xC5:
case 0xC6:
found = "a bin";
break;
case 0xC7:
case 0xC8:
case 0xC9:
case 0xD4:
case 0xD5:
case 0xD6:
case 0xD7:
case 0xD8:
found = "an ext";
break;
case 0xCC:
case 0xCD:
case 0xCE:
case 0xCF:
case 0xD0:
case 0xD1:
case 0xD2:
case 0xD3:
found = "an integer";
break;
case 0xDC:
case 0xDD:
found = "an array";
break;
case 0xDE:
case 0xDF:
found = "a map";
break;
default:
// fixint, fixmap, and fixarray; strings, EOF, and the unused
// byte 0xC1 are left to get_msgpack_string
if (current == char_traits<char_type>::eof())
{
return get_msgpack_string(result);
}
if (current <= 0x7F || current >= 0xE0)
{
found = "an integer";
}
else if (current <= 0x8F)
{
found = "a map";
}
else if (current <= 0x9F)
{
found = "an array";
}
else
{
return get_msgpack_string(result);
}
break;
}
auto last_token = get_token_string();
return sax->parse_error(chars_read, last_token, parse_error::create(113, chars_read,
exception_message(input_format_t::msgpack, concat("only string keys are supported, but found ", found, "; last byte: 0x", last_token), "object key"), nullptr));
}
/*!
@brief reads a MessagePack byte array
@@ -14969,7 +15132,7 @@ class binary_reader
{
get();
key.clear();
if (JSON_HEDLEY_UNLIKELY(!get_msgpack_string(key) || !sax->key(key)))
if (JSON_HEDLEY_UNLIKELY(!get_msgpack_object_key(key) || !sax->key(key)))
{
return false;
}
@@ -15494,7 +15657,7 @@ class binary_reader
{
return sax->parse_error(chars_read, get_token_string(), out_of_range::create(408, exception_message(input_format, "excessive ndarray size caused overflow", "size"), nullptr));
}
if (JSON_HEDLEY_UNLIKELY(!emit_unsigned(i)))
if (JSON_HEDLEY_UNLIKELY(!sax->number_unsigned(static_cast<number_unsigned_t>(i))))
{
return false;
}
@@ -15626,37 +15789,37 @@ class binary_reader
break;
}
std::uint8_t number{};
return get_number(input_format, number) && emit_unsigned(number);
return get_number(input_format, number) && sax->number_unsigned(number);
}
case 'U':
{
std::uint8_t number{};
return get_number(input_format, number) && emit_unsigned(number);
return get_number(input_format, number) && sax->number_unsigned(number);
}
case 'i':
{
std::int8_t number{};
return get_number(input_format, number) && emit_signed(number);
return get_number(input_format, number) && sax->number_integer(number);
}
case 'I':
{
std::int16_t number{};
return get_number(input_format, number) && emit_signed(number);
return get_number(input_format, number) && sax->number_integer(number);
}
case 'l':
{
std::int32_t number{};
return get_number(input_format, number) && emit_signed(number);
return get_number(input_format, number) && sax->number_integer(number);
}
case 'L':
{
std::int64_t number{};
return get_number(input_format, number) && emit_signed(number);
return get_number(input_format, number) && sax->number_integer(number);
}
case 'u':
@@ -15666,7 +15829,7 @@ class binary_reader
break;
}
std::uint16_t number{};
return get_number(input_format, number) && emit_unsigned(number);
return get_number(input_format, number) && sax->number_unsigned(number);
}
case 'm':
@@ -15676,7 +15839,7 @@ class binary_reader
break;
}
std::uint32_t number{};
return get_number(input_format, number) && emit_unsigned(number);
return get_number(input_format, number) && sax->number_unsigned(number);
}
case 'M':
@@ -15686,7 +15849,7 @@ class binary_reader
break;
}
std::uint64_t number{};
return get_number(input_format, number) && emit_unsigned(number);
return get_number(input_format, number) && sax->number_unsigned(number);
}
case 'h':
@@ -15744,13 +15907,13 @@ class binary_reader
case 'd':
{
float number{};
return get_number(input_format, number) && emit_float(input_format, number);
return get_number(input_format, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 'D':
{
double number{};
return get_number(input_format, number) && emit_float(input_format, number);
return get_number(input_format, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 'H':
@@ -16217,13 +16380,13 @@ class binary_reader
case 0x8E: // binary32
{
float number{};
return get_number(input_format_t::bon8, number) && emit_float(input_format_t::bon8, number);
return get_number(input_format_t::bon8, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 0x8F: // binary64
{
double number{};
return get_number(input_format_t::bon8, number) && emit_float(input_format_t::bon8, number);
return get_number(input_format_t::bon8, number) && sax->number_float(static_cast<number_float_t>(number), "");
}
case 0xF8:
@@ -16289,9 +16452,7 @@ class binary_reader
@brief pass an integer to the SAX parser
Non-negative integers are passed as unsigned, negative integers as signed
numbers, like the other binary formats do. A value that does not fit the
number type is passed as described for @ref emit_unsigned and
@ref emit_signed.
numbers, like the other binary formats do.
@param[in] number the integer
@return whether the SAX parser accepted the value
@@ -16300,9 +16461,9 @@ class binary_reader
{
if (number >= 0)
{
return emit_unsigned(static_cast<std::uint64_t>(number));
return sax->number_unsigned(static_cast<number_unsigned_t>(number));
}
return emit_signed(number);
return sax->number_integer(static_cast<number_integer_t>(number));
}
/*!
@@ -16359,7 +16520,8 @@ class binary_reader
value = (value << 8) | static_cast<std::int64_t>(current);
}
return emit_bon8_integer(negative ? -(value + offset) : value + offset);
return negative ? sax->number_integer(static_cast<number_integer_t>(-(value + offset)))
: sax->number_unsigned(static_cast<number_unsigned_t>(value + offset));
}
/*!
@@ -16656,79 +16818,6 @@ class binary_reader
return true;
}
/*!
@brief pass a signed integer read from the input to the SAX parser
Like the lexer does for JSON text, a value that does not fit into
number_integer_t is passed as number_unsigned_t if it is non-negative and
fits there, and as number_float_t otherwise. With the default number
types, every integer the binary formats can encode fits, so this only
matters for narrower custom number types.
@tparam NumberType a signed integer type
@param[in] number the integer
@return whether the SAX parser accepted the value
*/
template<typename NumberType>
bool emit_signed(const NumberType number)
{
if (JSON_HEDLEY_LIKELY(value_in_range_of<number_integer_t>(number)))
{
return sax->number_integer(static_cast<number_integer_t>(number));
}
if (value_in_range_of<number_unsigned_t>(number))
{
return sax->number_unsigned(static_cast<number_unsigned_t>(number));
}
return sax->number_float(static_cast<number_float_t>(number), "");
}
/*!
@brief pass an unsigned integer read from the input to the SAX parser
Like the lexer does for JSON text, a value that does not fit into
number_unsigned_t is passed as number_float_t.
@tparam NumberType an unsigned integer type
@param[in] number the integer
@return whether the SAX parser accepted the value
*/
template<typename NumberType>
bool emit_unsigned(const NumberType number)
{
if (JSON_HEDLEY_LIKELY(value_in_range_of<number_unsigned_t>(number)))
{
return sax->number_unsigned(static_cast<number_unsigned_t>(number));
}
return sax->number_float(static_cast<number_float_t>(number), "");
}
/*!
@brief pass a floating-point number read from the input to the SAX parser
Like the lexer does for JSON text, a finite value that overflows
number_float_t is rejected instead of silently becoming infinity. Infinity
and NaN in the input are passed on unchanged.
@tparam NumberType a floating-point type
@param[in] format the current format (for diagnostics)
@param[in] number the number
@return whether the SAX parser accepted the value
@throw out_of_range.406 if a finite @a number overflows number_float_t
*/
template<typename NumberType>
bool emit_float(const input_format_t format, const NumberType number)
{
const auto result = static_cast<number_float_t>(number);
if (JSON_HEDLEY_UNLIKELY(std::isfinite(number) && !std::isfinite(result)))
{
return sax->parse_error(chars_read, get_token_string(),
out_of_range::create(406, exception_message(format, "number overflow", "value"), nullptr));
}
return sax->number_float(result, "");
}
/*!
@brief create a string by reading characters from the input
-116
View File
@@ -11,12 +11,7 @@
#include <nlohmann/json.hpp>
using nlohmann::json;
#include <cmath>
#include <fstream>
#include <limits>
#include <map>
#include <string>
#include <vector>
#include "make_test_data_available.hpp"
TEST_CASE("Binary Formats" * doctest::skip())
@@ -229,114 +224,3 @@ TEST_CASE("Binary Formats" * doctest::skip())
CHECK((100.0 * double(ubjson_3_size) / double(json_size)) == Approx(89.450));
}
}
TEST_CASE("Binary formats with narrow number types")
{
// Numbers that do not fit the number types are handled like the lexer
// handles them in JSON text: an integer that fits neither integer type is
// stored as a floating-point number, and a finite floating-point number
// that overflows number_float_t is rejected with out_of_range.406.
using narrow_json = nlohmann::basic_json<std::map, std::vector, std::string, bool, std::int32_t, std::uint32_t, float>;
using bytes = std::vector<std::uint8_t>;
struct binary_format
{
const char* name;
bytes (*encode)(const json&);
narrow_json (*decode)(const bytes&, bool);
};
const std::vector<binary_format> formats =
{
{
"CBOR", [](const json & j) { return json::to_cbor(j); },
[](const bytes & v, bool allow_exceptions)
{
return narrow_json::from_cbor(v, true, allow_exceptions);
}
},
{
"MessagePack", [](const json & j) { return json::to_msgpack(j); },
[](const bytes & v, bool allow_exceptions)
{
return narrow_json::from_msgpack(v, true, allow_exceptions);
}
},
{
"UBJSON", [](const json & j) { return json::to_ubjson(j); },
[](const bytes & v, bool allow_exceptions)
{
return narrow_json::from_ubjson(v, true, allow_exceptions);
}
},
{
"BJData", [](const json & j) { return json::to_bjdata(j); },
[](const bytes & v, bool allow_exceptions)
{
return narrow_json::from_bjdata(v, true, allow_exceptions);
}
},
{
// BSON can only store numbers as object members
"BSON", [](const json & j) { return json::to_bson(json{{"a", j}}); },
[](const bytes & v, bool allow_exceptions)
{
const auto result = narrow_json::from_bson(v, true, allow_exceptions);
return result.is_discarded() ? result : result.at("a");
}
},
{
"BON8", [](const json & j) { return json::to_bon8(j); },
[](const bytes & v, bool allow_exceptions)
{
return narrow_json::from_bon8(v, true, allow_exceptions);
}
},
};
for (const auto& format : formats)
{
const std::string name = format.name;
INFO("format := ", name);
const auto roundtrip = [&format](const json & j)
{
return format.decode(format.encode(j), true);
};
// integers that fit keep their type
CHECK(roundtrip(json(-5)).is_number_integer());
CHECK(roundtrip(json(-5)).get<std::int32_t>() == -5);
CHECK(roundtrip(json(3000000000u)).is_number_unsigned());
CHECK(roundtrip(json(3000000000u)).get<std::uint32_t>() == 3000000000u);
// integers that fit neither integer type are stored as float
CHECK(roundtrip(json(5000000000u)).is_number_float());
CHECK(roundtrip(json(5000000000u)).get<float>() == 5000000000.0f);
if (name != "BON8") // BON8 cannot encode integers above INT64_MAX
{
CHECK(roundtrip(json(10000000000000000000u)).is_number_float());
CHECK(roundtrip(json(10000000000000000000u)).get<float>() == 10000000000000000000.0f);
}
CHECK(roundtrip(json(-3000000000)).is_number_float());
CHECK(roundtrip(json(-3000000000)).get<float>() == -3000000000.0f);
CHECK(roundtrip(json(-5000000000)).is_number_float());
CHECK(roundtrip(json(-5000000000)).get<float>() == -5000000000.0f);
// floating-point numbers that fit
CHECK(roundtrip(json(1.5)).get<float>() == 1.5f);
const auto just_above_max = std::nextafter(static_cast<double>((std::numeric_limits<float>::max)()),
std::numeric_limits<double>::infinity());
CHECK(roundtrip(json(just_above_max)).get<float>() == (std::numeric_limits<float>::max)());
// infinity and NaN are passed on
CHECK(std::isinf(roundtrip(json(std::numeric_limits<double>::infinity())).get<float>()));
CHECK(std::isnan(roundtrip(json(std::numeric_limits<double>::quiet_NaN())).get<float>()));
// finite floating-point numbers that overflow number_float_t are rejected
const std::string message = "[json.exception.out_of_range.406] syntax error while parsing " + name
+ " value: number overflow";
CHECK_THROWS_WITH_AS(roundtrip(json(1e300)), message.c_str(), narrow_json::out_of_range&);
CHECK_THROWS_WITH_AS(roundtrip(json(-1e300)), message.c_str(), narrow_json::out_of_range&);
CHECK(format.decode(format.encode(json(1e300)), false).is_discarded());
}
}
+57 -18
View File
@@ -1830,10 +1830,51 @@ TEST_CASE("CBOR")
SECTION("invalid string in map")
{
json _;
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0xa1, 0xff, 0x01})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR string: expected length specification (0x60-0x7B) or indefinite string type (0x7F); last byte: 0xFF", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0xa1, 0xff, 0x01})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR object key: only string keys are supported, but found a break stop code; last byte: 0xFF", json::parse_error&);
CHECK(json::from_cbor(std::vector<uint8_t>({0xa1, 0xff, 0x01}), true, false).is_discarded());
}
SECTION("non-string key (see #2766 and #3381)")
{
// only text strings map to JSON object keys; any other key is
// rejected with a message naming its type
const std::vector<std::pair<std::vector<std::uint8_t>, std::string>> cases =
{
{{0xA1, 0x01, 0x01}, "an unsigned integer; last byte: 0x01"},
{{0xA1, 0x20, 0x01}, "a negative integer; last byte: 0x20"},
{{0xA1, 0x41, 0x61, 0x01}, "a byte string; last byte: 0x41"},
{{0xA1, 0x80, 0x01}, "an array; last byte: 0x80"},
{{0xA1, 0xA0, 0x01}, "a map; last byte: 0xA0"},
{{0xA1, 0xC0, 0x61, 0x61, 0x01}, "a tag; last byte: 0xC0"},
{{0xA1, 0xF4, 0x01}, "a boolean; last byte: 0xF4"},
{{0xA1, 0xF5, 0x01}, "a boolean; last byte: 0xF5"},
{{0xA1, 0xF6, 0x01}, "null; last byte: 0xF6"},
{{0xA1, 0xF7, 0x01}, "undefined; last byte: 0xF7"},
{{0xA1, 0xF9, 0x3C, 0x00, 0x01}, "a floating-point number; last byte: 0xF9"},
{{0xA1, 0xFA, 0x3F, 0x80, 0x00, 0x00, 0x01}, "a floating-point number; last byte: 0xFA"},
{{0xA1, 0xFB, 0x3F, 0xF0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01}, "a floating-point number; last byte: 0xFB"},
{{0xA1, 0xE0, 0x01}, "a simple value; last byte: 0xE0"},
{{0xA1, 0xF8, 0x20, 0x01}, "a simple value; last byte: 0xF8"},
// indefinite-length map
{{0xBF, 0x01, 0x01, 0xFF}, "an unsigned integer; last byte: 0x01"},
};
for (const auto& c : cases)
{
CAPTURE(c.first)
const std::string expected = "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR object key: only string keys are supported, but found " + c.second;
json _;
CHECK_THROWS_WITH_AS(_ = json::from_cbor(c.first), expected.c_str(), json::parse_error&);
CHECK(json::from_cbor(c.first, true, false).is_discarded());
}
// a key of major type 3 with a reserved length is still reported as
// a malformed string, and a missing key as the end of input
json _;
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0xA1})), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing CBOR string: unexpected end of input", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0xA1, 0x7C, 0x01})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR string: expected length specification (0x60-0x7B) or indefinite string type (0x7F); last byte: 0x7C", json::parse_error&);
}
SECTION("invalid UTF-8 in string (see #5529)")
{
// a two-character text string (major type 3) whose bytes are not
@@ -2284,7 +2325,7 @@ TEST_CASE("CBOR indefinite-length strings do not recurse per chunk")
SECTION("a break marker outside an indefinite-length string is not a string")
{
// 0xFF only closes a string that was opened; on its own it is not one
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0xA1, 0xFF, 0x01})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR string: expected length specification (0x60-0x7B) or indefinite string type (0x7F); last byte: 0xFF", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0xA1, 0xFF, 0x01})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR object key: only string keys are supported, but found a break stop code; last byte: 0xFF", json::parse_error&);
}
}
@@ -3146,8 +3187,7 @@ TEST_CASE("Tagged values")
// CBOR encodes negative integers as: result = -1 - n
// For type 0x3B, n is an 8-byte uint64_t. Valid range for n with
// the default int64_t is [0, INT64_MAX], producing results in [INT64_MIN, -1].
// When n > INT64_MAX, the result exceeds int64_t range and is stored
// as a floating-point number, as the lexer does for JSON text.
// When n > INT64_MAX, the result exceeds int64_t range and is rejected.
SECTION("n = 0 is valid (result = -1)")
{
@@ -3168,34 +3208,33 @@ TEST_CASE("Tagged values")
CHECK(result.get<int64_t>() == (std::numeric_limits<int64_t>::min)());
}
SECTION("n = INT64_MAX + 1 is stored as float")
SECTION("n = INT64_MAX + 1 is rejected (overflow)")
{
// n = INT64_MAX + 1 (0x8000000000000000)
// result = -1 - n = -9223372036854775809, which exceeds int64_t range;
// the nearest double is -9223372036854775808.0
// result = -1 - n = -9223372036854775809, which exceeds int64_t range
const std::vector<uint8_t> input = {0x3B, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
const auto result = json::from_cbor(input);
CHECK(result.is_number_float());
CHECK(result.get<double>() == -9223372036854775808.0);
CHECK(result == json::parse("-9223372036854775809"));
json _;
CHECK_THROWS_WITH_AS(_ = json::from_cbor(input),
"[json.exception.parse_error.112] parse error at byte 9: syntax error while parsing CBOR value: negative integer overflow",
json::parse_error);
}
SECTION("n = UINT64_MAX is stored as float")
SECTION("n = UINT64_MAX is rejected (overflow)")
{
// n = UINT64_MAX (0xFFFFFFFFFFFFFFFF)
// result = -1 - n = -18446744073709551616, which exceeds int64_t range
const std::vector<uint8_t> input = {0x3B, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF};
const auto result = json::from_cbor(input);
CHECK(result.is_number_float());
CHECK(result.get<double>() == -18446744073709551616.0);
CHECK(result == json::parse("-18446744073709551616"));
json _;
CHECK_THROWS_WITH_AS(_ = json::from_cbor(input),
"[json.exception.parse_error.112] parse error at byte 9: syntax error while parsing CBOR value: negative integer overflow",
json::parse_error);
}
SECTION("overflow with allow_exceptions=false is not an error")
SECTION("overflow with allow_exceptions=false returns discarded")
{
const std::vector<uint8_t> input = {0x3B, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
const auto result = json::from_cbor(input, true, false);
CHECK(result.is_number_float());
CHECK(result.is_discarded());
}
}
+60 -1
View File
@@ -1551,10 +1551,69 @@ TEST_CASE("MessagePack")
SECTION("invalid string in map")
{
json _;
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0x81, 0xff, 0x01})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing MessagePack string: expected length specification (0xA0-0xBF, 0xD9-0xDB); last byte: 0xFF", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0x81, 0xff, 0x01})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing MessagePack object key: only string keys are supported, but found an integer; last byte: 0xFF", json::parse_error&);
CHECK(json::from_msgpack(std::vector<uint8_t>({0x81, 0xff, 0x01}), true, false).is_discarded());
}
SECTION("non-string key (see #3381)")
{
// only strings map to JSON object keys; any other key is rejected
// with a message naming its type
const std::vector<std::pair<std::vector<std::uint8_t>, std::string>> cases =
{
{{0x81, 0xC0, 0x01}, "nil; last byte: 0xC0"},
{{0x81, 0xC2, 0x01}, "a boolean; last byte: 0xC2"},
{{0x81, 0xC3, 0x01}, "a boolean; last byte: 0xC3"},
{{0x81, 0xCA, 0x3F, 0x80, 0x00, 0x00, 0x01}, "a float; last byte: 0xCA"},
{{0x81, 0xCB, 0x3F, 0xF0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01}, "a float; last byte: 0xCB"},
{{0x81, 0xC4, 0x00, 0x01}, "a bin; last byte: 0xC4"},
{{0x81, 0xC5, 0x00, 0x00, 0x01}, "a bin; last byte: 0xC5"},
{{0x81, 0xC6, 0x00, 0x00, 0x00, 0x00, 0x01}, "a bin; last byte: 0xC6"},
{{0x81, 0xC7, 0x00, 0x01, 0x01}, "an ext; last byte: 0xC7"},
{{0x81, 0xC8, 0x00, 0x00, 0x01, 0x01}, "an ext; last byte: 0xC8"},
{{0x81, 0xC9, 0x00, 0x00, 0x00, 0x00, 0x01, 0x01}, "an ext; last byte: 0xC9"},
{{0x81, 0xD4, 0x01, 0x00, 0x01}, "an ext; last byte: 0xD4"},
{{0x81, 0xD5, 0x01, 0x00, 0x00, 0x01}, "an ext; last byte: 0xD5"},
{{0x81, 0xD6, 0x01, 0x00, 0x00, 0x00, 0x00, 0x01}, "an ext; last byte: 0xD6"},
{{0x81, 0xD7, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01}, "an ext; last byte: 0xD7"},
{{0x81, 0xD8, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01}, "an ext; last byte: 0xD8"},
{{0x81, 0xCC, 0x01, 0x01}, "an integer; last byte: 0xCC"},
{{0x81, 0xCD, 0x00, 0x01, 0x01}, "an integer; last byte: 0xCD"},
{{0x81, 0xCE, 0x00, 0x00, 0x00, 0x01, 0x01}, "an integer; last byte: 0xCE"},
{{0x81, 0xCF, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x01}, "an integer; last byte: 0xCF"},
{{0x81, 0xD0, 0x01, 0x01}, "an integer; last byte: 0xD0"},
{{0x81, 0xD1, 0x00, 0x01, 0x01}, "an integer; last byte: 0xD1"},
{{0x81, 0xD2, 0x00, 0x00, 0x00, 0x01, 0x01}, "an integer; last byte: 0xD2"},
{{0x81, 0xD3, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x01}, "an integer; last byte: 0xD3"},
{{0x81, 0x00, 0x01}, "an integer; last byte: 0x00"},
{{0x81, 0x7F, 0x01}, "an integer; last byte: 0x7F"},
{{0x81, 0xE0, 0x01}, "an integer; last byte: 0xE0"},
{{0x81, 0x80, 0x01}, "a map; last byte: 0x80"},
{{0x81, 0x8F, 0x01}, "a map; last byte: 0x8F"},
{{0x81, 0xDE, 0x00, 0x00, 0x01}, "a map; last byte: 0xDE"},
{{0x81, 0xDF, 0x00, 0x00, 0x00, 0x00, 0x01}, "a map; last byte: 0xDF"},
{{0x81, 0x90, 0x01}, "an array; last byte: 0x90"},
{{0x81, 0x9F, 0x01}, "an array; last byte: 0x9F"},
{{0x81, 0xDC, 0x00, 0x00, 0x01}, "an array; last byte: 0xDC"},
{{0x81, 0xDD, 0x00, 0x00, 0x00, 0x00, 0x01}, "an array; last byte: 0xDD"},
};
for (const auto& c : cases)
{
CAPTURE(c.first)
const std::string expected = "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing MessagePack object key: only string keys are supported, but found " + c.second;
json _;
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(c.first), expected.c_str(), json::parse_error&);
CHECK(json::from_msgpack(c.first, true, false).is_discarded());
}
json _;
// the unused byte 0xC1 is still reported as a malformed string
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0x81, 0xC1, 0x01})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing MessagePack string: expected length specification (0xA0-0xBF, 0xD9-0xDB); last byte: 0xC1", json::parse_error&);
// a missing key is still reported as the end of input
CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0x81})), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing MessagePack string: unexpected end of input", json::parse_error&);
}
SECTION("invalid UTF-8 in string (see #5529)")
{
// a fixstr of length 2 (0xA0 | 2) whose bytes are not valid UTF-8
+3 -3
View File
@@ -1018,7 +1018,7 @@ TEST_CASE("regression tests 1")
};
json _;
CHECK_THROWS_WITH_AS(_ = json::from_cbor(vec), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR string: expected length specification (0x60-0x7B) or indefinite string type (0x7F); last byte: 0x98", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_cbor(vec), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR object key: only string keys are supported, but found an array; last byte: 0x98", json::parse_error&);
// related test case: nonempty UTF-8 string (indefinite length)
std::vector<uint8_t> const vec1 {0x7f, 0x61, 0x61};
@@ -1065,7 +1065,7 @@ TEST_CASE("regression tests 1")
};
json _;
CHECK_THROWS_WITH_AS(_ = json::from_cbor(vec1), "[json.exception.parse_error.113] parse error at byte 13: syntax error while parsing CBOR string: expected length specification (0x60-0x7B) or indefinite string type (0x7F); last byte: 0xB4", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_cbor(vec1), "[json.exception.parse_error.113] parse error at byte 13: syntax error while parsing CBOR object key: only string keys are supported, but found a map; last byte: 0xB4", json::parse_error&);
// related test case: double-precision
std::vector<uint8_t> const vec2
@@ -1077,7 +1077,7 @@ TEST_CASE("regression tests 1")
0x96, 0x96, 0xb4, 0xb4, 0xfa, 0x94, 0x94, 0x61,
0x61, 0x61, 0x61, 0x61, 0x61, 0x61, 0x61, 0xfb
};
CHECK_THROWS_WITH_AS(_ = json::from_cbor(vec2), "[json.exception.parse_error.113] parse error at byte 13: syntax error while parsing CBOR string: expected length specification (0x60-0x7B) or indefinite string type (0x7F); last byte: 0xB4", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_cbor(vec2), "[json.exception.parse_error.113] parse error at byte 13: syntax error while parsing CBOR object key: only string keys are supported, but found a map; last byte: 0xB4", json::parse_error&);
}
SECTION("issue #452 - Heap-buffer-overflow (OSS-Fuzz issue 585)")