Merge branch 'develop' into claude/binary-reader-narrow-numbers

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

README.md

@@ -363,7 +363,7 @@ std::cout << j_string << " == " << serialized_string << std::endl;
 
 [`.dump()`](https://json.nlohmann.me/api/basic_json/dump/) returns the originally stored string value.
 
-Note the library only supports UTF-8. When you store strings with different encodings in the library, calling [`dump()`](https://json.nlohmann.me/api/basic_json/dump/) may throw an exception unless `json::error_handler_t::replace`, `json::error_handler_t::ignore`, or `json::error_handler_t::keep` are used as error handlers.
+Note the library only supports UTF-8. When you store strings with different encodings in the library, calling [`dump()`](https://json.nlohmann.me/api/basic_json/dump/) may throw an exception unless `json::error_handler_t::replace` or `json::error_handler_t::ignore` are used as error handlers.
 
 #### To/from streams (e.g., files, string streams)
 
@@ -1915,7 +1915,7 @@ The library supports **Unicode input** as follows:
 - [Unicode noncharacters](https://www.unicode.org/faq/private_use.html#nonchar1) will not be replaced by the library.
 - Invalid surrogates (e.g., incomplete pairs such as `\uDEAD`) will yield parse errors.
 - The strings stored in the library are UTF-8 encoded. When using the default string type (`std::string`), note that its length/size functions return the number of stored bytes rather than the number of characters or glyphs.
-- When you store strings with different encodings in the library, calling [`dump()`](https://json.nlohmann.me/api/basic_json/dump/) may throw an exception unless `json::error_handler_t::replace`, `json::error_handler_t::ignore`, or `json::error_handler_t::keep` are used as error handlers.
+- When you store strings with different encodings in the library, calling [`dump()`](https://json.nlohmann.me/api/basic_json/dump/) may throw an exception unless `json::error_handler_t::replace` or `json::error_handler_t::ignore` are used as error handlers.
 - To store wide strings (e.g., `std::wstring`), you need to convert them to a UTF-8 encoded `std::string` before, see [an example](https://json.nlohmann.me/home/faq/#wide-string-handling).
 
 ### Comments in JSON

docs/mkdocs/docs/api/basic_json/dump.md

@@ -25,15 +25,10 @@ and `ensure_ascii` parameters.
     result consists of ASCII characters only.
 
 `error_handler` (in)
-:   how to react on decoding errors; there are four possible values (see [`error_handler_t`](error_handler_t.md)):
+:   how to react on decoding errors; there are three possible values (see [`error_handler_t`](error_handler_t.md):
-
+    `strict` (throws an exception in case a decoding error occurs; default), `replace` (replace invalid UTF-8 sequences
-    - `strict`: throw a [`type_error`](../../home/exceptions.md#type-errors) exception in case a decoding error occurs
+    with U+FFFD), and `ignore` (ignore invalid UTF-8 sequences during serialization; all valid bytes are copied to the
-      (default),
+    output unchanged, and invalid bytes are dropped)).
-    - `replace`: replace invalid UTF-8 sequences with U+FFFD (� REPLACEMENT CHARACTER),
-    - `ignore`: ignore invalid UTF-8 sequences during serialization; all valid bytes are copied to the output unchanged,
-      and invalid bytes are dropped, and
-    - `keep`: keep invalid UTF-8 sequences during serialization; all bytes are copied to the output unchanged, so the
-      result is not valid UTF-8.
     
 ## Return value
 
@@ -99,4 +94,3 @@ Binary values are serialized as an object containing two keys:
 - Indentation character `indent_char`, option `ensure_ascii` and exceptions added in version 3.0.0.
 - Error handlers added in version 3.4.0.
 - Serialization of binary values added in version 3.8.0.
-- Error handler value `keep` added in version 3.13.0.

docs/mkdocs/docs/api/basic_json/error_handler_t.md

@@ -4,13 +4,12 @@
 enum class error_handler_t {
     strict,
     replace,
-    ignore,
+    ignore
-    keep
 };
 ```
 
 This enumeration is used in the [`dump`](dump.md) function to choose how to treat decoding errors while serializing a
-`basic_json` value. Four values are differentiated:
+`basic_json` value. Three values are differentiated:
 
 strict
 :   throw a `type_error` exception in case of invalid UTF-8
@@ -21,12 +20,6 @@ replace
 ignore
 :   ignore invalid UTF-8 sequences; all valid bytes are copied to the output unchanged, and invalid bytes are dropped
 
-keep
-:   keep invalid UTF-8 sequences; all bytes are copied to the output unchanged. Valid characters are still escaped as
-    usual (e.g., `"`, `\\`, and control characters), so the result has valid JSON syntax, but it is not valid UTF-8.
-    In particular, [`parse`](parse.md) rejects it, and with `ensure_ascii` set to `true`, the invalid bytes are the
-    only non-ASCII bytes of the output.
-
 ## Examples
 
 ??? example
@@ -47,4 +40,3 @@ keep
 ## Version history
 
 - Added in version 3.4.0.
-- Added value `keep` in version 3.13.0.

docs/mkdocs/docs/api/basic_json/from_cbor.md

@@ -80,8 +80,8 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
   the end of the file was not reached when `strict` was set to true
 - Throws [parse_error.112](../../home/exceptions.md#jsonexceptionparse_error112) if unsupported features from CBOR were
   used in the given input or if the input is not valid CBOR
-- Throws [parse_error.113](../../home/exceptions.md#jsonexceptionparse_error113) if a string was expected as a map key,
+- Throws [parse_error.113](../../home/exceptions.md#jsonexceptionparse_error113) if a map key is not a string (keys of other
-  but not found
+  types are not supported, as JSON object keys are always strings) or a string is malformed
 
 ## Complexity
 

docs/mkdocs/docs/api/basic_json/from_msgpack.md

@@ -73,8 +73,8 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
   the end of the file was not reached when `strict` was set to true
 - Throws [parse_error.112](../../home/exceptions.md#jsonexceptionparse_error112) if unsupported features from
   MessagePack were used in the given input or if the input is not valid MessagePack
-- Throws [parse_error.113](../../home/exceptions.md#jsonexceptionparse_error113) if a string was expected as a map key,
+- Throws [parse_error.113](../../home/exceptions.md#jsonexceptionparse_error113) if a map key is not a string (keys of other
-  but not found
+  types are not supported, as JSON object keys are always strings) or a string is malformed
 
 ## Complexity
 

docs/mkdocs/docs/api/basic_json/number_float_t.md

@@ -55,6 +55,10 @@ This implementation does exactly follow this approach, as it uses double precisi
 smaller than `-1.79769313486232e+308` and values greater than `1.79769313486232e+308` will be stored as NaN internally
 and be serialized to `null`.
 
+During deserialization (from JSON text or any of the binary formats), a finite number that does not fit into
+`number_float_t` is rejected with [`out_of_range.406`](../../home/exceptions.md#jsonexceptionout_of_range406), for
+example a double-precision number in a binary format when `number_float_t` is `#!cpp float`.
+
 #### Storage
 
 Floating-point number values are stored directly inside a `basic_json` type.

docs/mkdocs/docs/api/basic_json/number_integer_t.md

@@ -47,8 +47,9 @@ With the default values for `NumberIntegerType` (`std::int64_t`), the default va
 
 When the default type is used, the maximal integer number that can be stored is `9223372036854775807` (INT64_MAX) and
 the minimal integer number that can be stored is `-9223372036854775808` (INT64_MIN). Integer numbers that are out of
-range will yield over/underflow when used in a constructor. During deserialization, too large or small integer numbers
+range will yield over/underflow when used in a constructor. During deserialization (from JSON text or any of the binary
-will automatically be stored as [`number_unsigned_t`](number_unsigned_t.md) or [`number_float_t`](number_float_t.md).
+formats), too large or small integer numbers will automatically be stored as [`number_unsigned_t`](number_unsigned_t.md)
+or [`number_float_t`](number_float_t.md).
 
 [RFC 8259](https://tools.ietf.org/html/rfc8259) further states:
 > Note that when such software is used, numbers that are integers and are in the range $[-2^{53}+1, 2^{53}-1]$ are

docs/mkdocs/docs/api/basic_json/number_unsigned_t.md

@@ -48,8 +48,9 @@ With the default values for `NumberUnsignedType` (`std::uint64_t`), the default
 
 When the default type is used, the maximal integer number that can be stored is `18446744073709551615` (UINT64_MAX) and
 the minimal integer number that can be stored is `0`. Integer numbers that are out of range will yield over/underflow
-when used in a constructor. During deserialization, too large or small integer numbers will automatically be stored
+when used in a constructor. During deserialization (from JSON text or any of the binary formats), too large or small
-as [`number_integer_t`](number_integer_t.md) or [`number_float_t`](number_float_t.md).
+integer numbers will automatically be stored as [`number_integer_t`](number_integer_t.md) or
+[`number_float_t`](number_float_t.md).
 
 [RFC 8259](https://tools.ietf.org/html/rfc8259) further states:
 > Note that when such software is used, numbers that are integers and are in the range $[-2^{53}+1, 2^{53}-1]$ are

docs/mkdocs/docs/examples/error_handler_t.cpp

@@ -1,4 +1,3 @@
-#include <iomanip>
 #include <iostream>
 #include <nlohmann/json.hpp>
 
@@ -22,12 +21,4 @@ int main()
               << "\nstring with ignored invalid characters: "
               << j_invalid.dump(-1, ' ', false, json::error_handler_t::ignore)
               << '\n';
-
-    // the invalid byte is kept; print the result byte-wise to make it visible
-    std::cout << "string with kept invalid characters:";
-    for (const unsigned char c : j_invalid.dump(-1, ' ', false, json::error_handler_t::keep))
-    {
-        std::cout << ' ' << std::hex << std::setw(2) << std::setfill('0') << static_cast<int>(c);
-    }
-    std::cout << '\n';
 }

docs/mkdocs/docs/examples/error_handler_t.output

@@ -1,4 +1,3 @@
 [json.exception.type_error.316] invalid UTF-8 byte at index 2: 0xA9
 string with replaced invalid characters: "ä�ü"
 string with ignored invalid characters: "äü"
-string with kept invalid characters: 22 c3 a4 a9 c3 bc 22

docs/mkdocs/docs/features/binary_formats/cbor.md

@@ -168,13 +168,26 @@ The library maps CBOR types to JSON value types as follows:
 !!! warning "Negative integer overflow"
 
     CBOR negative integers (major type 1) are decoded as `-1 - n`. If the encoded magnitude `n` is too large for the
-    result to fit into `number_integer_t` (`std::int64_t` by default), parsing fails with a
+    result to fit into `number_integer_t` (`std::int64_t` by default), the result is stored as `number_float_t`, like
-    [`parse_error.112`](../../home/exceptions.md#jsonexceptionparse_error112) exception rather than overflowing
+    a too small integer in JSON text. For example, `-18446744073709551616` (`0x3B` followed by eight `0xFF` bytes) is
-    silently.
+    stored as `-1.8446744073709552e+19`.
 
 !!! warning "Object keys"
 
-    CBOR allows map keys of any type, whereas JSON only allows strings as keys in object values. Therefore, CBOR maps with keys other than UTF-8 strings are rejected.
+    CBOR allows map keys of any type, whereas JSON only allows strings as keys in object values. Therefore, CBOR maps
+    with keys other than text strings (major type 3) are rejected with a
+    [`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113) exception (or, with `allow_exceptions` set
+    to `false`, a discarded value) naming the type of the key that was found, for instance:
+
+    ```
+    [json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR object key: only string keys are supported, but found an unsigned integer; last byte: 0x01
+    ```
+
+    This applies to the [SAX interface](../parsing/sax_interface.md) as well, as the key is read before it is passed
+    on. This is a deliberate restriction of the library's JSON value model, not an oversight: formats built on CBOR
+    maps with integer keys, such as COSE ([RFC 9052](https://www.rfc-editor.org/rfc/rfc9052.html)) or CWT
+    ([RFC 8392](https://www.rfc-editor.org/rfc/rfc8392.html)), cannot be read with this library and need a
+    general-purpose CBOR library instead.
 
 !!! warning "UTF-8 validation of text strings"
 

docs/mkdocs/docs/features/binary_formats/messagepack.md

@@ -138,6 +138,21 @@ The library maps MessagePack types to JSON value types as follows:
 
     Any MessagePack output created by `to_msgpack` can be successfully parsed by `from_msgpack`.
 
+!!! warning "Object keys"
+
+    MessagePack allows map keys of any type, whereas JSON only allows strings as keys in object values. Like the
+    JSON-compatible [profile](https://github.com/msgpack/msgpack/blob/master/spec.md#profile) sketched in the
+    MessagePack specification, this library restricts map keys to `str` values. Maps with keys of any other type are
+    rejected with a [`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113) exception (or, with
+    `allow_exceptions` set to `false`, a discarded value) naming the type of the key that was found, for instance:
+
+    ```
+    [json.exception.parse_error.113] parse error at byte 2: syntax error while parsing MessagePack object key: only string keys are supported, but found nil; last byte: 0xC0
+    ```
+
+    This applies to the [SAX interface](../parsing/sax_interface.md) as well, as the key is read before it is passed
+    on. Such input needs a general-purpose MessagePack library instead.
+
 !!! warning "UTF-8 validation of string values"
 
     The MessagePack specification requires `str` values (`fixstr`, `str 8`, `str 16`, `str 32`) to be valid UTF-8.

docs/mkdocs/docs/features/serialization.md

@@ -64,7 +64,6 @@ serialization fails by default. The fourth argument of `dump` selects an
 - `strict` (default) — throw a [`type_error.316`](../home/exceptions.md#jsonexceptiontype_error316) exception.
 - `replace` — replace invalid bytes with the Unicode replacement character U+FFFD (`�`).
 - `ignore` — silently drop invalid bytes.
-- `keep` — copy invalid bytes to the output unchanged; the result is not valid UTF-8.
 
 ??? example
 

docs/mkdocs/docs/home/exceptions.md

@@ -331,9 +331,6 @@ An unexpected byte was read in a [binary format](../features/binary_formats/inde
     [json.exception.parse_error.112] parse error at byte 15: syntax error while parsing BSON binary: byte array length cannot be negative, is -1
     ```
     ```
-    [json.exception.parse_error.112] parse error at byte 9: syntax error while parsing CBOR value: negative integer overflow
-    ```
-    ```
     [json.exception.parse_error.112] parse error at byte 5: syntax error while parsing BSON document: document size 6 does not match the number of bytes read (5)
     ```
 
@@ -343,13 +340,20 @@ A string could not be read from a [binary format](../features/binary_formats/ind
 string was read where one was required (for instance as a map key), the string's length specification is invalid, or
 the string's bytes are not valid UTF-8.
 
+CBOR and MessagePack allow map keys of any type, but JSON object keys are always strings. Maps with keys of any other
+type (for instance integers or `null`) are therefore not supported; see the notes on
+[CBOR](../features/binary_formats/cbor.md) and [MessagePack](../features/binary_formats/messagepack.md).
+
 !!! failure "Example messages"
 
     ```
-    [json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR string: expected length specification (0x60-0x7B) or indefinite string type (0x7F); last byte: 0xFF
+    [json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR object key: only string keys are supported, but found an unsigned integer; last byte: 0x01
     ```
     ```
-    [json.exception.parse_error.113] parse error at byte 2: syntax error while parsing MessagePack string: expected length specification (0xA0-0xBF, 0xD9-0xDB); last byte: 0xFF
+    [json.exception.parse_error.113] parse error at byte 2: syntax error while parsing MessagePack object key: only string keys are supported, but found nil; last byte: 0xC0
+    ```
+    ```
+    [json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR string: expected length specification (0x60-0x7B) or indefinite string type (0x7F); last byte: 0x7C
     ```
     ```
     [json.exception.parse_error.113] parse error at byte 2: syntax error while parsing UBJSON char: byte after 'C' must be in range 0x00..0x7F; last byte: 0x82
@@ -755,7 +759,6 @@ The `dump()` function only works with UTF-8 encoded strings; that is, if you ass
     - Pass an error handler as last parameter to the `dump()` function to avoid this exception:
         - `json::error_handler_t::replace` will replace invalid bytes sequences with `U+FFFD` 
         - `json::error_handler_t::ignore` will silently ignore invalid byte sequences
-        - `json::error_handler_t::keep` will copy invalid byte sequences to the output unchanged
 
 ### json.exception.type_error.317
 
@@ -848,13 +851,18 @@ The JSON Patch operations 'remove' and 'add' cannot be applied to the root eleme
 
 ### json.exception.out_of_range.406
 
-A parsed number could not be stored as without changing it to NaN or INF.
+A parsed number could not be stored without changing it to NaN or INF. For the binary formats, this happens when a
+finite floating-point number does not fit into [`number_float_t`](../api/basic_json/number_float_t.md), for example a
+double-precision number when `number_float_t` is `#!cpp float`.
 
-!!! failure "Example message"
+!!! failure "Example messages"
 
     ```
     number overflow parsing '10E1000'
     ```
+    ```
+    [json.exception.out_of_range.406] syntax error while parsing CBOR value: number overflow
+    ```
 
 ### json.exception.out_of_range.407
 

docs/mkdocs/docs/home/faq.md

@@ -85,7 +85,7 @@ The library supports **Unicode input** as follows:
 - The library will not replace [Unicode noncharacters](http://www.unicode.org/faq/private_use.html#nonchar1).
 - Invalid surrogates (e.g., incomplete pairs such as `\uDEAD`) will yield parse errors.
 - The strings stored in the library are UTF-8 encoded. When using the default string type (`std::string`), note that its length/size functions return the number of stored bytes rather than the number of characters or glyphs.
-- When you store strings with different encodings in the library, calling [`dump()`](https://nlohmann.github.io/json/classnlohmann_1_1basic__json_a50ec80b02d0f3f51130d4abb5d1cfdc5.html#a50ec80b02d0f3f51130d4abb5d1cfdc5) may throw an exception unless `json::error_handler_t::replace`, `json::error_handler_t::ignore`, or `json::error_handler_t::keep` are used as error handlers.
+- When you store strings with different encodings in the library, calling [`dump()`](https://nlohmann.github.io/json/classnlohmann_1_1basic__json_a50ec80b02d0f3f51130d4abb5d1cfdc5.html#a50ec80b02d0f3f51130d4abb5d1cfdc5) may throw an exception unless `json::error_handler_t::replace` or `json::error_handler_t::ignore` are used as error handlers.
 
 In most cases, the parser is right to complain, because the input is not UTF-8 encoded. This is especially true for Microsoft Windows, where Latin-1 or ISO 8859-1 is often the standard encoding.
 

include/nlohmann/detail/input/binary_reader.hpp

@@ -559,7 +559,7 @@ class binary_reader
             case 0x01: // double
             {
                 double number{};
-                return get_number<double, true>(input_format_t::bson, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number<double, true>(input_format_t::bson, number) && emit_float(input_format_t::bson, number);
             }
 
             case 0x02: // string
@@ -600,19 +600,19 @@ class binary_reader
             case 0x10: // int32
             {
                 std::int32_t value{};
-                return get_number<std::int32_t, true>(input_format_t::bson, value) && sax->number_integer(value);
+                return get_number<std::int32_t, true>(input_format_t::bson, value) && emit_signed(input_format_t::bson, value);
             }
 
             case 0x12: // int64
             {
                 std::int64_t value{};
-                return get_number<std::int64_t, true>(input_format_t::bson, value) && sax->number_integer(value);
+                return get_number<std::int64_t, true>(input_format_t::bson, value) && emit_signed(input_format_t::bson, value);
             }
 
             case 0x11: // uint64
             {
                 std::uint64_t value{};
-                return get_number<std::uint64_t, true>(input_format_t::bson, value) && sax->number_unsigned(value);
+                return get_number<std::uint64_t, true>(input_format_t::bson, value) && emit_unsigned(input_format_t::bson, value);
             }
 
             default: // anything else is not supported (yet)
@@ -638,14 +638,19 @@ class binary_reader
         {
             return false;
         }
-        const auto max_val = static_cast<NumberType>((std::numeric_limits<number_integer_t>::max)());
+
-        if (number > max_val)
+        // the value is -1 - number, which fits into number_integer_t
+        // whenever number does
+        if (JSON_HEDLEY_LIKELY(value_in_range_of<number_integer_t>(number)))
         {
-            return sax->parse_error(chars_read, get_token_string(),
+            return sax->number_integer(static_cast<number_integer_t>(-1) - static_cast<number_integer_t>(number));
-                                    parse_error::create(112, chars_read,
-                                            exception_message(input_format_t::cbor, "negative integer overflow", "value"), nullptr));
         }
-        return sax->number_integer(static_cast<number_integer_t>(-1) - static_cast<number_integer_t>(number));
+
+        // like the lexer does for JSON text, store a value too small for
+        // number_integer_t as number_float_t; compute it as long double so
+        // that emit_float sees a finite value and can detect an overflow of
+        // number_float_t
+        return emit_float(input_format_t::cbor, static_cast<long double>(-1) - static_cast<long double>(number));
     }
 
     /*!
@@ -702,25 +707,25 @@ class binary_reader
             case 0x18: // Unsigned integer (one-byte uint8_t follows)
             {
                 std::uint8_t number{};
-                return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::cbor, number) && emit_unsigned(input_format_t::cbor, number);
             }
 
             case 0x19: // Unsigned integer (two-byte uint16_t follows)
             {
                 std::uint16_t number{};
-                return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::cbor, number) && emit_unsigned(input_format_t::cbor, number);
             }
 
             case 0x1A: // Unsigned integer (four-byte uint32_t follows)
             {
                 std::uint32_t number{};
-                return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::cbor, number) && emit_unsigned(input_format_t::cbor, number);
             }
 
             case 0x1B: // Unsigned integer (eight-byte uint64_t follows)
             {
                 std::uint64_t number{};
-                return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::cbor, number) && emit_unsigned(input_format_t::cbor, number);
             }
 
             // Negative integer -1-0x00..-1-0x17 (-1..-24)
@@ -1165,13 +1170,13 @@ class binary_reader
             case 0xFA: // Single-Precision Float (four-byte IEEE 754)
             {
                 float number{};
-                return get_number(input_format_t::cbor, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format_t::cbor, number) && emit_float(input_format_t::cbor, number);
             }
 
             case 0xFB: // Double-Precision Float (eight-byte IEEE 754)
             {
                 double number{};
-                return get_number(input_format_t::cbor, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format_t::cbor, number) && emit_float(input_format_t::cbor, number);
             }
 
             default: // anything else (0xFF is handled inside the other types)
@@ -1324,6 +1329,80 @@ class binary_reader
         }
     }
 
+    /*!
+    @brief reads a CBOR object key
+
+    RFC 8949 allows any data item as a map key, but only strings have a
+    counterpart in JSON. A key of any other type is rejected with a message
+    naming that type, rather than the one @ref get_cbor_string gives for a
+    malformed string.
+
+    @param[out] result  created key
+
+    @return whether key creation completed
+    */
+    bool get_cbor_object_key(string_t& result)
+    {
+        // EOF and major type 3 (text string) are left to get_cbor_string
+        if (current == char_traits<char_type>::eof() || (static_cast<unsigned int>(current) & 0xE0u) == 0x60u)
+        {
+            return get_cbor_string(result);
+        }
+
+        const char* found = nullptr;
+        switch (static_cast<unsigned int>(current) >> 5u)
+        {
+            case 0:
+                found = "an unsigned integer";
+                break;
+            case 1:
+                found = "a negative integer";
+                break;
+            case 2:
+                found = "a byte string";
+                break;
+            case 4:
+                found = "an array";
+                break;
+            case 5:
+                found = "a map";
+                break;
+            case 6:
+                found = "a tag";
+                break;
+            default: // major type 7
+                switch (current)
+                {
+                    case 0xF4:
+                    case 0xF5:
+                        found = "a boolean";
+                        break;
+                    case 0xF6:
+                        found = "null";
+                        break;
+                    case 0xF7:
+                        found = "undefined";
+                        break;
+                    case 0xF9:
+                    case 0xFA:
+                    case 0xFB:
+                        found = "a floating-point number";
+                        break;
+                    case 0xFF:
+                        found = "a break stop code";
+                        break;
+                    default:
+                        found = "a simple value";
+                        break;
+                }
+                break;
+        }
+
+        auto last_token = get_token_string();
+        return sax->parse_error(chars_read, last_token, parse_error::create(113, chars_read,
+                                exception_message(input_format_t::cbor, concat("only string keys are supported, but found ", found, "; last byte: 0x", last_token), "object key"), nullptr));
+    }
+
     /*!
     @brief reads a definite-length CBOR byte array
 
@@ -1568,7 +1647,7 @@ class binary_reader
                 if (top.is_object)
                 {
                     key.clear();
-                    if (JSON_HEDLEY_UNLIKELY(!get_cbor_string(key) || !sax->key(key)))
+                    if (JSON_HEDLEY_UNLIKELY(!get_cbor_object_key(key) || !sax->key(key)))
                     {
                         return false;
                     }
@@ -1861,61 +1940,61 @@ class binary_reader
             case 0xCA: // float 32
             {
                 float number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format_t::msgpack, number) && emit_float(input_format_t::msgpack, number);
             }
 
             case 0xCB: // float 64
             {
                 double number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format_t::msgpack, number) && emit_float(input_format_t::msgpack, number);
             }
 
             case 0xCC: // uint 8
             {
                 std::uint8_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::msgpack, number) && emit_unsigned(input_format_t::msgpack, number);
             }
 
             case 0xCD: // uint 16
             {
                 std::uint16_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::msgpack, number) && emit_unsigned(input_format_t::msgpack, number);
             }
 
             case 0xCE: // uint 32
             {
                 std::uint32_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::msgpack, number) && emit_unsigned(input_format_t::msgpack, number);
             }
 
             case 0xCF: // uint 64
             {
                 std::uint64_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::msgpack, number) && emit_unsigned(input_format_t::msgpack, number);
             }
 
             case 0xD0: // int 8
             {
                 std::int8_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
+                return get_number(input_format_t::msgpack, number) && emit_signed(input_format_t::msgpack, number);
             }
 
             case 0xD1: // int 16
             {
                 std::int16_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
+                return get_number(input_format_t::msgpack, number) && emit_signed(input_format_t::msgpack, number);
             }
 
             case 0xD2: // int 32
             {
                 std::int32_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
+                return get_number(input_format_t::msgpack, number) && emit_signed(input_format_t::msgpack, number);
             }
 
             case 0xD3: // int 64
             {
                 std::int64_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
+                return get_number(input_format_t::msgpack, number) && emit_signed(input_format_t::msgpack, number);
             }
 
             case 0xDC: // array 16
@@ -2069,6 +2148,98 @@ class binary_reader
         }
     }
 
+    /*!
+    @brief reads a MessagePack object key
+
+    The MessagePack specification allows any type as a map key, but only
+    strings have a counterpart in JSON. A key of any other type is rejected
+    with a message naming that type, rather than the one @ref
+    get_msgpack_string gives for a malformed string.
+
+    @param[out] result  created key
+
+    @return whether key creation completed
+    */
+    bool get_msgpack_object_key(string_t& result)
+    {
+        const char* found = nullptr;
+        switch (current)
+        {
+            case 0xC0:
+                found = "nil";
+                break;
+            case 0xC2:
+            case 0xC3:
+                found = "a boolean";
+                break;
+            case 0xCA:
+            case 0xCB:
+                found = "a float";
+                break;
+            case 0xC4:
+            case 0xC5:
+            case 0xC6:
+                found = "a bin";
+                break;
+            case 0xC7:
+            case 0xC8:
+            case 0xC9:
+            case 0xD4:
+            case 0xD5:
+            case 0xD6:
+            case 0xD7:
+            case 0xD8:
+                found = "an ext";
+                break;
+            case 0xCC:
+            case 0xCD:
+            case 0xCE:
+            case 0xCF:
+            case 0xD0:
+            case 0xD1:
+            case 0xD2:
+            case 0xD3:
+                found = "an integer";
+                break;
+            case 0xDC:
+            case 0xDD:
+                found = "an array";
+                break;
+            case 0xDE:
+            case 0xDF:
+                found = "a map";
+                break;
+            default:
+                // fixint, fixmap, and fixarray; strings, EOF, and the unused
+                // byte 0xC1 are left to get_msgpack_string
+                if (current == char_traits<char_type>::eof())
+                {
+                    return get_msgpack_string(result);
+                }
+                if (current <= 0x7F || current >= 0xE0)
+                {
+                    found = "an integer";
+                }
+                else if (current <= 0x8F)
+                {
+                    found = "a map";
+                }
+                else if (current <= 0x9F)
+                {
+                    found = "an array";
+                }
+                else
+                {
+                    return get_msgpack_string(result);
+                }
+                break;
+        }
+
+        auto last_token = get_token_string();
+        return sax->parse_error(chars_read, last_token, parse_error::create(113, chars_read,
+                                exception_message(input_format_t::msgpack, concat("only string keys are supported, but found ", found, "; last byte: 0x", last_token), "object key"), nullptr));
+    }
+
     /*!
     @brief reads a MessagePack byte array
 
@@ -2231,7 +2402,7 @@ class binary_reader
                 {
                     get();
                     key.clear();
-                    if (JSON_HEDLEY_UNLIKELY(!get_msgpack_string(key) || !sax->key(key)))
+                    if (JSON_HEDLEY_UNLIKELY(!get_msgpack_object_key(key) || !sax->key(key)))
                     {
                         return false;
                     }
@@ -2756,7 +2927,7 @@ class binary_reader
                         {
                             return sax->parse_error(chars_read, get_token_string(), out_of_range::create(408, exception_message(input_format, "excessive ndarray size caused overflow", "size"), nullptr));
                         }
-                        if (JSON_HEDLEY_UNLIKELY(!sax->number_unsigned(static_cast<number_unsigned_t>(i))))
+                        if (JSON_HEDLEY_UNLIKELY(!emit_unsigned(input_format, i)))
                         {
                             return false;
                         }
@@ -2888,37 +3059,37 @@ class binary_reader
                     break;
                 }
                 std::uint8_t number{};
-                return get_number(input_format, number) && sax->number_unsigned(number);
+                return get_number(input_format, number) && emit_unsigned(input_format, number);
             }
 
             case 'U':
             {
                 std::uint8_t number{};
-                return get_number(input_format, number) && sax->number_unsigned(number);
+                return get_number(input_format, number) && emit_unsigned(input_format, number);
             }
 
             case 'i':
             {
                 std::int8_t number{};
-                return get_number(input_format, number) && sax->number_integer(number);
+                return get_number(input_format, number) && emit_signed(input_format, number);
             }
 
             case 'I':
             {
                 std::int16_t number{};
-                return get_number(input_format, number) && sax->number_integer(number);
+                return get_number(input_format, number) && emit_signed(input_format, number);
             }
 
             case 'l':
             {
                 std::int32_t number{};
-                return get_number(input_format, number) && sax->number_integer(number);
+                return get_number(input_format, number) && emit_signed(input_format, number);
             }
 
             case 'L':
             {
                 std::int64_t number{};
-                return get_number(input_format, number) && sax->number_integer(number);
+                return get_number(input_format, number) && emit_signed(input_format, number);
             }
 
             case 'u':
@@ -2928,7 +3099,7 @@ class binary_reader
                     break;
                 }
                 std::uint16_t number{};
-                return get_number(input_format, number) && sax->number_unsigned(number);
+                return get_number(input_format, number) && emit_unsigned(input_format, number);
             }
 
             case 'm':
@@ -2938,7 +3109,7 @@ class binary_reader
                     break;
                 }
                 std::uint32_t number{};
-                return get_number(input_format, number) && sax->number_unsigned(number);
+                return get_number(input_format, number) && emit_unsigned(input_format, number);
             }
 
             case 'M':
@@ -2948,7 +3119,7 @@ class binary_reader
                     break;
                 }
                 std::uint64_t number{};
-                return get_number(input_format, number) && sax->number_unsigned(number);
+                return get_number(input_format, number) && emit_unsigned(input_format, number);
             }
 
             case 'h':
@@ -3006,13 +3177,13 @@ class binary_reader
             case 'd':
             {
                 float number{};
-                return get_number(input_format, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format, number) && emit_float(input_format, number);
             }
 
             case 'D':
             {
                 double number{};
-                return get_number(input_format, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format, number) && emit_float(input_format, number);
             }
 
             case 'H':
@@ -3479,13 +3650,13 @@ class binary_reader
             case 0x8E: // binary32
             {
                 float number{};
-                return get_number(input_format_t::bon8, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format_t::bon8, number) && emit_float(input_format_t::bon8, number);
             }
 
             case 0x8F: // binary64
             {
                 double number{};
-                return get_number(input_format_t::bon8, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format_t::bon8, number) && emit_float(input_format_t::bon8, number);
             }
 
             case 0xF8:
@@ -3551,7 +3722,9 @@ class binary_reader
     @brief pass an integer to the SAX parser
 
     Non-negative integers are passed as unsigned, negative integers as signed
-    numbers, like the other binary formats do.
+    numbers, like the other binary formats do. A value that does not fit the
+    number type is passed as described for @ref emit_unsigned and
+    @ref emit_signed.
 
     @param[in] number  the integer
     @return whether the SAX parser accepted the value
@@ -3560,9 +3733,9 @@ class binary_reader
     {
         if (number >= 0)
         {
-            return sax->number_unsigned(static_cast<number_unsigned_t>(number));
+            return emit_unsigned(input_format_t::bon8, static_cast<std::uint64_t>(number));
         }
-        return sax->number_integer(static_cast<number_integer_t>(number));
+        return emit_signed(input_format_t::bon8, number);
     }
 
     /*!
@@ -3619,8 +3792,7 @@ class binary_reader
             value = (value << 8) | static_cast<std::int64_t>(current);
         }
 
-        return negative ? sax->number_integer(static_cast<number_integer_t>(-(value + offset)))
+        return emit_bon8_integer(negative ? -(value + offset) : value + offset);
-               : sax->number_unsigned(static_cast<number_unsigned_t>(value + offset));
     }
 
     /*!
@@ -3917,6 +4089,88 @@ class binary_reader
         return true;
     }
 
+    /*!
+    @brief pass a signed integer read from the input to the SAX parser
+
+    Like the lexer does for JSON text, a value that does not fit into
+    number_integer_t is passed as number_unsigned_t if it is non-negative and
+    fits there, and as number_float_t otherwise. With the default number
+    types, every integer the binary formats can encode fits, so this only
+    matters for narrower custom number types.
+
+    @tparam NumberType a signed integer type
+    @param[in] format  the current format (for diagnostics)
+    @param[in] number  the integer
+    @return whether the SAX parser accepted the value
+
+    @throw out_of_range.406 if @a number overflows number_float_t (see
+           @ref emit_float)
+    */
+    template<typename NumberType>
+    bool emit_signed(const input_format_t format, const NumberType number)
+    {
+        if (JSON_HEDLEY_LIKELY(value_in_range_of<number_integer_t>(number)))
+        {
+            return sax->number_integer(static_cast<number_integer_t>(number));
+        }
+        if (value_in_range_of<number_unsigned_t>(number))
+        {
+            return sax->number_unsigned(static_cast<number_unsigned_t>(number));
+        }
+        return emit_float(format, number);
+    }
+
+    /*!
+    @brief pass an unsigned integer read from the input to the SAX parser
+
+    Like the lexer does for JSON text, a value that does not fit into
+    number_unsigned_t is passed as number_float_t.
+
+    @tparam NumberType an unsigned integer type
+    @param[in] format  the current format (for diagnostics)
+    @param[in] number  the integer
+    @return whether the SAX parser accepted the value
+
+    @throw out_of_range.406 if @a number overflows number_float_t (see
+           @ref emit_float)
+    */
+    template<typename NumberType>
+    bool emit_unsigned(const input_format_t format, const NumberType number)
+    {
+        if (JSON_HEDLEY_LIKELY(value_in_range_of<number_unsigned_t>(number)))
+        {
+            return sax->number_unsigned(static_cast<number_unsigned_t>(number));
+        }
+        return emit_float(format, number);
+    }
+
+    /*!
+    @brief pass a floating-point number read from the input to the SAX parser
+
+    Like the lexer does for JSON text, a finite value that overflows
+    number_float_t is rejected instead of silently becoming infinity. Infinity
+    and NaN in the input are passed on unchanged. Integers only overflow if
+    number_float_t cannot represent 2^64, e.g., a half-precision type.
+
+    @tparam NumberType a floating-point or integer type
+    @param[in] format  the current format (for diagnostics)
+    @param[in] number  the number
+    @return whether the SAX parser accepted the value
+
+    @throw out_of_range.406 if a finite @a number overflows number_float_t
+    */
+    template<typename NumberType>
+    bool emit_float(const input_format_t format, const NumberType number)
+    {
+        const auto result = static_cast<number_float_t>(number);
+        if (JSON_HEDLEY_UNLIKELY(std::isfinite(number) && !std::isfinite(result)))
+        {
+            return sax->parse_error(chars_read, get_token_string(),
+                                    out_of_range::create(406, exception_message(format, "number overflow", "value"), nullptr));
+        }
+        return sax->number_float(result, "");
+    }
+
     /*!
     @brief create a string by reading characters from the input
 

include/nlohmann/detail/input/lexer.hpp

@@ -206,7 +206,6 @@ class lexer : public lexer_base<BasicJsonType>
     explicit lexer(InputAdapterType&& adapter, bool ignore_comments_ = false, bool discard_number_values_ = false) noexcept
         : ia(std::move(adapter))
         , ignore_comments(ignore_comments_)
-        , decimal_point_char(static_cast<char_int_type>(get_decimal_point()))
         , discard_number_values(discard_number_values_)
     {}
 
@@ -222,8 +221,7 @@ class lexer : public lexer_base<BasicJsonType>
     // locales
     /////////////////////
 
-    /// return the locale-dependent decimal point
+    /// return the decimal point of the current locale
-    JSON_HEDLEY_PURE
     static char get_decimal_point() noexcept
     {
         const auto* loc = localeconv();
@@ -1092,9 +1090,10 @@ class lexer : public lexer_base<BasicJsonType>
             token_type::value_float if number could be successfully scanned,
             token_type::parse_error otherwise
 
-    @note The scanner is independent of the current locale. Internally, the
+    @note The scanner is independent of the current locale: token_buffer
-          locale's decimal point is used instead of `.` to work with the
+          always holds `.`. Only the std::strtod fallback of convert_number()
-          locale-dependent converters.
+          depends on the locale, and it looks up the decimal point right
+          before converting (see convert_float_locale_aware()).
     */
     token_type scan_number()  // lgtm [cpp/use-of-goto] `goto` is used in this function to implement the number-parsing state machine described above. By design, any finite input will eventually reach the "done" state or return token_type::parse_error. In each intermediate state, 1 byte of the input is appended to the token_buffer vector, and only the already initialized variables token_buffer, number_type, and error_message are manipulated.
     {
@@ -1183,7 +1182,7 @@ scan_number_zero:
         {
             case '.':
             {
-                add(decimal_point_char);
+                add(current);
                 decimal_point_position = token_buffer.size() - 1;
                 goto scan_number_decimal1;
             }
@@ -1220,7 +1219,7 @@ scan_number_any1:
 
             case '.':
             {
-                add(decimal_point_char);
+                add(current);
                 decimal_point_position = token_buffer.size() - 1;
                 goto scan_number_decimal1;
             }
@@ -1462,9 +1461,9 @@ scan_number_done:
 
         // Only a number below 1 can carry further insignificant zeros, and only
         // while the count stays at the limit does removing them change the
-        // answer - so this loop is skipped for all but a few tokens. Note
+        // answer - so this loop is skipped for all but a few tokens. The
-        // token_buffer holds the locale's decimal point, so the fraction is
+        // fraction is located through decimal_point_position rather than by
-        // located through decimal_point_position rather than by searching '.'.
+        // searching '.'.
         if (lead_zero != 0)
         {
             JSON_ASSERT(has_dot != 0); // an integer "0" cannot reach the limit
@@ -1482,8 +1481,8 @@ scan_number_done:
     @brief convert the number text in token_buffer to its value and token type
 
     The digit sequence in token_buffer has already been validated (by the
-    scan_number() state machine or by the contiguous fast path) and holds the
+    scan_number() state machine or by the contiguous fast path) and holds '.'
-    locale decimal point in place of '.'. Integers are parsed first and fall
+    as decimal point, independent of the locale. Integers are parsed first and fall
     back to floating point on overflow. This is shared so both scanners produce
     identical results.
 
@@ -1563,7 +1562,7 @@ scan_number_done:
         // integer conversion above overflowed. Prefer std::from_chars
         // (Eisel-Lemire, locale-independent, correctly rounded) when available;
         // otherwise the exact Clinger fast path (double only); otherwise the
-        // locale-aware strtof/strtod.
+        // locale-aware strtof/strtod/strtold.
         if (parse_float_from_chars(num_begin, num_end, value_float))
         {
             return token_type::value_float;
@@ -1572,26 +1571,75 @@ scan_number_done:
         // extra pass over the token's bytes, which otherwise shows up on
         // high-precision inputs such as canada.json
         if (mantissa_fits_clinger(mantissa_end)
-                && parse_float_fast(num_begin, num_end, decimal_point_char, value_float))
+                && parse_float_fast(num_begin, num_end, value_float))
         {
             return token_type::value_float;
         }
 
-        char* endptr = nullptr; // NOLINT(misc-const-correctness,cppcoreguidelines-pro-type-vararg,hicpp-vararg)
+        convert_float_locale_aware();
-        strtof(value_float, token_buffer.data(), &endptr);
-
-        // we checked the number format before
-        JSON_ASSERT(endptr == token_buffer.data() + token_buffer.size());
-
         return token_type::value_float;
     }
 
+    /*!
+    @brief convert the float in token_buffer with strtof/strtod/strtold
+
+    These functions expect the decimal point of the *current* locale, so it is
+    looked up right before the conversion instead of once when the lexer is
+    constructed: a locale change in between (by a parser callback, a SAX
+    handler, or another thread) must not truncate the value (#5198). The
+    token has been validated before, so if the conversion stops early and the
+    decimal point changed in the meantime, the locale changed between the
+    lookup and the call, and the conversion is repeated with the new decimal
+    point. If the decimal point did not change, a retry cannot succeed: the
+    locale's decimal point is not a single character (e.g., the two-byte
+    U+066B of ar_EG.UTF-8 or fa_IR.UTF-8) and cannot be substituted in place.
+    The value strtod parsed up to that point is kept, as before this change.
+
+    Note that changing the locale in another thread *while* strtod runs is
+    undefined behavior of the C library, which this function cannot prevent.
+    */
+    void convert_float_locale_aware()
+    {
+        const bool has_dot = decimal_point_position != std::string::npos;
+        char decimal_point = get_decimal_point();
+        for (;;)
+        {
+            const bool substitute = has_dot && decimal_point != '.';
+            if (substitute)
+            {
+                token_buffer[decimal_point_position] = static_cast<typename string_t::value_type>(decimal_point);
+            }
+
+            char* endptr = nullptr; // NOLINT(misc-const-correctness,cppcoreguidelines-pro-type-vararg,hicpp-vararg)
+            strtof(value_float, token_buffer.data(), &endptr);
+
+            if (substitute)
+            {
+                // get_string() hands the token to the SAX interface with '.'
+                token_buffer[decimal_point_position] = '.';
+            }
+
+            if (JSON_HEDLEY_LIKELY(endptr == token_buffer.data() + token_buffer.size()))
+            {
+                return;
+            }
+
+            // retry only if the locale changed; otherwise, this would loop forever
+            const char current_decimal_point = get_decimal_point();
+            if (current_decimal_point == decimal_point)
+            {
+                return;
+            }
+            decimal_point = current_decimal_point;
+        }
+    }
+
     /*!
     @brief contiguous fast path for scanning a number
 
     Parses the whole number token straight from the input buffer, avoiding the
     per-character get()/add() of scan_number(). On success it fills token_buffer
-    (with the locale decimal point substituted, as scan_number() does) and
+    (as scan_number() does) and
     returns the token type. On anything it does not fully recognize as a
     well-formed number it makes no state change and returns
     token_type::uninitialized, so the caller falls back to scan_number(), which
@@ -1707,16 +1755,11 @@ scan_number_done:
         }
 #endif
 
-        // materialize the token exactly as scan_number() would, substituting the
+        // materialize the token exactly as scan_number() would. reset() already
-        // locale decimal point so convert_number()'s strtof fallback stays valid.
+        // cleared token_buffer, so append() fills it (assign() is avoided
-        // reset() already cleared token_buffer, so append() fills it (assign() is
+        // because custom string_t types need not provide it)
-        // avoided because custom string_t types need not provide it)
         token_buffer.append(reinterpret_cast<const typename string_t::value_type*>(data), len);
-        if (dot_index != std::string::npos)
+        decimal_point_position = dot_index;
-        {
-            token_buffer[dot_index] = static_cast<typename string_t::value_type>(decimal_point_char);
-            decimal_point_position = dot_index;
-        }
 
         ia.bulk_skip(len - 1);
         position.chars_read_total += (len - 1);
@@ -1983,11 +2026,7 @@ scan_number_done:
     /// return current string value (implicitly resets the token; useful only once)
     string_t& get_string()
     {
-        // translate decimal points from locale back to '.' (#4084)
+        // a number token holds '.' regardless of the locale (#4084)
-        if (decimal_point_char != '.' && decimal_point_position != std::string::npos)
-        {
-            token_buffer[decimal_point_position] = '.';
-        }
         return token_buffer;
     }
 
@@ -2283,9 +2322,7 @@ scan_number_done:
     number_unsigned_t value_unsigned = 0;
     number_float_t value_float = 0;
 
-    /// the decimal point
+    /// the position of the decimal point in token_buffer
-    const char_int_type decimal_point_char = '.';
-    /// the position of the decimal point in the input
     std::size_t decimal_point_position = std::string::npos;
 
     /// whether the caller (e.g. accept()/json_sax_acceptor) only needs the

include/nlohmann/detail/input/number_parse.hpp

@@ -118,14 +118,12 @@ std::strtod. The parser only activates for number_float_t == double; float and
 long double keep the std::strtof/std::strtold paths (see the templated overload
 below).
 
-@param[in]  first          pointer to the first character of the number
+@param[in]  first  pointer to the first character of the number
-@param[in]  last           pointer past the last character
+@param[in]  last   pointer past the last character
-@param[in]  decimal_point  the (locale-dependent) decimal point character
+@param[out] out    the parsed value on success
-@param[out] out            the parsed value on success
 @return true if the value was parsed exactly; false to fall back to strtod
 */
-template<typename DecimalPointType>
+inline bool parse_float_fast(const char* first, const char* last, double& out) noexcept
-bool parse_float_fast(const char* first, const char* last, DecimalPointType decimal_point, double& out) noexcept
 {
 #if defined(FLT_EVAL_METHOD) && FLT_EVAL_METHOD != 0
     // Clinger's fast path is only exact when double operations are evaluated in
@@ -136,7 +134,6 @@ bool parse_float_fast(const char* first, const char* last, DecimalPointType deci
     // std::from_chars / std::strtod path.
     static_cast<void>(first);
     static_cast<void>(last);
-    static_cast<void>(decimal_point);
     static_cast<void>(out);
     return false;
 #else
@@ -175,7 +172,7 @@ bool parse_float_fast(const char* first, const char* last, DecimalPointType deci
             ++num_digits;
             fractional_digits += static_cast<int>(seen_dot);
         }
-        else if (static_cast<DecimalPointType>(c) == decimal_point)
+        else if (c == '.')
         {
             if (JSON_HEDLEY_UNLIKELY(seen_dot))
             {
@@ -260,8 +257,8 @@ bool parse_float_fast(const char* first, const char* last, DecimalPointType deci
 }
 
 /// fast float path is only exact for `double`; decline for float/long double
-template<typename DecimalPointType, typename FloatType>
+template<typename FloatType>
-bool parse_float_fast(const char* /*first*/, const char* /*last*/, DecimalPointType /*decimal_point*/, FloatType& /*out*/) noexcept
+bool parse_float_fast(const char* /*first*/, const char* /*last*/, FloatType& /*out*/) noexcept
 {
     return false;
 }
@@ -273,9 +270,7 @@ std::from_chars is locale-independent, correctly rounded, and - via the
 Eisel-Lemire algorithm in modern standard libraries - much faster than strtod
 over the whole value range (not just the Clinger subset). It is used only when
 __cpp_lib_to_chars indicates full floating-point support and only when it
-consumes the entire token ([first, last)); a partial parse means the buffer
+consumes the entire token ([first, last)). An under-/overflow (result_out_of_range) also declines, so
-uses a non-'.' locale decimal point, in which case the caller falls back to the
-locale-aware path. An under-/overflow (result_out_of_range) also declines, so
 the caller's strtod fallback supplies the well-defined ±inf/0 result the parser
 expects (side-stepping the P4168 divergence between implementations).
 

include/nlohmann/detail/output/serializer.hpp

@@ -48,8 +48,7 @@ enum class error_handler_t
 {
     strict,  ///< throw a type_error exception in case of invalid UTF-8
     replace, ///< replace invalid UTF-8 sequences with U+FFFD
-    ignore,  ///< ignore invalid UTF-8 sequences
+    ignore   ///< ignore invalid UTF-8 sequences
-    keep     ///< keep invalid UTF-8 sequences; their bytes are copied unchanged
 };
 
 template<typename BasicJsonType>
@@ -1020,47 +1019,6 @@ class serializer
                             break;
                         }
 
-                        case error_handler_t::keep:
-                        {
-                            // drop whatever the incomplete sequence left in
-                            // the buffer (only copied if !EnsureAscii) and copy
-                            // the ill-formed bytes from the input instead
-                            bytes = bytes_after_last_accept;
-
-                            if (undumped_chars > 0)
-                            {
-                                // the pending bytes of the incomplete sequence
-                                // are ill-formed; the current byte may be OK for
-                                // itself, so we would like to read it again
-                                for (std::size_t j = i - undumped_chars; j < i; ++j)
-                                {
-                                    string_buffer[bytes++] = s[j];
-                                }
-                                --i;
-                            }
-                            else
-                            {
-                                // the current byte cannot start any sequence
-                                string_buffer[bytes++] = s[i];
-                            }
-
-                            // write buffer and reset index; there must be 13 bytes
-                            // left, as this is the maximal number of bytes to be
-                            // written ("\uxxxx\uxxxx\0") for one code point
-                            if (string_buffer.size() - bytes < 13)
-                            {
-                                put_buffer(string_buffer, bytes);
-                                bytes = 0;
-                            }
-
-                            bytes_after_last_accept = bytes;
-                            undumped_chars = 0;
-
-                            // continue processing the string
-                            state = UTF8_ACCEPT;
-                            break;
-                        }
-
                         default:            // LCOV_EXCL_LINE
                             JSON_ASSERT(false); // NOLINT(cert-dcl03-c,hicpp-static-assert,misc-static-assert) LCOV_EXCL_LINE
                     }
@@ -1106,15 +1064,6 @@ class serializer
                     break;
                 }
 
-                case error_handler_t::keep:
-                {
-                    // write all accepted bytes
-                    put_buffer(string_buffer, bytes_after_last_accept);
-                    // copy the bytes of the incomplete sequence unchanged
-                    put_string(s, s.size() - undumped_chars, s.size());
-                    break;
-                }
-
                 case error_handler_t::replace:
                 {
                     // write all accepted bytes

single_include/nlohmann/json.hpp

@@ -8605,14 +8605,12 @@ std::strtod. The parser only activates for number_float_t == double; float and
 long double keep the std::strtof/std::strtold paths (see the templated overload
 below).
 
-@param[in]  first          pointer to the first character of the number
+@param[in]  first  pointer to the first character of the number
-@param[in]  last           pointer past the last character
+@param[in]  last   pointer past the last character
-@param[in]  decimal_point  the (locale-dependent) decimal point character
+@param[out] out    the parsed value on success
-@param[out] out            the parsed value on success
 @return true if the value was parsed exactly; false to fall back to strtod
 */
-template<typename DecimalPointType>
+inline bool parse_float_fast(const char* first, const char* last, double& out) noexcept
-bool parse_float_fast(const char* first, const char* last, DecimalPointType decimal_point, double& out) noexcept
 {
 #if defined(FLT_EVAL_METHOD) && FLT_EVAL_METHOD != 0
     // Clinger's fast path is only exact when double operations are evaluated in
@@ -8623,7 +8621,6 @@ bool parse_float_fast(const char* first, const char* last, DecimalPointType deci
     // std::from_chars / std::strtod path.
     static_cast<void>(first);
     static_cast<void>(last);
-    static_cast<void>(decimal_point);
     static_cast<void>(out);
     return false;
 #else
@@ -8662,7 +8659,7 @@ bool parse_float_fast(const char* first, const char* last, DecimalPointType deci
             ++num_digits;
             fractional_digits += static_cast<int>(seen_dot);
         }
-        else if (static_cast<DecimalPointType>(c) == decimal_point)
+        else if (c == '.')
         {
             if (JSON_HEDLEY_UNLIKELY(seen_dot))
             {
@@ -8747,8 +8744,8 @@ bool parse_float_fast(const char* first, const char* last, DecimalPointType deci
 }
 
 /// fast float path is only exact for `double`; decline for float/long double
-template<typename DecimalPointType, typename FloatType>
+template<typename FloatType>
-bool parse_float_fast(const char* /*first*/, const char* /*last*/, DecimalPointType /*decimal_point*/, FloatType& /*out*/) noexcept
+bool parse_float_fast(const char* /*first*/, const char* /*last*/, FloatType& /*out*/) noexcept
 {
     return false;
 }
@@ -8760,9 +8757,7 @@ std::from_chars is locale-independent, correctly rounded, and - via the
 Eisel-Lemire algorithm in modern standard libraries - much faster than strtod
 over the whole value range (not just the Clinger subset). It is used only when
 __cpp_lib_to_chars indicates full floating-point support and only when it
-consumes the entire token ([first, last)); a partial parse means the buffer
+consumes the entire token ([first, last)). An under-/overflow (result_out_of_range) also declines, so
-uses a non-'.' locale decimal point, in which case the caller falls back to the
-locale-aware path. An under-/overflow (result_out_of_range) also declines, so
 the caller's strtod fallback supplies the well-defined ±inf/0 result the parser
 expects (side-stepping the P4168 divergence between implementations).
 
@@ -9303,7 +9298,6 @@ class lexer : public lexer_base<BasicJsonType>
     explicit lexer(InputAdapterType&& adapter, bool ignore_comments_ = false, bool discard_number_values_ = false) noexcept
         : ia(std::move(adapter))
         , ignore_comments(ignore_comments_)
-        , decimal_point_char(static_cast<char_int_type>(get_decimal_point()))
         , discard_number_values(discard_number_values_)
     {}
 
@@ -9319,8 +9313,7 @@ class lexer : public lexer_base<BasicJsonType>
     // locales
     /////////////////////
 
-    /// return the locale-dependent decimal point
+    /// return the decimal point of the current locale
-    JSON_HEDLEY_PURE
     static char get_decimal_point() noexcept
     {
         const auto* loc = localeconv();
@@ -10189,9 +10182,10 @@ class lexer : public lexer_base<BasicJsonType>
             token_type::value_float if number could be successfully scanned,
             token_type::parse_error otherwise
 
-    @note The scanner is independent of the current locale. Internally, the
+    @note The scanner is independent of the current locale: token_buffer
-          locale's decimal point is used instead of `.` to work with the
+          always holds `.`. Only the std::strtod fallback of convert_number()
-          locale-dependent converters.
+          depends on the locale, and it looks up the decimal point right
+          before converting (see convert_float_locale_aware()).
     */
     token_type scan_number()  // lgtm [cpp/use-of-goto] `goto` is used in this function to implement the number-parsing state machine described above. By design, any finite input will eventually reach the "done" state or return token_type::parse_error. In each intermediate state, 1 byte of the input is appended to the token_buffer vector, and only the already initialized variables token_buffer, number_type, and error_message are manipulated.
     {
@@ -10280,7 +10274,7 @@ scan_number_zero:
         {
             case '.':
             {
-                add(decimal_point_char);
+                add(current);
                 decimal_point_position = token_buffer.size() - 1;
                 goto scan_number_decimal1;
             }
@@ -10317,7 +10311,7 @@ scan_number_any1:
 
             case '.':
             {
-                add(decimal_point_char);
+                add(current);
                 decimal_point_position = token_buffer.size() - 1;
                 goto scan_number_decimal1;
             }
@@ -10559,9 +10553,9 @@ scan_number_done:
 
         // Only a number below 1 can carry further insignificant zeros, and only
         // while the count stays at the limit does removing them change the
-        // answer - so this loop is skipped for all but a few tokens. Note
+        // answer - so this loop is skipped for all but a few tokens. The
-        // token_buffer holds the locale's decimal point, so the fraction is
+        // fraction is located through decimal_point_position rather than by
-        // located through decimal_point_position rather than by searching '.'.
+        // searching '.'.
         if (lead_zero != 0)
         {
             JSON_ASSERT(has_dot != 0); // an integer "0" cannot reach the limit
@@ -10579,8 +10573,8 @@ scan_number_done:
     @brief convert the number text in token_buffer to its value and token type
 
     The digit sequence in token_buffer has already been validated (by the
-    scan_number() state machine or by the contiguous fast path) and holds the
+    scan_number() state machine or by the contiguous fast path) and holds '.'
-    locale decimal point in place of '.'. Integers are parsed first and fall
+    as decimal point, independent of the locale. Integers are parsed first and fall
     back to floating point on overflow. This is shared so both scanners produce
     identical results.
 
@@ -10660,7 +10654,7 @@ scan_number_done:
         // integer conversion above overflowed. Prefer std::from_chars
         // (Eisel-Lemire, locale-independent, correctly rounded) when available;
         // otherwise the exact Clinger fast path (double only); otherwise the
-        // locale-aware strtof/strtod.
+        // locale-aware strtof/strtod/strtold.
         if (parse_float_from_chars(num_begin, num_end, value_float))
         {
             return token_type::value_float;
@@ -10669,26 +10663,75 @@ scan_number_done:
         // extra pass over the token's bytes, which otherwise shows up on
         // high-precision inputs such as canada.json
         if (mantissa_fits_clinger(mantissa_end)
-                && parse_float_fast(num_begin, num_end, decimal_point_char, value_float))
+                && parse_float_fast(num_begin, num_end, value_float))
         {
             return token_type::value_float;
         }
 
-        char* endptr = nullptr; // NOLINT(misc-const-correctness,cppcoreguidelines-pro-type-vararg,hicpp-vararg)
+        convert_float_locale_aware();
-        strtof(value_float, token_buffer.data(), &endptr);
-
-        // we checked the number format before
-        JSON_ASSERT(endptr == token_buffer.data() + token_buffer.size());
-
         return token_type::value_float;
     }
 
+    /*!
+    @brief convert the float in token_buffer with strtof/strtod/strtold
+
+    These functions expect the decimal point of the *current* locale, so it is
+    looked up right before the conversion instead of once when the lexer is
+    constructed: a locale change in between (by a parser callback, a SAX
+    handler, or another thread) must not truncate the value (#5198). The
+    token has been validated before, so if the conversion stops early and the
+    decimal point changed in the meantime, the locale changed between the
+    lookup and the call, and the conversion is repeated with the new decimal
+    point. If the decimal point did not change, a retry cannot succeed: the
+    locale's decimal point is not a single character (e.g., the two-byte
+    U+066B of ar_EG.UTF-8 or fa_IR.UTF-8) and cannot be substituted in place.
+    The value strtod parsed up to that point is kept, as before this change.
+
+    Note that changing the locale in another thread *while* strtod runs is
+    undefined behavior of the C library, which this function cannot prevent.
+    */
+    void convert_float_locale_aware()
+    {
+        const bool has_dot = decimal_point_position != std::string::npos;
+        char decimal_point = get_decimal_point();
+        for (;;)
+        {
+            const bool substitute = has_dot && decimal_point != '.';
+            if (substitute)
+            {
+                token_buffer[decimal_point_position] = static_cast<typename string_t::value_type>(decimal_point);
+            }
+
+            char* endptr = nullptr; // NOLINT(misc-const-correctness,cppcoreguidelines-pro-type-vararg,hicpp-vararg)
+            strtof(value_float, token_buffer.data(), &endptr);
+
+            if (substitute)
+            {
+                // get_string() hands the token to the SAX interface with '.'
+                token_buffer[decimal_point_position] = '.';
+            }
+
+            if (JSON_HEDLEY_LIKELY(endptr == token_buffer.data() + token_buffer.size()))
+            {
+                return;
+            }
+
+            // retry only if the locale changed; otherwise, this would loop forever
+            const char current_decimal_point = get_decimal_point();
+            if (current_decimal_point == decimal_point)
+            {
+                return;
+            }
+            decimal_point = current_decimal_point;
+        }
+    }
+
     /*!
     @brief contiguous fast path for scanning a number
 
     Parses the whole number token straight from the input buffer, avoiding the
     per-character get()/add() of scan_number(). On success it fills token_buffer
-    (with the locale decimal point substituted, as scan_number() does) and
+    (as scan_number() does) and
     returns the token type. On anything it does not fully recognize as a
     well-formed number it makes no state change and returns
     token_type::uninitialized, so the caller falls back to scan_number(), which
@@ -10804,16 +10847,11 @@ scan_number_done:
         }
 #endif
 
-        // materialize the token exactly as scan_number() would, substituting the
+        // materialize the token exactly as scan_number() would. reset() already
-        // locale decimal point so convert_number()'s strtof fallback stays valid.
+        // cleared token_buffer, so append() fills it (assign() is avoided
-        // reset() already cleared token_buffer, so append() fills it (assign() is
+        // because custom string_t types need not provide it)
-        // avoided because custom string_t types need not provide it)
         token_buffer.append(reinterpret_cast<const typename string_t::value_type*>(data), len);
-        if (dot_index != std::string::npos)
+        decimal_point_position = dot_index;
-        {
-            token_buffer[dot_index] = static_cast<typename string_t::value_type>(decimal_point_char);
-            decimal_point_position = dot_index;
-        }
 
         ia.bulk_skip(len - 1);
         position.chars_read_total += (len - 1);
@@ -11080,11 +11118,7 @@ scan_number_done:
     /// return current string value (implicitly resets the token; useful only once)
     string_t& get_string()
     {
-        // translate decimal points from locale back to '.' (#4084)
+        // a number token holds '.' regardless of the locale (#4084)
-        if (decimal_point_char != '.' && decimal_point_position != std::string::npos)
-        {
-            token_buffer[decimal_point_position] = '.';
-        }
         return token_buffer;
     }
 
@@ -11380,9 +11414,7 @@ scan_number_done:
     number_unsigned_t value_unsigned = 0;
     number_float_t value_float = 0;
 
-    /// the decimal point
+    /// the position of the decimal point in token_buffer
-    const char_int_type decimal_point_char = '.';
-    /// the position of the decimal point in the input
     std::size_t decimal_point_position = std::string::npos;
 
     /// whether the caller (e.g. accept()/json_sax_acceptor) only needs the
@@ -13294,7 +13326,7 @@ class binary_reader
             case 0x01: // double
             {
                 double number{};
-                return get_number<double, true>(input_format_t::bson, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number<double, true>(input_format_t::bson, number) && emit_float(input_format_t::bson, number);
             }
 
             case 0x02: // string
@@ -13335,19 +13367,19 @@ class binary_reader
             case 0x10: // int32
             {
                 std::int32_t value{};
-                return get_number<std::int32_t, true>(input_format_t::bson, value) && sax->number_integer(value);
+                return get_number<std::int32_t, true>(input_format_t::bson, value) && emit_signed(input_format_t::bson, value);
             }
 
             case 0x12: // int64
             {
                 std::int64_t value{};
-                return get_number<std::int64_t, true>(input_format_t::bson, value) && sax->number_integer(value);
+                return get_number<std::int64_t, true>(input_format_t::bson, value) && emit_signed(input_format_t::bson, value);
             }
 
             case 0x11: // uint64
             {
                 std::uint64_t value{};
-                return get_number<std::uint64_t, true>(input_format_t::bson, value) && sax->number_unsigned(value);
+                return get_number<std::uint64_t, true>(input_format_t::bson, value) && emit_unsigned(input_format_t::bson, value);
             }
 
             default: // anything else is not supported (yet)
@@ -13373,14 +13405,19 @@ class binary_reader
         {
             return false;
         }
-        const auto max_val = static_cast<NumberType>((std::numeric_limits<number_integer_t>::max)());
+
-        if (number > max_val)
+        // the value is -1 - number, which fits into number_integer_t
+        // whenever number does
+        if (JSON_HEDLEY_LIKELY(value_in_range_of<number_integer_t>(number)))
         {
-            return sax->parse_error(chars_read, get_token_string(),
+            return sax->number_integer(static_cast<number_integer_t>(-1) - static_cast<number_integer_t>(number));
-                                    parse_error::create(112, chars_read,
-                                            exception_message(input_format_t::cbor, "negative integer overflow", "value"), nullptr));
         }
-        return sax->number_integer(static_cast<number_integer_t>(-1) - static_cast<number_integer_t>(number));
+
+        // like the lexer does for JSON text, store a value too small for
+        // number_integer_t as number_float_t; compute it as long double so
+        // that emit_float sees a finite value and can detect an overflow of
+        // number_float_t
+        return emit_float(input_format_t::cbor, static_cast<long double>(-1) - static_cast<long double>(number));
     }
 
     /*!
@@ -13437,25 +13474,25 @@ class binary_reader
             case 0x18: // Unsigned integer (one-byte uint8_t follows)
             {
                 std::uint8_t number{};
-                return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::cbor, number) && emit_unsigned(input_format_t::cbor, number);
             }
 
             case 0x19: // Unsigned integer (two-byte uint16_t follows)
             {
                 std::uint16_t number{};
-                return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::cbor, number) && emit_unsigned(input_format_t::cbor, number);
             }
 
             case 0x1A: // Unsigned integer (four-byte uint32_t follows)
             {
                 std::uint32_t number{};
-                return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::cbor, number) && emit_unsigned(input_format_t::cbor, number);
             }
 
             case 0x1B: // Unsigned integer (eight-byte uint64_t follows)
             {
                 std::uint64_t number{};
-                return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::cbor, number) && emit_unsigned(input_format_t::cbor, number);
             }
 
             // Negative integer -1-0x00..-1-0x17 (-1..-24)
@@ -13900,13 +13937,13 @@ class binary_reader
             case 0xFA: // Single-Precision Float (four-byte IEEE 754)
             {
                 float number{};
-                return get_number(input_format_t::cbor, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format_t::cbor, number) && emit_float(input_format_t::cbor, number);
             }
 
             case 0xFB: // Double-Precision Float (eight-byte IEEE 754)
             {
                 double number{};
-                return get_number(input_format_t::cbor, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format_t::cbor, number) && emit_float(input_format_t::cbor, number);
             }
 
             default: // anything else (0xFF is handled inside the other types)
@@ -14059,6 +14096,80 @@ class binary_reader
         }
     }
 
+    /*!
+    @brief reads a CBOR object key
+
+    RFC 8949 allows any data item as a map key, but only strings have a
+    counterpart in JSON. A key of any other type is rejected with a message
+    naming that type, rather than the one @ref get_cbor_string gives for a
+    malformed string.
+
+    @param[out] result  created key
+
+    @return whether key creation completed
+    */
+    bool get_cbor_object_key(string_t& result)
+    {
+        // EOF and major type 3 (text string) are left to get_cbor_string
+        if (current == char_traits<char_type>::eof() || (static_cast<unsigned int>(current) & 0xE0u) == 0x60u)
+        {
+            return get_cbor_string(result);
+        }
+
+        const char* found = nullptr;
+        switch (static_cast<unsigned int>(current) >> 5u)
+        {
+            case 0:
+                found = "an unsigned integer";
+                break;
+            case 1:
+                found = "a negative integer";
+                break;
+            case 2:
+                found = "a byte string";
+                break;
+            case 4:
+                found = "an array";
+                break;
+            case 5:
+                found = "a map";
+                break;
+            case 6:
+                found = "a tag";
+                break;
+            default: // major type 7
+                switch (current)
+                {
+                    case 0xF4:
+                    case 0xF5:
+                        found = "a boolean";
+                        break;
+                    case 0xF6:
+                        found = "null";
+                        break;
+                    case 0xF7:
+                        found = "undefined";
+                        break;
+                    case 0xF9:
+                    case 0xFA:
+                    case 0xFB:
+                        found = "a floating-point number";
+                        break;
+                    case 0xFF:
+                        found = "a break stop code";
+                        break;
+                    default:
+                        found = "a simple value";
+                        break;
+                }
+                break;
+        }
+
+        auto last_token = get_token_string();
+        return sax->parse_error(chars_read, last_token, parse_error::create(113, chars_read,
+                                exception_message(input_format_t::cbor, concat("only string keys are supported, but found ", found, "; last byte: 0x", last_token), "object key"), nullptr));
+    }
+
     /*!
     @brief reads a definite-length CBOR byte array
 
@@ -14303,7 +14414,7 @@ class binary_reader
                 if (top.is_object)
                 {
                     key.clear();
-                    if (JSON_HEDLEY_UNLIKELY(!get_cbor_string(key) || !sax->key(key)))
+                    if (JSON_HEDLEY_UNLIKELY(!get_cbor_object_key(key) || !sax->key(key)))
                     {
                         return false;
                     }
@@ -14596,61 +14707,61 @@ class binary_reader
             case 0xCA: // float 32
             {
                 float number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format_t::msgpack, number) && emit_float(input_format_t::msgpack, number);
             }
 
             case 0xCB: // float 64
             {
                 double number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format_t::msgpack, number) && emit_float(input_format_t::msgpack, number);
             }
 
             case 0xCC: // uint 8
             {
                 std::uint8_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::msgpack, number) && emit_unsigned(input_format_t::msgpack, number);
             }
 
             case 0xCD: // uint 16
             {
                 std::uint16_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::msgpack, number) && emit_unsigned(input_format_t::msgpack, number);
             }
 
             case 0xCE: // uint 32
             {
                 std::uint32_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::msgpack, number) && emit_unsigned(input_format_t::msgpack, number);
             }
 
             case 0xCF: // uint 64
             {
                 std::uint64_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::msgpack, number) && emit_unsigned(input_format_t::msgpack, number);
             }
 
             case 0xD0: // int 8
             {
                 std::int8_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
+                return get_number(input_format_t::msgpack, number) && emit_signed(input_format_t::msgpack, number);
             }
 
             case 0xD1: // int 16
             {
                 std::int16_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
+                return get_number(input_format_t::msgpack, number) && emit_signed(input_format_t::msgpack, number);
             }
 
             case 0xD2: // int 32
             {
                 std::int32_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
+                return get_number(input_format_t::msgpack, number) && emit_signed(input_format_t::msgpack, number);
             }
 
             case 0xD3: // int 64
             {
                 std::int64_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
+                return get_number(input_format_t::msgpack, number) && emit_signed(input_format_t::msgpack, number);
             }
 
             case 0xDC: // array 16
@@ -14804,6 +14915,98 @@ class binary_reader
         }
     }
 
+    /*!
+    @brief reads a MessagePack object key
+
+    The MessagePack specification allows any type as a map key, but only
+    strings have a counterpart in JSON. A key of any other type is rejected
+    with a message naming that type, rather than the one @ref
+    get_msgpack_string gives for a malformed string.
+
+    @param[out] result  created key
+
+    @return whether key creation completed
+    */
+    bool get_msgpack_object_key(string_t& result)
+    {
+        const char* found = nullptr;
+        switch (current)
+        {
+            case 0xC0:
+                found = "nil";
+                break;
+            case 0xC2:
+            case 0xC3:
+                found = "a boolean";
+                break;
+            case 0xCA:
+            case 0xCB:
+                found = "a float";
+                break;
+            case 0xC4:
+            case 0xC5:
+            case 0xC6:
+                found = "a bin";
+                break;
+            case 0xC7:
+            case 0xC8:
+            case 0xC9:
+            case 0xD4:
+            case 0xD5:
+            case 0xD6:
+            case 0xD7:
+            case 0xD8:
+                found = "an ext";
+                break;
+            case 0xCC:
+            case 0xCD:
+            case 0xCE:
+            case 0xCF:
+            case 0xD0:
+            case 0xD1:
+            case 0xD2:
+            case 0xD3:
+                found = "an integer";
+                break;
+            case 0xDC:
+            case 0xDD:
+                found = "an array";
+                break;
+            case 0xDE:
+            case 0xDF:
+                found = "a map";
+                break;
+            default:
+                // fixint, fixmap, and fixarray; strings, EOF, and the unused
+                // byte 0xC1 are left to get_msgpack_string
+                if (current == char_traits<char_type>::eof())
+                {
+                    return get_msgpack_string(result);
+                }
+                if (current <= 0x7F || current >= 0xE0)
+                {
+                    found = "an integer";
+                }
+                else if (current <= 0x8F)
+                {
+                    found = "a map";
+                }
+                else if (current <= 0x9F)
+                {
+                    found = "an array";
+                }
+                else
+                {
+                    return get_msgpack_string(result);
+                }
+                break;
+        }
+
+        auto last_token = get_token_string();
+        return sax->parse_error(chars_read, last_token, parse_error::create(113, chars_read,
+                                exception_message(input_format_t::msgpack, concat("only string keys are supported, but found ", found, "; last byte: 0x", last_token), "object key"), nullptr));
+    }
+
     /*!
     @brief reads a MessagePack byte array
 
@@ -14966,7 +15169,7 @@ class binary_reader
                 {
                     get();
                     key.clear();
-                    if (JSON_HEDLEY_UNLIKELY(!get_msgpack_string(key) || !sax->key(key)))
+                    if (JSON_HEDLEY_UNLIKELY(!get_msgpack_object_key(key) || !sax->key(key)))
                     {
                         return false;
                     }
@@ -15491,7 +15694,7 @@ class binary_reader
                         {
                             return sax->parse_error(chars_read, get_token_string(), out_of_range::create(408, exception_message(input_format, "excessive ndarray size caused overflow", "size"), nullptr));
                         }
-                        if (JSON_HEDLEY_UNLIKELY(!sax->number_unsigned(static_cast<number_unsigned_t>(i))))
+                        if (JSON_HEDLEY_UNLIKELY(!emit_unsigned(input_format, i)))
                         {
                             return false;
                         }
@@ -15623,37 +15826,37 @@ class binary_reader
                     break;
                 }
                 std::uint8_t number{};
-                return get_number(input_format, number) && sax->number_unsigned(number);
+                return get_number(input_format, number) && emit_unsigned(input_format, number);
             }
 
             case 'U':
             {
                 std::uint8_t number{};
-                return get_number(input_format, number) && sax->number_unsigned(number);
+                return get_number(input_format, number) && emit_unsigned(input_format, number);
             }
 
             case 'i':
             {
                 std::int8_t number{};
-                return get_number(input_format, number) && sax->number_integer(number);
+                return get_number(input_format, number) && emit_signed(input_format, number);
             }
 
             case 'I':
             {
                 std::int16_t number{};
-                return get_number(input_format, number) && sax->number_integer(number);
+                return get_number(input_format, number) && emit_signed(input_format, number);
             }
 
             case 'l':
             {
                 std::int32_t number{};
-                return get_number(input_format, number) && sax->number_integer(number);
+                return get_number(input_format, number) && emit_signed(input_format, number);
             }
 
             case 'L':
             {
                 std::int64_t number{};
-                return get_number(input_format, number) && sax->number_integer(number);
+                return get_number(input_format, number) && emit_signed(input_format, number);
             }
 
             case 'u':
@@ -15663,7 +15866,7 @@ class binary_reader
                     break;
                 }
                 std::uint16_t number{};
-                return get_number(input_format, number) && sax->number_unsigned(number);
+                return get_number(input_format, number) && emit_unsigned(input_format, number);
             }
 
             case 'm':
@@ -15673,7 +15876,7 @@ class binary_reader
                     break;
                 }
                 std::uint32_t number{};
-                return get_number(input_format, number) && sax->number_unsigned(number);
+                return get_number(input_format, number) && emit_unsigned(input_format, number);
             }
 
             case 'M':
@@ -15683,7 +15886,7 @@ class binary_reader
                     break;
                 }
                 std::uint64_t number{};
-                return get_number(input_format, number) && sax->number_unsigned(number);
+                return get_number(input_format, number) && emit_unsigned(input_format, number);
             }
 
             case 'h':
@@ -15741,13 +15944,13 @@ class binary_reader
             case 'd':
             {
                 float number{};
-                return get_number(input_format, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format, number) && emit_float(input_format, number);
             }
 
             case 'D':
             {
                 double number{};
-                return get_number(input_format, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format, number) && emit_float(input_format, number);
             }
 
             case 'H':
@@ -16214,13 +16417,13 @@ class binary_reader
             case 0x8E: // binary32
             {
                 float number{};
-                return get_number(input_format_t::bon8, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format_t::bon8, number) && emit_float(input_format_t::bon8, number);
             }
 
             case 0x8F: // binary64
             {
                 double number{};
-                return get_number(input_format_t::bon8, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format_t::bon8, number) && emit_float(input_format_t::bon8, number);
             }
 
             case 0xF8:
@@ -16286,7 +16489,9 @@ class binary_reader
     @brief pass an integer to the SAX parser
 
     Non-negative integers are passed as unsigned, negative integers as signed
-    numbers, like the other binary formats do.
+    numbers, like the other binary formats do. A value that does not fit the
+    number type is passed as described for @ref emit_unsigned and
+    @ref emit_signed.
 
     @param[in] number  the integer
     @return whether the SAX parser accepted the value
@@ -16295,9 +16500,9 @@ class binary_reader
     {
         if (number >= 0)
         {
-            return sax->number_unsigned(static_cast<number_unsigned_t>(number));
+            return emit_unsigned(input_format_t::bon8, static_cast<std::uint64_t>(number));
         }
-        return sax->number_integer(static_cast<number_integer_t>(number));
+        return emit_signed(input_format_t::bon8, number);
     }
 
     /*!
@@ -16354,8 +16559,7 @@ class binary_reader
             value = (value << 8) | static_cast<std::int64_t>(current);
         }
 
-        return negative ? sax->number_integer(static_cast<number_integer_t>(-(value + offset)))
+        return emit_bon8_integer(negative ? -(value + offset) : value + offset);
-               : sax->number_unsigned(static_cast<number_unsigned_t>(value + offset));
     }
 
     /*!
@@ -16652,6 +16856,88 @@ class binary_reader
         return true;
     }
 
+    /*!
+    @brief pass a signed integer read from the input to the SAX parser
+
+    Like the lexer does for JSON text, a value that does not fit into
+    number_integer_t is passed as number_unsigned_t if it is non-negative and
+    fits there, and as number_float_t otherwise. With the default number
+    types, every integer the binary formats can encode fits, so this only
+    matters for narrower custom number types.
+
+    @tparam NumberType a signed integer type
+    @param[in] format  the current format (for diagnostics)
+    @param[in] number  the integer
+    @return whether the SAX parser accepted the value
+
+    @throw out_of_range.406 if @a number overflows number_float_t (see
+           @ref emit_float)
+    */
+    template<typename NumberType>
+    bool emit_signed(const input_format_t format, const NumberType number)
+    {
+        if (JSON_HEDLEY_LIKELY(value_in_range_of<number_integer_t>(number)))
+        {
+            return sax->number_integer(static_cast<number_integer_t>(number));
+        }
+        if (value_in_range_of<number_unsigned_t>(number))
+        {
+            return sax->number_unsigned(static_cast<number_unsigned_t>(number));
+        }
+        return emit_float(format, number);
+    }
+
+    /*!
+    @brief pass an unsigned integer read from the input to the SAX parser
+
+    Like the lexer does for JSON text, a value that does not fit into
+    number_unsigned_t is passed as number_float_t.
+
+    @tparam NumberType an unsigned integer type
+    @param[in] format  the current format (for diagnostics)
+    @param[in] number  the integer
+    @return whether the SAX parser accepted the value
+
+    @throw out_of_range.406 if @a number overflows number_float_t (see
+           @ref emit_float)
+    */
+    template<typename NumberType>
+    bool emit_unsigned(const input_format_t format, const NumberType number)
+    {
+        if (JSON_HEDLEY_LIKELY(value_in_range_of<number_unsigned_t>(number)))
+        {
+            return sax->number_unsigned(static_cast<number_unsigned_t>(number));
+        }
+        return emit_float(format, number);
+    }
+
+    /*!
+    @brief pass a floating-point number read from the input to the SAX parser
+
+    Like the lexer does for JSON text, a finite value that overflows
+    number_float_t is rejected instead of silently becoming infinity. Infinity
+    and NaN in the input are passed on unchanged. Integers only overflow if
+    number_float_t cannot represent 2^64, e.g., a half-precision type.
+
+    @tparam NumberType a floating-point or integer type
+    @param[in] format  the current format (for diagnostics)
+    @param[in] number  the number
+    @return whether the SAX parser accepted the value
+
+    @throw out_of_range.406 if a finite @a number overflows number_float_t
+    */
+    template<typename NumberType>
+    bool emit_float(const input_format_t format, const NumberType number)
+    {
+        const auto result = static_cast<number_float_t>(number);
+        if (JSON_HEDLEY_UNLIKELY(std::isfinite(number) && !std::isfinite(result)))
+        {
+            return sax->parse_error(chars_read, get_token_string(),
+                                    out_of_range::create(406, exception_message(format, "number overflow", "value"), nullptr));
+        }
+        return sax->number_float(result, "");
+    }
+
     /*!
     @brief create a string by reading characters from the input
 
@@ -23931,8 +24217,7 @@ enum class error_handler_t
 {
     strict,  ///< throw a type_error exception in case of invalid UTF-8
     replace, ///< replace invalid UTF-8 sequences with U+FFFD
-    ignore,  ///< ignore invalid UTF-8 sequences
+    ignore   ///< ignore invalid UTF-8 sequences
-    keep     ///< keep invalid UTF-8 sequences; their bytes are copied unchanged
 };
 
 template<typename BasicJsonType>
@@ -24903,47 +25188,6 @@ class serializer
                             break;
                         }
 
-                        case error_handler_t::keep:
-                        {
-                            // drop whatever the incomplete sequence left in
-                            // the buffer (only copied if !EnsureAscii) and copy
-                            // the ill-formed bytes from the input instead
-                            bytes = bytes_after_last_accept;
-
-                            if (undumped_chars > 0)
-                            {
-                                // the pending bytes of the incomplete sequence
-                                // are ill-formed; the current byte may be OK for
-                                // itself, so we would like to read it again
-                                for (std::size_t j = i - undumped_chars; j < i; ++j)
-                                {
-                                    string_buffer[bytes++] = s[j];
-                                }
-                                --i;
-                            }
-                            else
-                            {
-                                // the current byte cannot start any sequence
-                                string_buffer[bytes++] = s[i];
-                            }
-
-                            // write buffer and reset index; there must be 13 bytes
-                            // left, as this is the maximal number of bytes to be
-                            // written ("\uxxxx\uxxxx\0") for one code point
-                            if (string_buffer.size() - bytes < 13)
-                            {
-                                put_buffer(string_buffer, bytes);
-                                bytes = 0;
-                            }
-
-                            bytes_after_last_accept = bytes;
-                            undumped_chars = 0;
-
-                            // continue processing the string
-                            state = UTF8_ACCEPT;
-                            break;
-                        }
-
                         default:            // LCOV_EXCL_LINE
                             JSON_ASSERT(false); // NOLINT(cert-dcl03-c,hicpp-static-assert,misc-static-assert) LCOV_EXCL_LINE
                     }
@@ -24989,15 +25233,6 @@ class serializer
                     break;
                 }
 
-                case error_handler_t::keep:
-                {
-                    // write all accepted bytes
-                    put_buffer(string_buffer, bytes_after_last_accept);
-                    // copy the bytes of the incomplete sequence unchanged
-                    put_string(s, s.size() - undumped_chars, s.size());
-                    break;
-                }
-
                 case error_handler_t::replace:
                 {
                     // write all accepted bytes

tests/src/unit-binary_formats.cpp

@@ -11,7 +11,12 @@
 #include <nlohmann/json.hpp>
 using nlohmann::json;
 
+#include <cmath>
 #include <fstream>
+#include <limits>
+#include <map>
+#include <string>
+#include <vector>
 #include "make_test_data_available.hpp"
 
 TEST_CASE("Binary Formats" * doctest::skip())
@@ -224,3 +229,139 @@ TEST_CASE("Binary Formats" * doctest::skip())
         CHECK((100.0 * double(ubjson_3_size) / double(json_size)) == Approx(89.450));
     }
 }
+
+namespace
+{
+// the binary formats as function pointers for "Binary formats with narrow number types";
+// named functions rather than lambdas, because clang 3.5 cannot convert a lambda
+// to a function pointer in the braced initializer of the format table
+using narrow_json = nlohmann::basic_json<std::map, std::vector, std::string, bool, std::int32_t, std::uint32_t, float>;
+using bytes = std::vector<std::uint8_t>;
+
+bytes encode_cbor(const json& j)
+{
+    return json::to_cbor(j);
+}
+narrow_json decode_cbor(const bytes& v, bool allow_exceptions)
+{
+    return narrow_json::from_cbor(v, true, allow_exceptions);
+}
+
+bytes encode_msgpack(const json& j)
+{
+    return json::to_msgpack(j);
+}
+narrow_json decode_msgpack(const bytes& v, bool allow_exceptions)
+{
+    return narrow_json::from_msgpack(v, true, allow_exceptions);
+}
+
+bytes encode_ubjson(const json& j)
+{
+    return json::to_ubjson(j);
+}
+narrow_json decode_ubjson(const bytes& v, bool allow_exceptions)
+{
+    return narrow_json::from_ubjson(v, true, allow_exceptions);
+}
+
+bytes encode_bjdata(const json& j)
+{
+    return json::to_bjdata(j);
+}
+narrow_json decode_bjdata(const bytes& v, bool allow_exceptions)
+{
+    return narrow_json::from_bjdata(v, true, allow_exceptions);
+}
+
+// BSON can only store numbers as object members
+bytes encode_bson(const json& j)
+{
+    return json::to_bson(json{{"a", j}});
+}
+narrow_json decode_bson(const bytes& v, bool allow_exceptions)
+{
+    const auto result = narrow_json::from_bson(v, true, allow_exceptions);
+    return result.is_discarded() ? result : result.at("a");
+}
+
+bytes encode_bon8(const json& j)
+{
+    return json::to_bon8(j);
+}
+narrow_json decode_bon8(const bytes& v, bool allow_exceptions)
+{
+    return narrow_json::from_bon8(v, true, allow_exceptions);
+}
+
+} // namespace
+
+TEST_CASE("Binary formats with narrow number types")
+{
+    // Numbers that do not fit the number types are handled like the lexer
+    // handles them in JSON text: an integer that fits neither integer type is
+    // stored as a floating-point number, and a finite floating-point number
+    // that overflows number_float_t is rejected with out_of_range.406.
+    struct binary_format
+    {
+        const char* name;
+        bytes (*encode)(const json&);
+        narrow_json (*decode)(const bytes&, bool);
+    };
+
+    const std::vector<binary_format> formats =
+    {
+        {"CBOR", encode_cbor, decode_cbor},
+        {"MessagePack", encode_msgpack, decode_msgpack},
+        {"UBJSON", encode_ubjson, decode_ubjson},
+        {"BJData", encode_bjdata, decode_bjdata},
+        {"BSON", encode_bson, decode_bson},
+        {"BON8", encode_bon8, decode_bon8},
+    };
+
+    for (const auto& format : formats)
+    {
+        const std::string name = format.name;
+        INFO("format := ", name);
+        const auto roundtrip = [&format](const json & j)
+        {
+            return format.decode(format.encode(j), true);
+        };
+
+        // integers that fit keep their type
+        CHECK(roundtrip(json(-5)).is_number_integer());
+        CHECK(roundtrip(json(-5)).get<std::int32_t>() == -5);
+        CHECK(roundtrip(json(3000000000u)).is_number_unsigned());
+        CHECK(roundtrip(json(3000000000u)).get<std::uint32_t>() == 3000000000u);
+
+        // integers that fit neither integer type are stored as float
+        CHECK(roundtrip(json(5000000000u)).is_number_float());
+        CHECK(roundtrip(json(5000000000u)).get<float>() == 5000000000.0f);
+        if (name != "BON8") // BON8 cannot encode integers above INT64_MAX
+        {
+            CHECK(roundtrip(json(10000000000000000000u)).is_number_float());
+            CHECK(roundtrip(json(10000000000000000000u)).get<float>() == 10000000000000000000.0f);
+        }
+        CHECK(roundtrip(json(-3000000000LL)).is_number_float());
+        CHECK(roundtrip(json(-3000000000LL)).get<float>() == -3000000000.0f);
+        CHECK(roundtrip(json(-5000000000LL)).is_number_float());
+        CHECK(roundtrip(json(-5000000000LL)).get<float>() == -5000000000.0f);
+
+        // floating-point numbers that fit
+        CHECK(roundtrip(json(1.5)).get<float>() == 1.5f);
+        const auto just_above_max = std::nextafter(static_cast<double>((std::numeric_limits<float>::max)()),
+                                    std::numeric_limits<double>::infinity());
+        CHECK(roundtrip(json(just_above_max)).get<float>() == (std::numeric_limits<float>::max)());
+
+        // infinity and NaN are passed on
+        CHECK(std::isinf(roundtrip(json(std::numeric_limits<double>::infinity())).get<float>()));
+        CHECK(std::isnan(roundtrip(json(std::numeric_limits<double>::quiet_NaN())).get<float>()));
+
+        // finite floating-point numbers that overflow number_float_t are rejected
+        const std::string message = "[json.exception.out_of_range.406] syntax error while parsing " + name
+                                    + " value: number overflow";
+        CHECK_THROWS_WITH_AS(roundtrip(json(1e300)), message.c_str(), narrow_json::out_of_range&);
+        CHECK_THROWS_WITH_AS(roundtrip(json(-1e300)), message.c_str(), narrow_json::out_of_range&);
+        CHECK(format.decode(format.encode(json(1e300)), false).is_discarded());
+    }
+}

tests/src/unit-cbor.cpp

@@ -1830,10 +1830,51 @@ TEST_CASE("CBOR")
         SECTION("invalid string in map")
         {
             json _;
-            CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0xa1, 0xff, 0x01})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR string: expected length specification (0x60-0x7B) or indefinite string type (0x7F); last byte: 0xFF", json::parse_error&);
+            CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0xa1, 0xff, 0x01})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR object key: only string keys are supported, but found a break stop code; last byte: 0xFF", json::parse_error&);
             CHECK(json::from_cbor(std::vector<uint8_t>({0xa1, 0xff, 0x01}), true, false).is_discarded());
         }
 
+        SECTION("non-string key (see #2766 and #3381)")
+        {
+            // only text strings map to JSON object keys; any other key is
+            // rejected with a message naming its type
+            const std::vector<std::pair<std::vector<std::uint8_t>, std::string>> cases =
+            {
+                {{0xA1, 0x01, 0x01}, "an unsigned integer; last byte: 0x01"},
+                {{0xA1, 0x20, 0x01}, "a negative integer; last byte: 0x20"},
+                {{0xA1, 0x41, 0x61, 0x01}, "a byte string; last byte: 0x41"},
+                {{0xA1, 0x80, 0x01}, "an array; last byte: 0x80"},
+                {{0xA1, 0xA0, 0x01}, "a map; last byte: 0xA0"},
+                {{0xA1, 0xC0, 0x61, 0x61, 0x01}, "a tag; last byte: 0xC0"},
+                {{0xA1, 0xF4, 0x01}, "a boolean; last byte: 0xF4"},
+                {{0xA1, 0xF5, 0x01}, "a boolean; last byte: 0xF5"},
+                {{0xA1, 0xF6, 0x01}, "null; last byte: 0xF6"},
+                {{0xA1, 0xF7, 0x01}, "undefined; last byte: 0xF7"},
+                {{0xA1, 0xF9, 0x3C, 0x00, 0x01}, "a floating-point number; last byte: 0xF9"},
+                {{0xA1, 0xFA, 0x3F, 0x80, 0x00, 0x00, 0x01}, "a floating-point number; last byte: 0xFA"},
+                {{0xA1, 0xFB, 0x3F, 0xF0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01}, "a floating-point number; last byte: 0xFB"},
+                {{0xA1, 0xE0, 0x01}, "a simple value; last byte: 0xE0"},
+                {{0xA1, 0xF8, 0x20, 0x01}, "a simple value; last byte: 0xF8"},
+                // indefinite-length map
+                {{0xBF, 0x01, 0x01, 0xFF}, "an unsigned integer; last byte: 0x01"},
+            };
+
+            for (const auto& c : cases)
+            {
+                CAPTURE(c.first)
+                const std::string expected = "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR object key: only string keys are supported, but found " + c.second;
+                json _;
+                CHECK_THROWS_WITH_AS(_ = json::from_cbor(c.first), expected.c_str(), json::parse_error&);
+                CHECK(json::from_cbor(c.first, true, false).is_discarded());
+            }
+
+            // a key of major type 3 with a reserved length is still reported as
+            // a malformed string, and a missing key as the end of input
+            json _;
+            CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0xA1})), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing CBOR string: unexpected end of input", json::parse_error&);
+            CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0xA1, 0x7C, 0x01})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR string: expected length specification (0x60-0x7B) or indefinite string type (0x7F); last byte: 0x7C", json::parse_error&);
+        }
+
         SECTION("invalid UTF-8 in string (see #5529)")
         {
             // a two-character text string (major type 3) whose bytes are not
@@ -2284,7 +2325,7 @@ TEST_CASE("CBOR indefinite-length strings do not recurse per chunk")
     SECTION("a break marker outside an indefinite-length string is not a string")
     {
         // 0xFF only closes a string that was opened; on its own it is not one
-        CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0xA1, 0xFF, 0x01})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR string: expected length specification (0x60-0x7B) or indefinite string type (0x7F); last byte: 0xFF", json::parse_error&);
+        CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0xA1, 0xFF, 0x01})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR object key: only string keys are supported, but found a break stop code; last byte: 0xFF", json::parse_error&);
     }
 }
 
@@ -3146,7 +3187,8 @@ TEST_CASE("Tagged values")
         // CBOR encodes negative integers as: result = -1 - n
         // For type 0x3B, n is an 8-byte uint64_t. Valid range for n with
         // the default int64_t is [0, INT64_MAX], producing results in [INT64_MIN, -1].
-        // When n > INT64_MAX, the result exceeds int64_t range and is rejected.
+        // When n > INT64_MAX, the result exceeds int64_t range and is stored
+        // as a floating-point number, as the lexer does for JSON text.
 
         SECTION("n = 0 is valid (result = -1)")
         {
@@ -3167,33 +3209,34 @@ TEST_CASE("Tagged values")
             CHECK(result.get<int64_t>() == (std::numeric_limits<int64_t>::min)());
         }
 
-        SECTION("n = INT64_MAX + 1 is rejected (overflow)")
+        SECTION("n = INT64_MAX + 1 is stored as float")
         {
             // n = INT64_MAX + 1 (0x8000000000000000)
-            // result = -1 - n = -9223372036854775809, which exceeds int64_t range
+            // result = -1 - n = -9223372036854775809, which exceeds int64_t range;
+            // the nearest double is -9223372036854775808.0
             const std::vector<uint8_t> input = {0x3B, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
-            json _;
+            const auto result = json::from_cbor(input);
-            CHECK_THROWS_WITH_AS(_ = json::from_cbor(input),
+            CHECK(result.is_number_float());
-                                 "[json.exception.parse_error.112] parse error at byte 9: syntax error while parsing CBOR value: negative integer overflow",
+            CHECK(result.get<double>() == -9223372036854775808.0);
-                                 json::parse_error);
+            CHECK(result == json::parse("-9223372036854775809"));
         }
 
-        SECTION("n = UINT64_MAX is rejected (overflow)")
+        SECTION("n = UINT64_MAX is stored as float")
         {
             // n = UINT64_MAX (0xFFFFFFFFFFFFFFFF)
             // result = -1 - n = -18446744073709551616, which exceeds int64_t range
             const std::vector<uint8_t> input = {0x3B, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF};
-            json _;
+            const auto result = json::from_cbor(input);
-            CHECK_THROWS_WITH_AS(_ = json::from_cbor(input),
+            CHECK(result.is_number_float());
-                                 "[json.exception.parse_error.112] parse error at byte 9: syntax error while parsing CBOR value: negative integer overflow",
+            CHECK(result.get<double>() == -18446744073709551616.0);
-                                 json::parse_error);
+            CHECK(result == json::parse("-18446744073709551616"));
         }
 
-        SECTION("overflow with allow_exceptions=false returns discarded")
+        SECTION("overflow with allow_exceptions=false is not an error")
         {
             const std::vector<uint8_t> input = {0x3B, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
             const auto result = json::from_cbor(input, true, false);
-            CHECK(result.is_discarded());
+            CHECK(result.is_number_float());
         }
     }
 

tests/src/unit-class_lexer.cpp

@@ -666,7 +666,7 @@ TEST_CASE("parse_float_fast declines what it cannot convert exactly")
     // always safe: the caller then falls back to a slower, exact conversion.
     const auto fast = [](const std::string & s, double & out)
     {
-        return nlohmann::detail::parse_float_fast(s.data(), s.data() + s.size(), '.', out);
+        return nlohmann::detail::parse_float_fast(s.data(), s.data() + s.size(), out);
     };
     double out = 0;
 

tests/src/unit-locale-cpp.cpp

@@ -12,7 +12,12 @@
 #include <nlohmann/json.hpp>
 using nlohmann::json;
 
+#include <array>
 #include <clocale>
+#include <map>
+#include <string>
+#include <utility>
+#include <vector>
 
 struct ParserImpl final: public nlohmann::json_sax<json>
 {
@@ -175,3 +180,208 @@ TEST_CASE("locale-dependent test (LC_NUMERIC=de_DE)")
         MESSAGE("locale de_DE is not usable");
     }
 }
+
+namespace
+{
+// records the numbers of a flat array and switches LC_NUMERIC to the given
+// locale once the array opens - after the lexer was constructed, but before
+// any number in the array is lexed
+struct LocaleSwitchingSax final: public nlohmann::json_sax<json>
+{
+    explicit LocaleSwitchingSax(const char* switch_to)
+        : locale_after_open(switch_to)
+    {}
+
+    bool null() override
+    {
+        return true;
+    }
+    bool boolean(bool /*val*/) override
+    {
+        return true;
+    }
+    bool number_integer(json::number_integer_t /*val*/) override
+    {
+        return true;
+    }
+    bool number_unsigned(json::number_unsigned_t /*val*/) override
+    {
+        return true;
+    }
+    bool number_float(json::number_float_t val, const json::string_t& s) override
+    {
+        values.push_back(val);
+        strings.push_back(s);
+        return true;
+    }
+    bool string(json::string_t& /*val*/) override
+    {
+        return true;
+    }
+    bool binary(json::binary_t& /*val*/) override
+    {
+        return true;
+    }
+    bool start_object(std::size_t /*val*/) override
+    {
+        return true;
+    }
+    bool key(json::string_t& /*val*/) override
+    {
+        return true;
+    }
+    bool end_object() override
+    {
+        return true;
+    }
+    bool start_array(std::size_t /*val*/) override
+    {
+        switched = std::setlocale(LC_NUMERIC, locale_after_open.c_str()) != nullptr;
+        return true;
+    }
+    bool end_array() override
+    {
+        return true;
+    }
+    bool parse_error(std::size_t /*val*/, const std::string& /*val*/, const nlohmann::detail::exception& /*val*/) override
+    {
+        return false;
+    }
+
+    std::string locale_after_open;
+    bool switched = false;
+    std::vector<json::number_float_t> values {}; // NOLINT(readability-redundant-member-init)
+    std::vector<json::string_t> strings {}; // NOLINT(readability-redundant-member-init)
+};
+} // namespace
+
+TEST_CASE("locale changes between lexer construction and number conversion (#5198)")
+{
+    // The numbers are chosen so that the conversion also takes the strtod
+    // fallback, which honors the locale that is current at conversion time:
+    // too many significant digits for Clinger's fast path, an underflow that
+    // std::from_chars rejects, and a plain value.
+    const std::vector<std::string> numbers = {"3.14159265358979323846", "1.5e-400", "12.34", "-0.000123456789012345678"};
+    std::string text = "[";
+    for (const auto& n : numbers)
+    {
+        text += (text.size() == 1 ? "" : ",") + n;
+    }
+    text += "]";
+
+    using long_double_json = nlohmann::basic_json<std::map, std::vector, std::string, bool, std::int64_t, std::uint64_t, long double>;
+
+    // reference values, parsed without a locale switch
+    REQUIRE(std::setlocale(LC_NUMERIC, "C") != nullptr);
+    const json expected = json::parse(text);
+    const long_double_json expected_ld = long_double_json::parse(text);
+
+    const std::array<std::pair<const char*, const char*>, 2> transitions =
+    {
+        {
+            {"C", "de_DE"},
+            {"de_DE", "C"}
+        }
+    };
+
+    for (const auto& transition : transitions)
+    {
+        CAPTURE(transition.first);
+        CAPTURE(transition.second);
+
+        if (std::setlocale(LC_NUMERIC, transition.first) == nullptr)
+        {
+            MESSAGE("locale is not usable");
+            continue;
+        }
+
+        // SAX parsing
+        {
+            LocaleSwitchingSax sax(transition.second);
+            CHECK(json::sax_parse(text, &sax));
+            if (sax.switched)
+            {
+                CHECK(sax.values == expected.get<std::vector<json::number_float_t>>());
+                CHECK(sax.strings == numbers);
+            }
+        }
+
+        // DOM parsing with a callback
+        {
+            bool switched = false;
+            const auto cb = [&](int /*depth*/, json::parse_event_t event, json& /*parsed*/) noexcept
+            {
+                if (event == json::parse_event_t::array_start)
+                {
+                    switched = std::setlocale(LC_NUMERIC, transition.second) != nullptr;
+                }
+                return true;
+            };
+            const json j = json::parse(text, cb);
+            if (switched)
+            {
+                CHECK(j == expected);
+            }
+        }
+
+        // a long double goes through std::strtold unless std::from_chars supports it
+        {
+            bool switched = false;
+            const auto cb = [&](int /*depth*/, long_double_json::parse_event_t event, long_double_json& /*parsed*/) noexcept
+            {
+                if (event == long_double_json::parse_event_t::array_start)
+                {
+                    switched = std::setlocale(LC_NUMERIC, transition.second) != nullptr;
+                }
+                return true;
+            };
+            const long_double_json j = long_double_json::parse(text, cb);
+            if (switched)
+            {
+                CHECK(j == expected_ld);
+            }
+        }
+    }
+
+    CHECK(std::setlocale(LC_NUMERIC, "C") != nullptr);
+}
+
+TEST_CASE("locale with a multi-byte decimal point")
+{
+    // Some locales use a decimal point that is not a single character, e.g.
+    // U+066B ARABIC DECIMAL SEPARATOR (two bytes in UTF-8). It cannot be
+    // substituted in place for '.', so the strtod fallback stops early. The
+    // conversion must still terminate rather than retry forever.
+    const std::array<const char*, 6> names = {{"ar_EG.UTF-8", "ar_SA.UTF-8", "fa_IR.UTF-8", "ps_AF.UTF-8", "ar_EG", "fa_IR"}};
+    bool tested = false;
+    for (const char* name : names)
+    {
+        if (std::setlocale(LC_NUMERIC, name) == nullptr)
+        {
+            continue;
+        }
+        const std::string decimal_point = std::localeconv()->decimal_point;
+        if (decimal_point.size() < 2)
+        {
+            continue;
+        }
+        CAPTURE(name);
+        tested = true;
+
+        // too many significant digits for Clinger's fast path, and an underflow
+        // that std::from_chars rejects: both reach the strtod fallback
+        json j;
+        CHECK_NOTHROW(j = json::parse("[3.14159265358979323846, 1.5e-400, -0.000123456789012345678]"));
+        CHECK(j.is_array());
+        CHECK(json::accept("3.14159265358979323846"));
+
+        // a value the locale-independent paths convert is not affected
+        CHECK(json::parse("12.5") == 12.5);
+    }
+    if (!tested)
+    {
+        MESSAGE("no locale with a multi-byte decimal point is usable");
+    }
+
+    CHECK(std::setlocale(LC_NUMERIC, "C") != nullptr);
+}

tests/src/unit-msgpack.cpp

@@ -1551,10 +1551,69 @@ TEST_CASE("MessagePack")
         SECTION("invalid string in map")
         {
             json _;
-            CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0x81, 0xff, 0x01})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing MessagePack string: expected length specification (0xA0-0xBF, 0xD9-0xDB); last byte: 0xFF", json::parse_error&);
+            CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0x81, 0xff, 0x01})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing MessagePack object key: only string keys are supported, but found an integer; last byte: 0xFF", json::parse_error&);
             CHECK(json::from_msgpack(std::vector<uint8_t>({0x81, 0xff, 0x01}), true, false).is_discarded());
         }
 
+        SECTION("non-string key (see #3381)")
+        {
+            // only strings map to JSON object keys; any other key is rejected
+            // with a message naming its type
+            const std::vector<std::pair<std::vector<std::uint8_t>, std::string>> cases =
+            {
+                {{0x81, 0xC0, 0x01}, "nil; last byte: 0xC0"},
+                {{0x81, 0xC2, 0x01}, "a boolean; last byte: 0xC2"},
+                {{0x81, 0xC3, 0x01}, "a boolean; last byte: 0xC3"},
+                {{0x81, 0xCA, 0x3F, 0x80, 0x00, 0x00, 0x01}, "a float; last byte: 0xCA"},
+                {{0x81, 0xCB, 0x3F, 0xF0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01}, "a float; last byte: 0xCB"},
+                {{0x81, 0xC4, 0x00, 0x01}, "a bin; last byte: 0xC4"},
+                {{0x81, 0xC5, 0x00, 0x00, 0x01}, "a bin; last byte: 0xC5"},
+                {{0x81, 0xC6, 0x00, 0x00, 0x00, 0x00, 0x01}, "a bin; last byte: 0xC6"},
+                {{0x81, 0xC7, 0x00, 0x01, 0x01}, "an ext; last byte: 0xC7"},
+                {{0x81, 0xC8, 0x00, 0x00, 0x01, 0x01}, "an ext; last byte: 0xC8"},
+                {{0x81, 0xC9, 0x00, 0x00, 0x00, 0x00, 0x01, 0x01}, "an ext; last byte: 0xC9"},
+                {{0x81, 0xD4, 0x01, 0x00, 0x01}, "an ext; last byte: 0xD4"},
+                {{0x81, 0xD5, 0x01, 0x00, 0x00, 0x01}, "an ext; last byte: 0xD5"},
+                {{0x81, 0xD6, 0x01, 0x00, 0x00, 0x00, 0x00, 0x01}, "an ext; last byte: 0xD6"},
+                {{0x81, 0xD7, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01}, "an ext; last byte: 0xD7"},
+                {{0x81, 0xD8, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01}, "an ext; last byte: 0xD8"},
+                {{0x81, 0xCC, 0x01, 0x01}, "an integer; last byte: 0xCC"},
+                {{0x81, 0xCD, 0x00, 0x01, 0x01}, "an integer; last byte: 0xCD"},
+                {{0x81, 0xCE, 0x00, 0x00, 0x00, 0x01, 0x01}, "an integer; last byte: 0xCE"},
+                {{0x81, 0xCF, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x01}, "an integer; last byte: 0xCF"},
+                {{0x81, 0xD0, 0x01, 0x01}, "an integer; last byte: 0xD0"},
+                {{0x81, 0xD1, 0x00, 0x01, 0x01}, "an integer; last byte: 0xD1"},
+                {{0x81, 0xD2, 0x00, 0x00, 0x00, 0x01, 0x01}, "an integer; last byte: 0xD2"},
+                {{0x81, 0xD3, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x01}, "an integer; last byte: 0xD3"},
+                {{0x81, 0x00, 0x01}, "an integer; last byte: 0x00"},
+                {{0x81, 0x7F, 0x01}, "an integer; last byte: 0x7F"},
+                {{0x81, 0xE0, 0x01}, "an integer; last byte: 0xE0"},
+                {{0x81, 0x80, 0x01}, "a map; last byte: 0x80"},
+                {{0x81, 0x8F, 0x01}, "a map; last byte: 0x8F"},
+                {{0x81, 0xDE, 0x00, 0x00, 0x01}, "a map; last byte: 0xDE"},
+                {{0x81, 0xDF, 0x00, 0x00, 0x00, 0x00, 0x01}, "a map; last byte: 0xDF"},
+                {{0x81, 0x90, 0x01}, "an array; last byte: 0x90"},
+                {{0x81, 0x9F, 0x01}, "an array; last byte: 0x9F"},
+                {{0x81, 0xDC, 0x00, 0x00, 0x01}, "an array; last byte: 0xDC"},
+                {{0x81, 0xDD, 0x00, 0x00, 0x00, 0x00, 0x01}, "an array; last byte: 0xDD"},
+            };
+
+            for (const auto& c : cases)
+            {
+                CAPTURE(c.first)
+                const std::string expected = "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing MessagePack object key: only string keys are supported, but found " + c.second;
+                json _;
+                CHECK_THROWS_WITH_AS(_ = json::from_msgpack(c.first), expected.c_str(), json::parse_error&);
+                CHECK(json::from_msgpack(c.first, true, false).is_discarded());
+            }
+
+            json _;
+            // the unused byte 0xC1 is still reported as a malformed string
+            CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0x81, 0xC1, 0x01})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing MessagePack string: expected length specification (0xA0-0xBF, 0xD9-0xDB); last byte: 0xC1", json::parse_error&);
+            // a missing key is still reported as the end of input
+            CHECK_THROWS_WITH_AS(_ = json::from_msgpack(std::vector<uint8_t>({0x81})), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing MessagePack string: unexpected end of input", json::parse_error&);
+        }
+
         SECTION("invalid UTF-8 in string (see #5529)")
         {
             // a fixstr of length 2 (0xA0 | 2) whose bytes are not valid UTF-8

tests/src/unit-regression1.cpp

@@ -1018,7 +1018,7 @@ TEST_CASE("regression tests 1")
         };
 
         json _;
-        CHECK_THROWS_WITH_AS(_ = json::from_cbor(vec), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR string: expected length specification (0x60-0x7B) or indefinite string type (0x7F); last byte: 0x98", json::parse_error&);
+        CHECK_THROWS_WITH_AS(_ = json::from_cbor(vec), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR object key: only string keys are supported, but found an array; last byte: 0x98", json::parse_error&);
 
         // related test case: nonempty UTF-8 string (indefinite length)
         std::vector<uint8_t> const vec1 {0x7f, 0x61, 0x61};
@@ -1065,7 +1065,7 @@ TEST_CASE("regression tests 1")
         };
 
         json _;
-        CHECK_THROWS_WITH_AS(_ = json::from_cbor(vec1), "[json.exception.parse_error.113] parse error at byte 13: syntax error while parsing CBOR string: expected length specification (0x60-0x7B) or indefinite string type (0x7F); last byte: 0xB4", json::parse_error&);
+        CHECK_THROWS_WITH_AS(_ = json::from_cbor(vec1), "[json.exception.parse_error.113] parse error at byte 13: syntax error while parsing CBOR object key: only string keys are supported, but found a map; last byte: 0xB4", json::parse_error&);
 
         // related test case: double-precision
         std::vector<uint8_t> const vec2
@@ -1077,7 +1077,7 @@ TEST_CASE("regression tests 1")
             0x96, 0x96, 0xb4, 0xb4, 0xfa, 0x94, 0x94, 0x61,
             0x61, 0x61, 0x61, 0x61, 0x61, 0x61, 0x61, 0xfb
         };
-        CHECK_THROWS_WITH_AS(_ = json::from_cbor(vec2), "[json.exception.parse_error.113] parse error at byte 13: syntax error while parsing CBOR string: expected length specification (0x60-0x7B) or indefinite string type (0x7F); last byte: 0xB4", json::parse_error&);
+        CHECK_THROWS_WITH_AS(_ = json::from_cbor(vec2), "[json.exception.parse_error.113] parse error at byte 13: syntax error while parsing CBOR object key: only string keys are supported, but found a map; last byte: 0xB4", json::parse_error&);
     }
 
     SECTION("issue #452 - Heap-buffer-overflow (OSS-Fuzz issue 585)")

tests/src/unit-regression2.cpp

@@ -766,15 +766,6 @@ TEST_CASE("regression tests 2")
         CHECK(j == k);
     }
 
-    SECTION("issue #4552 - UTF-8 invalid characters are not always ignored when dumping with error_handler_t::ignore")
-    {
-        json node;
-        node["test"] = "test\334\005";
-        CHECK(node.dump(-1, ' ', false, json::error_handler_t::ignore) == "{\"test\":\"test\\u0005\"}");
-        CHECK(node.dump(-1, ' ', false, json::error_handler_t::keep) == "{\"test\":\"test\334\\u0005\"}");
-        CHECK(node.dump(-1, ' ', true, json::error_handler_t::keep) == "{\"test\":\"test\334\\u0005\"}");
-    }
-
 }
 
 TEST_CASE("regression test - parser callback must not lose a duplicate key's prior value")

tests/src/unit-serialization.cpp

@@ -92,8 +92,6 @@ TEST_CASE("serialization")
             CHECK(j.dump(-1, ' ', false, json::error_handler_t::ignore) == "\"äü\"");
             CHECK(j.dump(-1, ' ', false, json::error_handler_t::replace) == "\"ä\xEF\xBF\xBDü\"");
             CHECK(j.dump(-1, ' ', true, json::error_handler_t::replace) == "\"\\u00e4\\ufffd\\u00fc\"");
-            CHECK(j.dump(-1, ' ', false, json::error_handler_t::keep) == "\"ä\xA9ü\"");
-            CHECK(j.dump(-1, ' ', true, json::error_handler_t::keep) == "\"\\u00e4\xA9\\u00fc\"");
         }
 
         SECTION("invalid character (regression guard for shared UTF-8 decoder, see #5529)")
@@ -116,8 +114,6 @@ TEST_CASE("serialization")
             CHECK(j.dump(-1, ' ', false, json::error_handler_t::ignore) == "\"123\"");
             CHECK(j.dump(-1, ' ', false, json::error_handler_t::replace) == "\"123\xEF\xBF\xBD\"");
             CHECK(j.dump(-1, ' ', true, json::error_handler_t::replace) == "\"123\\ufffd\"");
-            CHECK(j.dump(-1, ' ', false, json::error_handler_t::keep) == "\"123\xC2\"");
-            CHECK(j.dump(-1, ' ', true, json::error_handler_t::keep) == "\"123\xC2\"");
         }
 
         SECTION("unexpected character")
@@ -130,39 +126,6 @@ TEST_CASE("serialization")
             CHECK(j.dump(-1, ' ', false, json::error_handler_t::ignore) == "\"123456\"");
             CHECK(j.dump(-1, ' ', false, json::error_handler_t::replace) == "\"123\xEF\xBF\xBD\x34\x35\x36\"");
             CHECK(j.dump(-1, ' ', true, json::error_handler_t::replace) == "\"123\\ufffd456\"");
-            CHECK(j.dump(-1, ' ', false, json::error_handler_t::keep) == "\"123\xF1\xB0\x34\x35\x36\"");
-            CHECK(j.dump(-1, ' ', true, json::error_handler_t::keep) == "\"123\xF1\xB0\x34\x35\x36\"");
-        }
-
-        SECTION("keep: valid characters are still escaped")
-        {
-            // an invalid byte followed by characters that must be escaped
-            const json j = "\xC2\"\\\n\xFF\x05";
-            CHECK(j.dump(-1, ' ', false, json::error_handler_t::keep) == "\"\xC2\\\"\\\\\\n\xFF\\u0005\"");
-            CHECK(j.dump(-1, ' ', true, json::error_handler_t::keep) == "\"\xC2\\\"\\\\\\n\xFF\\u0005\"");
-        }
-
-        SECTION("keep: truncated multibyte sequences")
-        {
-            CHECK(json("\xF0\x9F\x98").dump(-1, ' ', false, json::error_handler_t::keep) == "\"\xF0\x9F\x98\"");
-            CHECK(json("\xF0\x9F\x98").dump(-1, ' ', true, json::error_handler_t::keep) == "\"\xF0\x9F\x98\"");
-            CHECK(json("\xF0\x9F\x98" "a").dump(-1, ' ', false, json::error_handler_t::keep) == "\"\xF0\x9F\x98" "a\"");
-            CHECK(json("\xF0\x9F\x98" "a").dump(-1, ' ', true, json::error_handler_t::keep) == "\"\xF0\x9F\x98" "a\"");
-        }
-
-        SECTION("keep: long string with many invalid bytes")
-        {
-            // exceeds the internal string buffer several times
-            std::string input;
-            std::string expected = "\"";
-            for (int i = 0; i < 2000; ++i)
-            {
-                input += "\xFF\xE2\x82\n\xC3\xA4";
-                expected += "\xFF\xE2\x82\\n\xC3\xA4";
-            }
-            expected += "\"";
-            const json j = input;
-            CHECK(j.dump(-1, ' ', false, json::error_handler_t::keep) == expected);
         }
 
         SECTION("U+FFFD Substitution of Maximal Subparts")

tests/src/unit-unicode2.cpp

@@ -14,7 +14,6 @@
 #include <nlohmann/json.hpp>
 using nlohmann::json;
 
-#include <algorithm>
 #include <fstream>
 #include <sstream>
 #include <iostream>
@@ -76,11 +75,8 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
     static std::string s_replaced2;
     static std::string s_replaced_ascii;
     static std::string s_replaced2_ascii;
-    static std::string s_kept;
-    static std::string s_kept2;
-    static std::string s_kept_ascii;
 
-    // dumping with ignore/replace/keep must not throw in any case
+    // dumping with ignore/replace must not throw in any case
     s_ignored = j.dump(-1, ' ', false, json::error_handler_t::ignore);
     s_ignored2 = j2.dump(-1, ' ', false, json::error_handler_t::ignore);
     s_ignored_ascii = j.dump(-1, ' ', true, json::error_handler_t::ignore);
@@ -89,9 +85,6 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
     s_replaced2 = j2.dump(-1, ' ', false, json::error_handler_t::replace);
     s_replaced_ascii = j.dump(-1, ' ', true, json::error_handler_t::replace);
     s_replaced2_ascii = j2.dump(-1, ' ', true, json::error_handler_t::replace);
-    s_kept = j.dump(-1, ' ', false, json::error_handler_t::keep);
-    s_kept2 = j2.dump(-1, ' ', false, json::error_handler_t::keep);
-    s_kept_ascii = j.dump(-1, ' ', true, json::error_handler_t::keep);
 
     if (success_expected)
     {
@@ -101,7 +94,6 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
         // all dumps should agree on the string
         CHECK(s_strict == s_ignored);
         CHECK(s_strict == s_replaced);
-        CHECK(s_strict == s_kept);
     }
     else
     {
@@ -113,20 +105,6 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
 
         // check that replace string contains a replacement character
         CHECK(s_replaced.find("\xEF\xBF\xBD") != std::string::npos);
-
-        // ignore drops the invalid bytes, keep copies them
-        CHECK(s_ignored != s_kept);
-        CHECK(s_ignored_ascii != s_kept_ascii);
-
-        // unless a byte needs escaping, keep copies the input unchanged
-        const bool needs_escaping = std::any_of(json_string.begin(), json_string.end(), [](char c)
-        {
-            return static_cast<unsigned char>(c) < 0x20 || c == '"' || c == '\\';
-        });
-        if (!needs_escaping)
-        {
-            CHECK(s_kept == "\"" + json_string + "\"");
-        }
     }
 
     // check that prefix and suffix are preserved
@@ -138,8 +116,6 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
     CHECK(s_replaced2.substr(s_replaced2.size() - 4, 3) == "xyz");
     CHECK(s_replaced2_ascii.substr(1, 3) == "abc");
     CHECK(s_replaced2_ascii.substr(s_replaced2_ascii.size() - 4, 3) == "xyz");
-    CHECK(s_kept2.substr(1, 3) == "abc");
-    CHECK(s_kept2.substr(s_kept2.size() - 4, 3) == "xyz");
 }
 
 void check_utf8string(bool success_expected, int byte1, int byte2, int byte3, int byte4);

tests/src/unit-unicode3.cpp

@@ -14,7 +14,6 @@
 #include <nlohmann/json.hpp>
 using nlohmann::json;
 
-#include <algorithm>
 #include <fstream>
 #include <sstream>
 #include <iostream>
@@ -76,11 +75,8 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
     static std::string s_replaced2;
     static std::string s_replaced_ascii;
     static std::string s_replaced2_ascii;
-    static std::string s_kept;
-    static std::string s_kept2;
-    static std::string s_kept_ascii;
 
-    // dumping with ignore/replace/keep must not throw in any case
+    // dumping with ignore/replace must not throw in any case
     s_ignored = j.dump(-1, ' ', false, json::error_handler_t::ignore);
     s_ignored2 = j2.dump(-1, ' ', false, json::error_handler_t::ignore);
     s_ignored_ascii = j.dump(-1, ' ', true, json::error_handler_t::ignore);
@@ -89,9 +85,6 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
     s_replaced2 = j2.dump(-1, ' ', false, json::error_handler_t::replace);
     s_replaced_ascii = j.dump(-1, ' ', true, json::error_handler_t::replace);
     s_replaced2_ascii = j2.dump(-1, ' ', true, json::error_handler_t::replace);
-    s_kept = j.dump(-1, ' ', false, json::error_handler_t::keep);
-    s_kept2 = j2.dump(-1, ' ', false, json::error_handler_t::keep);
-    s_kept_ascii = j.dump(-1, ' ', true, json::error_handler_t::keep);
 
     if (success_expected)
     {
@@ -101,7 +94,6 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
         // all dumps should agree on the string
         CHECK(s_strict == s_ignored);
         CHECK(s_strict == s_replaced);
-        CHECK(s_strict == s_kept);
     }
     else
     {
@@ -113,20 +105,6 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
 
         // check that replace string contains a replacement character
         CHECK(s_replaced.find("\xEF\xBF\xBD") != std::string::npos);
-
-        // ignore drops the invalid bytes, keep copies them
-        CHECK(s_ignored != s_kept);
-        CHECK(s_ignored_ascii != s_kept_ascii);
-
-        // unless a byte needs escaping, keep copies the input unchanged
-        const bool needs_escaping = std::any_of(json_string.begin(), json_string.end(), [](char c)
-        {
-            return static_cast<unsigned char>(c) < 0x20 || c == '"' || c == '\\';
-        });
-        if (!needs_escaping)
-        {
-            CHECK(s_kept == "\"" + json_string + "\"");
-        }
     }
 
     // check that prefix and suffix are preserved
@@ -138,8 +116,6 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
     CHECK(s_replaced2.substr(s_replaced2.size() - 4, 3) == "xyz");
     CHECK(s_replaced2_ascii.substr(1, 3) == "abc");
     CHECK(s_replaced2_ascii.substr(s_replaced2_ascii.size() - 4, 3) == "xyz");
-    CHECK(s_kept2.substr(1, 3) == "abc");
-    CHECK(s_kept2.substr(s_kept2.size() - 4, 3) == "xyz");
 }
 
 void check_utf8string(bool success_expected, int byte1, int byte2, int byte3, int byte4);

tests/src/unit-unicode4.cpp

@@ -14,7 +14,6 @@
 #include <nlohmann/json.hpp>
 using nlohmann::json;
 
-#include <algorithm>
 #include <fstream>
 #include <sstream>
 #include <iostream>
@@ -76,11 +75,8 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
     static std::string s_replaced2;
     static std::string s_replaced_ascii;
     static std::string s_replaced2_ascii;
-    static std::string s_kept;
-    static std::string s_kept2;
-    static std::string s_kept_ascii;
 
-    // dumping with ignore/replace/keep must not throw in any case
+    // dumping with ignore/replace must not throw in any case
     s_ignored = j.dump(-1, ' ', false, json::error_handler_t::ignore);
     s_ignored2 = j2.dump(-1, ' ', false, json::error_handler_t::ignore);
     s_ignored_ascii = j.dump(-1, ' ', true, json::error_handler_t::ignore);
@@ -89,9 +85,6 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
     s_replaced2 = j2.dump(-1, ' ', false, json::error_handler_t::replace);
     s_replaced_ascii = j.dump(-1, ' ', true, json::error_handler_t::replace);
     s_replaced2_ascii = j2.dump(-1, ' ', true, json::error_handler_t::replace);
-    s_kept = j.dump(-1, ' ', false, json::error_handler_t::keep);
-    s_kept2 = j2.dump(-1, ' ', false, json::error_handler_t::keep);
-    s_kept_ascii = j.dump(-1, ' ', true, json::error_handler_t::keep);
 
     if (success_expected)
     {
@@ -101,7 +94,6 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
         // all dumps should agree on the string
         CHECK(s_strict == s_ignored);
         CHECK(s_strict == s_replaced);
-        CHECK(s_strict == s_kept);
     }
     else
     {
@@ -113,20 +105,6 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
 
         // check that replace string contains a replacement character
         CHECK(s_replaced.find("\xEF\xBF\xBD") != std::string::npos);
-
-        // ignore drops the invalid bytes, keep copies them
-        CHECK(s_ignored != s_kept);
-        CHECK(s_ignored_ascii != s_kept_ascii);
-
-        // unless a byte needs escaping, keep copies the input unchanged
-        const bool needs_escaping = std::any_of(json_string.begin(), json_string.end(), [](char c)
-        {
-            return static_cast<unsigned char>(c) < 0x20 || c == '"' || c == '\\';
-        });
-        if (!needs_escaping)
-        {
-            CHECK(s_kept == "\"" + json_string + "\"");
-        }
     }
 
     // check that prefix and suffix are preserved
@@ -138,8 +116,6 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
     CHECK(s_replaced2.substr(s_replaced2.size() - 4, 3) == "xyz");
     CHECK(s_replaced2_ascii.substr(1, 3) == "abc");
     CHECK(s_replaced2_ascii.substr(s_replaced2_ascii.size() - 4, 3) == "xyz");
-    CHECK(s_kept2.substr(1, 3) == "abc");
-    CHECK(s_kept2.substr(s_kept2.size() - 4, 3) == "xyz");
 }
 
 void check_utf8string(bool success_expected, int byte1, int byte2, int byte3, int byte4);

tests/src/unit-unicode5.cpp

@@ -14,7 +14,6 @@
 #include <nlohmann/json.hpp>
 using nlohmann::json;
 
-#include <algorithm>
 #include <fstream>
 #include <sstream>
 #include <iostream>
@@ -76,11 +75,8 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
     static std::string s_replaced2;
     static std::string s_replaced_ascii;
     static std::string s_replaced2_ascii;
-    static std::string s_kept;
-    static std::string s_kept2;
-    static std::string s_kept_ascii;
 
-    // dumping with ignore/replace/keep must not throw in any case
+    // dumping with ignore/replace must not throw in any case
     s_ignored = j.dump(-1, ' ', false, json::error_handler_t::ignore);
     s_ignored2 = j2.dump(-1, ' ', false, json::error_handler_t::ignore);
     s_ignored_ascii = j.dump(-1, ' ', true, json::error_handler_t::ignore);
@@ -89,9 +85,6 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
     s_replaced2 = j2.dump(-1, ' ', false, json::error_handler_t::replace);
     s_replaced_ascii = j.dump(-1, ' ', true, json::error_handler_t::replace);
     s_replaced2_ascii = j2.dump(-1, ' ', true, json::error_handler_t::replace);
-    s_kept = j.dump(-1, ' ', false, json::error_handler_t::keep);
-    s_kept2 = j2.dump(-1, ' ', false, json::error_handler_t::keep);
-    s_kept_ascii = j.dump(-1, ' ', true, json::error_handler_t::keep);
 
     if (success_expected)
     {
@@ -101,7 +94,6 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
         // all dumps should agree on the string
         CHECK(s_strict == s_ignored);
         CHECK(s_strict == s_replaced);
-        CHECK(s_strict == s_kept);
     }
     else
     {
@@ -113,20 +105,6 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
 
         // check that replace string contains a replacement character
         CHECK(s_replaced.find("\xEF\xBF\xBD") != std::string::npos);
-
-        // ignore drops the invalid bytes, keep copies them
-        CHECK(s_ignored != s_kept);
-        CHECK(s_ignored_ascii != s_kept_ascii);
-
-        // unless a byte needs escaping, keep copies the input unchanged
-        const bool needs_escaping = std::any_of(json_string.begin(), json_string.end(), [](char c)
-        {
-            return static_cast<unsigned char>(c) < 0x20 || c == '"' || c == '\\';
-        });
-        if (!needs_escaping)
-        {
-            CHECK(s_kept == "\"" + json_string + "\"");
-        }
     }
 
     // check that prefix and suffix are preserved
@@ -138,8 +116,6 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
     CHECK(s_replaced2.substr(s_replaced2.size() - 4, 3) == "xyz");
     CHECK(s_replaced2_ascii.substr(1, 3) == "abc");
     CHECK(s_replaced2_ascii.substr(s_replaced2_ascii.size() - 4, 3) == "xyz");
-    CHECK(s_kept2.substr(1, 3) == "abc");
-    CHECK(s_kept2.substr(s_kept2.size() - 4, 3) == "xyz");
 }
 
 void check_utf8string(bool success_expected, int byte1, int byte2, int byte3, int byte4);