Handle numbers that do not fit narrow number types in the binary readers

With custom number types narrower than the values in a binary document,
for example basic_json<..., std::int32_t, std::uint32_t, float>, every
binary reader (CBOR, MessagePack, UBJSON, BJData, BSON, BON8) passed the
decoded number to the SAX interface with an implicit conversion: the
integer 5000000000 silently became 705032704, and a finite double such as
1e300 became infinity. The lexer handles the same values in JSON text: an
integer that fits neither integer type is stored as number_float_t, and a
finite number that overflows number_float_t is rejected with
out_of_range.406.

Pass every number read from binary input through three helpers that
apply the lexer's rules:
- emit_signed(): number_integer_t, else number_unsigned_t for a
  non-negative value, else number_float_t
- emit_unsigned(): number_unsigned_t, else number_float_t
- emit_float(): out_of_range.406 if a finite value overflows
  number_float_t; infinity and NaN are passed on

For consistency, a CBOR negative integer below the range of
number_integer_t is now stored as number_float_t, like a too small
integer in JSON text, instead of being rejected with parse_error.112.
With the default number types, this is the only change in behavior.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/number_float_t.md

@@ -55,6 +55,10 @@ This implementation does exactly follow this approach, as it uses double precisi
 smaller than `-1.79769313486232e+308` and values greater than `1.79769313486232e+308` will be stored as NaN internally
 and be serialized to `null`.
 
+During deserialization (from JSON text or any of the binary formats), a finite number that does not fit into
+`number_float_t` is rejected with [`out_of_range.406`](../../home/exceptions.md#jsonexceptionout_of_range406), for
+example a double-precision number in a binary format when `number_float_t` is `#!cpp float`.
+
 #### Storage
 
 Floating-point number values are stored directly inside a `basic_json` type.

docs/mkdocs/docs/api/basic_json/number_integer_t.md

@@ -47,8 +47,9 @@ With the default values for `NumberIntegerType` (`std::int64_t`), the default va
 
 When the default type is used, the maximal integer number that can be stored is `9223372036854775807` (INT64_MAX) and
 the minimal integer number that can be stored is `-9223372036854775808` (INT64_MIN). Integer numbers that are out of
-range will yield over/underflow when used in a constructor. During deserialization, too large or small integer numbers
+range will yield over/underflow when used in a constructor. During deserialization (from JSON text or any of the binary
-will automatically be stored as [`number_unsigned_t`](number_unsigned_t.md) or [`number_float_t`](number_float_t.md).
+formats), too large or small integer numbers will automatically be stored as [`number_unsigned_t`](number_unsigned_t.md)
+or [`number_float_t`](number_float_t.md).
 
 [RFC 8259](https://tools.ietf.org/html/rfc8259) further states:
 > Note that when such software is used, numbers that are integers and are in the range $[-2^{53}+1, 2^{53}-1]$ are

docs/mkdocs/docs/api/basic_json/number_unsigned_t.md

@@ -48,8 +48,9 @@ With the default values for `NumberUnsignedType` (`std::uint64_t`), the default
 
 When the default type is used, the maximal integer number that can be stored is `18446744073709551615` (UINT64_MAX) and
 the minimal integer number that can be stored is `0`. Integer numbers that are out of range will yield over/underflow
-when used in a constructor. During deserialization, too large or small integer numbers will automatically be stored
+when used in a constructor. During deserialization (from JSON text or any of the binary formats), too large or small
-as [`number_integer_t`](number_integer_t.md) or [`number_float_t`](number_float_t.md).
+integer numbers will automatically be stored as [`number_integer_t`](number_integer_t.md) or
+[`number_float_t`](number_float_t.md).
 
 [RFC 8259](https://tools.ietf.org/html/rfc8259) further states:
 > Note that when such software is used, numbers that are integers and are in the range $[-2^{53}+1, 2^{53}-1]$ are

docs/mkdocs/docs/features/binary_formats/cbor.md

@@ -168,9 +168,9 @@ The library maps CBOR types to JSON value types as follows:
 !!! warning "Negative integer overflow"
 
     CBOR negative integers (major type 1) are decoded as `-1 - n`. If the encoded magnitude `n` is too large for the
-    result to fit into `number_integer_t` (`std::int64_t` by default), parsing fails with a
+    result to fit into `number_integer_t` (`std::int64_t` by default), the result is stored as `number_float_t`, like
-    [`parse_error.112`](../../home/exceptions.md#jsonexceptionparse_error112) exception rather than overflowing
+    a too small integer in JSON text. For example, `-18446744073709551616` (`0x3B` followed by eight `0xFF` bytes) is
-    silently.
+    stored as `-1.8446744073709552e+19`.
 
 !!! warning "Object keys"
 

docs/mkdocs/docs/home/exceptions.md

@@ -331,9 +331,6 @@ An unexpected byte was read in a [binary format](../features/binary_formats/inde
     [json.exception.parse_error.112] parse error at byte 15: syntax error while parsing BSON binary: byte array length cannot be negative, is -1
     ```
     ```
-    [json.exception.parse_error.112] parse error at byte 9: syntax error while parsing CBOR value: negative integer overflow
-    ```
-    ```
     [json.exception.parse_error.112] parse error at byte 5: syntax error while parsing BSON document: document size 6 does not match the number of bytes read (5)
     ```
 
@@ -847,13 +844,18 @@ The JSON Patch operations 'remove' and 'add' cannot be applied to the root eleme
 
 ### json.exception.out_of_range.406
 
-A parsed number could not be stored as without changing it to NaN or INF.
+A parsed number could not be stored without changing it to NaN or INF. For the binary formats, this happens when a
+finite floating-point number does not fit into [`number_float_t`](../api/basic_json/number_float_t.md), for example a
+double-precision number when `number_float_t` is `#!cpp float`.
 
-!!! failure "Example message"
+!!! failure "Example messages"
 
     ```
     number overflow parsing '10E1000'
     ```
+    ```
+    [json.exception.out_of_range.406] syntax error while parsing CBOR value: number overflow
+    ```
 
 ### json.exception.out_of_range.407
 

include/nlohmann/detail/input/binary_reader.hpp

@@ -559,7 +559,7 @@ class binary_reader
             case 0x01: // double
             {
                 double number{};
-                return get_number<double, true>(input_format_t::bson, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number<double, true>(input_format_t::bson, number) && emit_float(input_format_t::bson, number);
             }
 
             case 0x02: // string
@@ -600,19 +600,19 @@ class binary_reader
             case 0x10: // int32
             {
                 std::int32_t value{};
-                return get_number<std::int32_t, true>(input_format_t::bson, value) && sax->number_integer(value);
+                return get_number<std::int32_t, true>(input_format_t::bson, value) && emit_signed(value);
             }
 
             case 0x12: // int64
             {
                 std::int64_t value{};
-                return get_number<std::int64_t, true>(input_format_t::bson, value) && sax->number_integer(value);
+                return get_number<std::int64_t, true>(input_format_t::bson, value) && emit_signed(value);
             }
 
             case 0x11: // uint64
             {
                 std::uint64_t value{};
-                return get_number<std::uint64_t, true>(input_format_t::bson, value) && sax->number_unsigned(value);
+                return get_number<std::uint64_t, true>(input_format_t::bson, value) && emit_unsigned(value);
             }
 
             default: // anything else is not supported (yet)
@@ -638,14 +638,17 @@ class binary_reader
         {
             return false;
         }
-        const auto max_val = static_cast<NumberType>((std::numeric_limits<number_integer_t>::max)());
+
-        if (number > max_val)
+        // the value is -1 - number, which fits into number_integer_t
+        // whenever number does
+        if (JSON_HEDLEY_LIKELY(value_in_range_of<number_integer_t>(number)))
         {
-            return sax->parse_error(chars_read, get_token_string(),
+            return sax->number_integer(static_cast<number_integer_t>(-1) - static_cast<number_integer_t>(number));
-                                    parse_error::create(112, chars_read,
-                                            exception_message(input_format_t::cbor, "negative integer overflow", "value"), nullptr));
         }
-        return sax->number_integer(static_cast<number_integer_t>(-1) - static_cast<number_integer_t>(number));
+
+        // like the lexer does for JSON text, store a value too small for
+        // number_integer_t as number_float_t
+        return sax->number_float(static_cast<number_float_t>(-1) - static_cast<number_float_t>(number), "");
     }
 
     /*!
@@ -702,25 +705,25 @@ class binary_reader
             case 0x18: // Unsigned integer (one-byte uint8_t follows)
             {
                 std::uint8_t number{};
-                return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::cbor, number) && emit_unsigned(number);
             }
 
             case 0x19: // Unsigned integer (two-byte uint16_t follows)
             {
                 std::uint16_t number{};
-                return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::cbor, number) && emit_unsigned(number);
             }
 
             case 0x1A: // Unsigned integer (four-byte uint32_t follows)
             {
                 std::uint32_t number{};
-                return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::cbor, number) && emit_unsigned(number);
             }
 
             case 0x1B: // Unsigned integer (eight-byte uint64_t follows)
             {
                 std::uint64_t number{};
-                return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::cbor, number) && emit_unsigned(number);
             }
 
             // Negative integer -1-0x00..-1-0x17 (-1..-24)
@@ -1165,13 +1168,13 @@ class binary_reader
             case 0xFA: // Single-Precision Float (four-byte IEEE 754)
             {
                 float number{};
-                return get_number(input_format_t::cbor, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format_t::cbor, number) && emit_float(input_format_t::cbor, number);
             }
 
             case 0xFB: // Double-Precision Float (eight-byte IEEE 754)
             {
                 double number{};
-                return get_number(input_format_t::cbor, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format_t::cbor, number) && emit_float(input_format_t::cbor, number);
             }
 
             default: // anything else (0xFF is handled inside the other types)
@@ -1861,61 +1864,61 @@ class binary_reader
             case 0xCA: // float 32
             {
                 float number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format_t::msgpack, number) && emit_float(input_format_t::msgpack, number);
             }
 
             case 0xCB: // float 64
             {
                 double number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format_t::msgpack, number) && emit_float(input_format_t::msgpack, number);
             }
 
             case 0xCC: // uint 8
             {
                 std::uint8_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::msgpack, number) && emit_unsigned(number);
             }
 
             case 0xCD: // uint 16
             {
                 std::uint16_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::msgpack, number) && emit_unsigned(number);
             }
 
             case 0xCE: // uint 32
             {
                 std::uint32_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::msgpack, number) && emit_unsigned(number);
             }
 
             case 0xCF: // uint 64
             {
                 std::uint64_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::msgpack, number) && emit_unsigned(number);
             }
 
             case 0xD0: // int 8
             {
                 std::int8_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
+                return get_number(input_format_t::msgpack, number) && emit_signed(number);
             }
 
             case 0xD1: // int 16
             {
                 std::int16_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
+                return get_number(input_format_t::msgpack, number) && emit_signed(number);
             }
 
             case 0xD2: // int 32
             {
                 std::int32_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
+                return get_number(input_format_t::msgpack, number) && emit_signed(number);
             }
 
             case 0xD3: // int 64
             {
                 std::int64_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
+                return get_number(input_format_t::msgpack, number) && emit_signed(number);
             }
 
             case 0xDC: // array 16
@@ -2756,7 +2759,7 @@ class binary_reader
                         {
                             return sax->parse_error(chars_read, get_token_string(), out_of_range::create(408, exception_message(input_format, "excessive ndarray size caused overflow", "size"), nullptr));
                         }
-                        if (JSON_HEDLEY_UNLIKELY(!sax->number_unsigned(static_cast<number_unsigned_t>(i))))
+                        if (JSON_HEDLEY_UNLIKELY(!emit_unsigned(i)))
                         {
                             return false;
                         }
@@ -2888,37 +2891,37 @@ class binary_reader
                     break;
                 }
                 std::uint8_t number{};
-                return get_number(input_format, number) && sax->number_unsigned(number);
+                return get_number(input_format, number) && emit_unsigned(number);
             }
 
             case 'U':
             {
                 std::uint8_t number{};
-                return get_number(input_format, number) && sax->number_unsigned(number);
+                return get_number(input_format, number) && emit_unsigned(number);
             }
 
             case 'i':
             {
                 std::int8_t number{};
-                return get_number(input_format, number) && sax->number_integer(number);
+                return get_number(input_format, number) && emit_signed(number);
             }
 
             case 'I':
             {
                 std::int16_t number{};
-                return get_number(input_format, number) && sax->number_integer(number);
+                return get_number(input_format, number) && emit_signed(number);
             }
 
             case 'l':
             {
                 std::int32_t number{};
-                return get_number(input_format, number) && sax->number_integer(number);
+                return get_number(input_format, number) && emit_signed(number);
             }
 
             case 'L':
             {
                 std::int64_t number{};
-                return get_number(input_format, number) && sax->number_integer(number);
+                return get_number(input_format, number) && emit_signed(number);
             }
 
             case 'u':
@@ -2928,7 +2931,7 @@ class binary_reader
                     break;
                 }
                 std::uint16_t number{};
-                return get_number(input_format, number) && sax->number_unsigned(number);
+                return get_number(input_format, number) && emit_unsigned(number);
             }
 
             case 'm':
@@ -2938,7 +2941,7 @@ class binary_reader
                     break;
                 }
                 std::uint32_t number{};
-                return get_number(input_format, number) && sax->number_unsigned(number);
+                return get_number(input_format, number) && emit_unsigned(number);
             }
 
             case 'M':
@@ -2948,7 +2951,7 @@ class binary_reader
                     break;
                 }
                 std::uint64_t number{};
-                return get_number(input_format, number) && sax->number_unsigned(number);
+                return get_number(input_format, number) && emit_unsigned(number);
             }
 
             case 'h':
@@ -3006,13 +3009,13 @@ class binary_reader
             case 'd':
             {
                 float number{};
-                return get_number(input_format, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format, number) && emit_float(input_format, number);
             }
 
             case 'D':
             {
                 double number{};
-                return get_number(input_format, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format, number) && emit_float(input_format, number);
             }
 
             case 'H':
@@ -3479,13 +3482,13 @@ class binary_reader
             case 0x8E: // binary32
             {
                 float number{};
-                return get_number(input_format_t::bon8, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format_t::bon8, number) && emit_float(input_format_t::bon8, number);
             }
 
             case 0x8F: // binary64
             {
                 double number{};
-                return get_number(input_format_t::bon8, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format_t::bon8, number) && emit_float(input_format_t::bon8, number);
             }
 
             case 0xF8:
@@ -3551,7 +3554,9 @@ class binary_reader
     @brief pass an integer to the SAX parser
 
     Non-negative integers are passed as unsigned, negative integers as signed
-    numbers, like the other binary formats do.
+    numbers, like the other binary formats do. A value that does not fit the
+    number type is passed as described for @ref emit_unsigned and
+    @ref emit_signed.
 
     @param[in] number  the integer
     @return whether the SAX parser accepted the value
@@ -3560,9 +3565,9 @@ class binary_reader
     {
         if (number >= 0)
         {
-            return sax->number_unsigned(static_cast<number_unsigned_t>(number));
+            return emit_unsigned(static_cast<std::uint64_t>(number));
         }
-        return sax->number_integer(static_cast<number_integer_t>(number));
+        return emit_signed(number);
     }
 
     /*!
@@ -3619,8 +3624,7 @@ class binary_reader
             value = (value << 8) | static_cast<std::int64_t>(current);
         }
 
-        return negative ? sax->number_integer(static_cast<number_integer_t>(-(value + offset)))
+        return emit_bon8_integer(negative ? -(value + offset) : value + offset);
-               : sax->number_unsigned(static_cast<number_unsigned_t>(value + offset));
     }
 
     /*!
@@ -3917,6 +3921,79 @@ class binary_reader
         return true;
     }
 
+    /*!
+    @brief pass a signed integer read from the input to the SAX parser
+
+    Like the lexer does for JSON text, a value that does not fit into
+    number_integer_t is passed as number_unsigned_t if it is non-negative and
+    fits there, and as number_float_t otherwise. With the default number
+    types, every integer the binary formats can encode fits, so this only
+    matters for narrower custom number types.
+
+    @tparam NumberType a signed integer type
+    @param[in] number  the integer
+    @return whether the SAX parser accepted the value
+    */
+    template<typename NumberType>
+    bool emit_signed(const NumberType number)
+    {
+        if (JSON_HEDLEY_LIKELY(value_in_range_of<number_integer_t>(number)))
+        {
+            return sax->number_integer(static_cast<number_integer_t>(number));
+        }
+        if (value_in_range_of<number_unsigned_t>(number))
+        {
+            return sax->number_unsigned(static_cast<number_unsigned_t>(number));
+        }
+        return sax->number_float(static_cast<number_float_t>(number), "");
+    }
+
+    /*!
+    @brief pass an unsigned integer read from the input to the SAX parser
+
+    Like the lexer does for JSON text, a value that does not fit into
+    number_unsigned_t is passed as number_float_t.
+
+    @tparam NumberType an unsigned integer type
+    @param[in] number  the integer
+    @return whether the SAX parser accepted the value
+    */
+    template<typename NumberType>
+    bool emit_unsigned(const NumberType number)
+    {
+        if (JSON_HEDLEY_LIKELY(value_in_range_of<number_unsigned_t>(number)))
+        {
+            return sax->number_unsigned(static_cast<number_unsigned_t>(number));
+        }
+        return sax->number_float(static_cast<number_float_t>(number), "");
+    }
+
+    /*!
+    @brief pass a floating-point number read from the input to the SAX parser
+
+    Like the lexer does for JSON text, a finite value that overflows
+    number_float_t is rejected instead of silently becoming infinity. Infinity
+    and NaN in the input are passed on unchanged.
+
+    @tparam NumberType a floating-point type
+    @param[in] format  the current format (for diagnostics)
+    @param[in] number  the number
+    @return whether the SAX parser accepted the value
+
+    @throw out_of_range.406 if a finite @a number overflows number_float_t
+    */
+    template<typename NumberType>
+    bool emit_float(const input_format_t format, const NumberType number)
+    {
+        const auto result = static_cast<number_float_t>(number);
+        if (JSON_HEDLEY_UNLIKELY(std::isfinite(number) && !std::isfinite(result)))
+        {
+            return sax->parse_error(chars_read, get_token_string(),
+                                    out_of_range::create(406, exception_message(format, "number overflow", "value"), nullptr));
+        }
+        return sax->number_float(result, "");
+    }
+
     /*!
     @brief create a string by reading characters from the input
 

include/nlohmann/detail/input/lexer.hpp

@@ -206,6 +206,7 @@ class lexer : public lexer_base<BasicJsonType>
     explicit lexer(InputAdapterType&& adapter, bool ignore_comments_ = false, bool discard_number_values_ = false) noexcept
         : ia(std::move(adapter))
         , ignore_comments(ignore_comments_)
+        , decimal_point_char(static_cast<char_int_type>(get_decimal_point()))
         , discard_number_values(discard_number_values_)
     {}
 
@@ -221,7 +222,8 @@ class lexer : public lexer_base<BasicJsonType>
     // locales
     /////////////////////
 
-    /// return the decimal point of the current locale
+    /// return the locale-dependent decimal point
+    JSON_HEDLEY_PURE
     static char get_decimal_point() noexcept
     {
         const auto* loc = localeconv();
@@ -1090,10 +1092,9 @@ class lexer : public lexer_base<BasicJsonType>
             token_type::value_float if number could be successfully scanned,
             token_type::parse_error otherwise
 
-    @note The scanner is independent of the current locale: token_buffer
+    @note The scanner is independent of the current locale. Internally, the
-          always holds `.`. Only the std::strtod fallback of convert_number()
+          locale's decimal point is used instead of `.` to work with the
-          depends on the locale, and it looks up the decimal point right
+          locale-dependent converters.
-          before converting (see convert_float_locale_aware()).
     */
     token_type scan_number()  // lgtm [cpp/use-of-goto] `goto` is used in this function to implement the number-parsing state machine described above. By design, any finite input will eventually reach the "done" state or return token_type::parse_error. In each intermediate state, 1 byte of the input is appended to the token_buffer vector, and only the already initialized variables token_buffer, number_type, and error_message are manipulated.
     {
@@ -1182,7 +1183,7 @@ scan_number_zero:
         {
             case '.':
             {
-                add(current);
+                add(decimal_point_char);
                 decimal_point_position = token_buffer.size() - 1;
                 goto scan_number_decimal1;
             }
@@ -1219,7 +1220,7 @@ scan_number_any1:
 
             case '.':
             {
-                add(current);
+                add(decimal_point_char);
                 decimal_point_position = token_buffer.size() - 1;
                 goto scan_number_decimal1;
             }
@@ -1461,9 +1462,9 @@ scan_number_done:
 
         // Only a number below 1 can carry further insignificant zeros, and only
         // while the count stays at the limit does removing them change the
-        // answer - so this loop is skipped for all but a few tokens. The
+        // answer - so this loop is skipped for all but a few tokens. Note
-        // fraction is located through decimal_point_position rather than by
+        // token_buffer holds the locale's decimal point, so the fraction is
-        // searching '.'.
+        // located through decimal_point_position rather than by searching '.'.
         if (lead_zero != 0)
         {
             JSON_ASSERT(has_dot != 0); // an integer "0" cannot reach the limit
@@ -1481,8 +1482,8 @@ scan_number_done:
     @brief convert the number text in token_buffer to its value and token type
 
     The digit sequence in token_buffer has already been validated (by the
-    scan_number() state machine or by the contiguous fast path) and holds '.'
+    scan_number() state machine or by the contiguous fast path) and holds the
-    as decimal point, independent of the locale. Integers are parsed first and fall
+    locale decimal point in place of '.'. Integers are parsed first and fall
     back to floating point on overflow. This is shared so both scanners produce
     identical results.
 
@@ -1562,7 +1563,7 @@ scan_number_done:
         // integer conversion above overflowed. Prefer std::from_chars
         // (Eisel-Lemire, locale-independent, correctly rounded) when available;
         // otherwise the exact Clinger fast path (double only); otherwise the
-        // locale-aware strtof/strtod/strtold.
+        // locale-aware strtof/strtod.
         if (parse_float_from_chars(num_begin, num_end, value_float))
         {
             return token_type::value_float;
@@ -1571,75 +1572,26 @@ scan_number_done:
         // extra pass over the token's bytes, which otherwise shows up on
         // high-precision inputs such as canada.json
         if (mantissa_fits_clinger(mantissa_end)
-                && parse_float_fast(num_begin, num_end, value_float))
+                && parse_float_fast(num_begin, num_end, decimal_point_char, value_float))
         {
             return token_type::value_float;
         }
 
-        convert_float_locale_aware();
+        char* endptr = nullptr; // NOLINT(misc-const-correctness,cppcoreguidelines-pro-type-vararg,hicpp-vararg)
+        strtof(value_float, token_buffer.data(), &endptr);
+
+        // we checked the number format before
+        JSON_ASSERT(endptr == token_buffer.data() + token_buffer.size());
+
         return token_type::value_float;
     }
 
-    /*!
-    @brief convert the float in token_buffer with strtof/strtod/strtold
-
-    These functions expect the decimal point of the *current* locale, so it is
-    looked up right before the conversion instead of once when the lexer is
-    constructed: a locale change in between (by a parser callback, a SAX
-    handler, or another thread) must not truncate the value (#5198). The
-    token has been validated before, so if the conversion stops early and the
-    decimal point changed in the meantime, the locale changed between the
-    lookup and the call, and the conversion is repeated with the new decimal
-    point. If the decimal point did not change, a retry cannot succeed: the
-    locale's decimal point is not a single character (e.g., the two-byte
-    U+066B of ar_EG.UTF-8 or fa_IR.UTF-8) and cannot be substituted in place.
-    The value strtod parsed up to that point is kept, as before this change.
-
-    Note that changing the locale in another thread *while* strtod runs is
-    undefined behavior of the C library, which this function cannot prevent.
-    */
-    void convert_float_locale_aware()
-    {
-        const bool has_dot = decimal_point_position != std::string::npos;
-        char decimal_point = get_decimal_point();
-        for (;;)
-        {
-            const bool substitute = has_dot && decimal_point != '.';
-            if (substitute)
-            {
-                token_buffer[decimal_point_position] = static_cast<typename string_t::value_type>(decimal_point);
-            }
-
-            char* endptr = nullptr; // NOLINT(misc-const-correctness,cppcoreguidelines-pro-type-vararg,hicpp-vararg)
-            strtof(value_float, token_buffer.data(), &endptr);
-
-            if (substitute)
-            {
-                // get_string() hands the token to the SAX interface with '.'
-                token_buffer[decimal_point_position] = '.';
-            }
-
-            if (JSON_HEDLEY_LIKELY(endptr == token_buffer.data() + token_buffer.size()))
-            {
-                return;
-            }
-
-            // retry only if the locale changed; otherwise, this would loop forever
-            const char current_decimal_point = get_decimal_point();
-            if (current_decimal_point == decimal_point)
-            {
-                return;
-            }
-            decimal_point = current_decimal_point;
-        }
-    }
-
     /*!
     @brief contiguous fast path for scanning a number
 
     Parses the whole number token straight from the input buffer, avoiding the
     per-character get()/add() of scan_number(). On success it fills token_buffer
-    (as scan_number() does) and
+    (with the locale decimal point substituted, as scan_number() does) and
     returns the token type. On anything it does not fully recognize as a
     well-formed number it makes no state change and returns
     token_type::uninitialized, so the caller falls back to scan_number(), which
@@ -1755,11 +1707,16 @@ scan_number_done:
         }
 #endif
 
-        // materialize the token exactly as scan_number() would. reset() already
+        // materialize the token exactly as scan_number() would, substituting the
-        // cleared token_buffer, so append() fills it (assign() is avoided
+        // locale decimal point so convert_number()'s strtof fallback stays valid.
-        // because custom string_t types need not provide it)
+        // reset() already cleared token_buffer, so append() fills it (assign() is
+        // avoided because custom string_t types need not provide it)
         token_buffer.append(reinterpret_cast<const typename string_t::value_type*>(data), len);
-        decimal_point_position = dot_index;
+        if (dot_index != std::string::npos)
+        {
+            token_buffer[dot_index] = static_cast<typename string_t::value_type>(decimal_point_char);
+            decimal_point_position = dot_index;
+        }
 
         ia.bulk_skip(len - 1);
         position.chars_read_total += (len - 1);
@@ -2026,7 +1983,11 @@ scan_number_done:
     /// return current string value (implicitly resets the token; useful only once)
     string_t& get_string()
     {
-        // a number token holds '.' regardless of the locale (#4084)
+        // translate decimal points from locale back to '.' (#4084)
+        if (decimal_point_char != '.' && decimal_point_position != std::string::npos)
+        {
+            token_buffer[decimal_point_position] = '.';
+        }
         return token_buffer;
     }
 
@@ -2322,7 +2283,9 @@ scan_number_done:
     number_unsigned_t value_unsigned = 0;
     number_float_t value_float = 0;
 
-    /// the position of the decimal point in token_buffer
+    /// the decimal point
+    const char_int_type decimal_point_char = '.';
+    /// the position of the decimal point in the input
     std::size_t decimal_point_position = std::string::npos;
 
     /// whether the caller (e.g. accept()/json_sax_acceptor) only needs the

include/nlohmann/detail/input/number_parse.hpp

@@ -118,12 +118,14 @@ std::strtod. The parser only activates for number_float_t == double; float and
 long double keep the std::strtof/std::strtold paths (see the templated overload
 below).
 
-@param[in]  first  pointer to the first character of the number
+@param[in]  first          pointer to the first character of the number
-@param[in]  last   pointer past the last character
+@param[in]  last           pointer past the last character
-@param[out] out    the parsed value on success
+@param[in]  decimal_point  the (locale-dependent) decimal point character
+@param[out] out            the parsed value on success
 @return true if the value was parsed exactly; false to fall back to strtod
 */
-inline bool parse_float_fast(const char* first, const char* last, double& out) noexcept
+template<typename DecimalPointType>
+bool parse_float_fast(const char* first, const char* last, DecimalPointType decimal_point, double& out) noexcept
 {
 #if defined(FLT_EVAL_METHOD) && FLT_EVAL_METHOD != 0
     // Clinger's fast path is only exact when double operations are evaluated in
@@ -134,6 +136,7 @@ inline bool parse_float_fast(const char* first, const char* last, double& out) n
     // std::from_chars / std::strtod path.
     static_cast<void>(first);
     static_cast<void>(last);
+    static_cast<void>(decimal_point);
     static_cast<void>(out);
     return false;
 #else
@@ -172,7 +175,7 @@ inline bool parse_float_fast(const char* first, const char* last, double& out) n
             ++num_digits;
             fractional_digits += static_cast<int>(seen_dot);
         }
-        else if (c == '.')
+        else if (static_cast<DecimalPointType>(c) == decimal_point)
         {
             if (JSON_HEDLEY_UNLIKELY(seen_dot))
             {
@@ -257,8 +260,8 @@ inline bool parse_float_fast(const char* first, const char* last, double& out) n
 }
 
 /// fast float path is only exact for `double`; decline for float/long double
-template<typename FloatType>
+template<typename DecimalPointType, typename FloatType>
-bool parse_float_fast(const char* /*first*/, const char* /*last*/, FloatType& /*out*/) noexcept
+bool parse_float_fast(const char* /*first*/, const char* /*last*/, DecimalPointType /*decimal_point*/, FloatType& /*out*/) noexcept
 {
     return false;
 }
@@ -270,7 +273,9 @@ std::from_chars is locale-independent, correctly rounded, and - via the
 Eisel-Lemire algorithm in modern standard libraries - much faster than strtod
 over the whole value range (not just the Clinger subset). It is used only when
 __cpp_lib_to_chars indicates full floating-point support and only when it
-consumes the entire token ([first, last)). An under-/overflow (result_out_of_range) also declines, so
+consumes the entire token ([first, last)); a partial parse means the buffer
+uses a non-'.' locale decimal point, in which case the caller falls back to the
+locale-aware path. An under-/overflow (result_out_of_range) also declines, so
 the caller's strtod fallback supplies the well-defined ±inf/0 result the parser
 expects (side-stepping the P4168 divergence between implementations).
 

single_include/nlohmann/json.hpp

@@ -8605,12 +8605,14 @@ std::strtod. The parser only activates for number_float_t == double; float and
 long double keep the std::strtof/std::strtold paths (see the templated overload
 below).
 
-@param[in]  first  pointer to the first character of the number
+@param[in]  first          pointer to the first character of the number
-@param[in]  last   pointer past the last character
+@param[in]  last           pointer past the last character
-@param[out] out    the parsed value on success
+@param[in]  decimal_point  the (locale-dependent) decimal point character
+@param[out] out            the parsed value on success
 @return true if the value was parsed exactly; false to fall back to strtod
 */
-inline bool parse_float_fast(const char* first, const char* last, double& out) noexcept
+template<typename DecimalPointType>
+bool parse_float_fast(const char* first, const char* last, DecimalPointType decimal_point, double& out) noexcept
 {
 #if defined(FLT_EVAL_METHOD) && FLT_EVAL_METHOD != 0
     // Clinger's fast path is only exact when double operations are evaluated in
@@ -8621,6 +8623,7 @@ inline bool parse_float_fast(const char* first, const char* last, double& out) n
     // std::from_chars / std::strtod path.
     static_cast<void>(first);
     static_cast<void>(last);
+    static_cast<void>(decimal_point);
     static_cast<void>(out);
     return false;
 #else
@@ -8659,7 +8662,7 @@ inline bool parse_float_fast(const char* first, const char* last, double& out) n
             ++num_digits;
             fractional_digits += static_cast<int>(seen_dot);
         }
-        else if (c == '.')
+        else if (static_cast<DecimalPointType>(c) == decimal_point)
         {
             if (JSON_HEDLEY_UNLIKELY(seen_dot))
             {
@@ -8744,8 +8747,8 @@ inline bool parse_float_fast(const char* first, const char* last, double& out) n
 }
 
 /// fast float path is only exact for `double`; decline for float/long double
-template<typename FloatType>
+template<typename DecimalPointType, typename FloatType>
-bool parse_float_fast(const char* /*first*/, const char* /*last*/, FloatType& /*out*/) noexcept
+bool parse_float_fast(const char* /*first*/, const char* /*last*/, DecimalPointType /*decimal_point*/, FloatType& /*out*/) noexcept
 {
     return false;
 }
@@ -8757,7 +8760,9 @@ std::from_chars is locale-independent, correctly rounded, and - via the
 Eisel-Lemire algorithm in modern standard libraries - much faster than strtod
 over the whole value range (not just the Clinger subset). It is used only when
 __cpp_lib_to_chars indicates full floating-point support and only when it
-consumes the entire token ([first, last)). An under-/overflow (result_out_of_range) also declines, so
+consumes the entire token ([first, last)); a partial parse means the buffer
+uses a non-'.' locale decimal point, in which case the caller falls back to the
+locale-aware path. An under-/overflow (result_out_of_range) also declines, so
 the caller's strtod fallback supplies the well-defined ±inf/0 result the parser
 expects (side-stepping the P4168 divergence between implementations).
 
@@ -9298,6 +9303,7 @@ class lexer : public lexer_base<BasicJsonType>
     explicit lexer(InputAdapterType&& adapter, bool ignore_comments_ = false, bool discard_number_values_ = false) noexcept
         : ia(std::move(adapter))
         , ignore_comments(ignore_comments_)
+        , decimal_point_char(static_cast<char_int_type>(get_decimal_point()))
         , discard_number_values(discard_number_values_)
     {}
 
@@ -9313,7 +9319,8 @@ class lexer : public lexer_base<BasicJsonType>
     // locales
     /////////////////////
 
-    /// return the decimal point of the current locale
+    /// return the locale-dependent decimal point
+    JSON_HEDLEY_PURE
     static char get_decimal_point() noexcept
     {
         const auto* loc = localeconv();
@@ -10182,10 +10189,9 @@ class lexer : public lexer_base<BasicJsonType>
             token_type::value_float if number could be successfully scanned,
             token_type::parse_error otherwise
 
-    @note The scanner is independent of the current locale: token_buffer
+    @note The scanner is independent of the current locale. Internally, the
-          always holds `.`. Only the std::strtod fallback of convert_number()
+          locale's decimal point is used instead of `.` to work with the
-          depends on the locale, and it looks up the decimal point right
+          locale-dependent converters.
-          before converting (see convert_float_locale_aware()).
     */
     token_type scan_number()  // lgtm [cpp/use-of-goto] `goto` is used in this function to implement the number-parsing state machine described above. By design, any finite input will eventually reach the "done" state or return token_type::parse_error. In each intermediate state, 1 byte of the input is appended to the token_buffer vector, and only the already initialized variables token_buffer, number_type, and error_message are manipulated.
     {
@@ -10274,7 +10280,7 @@ scan_number_zero:
         {
             case '.':
             {
-                add(current);
+                add(decimal_point_char);
                 decimal_point_position = token_buffer.size() - 1;
                 goto scan_number_decimal1;
             }
@@ -10311,7 +10317,7 @@ scan_number_any1:
 
             case '.':
             {
-                add(current);
+                add(decimal_point_char);
                 decimal_point_position = token_buffer.size() - 1;
                 goto scan_number_decimal1;
             }
@@ -10553,9 +10559,9 @@ scan_number_done:
 
         // Only a number below 1 can carry further insignificant zeros, and only
         // while the count stays at the limit does removing them change the
-        // answer - so this loop is skipped for all but a few tokens. The
+        // answer - so this loop is skipped for all but a few tokens. Note
-        // fraction is located through decimal_point_position rather than by
+        // token_buffer holds the locale's decimal point, so the fraction is
-        // searching '.'.
+        // located through decimal_point_position rather than by searching '.'.
         if (lead_zero != 0)
         {
             JSON_ASSERT(has_dot != 0); // an integer "0" cannot reach the limit
@@ -10573,8 +10579,8 @@ scan_number_done:
     @brief convert the number text in token_buffer to its value and token type
 
     The digit sequence in token_buffer has already been validated (by the
-    scan_number() state machine or by the contiguous fast path) and holds '.'
+    scan_number() state machine or by the contiguous fast path) and holds the
-    as decimal point, independent of the locale. Integers are parsed first and fall
+    locale decimal point in place of '.'. Integers are parsed first and fall
     back to floating point on overflow. This is shared so both scanners produce
     identical results.
 
@@ -10654,7 +10660,7 @@ scan_number_done:
         // integer conversion above overflowed. Prefer std::from_chars
         // (Eisel-Lemire, locale-independent, correctly rounded) when available;
         // otherwise the exact Clinger fast path (double only); otherwise the
-        // locale-aware strtof/strtod/strtold.
+        // locale-aware strtof/strtod.
         if (parse_float_from_chars(num_begin, num_end, value_float))
         {
             return token_type::value_float;
@@ -10663,75 +10669,26 @@ scan_number_done:
         // extra pass over the token's bytes, which otherwise shows up on
         // high-precision inputs such as canada.json
         if (mantissa_fits_clinger(mantissa_end)
-                && parse_float_fast(num_begin, num_end, value_float))
+                && parse_float_fast(num_begin, num_end, decimal_point_char, value_float))
         {
             return token_type::value_float;
         }
 
-        convert_float_locale_aware();
+        char* endptr = nullptr; // NOLINT(misc-const-correctness,cppcoreguidelines-pro-type-vararg,hicpp-vararg)
+        strtof(value_float, token_buffer.data(), &endptr);
+
+        // we checked the number format before
+        JSON_ASSERT(endptr == token_buffer.data() + token_buffer.size());
+
         return token_type::value_float;
     }
 
-    /*!
-    @brief convert the float in token_buffer with strtof/strtod/strtold
-
-    These functions expect the decimal point of the *current* locale, so it is
-    looked up right before the conversion instead of once when the lexer is
-    constructed: a locale change in between (by a parser callback, a SAX
-    handler, or another thread) must not truncate the value (#5198). The
-    token has been validated before, so if the conversion stops early and the
-    decimal point changed in the meantime, the locale changed between the
-    lookup and the call, and the conversion is repeated with the new decimal
-    point. If the decimal point did not change, a retry cannot succeed: the
-    locale's decimal point is not a single character (e.g., the two-byte
-    U+066B of ar_EG.UTF-8 or fa_IR.UTF-8) and cannot be substituted in place.
-    The value strtod parsed up to that point is kept, as before this change.
-
-    Note that changing the locale in another thread *while* strtod runs is
-    undefined behavior of the C library, which this function cannot prevent.
-    */
-    void convert_float_locale_aware()
-    {
-        const bool has_dot = decimal_point_position != std::string::npos;
-        char decimal_point = get_decimal_point();
-        for (;;)
-        {
-            const bool substitute = has_dot && decimal_point != '.';
-            if (substitute)
-            {
-                token_buffer[decimal_point_position] = static_cast<typename string_t::value_type>(decimal_point);
-            }
-
-            char* endptr = nullptr; // NOLINT(misc-const-correctness,cppcoreguidelines-pro-type-vararg,hicpp-vararg)
-            strtof(value_float, token_buffer.data(), &endptr);
-
-            if (substitute)
-            {
-                // get_string() hands the token to the SAX interface with '.'
-                token_buffer[decimal_point_position] = '.';
-            }
-
-            if (JSON_HEDLEY_LIKELY(endptr == token_buffer.data() + token_buffer.size()))
-            {
-                return;
-            }
-
-            // retry only if the locale changed; otherwise, this would loop forever
-            const char current_decimal_point = get_decimal_point();
-            if (current_decimal_point == decimal_point)
-            {
-                return;
-            }
-            decimal_point = current_decimal_point;
-        }
-    }
-
     /*!
     @brief contiguous fast path for scanning a number
 
     Parses the whole number token straight from the input buffer, avoiding the
     per-character get()/add() of scan_number(). On success it fills token_buffer
-    (as scan_number() does) and
+    (with the locale decimal point substituted, as scan_number() does) and
     returns the token type. On anything it does not fully recognize as a
     well-formed number it makes no state change and returns
     token_type::uninitialized, so the caller falls back to scan_number(), which
@@ -10847,11 +10804,16 @@ scan_number_done:
         }
 #endif
 
-        // materialize the token exactly as scan_number() would. reset() already
+        // materialize the token exactly as scan_number() would, substituting the
-        // cleared token_buffer, so append() fills it (assign() is avoided
+        // locale decimal point so convert_number()'s strtof fallback stays valid.
-        // because custom string_t types need not provide it)
+        // reset() already cleared token_buffer, so append() fills it (assign() is
+        // avoided because custom string_t types need not provide it)
         token_buffer.append(reinterpret_cast<const typename string_t::value_type*>(data), len);
-        decimal_point_position = dot_index;
+        if (dot_index != std::string::npos)
+        {
+            token_buffer[dot_index] = static_cast<typename string_t::value_type>(decimal_point_char);
+            decimal_point_position = dot_index;
+        }
 
         ia.bulk_skip(len - 1);
         position.chars_read_total += (len - 1);
@@ -11118,7 +11080,11 @@ scan_number_done:
     /// return current string value (implicitly resets the token; useful only once)
     string_t& get_string()
     {
-        // a number token holds '.' regardless of the locale (#4084)
+        // translate decimal points from locale back to '.' (#4084)
+        if (decimal_point_char != '.' && decimal_point_position != std::string::npos)
+        {
+            token_buffer[decimal_point_position] = '.';
+        }
         return token_buffer;
     }
 
@@ -11414,7 +11380,9 @@ scan_number_done:
     number_unsigned_t value_unsigned = 0;
     number_float_t value_float = 0;
 
-    /// the position of the decimal point in token_buffer
+    /// the decimal point
+    const char_int_type decimal_point_char = '.';
+    /// the position of the decimal point in the input
     std::size_t decimal_point_position = std::string::npos;
 
     /// whether the caller (e.g. accept()/json_sax_acceptor) only needs the
@@ -13326,7 +13294,7 @@ class binary_reader
             case 0x01: // double
             {
                 double number{};
-                return get_number<double, true>(input_format_t::bson, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number<double, true>(input_format_t::bson, number) && emit_float(input_format_t::bson, number);
             }
 
             case 0x02: // string
@@ -13367,19 +13335,19 @@ class binary_reader
             case 0x10: // int32
             {
                 std::int32_t value{};
-                return get_number<std::int32_t, true>(input_format_t::bson, value) && sax->number_integer(value);
+                return get_number<std::int32_t, true>(input_format_t::bson, value) && emit_signed(value);
             }
 
             case 0x12: // int64
             {
                 std::int64_t value{};
-                return get_number<std::int64_t, true>(input_format_t::bson, value) && sax->number_integer(value);
+                return get_number<std::int64_t, true>(input_format_t::bson, value) && emit_signed(value);
             }
 
             case 0x11: // uint64
             {
                 std::uint64_t value{};
-                return get_number<std::uint64_t, true>(input_format_t::bson, value) && sax->number_unsigned(value);
+                return get_number<std::uint64_t, true>(input_format_t::bson, value) && emit_unsigned(value);
             }
 
             default: // anything else is not supported (yet)
@@ -13405,14 +13373,17 @@ class binary_reader
         {
             return false;
         }
-        const auto max_val = static_cast<NumberType>((std::numeric_limits<number_integer_t>::max)());
+
-        if (number > max_val)
+        // the value is -1 - number, which fits into number_integer_t
+        // whenever number does
+        if (JSON_HEDLEY_LIKELY(value_in_range_of<number_integer_t>(number)))
         {
-            return sax->parse_error(chars_read, get_token_string(),
+            return sax->number_integer(static_cast<number_integer_t>(-1) - static_cast<number_integer_t>(number));
-                                    parse_error::create(112, chars_read,
-                                            exception_message(input_format_t::cbor, "negative integer overflow", "value"), nullptr));
         }
-        return sax->number_integer(static_cast<number_integer_t>(-1) - static_cast<number_integer_t>(number));
+
+        // like the lexer does for JSON text, store a value too small for
+        // number_integer_t as number_float_t
+        return sax->number_float(static_cast<number_float_t>(-1) - static_cast<number_float_t>(number), "");
     }
 
     /*!
@@ -13469,25 +13440,25 @@ class binary_reader
             case 0x18: // Unsigned integer (one-byte uint8_t follows)
             {
                 std::uint8_t number{};
-                return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::cbor, number) && emit_unsigned(number);
             }
 
             case 0x19: // Unsigned integer (two-byte uint16_t follows)
             {
                 std::uint16_t number{};
-                return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::cbor, number) && emit_unsigned(number);
             }
 
             case 0x1A: // Unsigned integer (four-byte uint32_t follows)
             {
                 std::uint32_t number{};
-                return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::cbor, number) && emit_unsigned(number);
             }
 
             case 0x1B: // Unsigned integer (eight-byte uint64_t follows)
             {
                 std::uint64_t number{};
-                return get_number(input_format_t::cbor, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::cbor, number) && emit_unsigned(number);
             }
 
             // Negative integer -1-0x00..-1-0x17 (-1..-24)
@@ -13932,13 +13903,13 @@ class binary_reader
             case 0xFA: // Single-Precision Float (four-byte IEEE 754)
             {
                 float number{};
-                return get_number(input_format_t::cbor, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format_t::cbor, number) && emit_float(input_format_t::cbor, number);
             }
 
             case 0xFB: // Double-Precision Float (eight-byte IEEE 754)
             {
                 double number{};
-                return get_number(input_format_t::cbor, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format_t::cbor, number) && emit_float(input_format_t::cbor, number);
             }
 
             default: // anything else (0xFF is handled inside the other types)
@@ -14628,61 +14599,61 @@ class binary_reader
             case 0xCA: // float 32
             {
                 float number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format_t::msgpack, number) && emit_float(input_format_t::msgpack, number);
             }
 
             case 0xCB: // float 64
             {
                 double number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format_t::msgpack, number) && emit_float(input_format_t::msgpack, number);
             }
 
             case 0xCC: // uint 8
             {
                 std::uint8_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::msgpack, number) && emit_unsigned(number);
             }
 
             case 0xCD: // uint 16
             {
                 std::uint16_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::msgpack, number) && emit_unsigned(number);
             }
 
             case 0xCE: // uint 32
             {
                 std::uint32_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::msgpack, number) && emit_unsigned(number);
             }
 
             case 0xCF: // uint 64
             {
                 std::uint64_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number);
+                return get_number(input_format_t::msgpack, number) && emit_unsigned(number);
             }
 
             case 0xD0: // int 8
             {
                 std::int8_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
+                return get_number(input_format_t::msgpack, number) && emit_signed(number);
             }
 
             case 0xD1: // int 16
             {
                 std::int16_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
+                return get_number(input_format_t::msgpack, number) && emit_signed(number);
             }
 
             case 0xD2: // int 32
             {
                 std::int32_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
+                return get_number(input_format_t::msgpack, number) && emit_signed(number);
             }
 
             case 0xD3: // int 64
             {
                 std::int64_t number{};
-                return get_number(input_format_t::msgpack, number) && sax->number_integer(number);
+                return get_number(input_format_t::msgpack, number) && emit_signed(number);
             }
 
             case 0xDC: // array 16
@@ -15523,7 +15494,7 @@ class binary_reader
                         {
                             return sax->parse_error(chars_read, get_token_string(), out_of_range::create(408, exception_message(input_format, "excessive ndarray size caused overflow", "size"), nullptr));
                         }
-                        if (JSON_HEDLEY_UNLIKELY(!sax->number_unsigned(static_cast<number_unsigned_t>(i))))
+                        if (JSON_HEDLEY_UNLIKELY(!emit_unsigned(i)))
                         {
                             return false;
                         }
@@ -15655,37 +15626,37 @@ class binary_reader
                     break;
                 }
                 std::uint8_t number{};
-                return get_number(input_format, number) && sax->number_unsigned(number);
+                return get_number(input_format, number) && emit_unsigned(number);
             }
 
             case 'U':
             {
                 std::uint8_t number{};
-                return get_number(input_format, number) && sax->number_unsigned(number);
+                return get_number(input_format, number) && emit_unsigned(number);
             }
 
             case 'i':
             {
                 std::int8_t number{};
-                return get_number(input_format, number) && sax->number_integer(number);
+                return get_number(input_format, number) && emit_signed(number);
             }
 
             case 'I':
             {
                 std::int16_t number{};
-                return get_number(input_format, number) && sax->number_integer(number);
+                return get_number(input_format, number) && emit_signed(number);
             }
 
             case 'l':
             {
                 std::int32_t number{};
-                return get_number(input_format, number) && sax->number_integer(number);
+                return get_number(input_format, number) && emit_signed(number);
             }
 
             case 'L':
             {
                 std::int64_t number{};
-                return get_number(input_format, number) && sax->number_integer(number);
+                return get_number(input_format, number) && emit_signed(number);
             }
 
             case 'u':
@@ -15695,7 +15666,7 @@ class binary_reader
                     break;
                 }
                 std::uint16_t number{};
-                return get_number(input_format, number) && sax->number_unsigned(number);
+                return get_number(input_format, number) && emit_unsigned(number);
             }
 
             case 'm':
@@ -15705,7 +15676,7 @@ class binary_reader
                     break;
                 }
                 std::uint32_t number{};
-                return get_number(input_format, number) && sax->number_unsigned(number);
+                return get_number(input_format, number) && emit_unsigned(number);
             }
 
             case 'M':
@@ -15715,7 +15686,7 @@ class binary_reader
                     break;
                 }
                 std::uint64_t number{};
-                return get_number(input_format, number) && sax->number_unsigned(number);
+                return get_number(input_format, number) && emit_unsigned(number);
             }
 
             case 'h':
@@ -15773,13 +15744,13 @@ class binary_reader
             case 'd':
             {
                 float number{};
-                return get_number(input_format, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format, number) && emit_float(input_format, number);
             }
 
             case 'D':
             {
                 double number{};
-                return get_number(input_format, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format, number) && emit_float(input_format, number);
             }
 
             case 'H':
@@ -16246,13 +16217,13 @@ class binary_reader
             case 0x8E: // binary32
             {
                 float number{};
-                return get_number(input_format_t::bon8, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format_t::bon8, number) && emit_float(input_format_t::bon8, number);
             }
 
             case 0x8F: // binary64
             {
                 double number{};
-                return get_number(input_format_t::bon8, number) && sax->number_float(static_cast<number_float_t>(number), "");
+                return get_number(input_format_t::bon8, number) && emit_float(input_format_t::bon8, number);
             }
 
             case 0xF8:
@@ -16318,7 +16289,9 @@ class binary_reader
     @brief pass an integer to the SAX parser
 
     Non-negative integers are passed as unsigned, negative integers as signed
-    numbers, like the other binary formats do.
+    numbers, like the other binary formats do. A value that does not fit the
+    number type is passed as described for @ref emit_unsigned and
+    @ref emit_signed.
 
     @param[in] number  the integer
     @return whether the SAX parser accepted the value
@@ -16327,9 +16300,9 @@ class binary_reader
     {
         if (number >= 0)
         {
-            return sax->number_unsigned(static_cast<number_unsigned_t>(number));
+            return emit_unsigned(static_cast<std::uint64_t>(number));
         }
-        return sax->number_integer(static_cast<number_integer_t>(number));
+        return emit_signed(number);
     }
 
     /*!
@@ -16386,8 +16359,7 @@ class binary_reader
             value = (value << 8) | static_cast<std::int64_t>(current);
         }
 
-        return negative ? sax->number_integer(static_cast<number_integer_t>(-(value + offset)))
+        return emit_bon8_integer(negative ? -(value + offset) : value + offset);
-               : sax->number_unsigned(static_cast<number_unsigned_t>(value + offset));
     }
 
     /*!
@@ -16684,6 +16656,79 @@ class binary_reader
         return true;
     }
 
+    /*!
+    @brief pass a signed integer read from the input to the SAX parser
+
+    Like the lexer does for JSON text, a value that does not fit into
+    number_integer_t is passed as number_unsigned_t if it is non-negative and
+    fits there, and as number_float_t otherwise. With the default number
+    types, every integer the binary formats can encode fits, so this only
+    matters for narrower custom number types.
+
+    @tparam NumberType a signed integer type
+    @param[in] number  the integer
+    @return whether the SAX parser accepted the value
+    */
+    template<typename NumberType>
+    bool emit_signed(const NumberType number)
+    {
+        if (JSON_HEDLEY_LIKELY(value_in_range_of<number_integer_t>(number)))
+        {
+            return sax->number_integer(static_cast<number_integer_t>(number));
+        }
+        if (value_in_range_of<number_unsigned_t>(number))
+        {
+            return sax->number_unsigned(static_cast<number_unsigned_t>(number));
+        }
+        return sax->number_float(static_cast<number_float_t>(number), "");
+    }
+
+    /*!
+    @brief pass an unsigned integer read from the input to the SAX parser
+
+    Like the lexer does for JSON text, a value that does not fit into
+    number_unsigned_t is passed as number_float_t.
+
+    @tparam NumberType an unsigned integer type
+    @param[in] number  the integer
+    @return whether the SAX parser accepted the value
+    */
+    template<typename NumberType>
+    bool emit_unsigned(const NumberType number)
+    {
+        if (JSON_HEDLEY_LIKELY(value_in_range_of<number_unsigned_t>(number)))
+        {
+            return sax->number_unsigned(static_cast<number_unsigned_t>(number));
+        }
+        return sax->number_float(static_cast<number_float_t>(number), "");
+    }
+
+    /*!
+    @brief pass a floating-point number read from the input to the SAX parser
+
+    Like the lexer does for JSON text, a finite value that overflows
+    number_float_t is rejected instead of silently becoming infinity. Infinity
+    and NaN in the input are passed on unchanged.
+
+    @tparam NumberType a floating-point type
+    @param[in] format  the current format (for diagnostics)
+    @param[in] number  the number
+    @return whether the SAX parser accepted the value
+
+    @throw out_of_range.406 if a finite @a number overflows number_float_t
+    */
+    template<typename NumberType>
+    bool emit_float(const input_format_t format, const NumberType number)
+    {
+        const auto result = static_cast<number_float_t>(number);
+        if (JSON_HEDLEY_UNLIKELY(std::isfinite(number) && !std::isfinite(result)))
+        {
+            return sax->parse_error(chars_read, get_token_string(),
+                                    out_of_range::create(406, exception_message(format, "number overflow", "value"), nullptr));
+        }
+        return sax->number_float(result, "");
+    }
+
     /*!
     @brief create a string by reading characters from the input
 

tests/src/unit-binary_formats.cpp

@@ -11,7 +11,12 @@
 #include <nlohmann/json.hpp>
 using nlohmann::json;
 
+#include <cmath>
 #include <fstream>
+#include <limits>
+#include <map>
+#include <string>
+#include <vector>
 #include "make_test_data_available.hpp"
 
 TEST_CASE("Binary Formats" * doctest::skip())
@@ -224,3 +229,114 @@ TEST_CASE("Binary Formats" * doctest::skip())
         CHECK((100.0 * double(ubjson_3_size) / double(json_size)) == Approx(89.450));
     }
 }
+
+TEST_CASE("Binary formats with narrow number types")
+{
+    // Numbers that do not fit the number types are handled like the lexer
+    // handles them in JSON text: an integer that fits neither integer type is
+    // stored as a floating-point number, and a finite floating-point number
+    // that overflows number_float_t is rejected with out_of_range.406.
+    using narrow_json = nlohmann::basic_json<std::map, std::vector, std::string, bool, std::int32_t, std::uint32_t, float>;
+    using bytes = std::vector<std::uint8_t>;
+
+    struct binary_format
+    {
+        const char* name;
+        bytes (*encode)(const json&);
+        narrow_json (*decode)(const bytes&, bool);
+    };
+
+    const std::vector<binary_format> formats =
+    {
+        {
+            "CBOR", [](const json & j) { return json::to_cbor(j); },
+            [](const bytes & v, bool allow_exceptions)
+            {
+                return narrow_json::from_cbor(v, true, allow_exceptions);
+            }
+        },
+        {
+            "MessagePack", [](const json & j) { return json::to_msgpack(j); },
+            [](const bytes & v, bool allow_exceptions)
+            {
+                return narrow_json::from_msgpack(v, true, allow_exceptions);
+            }
+        },
+        {
+            "UBJSON", [](const json & j) { return json::to_ubjson(j); },
+            [](const bytes & v, bool allow_exceptions)
+            {
+                return narrow_json::from_ubjson(v, true, allow_exceptions);
+            }
+        },
+        {
+            "BJData", [](const json & j) { return json::to_bjdata(j); },
+            [](const bytes & v, bool allow_exceptions)
+            {
+                return narrow_json::from_bjdata(v, true, allow_exceptions);
+            }
+        },
+        {
+            // BSON can only store numbers as object members
+            "BSON", [](const json & j) { return json::to_bson(json{{"a", j}}); },
+            [](const bytes & v, bool allow_exceptions)
+            {
+                const auto result = narrow_json::from_bson(v, true, allow_exceptions);
+                return result.is_discarded() ? result : result.at("a");
+            }
+        },
+        {
+            "BON8", [](const json & j) { return json::to_bon8(j); },
+            [](const bytes & v, bool allow_exceptions)
+            {
+                return narrow_json::from_bon8(v, true, allow_exceptions);
+            }
+        },
+    };
+
+    for (const auto& format : formats)
+    {
+        const std::string name = format.name;
+        INFO("format := ", name);
+        const auto roundtrip = [&format](const json & j)
+        {
+            return format.decode(format.encode(j), true);
+        };
+
+        // integers that fit keep their type
+        CHECK(roundtrip(json(-5)).is_number_integer());
+        CHECK(roundtrip(json(-5)).get<std::int32_t>() == -5);
+        CHECK(roundtrip(json(3000000000u)).is_number_unsigned());
+        CHECK(roundtrip(json(3000000000u)).get<std::uint32_t>() == 3000000000u);
+
+        // integers that fit neither integer type are stored as float
+        CHECK(roundtrip(json(5000000000u)).is_number_float());
+        CHECK(roundtrip(json(5000000000u)).get<float>() == 5000000000.0f);
+        if (name != "BON8") // BON8 cannot encode integers above INT64_MAX
+        {
+            CHECK(roundtrip(json(10000000000000000000u)).is_number_float());
+            CHECK(roundtrip(json(10000000000000000000u)).get<float>() == 10000000000000000000.0f);
+        }
+        CHECK(roundtrip(json(-3000000000)).is_number_float());
+        CHECK(roundtrip(json(-3000000000)).get<float>() == -3000000000.0f);
+        CHECK(roundtrip(json(-5000000000)).is_number_float());
+        CHECK(roundtrip(json(-5000000000)).get<float>() == -5000000000.0f);
+
+        // floating-point numbers that fit
+        CHECK(roundtrip(json(1.5)).get<float>() == 1.5f);
+        const auto just_above_max = std::nextafter(static_cast<double>((std::numeric_limits<float>::max)()),
+                                    std::numeric_limits<double>::infinity());
+        CHECK(roundtrip(json(just_above_max)).get<float>() == (std::numeric_limits<float>::max)());
+
+        // infinity and NaN are passed on
+        CHECK(std::isinf(roundtrip(json(std::numeric_limits<double>::infinity())).get<float>()));
+        CHECK(std::isnan(roundtrip(json(std::numeric_limits<double>::quiet_NaN())).get<float>()));
+
+        // finite floating-point numbers that overflow number_float_t are rejected
+        const std::string message = "[json.exception.out_of_range.406] syntax error while parsing " + name
+                                    + " value: number overflow";
+        CHECK_THROWS_WITH_AS(roundtrip(json(1e300)), message.c_str(), narrow_json::out_of_range&);
+        CHECK_THROWS_WITH_AS(roundtrip(json(-1e300)), message.c_str(), narrow_json::out_of_range&);
+        CHECK(format.decode(format.encode(json(1e300)), false).is_discarded());
+    }
+}

tests/src/unit-cbor.cpp

@@ -3146,7 +3146,8 @@ TEST_CASE("Tagged values")
         // CBOR encodes negative integers as: result = -1 - n
         // For type 0x3B, n is an 8-byte uint64_t. Valid range for n with
         // the default int64_t is [0, INT64_MAX], producing results in [INT64_MIN, -1].
-        // When n > INT64_MAX, the result exceeds int64_t range and is rejected.
+        // When n > INT64_MAX, the result exceeds int64_t range and is stored
+        // as a floating-point number, as the lexer does for JSON text.
 
         SECTION("n = 0 is valid (result = -1)")
         {
@@ -3167,33 +3168,34 @@ TEST_CASE("Tagged values")
             CHECK(result.get<int64_t>() == (std::numeric_limits<int64_t>::min)());
         }
 
-        SECTION("n = INT64_MAX + 1 is rejected (overflow)")
+        SECTION("n = INT64_MAX + 1 is stored as float")
         {
             // n = INT64_MAX + 1 (0x8000000000000000)
-            // result = -1 - n = -9223372036854775809, which exceeds int64_t range
+            // result = -1 - n = -9223372036854775809, which exceeds int64_t range;
+            // the nearest double is -9223372036854775808.0
             const std::vector<uint8_t> input = {0x3B, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
-            json _;
+            const auto result = json::from_cbor(input);
-            CHECK_THROWS_WITH_AS(_ = json::from_cbor(input),
+            CHECK(result.is_number_float());
-                                 "[json.exception.parse_error.112] parse error at byte 9: syntax error while parsing CBOR value: negative integer overflow",
+            CHECK(result.get<double>() == -9223372036854775808.0);
-                                 json::parse_error);
+            CHECK(result == json::parse("-9223372036854775809"));
         }
 
-        SECTION("n = UINT64_MAX is rejected (overflow)")
+        SECTION("n = UINT64_MAX is stored as float")
         {
             // n = UINT64_MAX (0xFFFFFFFFFFFFFFFF)
             // result = -1 - n = -18446744073709551616, which exceeds int64_t range
             const std::vector<uint8_t> input = {0x3B, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF};
-            json _;
+            const auto result = json::from_cbor(input);
-            CHECK_THROWS_WITH_AS(_ = json::from_cbor(input),
+            CHECK(result.is_number_float());
-                                 "[json.exception.parse_error.112] parse error at byte 9: syntax error while parsing CBOR value: negative integer overflow",
+            CHECK(result.get<double>() == -18446744073709551616.0);
-                                 json::parse_error);
+            CHECK(result == json::parse("-18446744073709551616"));
         }
 
-        SECTION("overflow with allow_exceptions=false returns discarded")
+        SECTION("overflow with allow_exceptions=false is not an error")
         {
             const std::vector<uint8_t> input = {0x3B, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
             const auto result = json::from_cbor(input, true, false);
-            CHECK(result.is_discarded());
+            CHECK(result.is_number_float());
         }
     }
 

tests/src/unit-class_lexer.cpp

@@ -666,7 +666,7 @@ TEST_CASE("parse_float_fast declines what it cannot convert exactly")
     // always safe: the caller then falls back to a slower, exact conversion.
     const auto fast = [](const std::string & s, double & out)
     {
-        return nlohmann::detail::parse_float_fast(s.data(), s.data() + s.size(), out);
+        return nlohmann::detail::parse_float_fast(s.data(), s.data() + s.size(), '.', out);
     };
     double out = 0;
 

tests/src/unit-locale-cpp.cpp

@@ -12,12 +12,7 @@
 #include <nlohmann/json.hpp>
 using nlohmann::json;
 
-#include <array>
 #include <clocale>
-#include <map>
-#include <string>
-#include <utility>
-#include <vector>
 
 struct ParserImpl final: public nlohmann::json_sax<json>
 {
@@ -180,208 +175,3 @@ TEST_CASE("locale-dependent test (LC_NUMERIC=de_DE)")
         MESSAGE("locale de_DE is not usable");
     }
 }
-
-namespace
-{
-// records the numbers of a flat array and switches LC_NUMERIC to the given
-// locale once the array opens - after the lexer was constructed, but before
-// any number in the array is lexed
-struct LocaleSwitchingSax final: public nlohmann::json_sax<json>
-{
-    explicit LocaleSwitchingSax(const char* switch_to)
-        : locale_after_open(switch_to)
-    {}
-
-    bool null() override
-    {
-        return true;
-    }
-    bool boolean(bool /*val*/) override
-    {
-        return true;
-    }
-    bool number_integer(json::number_integer_t /*val*/) override
-    {
-        return true;
-    }
-    bool number_unsigned(json::number_unsigned_t /*val*/) override
-    {
-        return true;
-    }
-    bool number_float(json::number_float_t val, const json::string_t& s) override
-    {
-        values.push_back(val);
-        strings.push_back(s);
-        return true;
-    }
-    bool string(json::string_t& /*val*/) override
-    {
-        return true;
-    }
-    bool binary(json::binary_t& /*val*/) override
-    {
-        return true;
-    }
-    bool start_object(std::size_t /*val*/) override
-    {
-        return true;
-    }
-    bool key(json::string_t& /*val*/) override
-    {
-        return true;
-    }
-    bool end_object() override
-    {
-        return true;
-    }
-    bool start_array(std::size_t /*val*/) override
-    {
-        switched = std::setlocale(LC_NUMERIC, locale_after_open.c_str()) != nullptr;
-        return true;
-    }
-    bool end_array() override
-    {
-        return true;
-    }
-    bool parse_error(std::size_t /*val*/, const std::string& /*val*/, const nlohmann::detail::exception& /*val*/) override
-    {
-        return false;
-    }
-
-    std::string locale_after_open;
-    bool switched = false;
-    std::vector<json::number_float_t> values {}; // NOLINT(readability-redundant-member-init)
-    std::vector<json::string_t> strings {}; // NOLINT(readability-redundant-member-init)
-};
-} // namespace
-
-TEST_CASE("locale changes between lexer construction and number conversion (#5198)")
-{
-    // The numbers are chosen so that the conversion also takes the strtod
-    // fallback, which honors the locale that is current at conversion time:
-    // too many significant digits for Clinger's fast path, an underflow that
-    // std::from_chars rejects, and a plain value.
-    const std::vector<std::string> numbers = {"3.14159265358979323846", "1.5e-400", "12.34", "-0.000123456789012345678"};
-    std::string text = "[";
-    for (const auto& n : numbers)
-    {
-        text += (text.size() == 1 ? "" : ",") + n;
-    }
-    text += "]";
-
-    using long_double_json = nlohmann::basic_json<std::map, std::vector, std::string, bool, std::int64_t, std::uint64_t, long double>;
-
-    // reference values, parsed without a locale switch
-    REQUIRE(std::setlocale(LC_NUMERIC, "C") != nullptr);
-    const json expected = json::parse(text);
-    const long_double_json expected_ld = long_double_json::parse(text);
-
-    const std::array<std::pair<const char*, const char*>, 2> transitions =
-    {
-        {
-            {"C", "de_DE"},
-            {"de_DE", "C"}
-        }
-    };
-
-    for (const auto& transition : transitions)
-    {
-        CAPTURE(transition.first);
-        CAPTURE(transition.second);
-
-        if (std::setlocale(LC_NUMERIC, transition.first) == nullptr)
-        {
-            MESSAGE("locale is not usable");
-            continue;
-        }
-
-        // SAX parsing
-        {
-            LocaleSwitchingSax sax(transition.second);
-            CHECK(json::sax_parse(text, &sax));
-            if (sax.switched)
-            {
-                CHECK(sax.values == expected.get<std::vector<json::number_float_t>>());
-                CHECK(sax.strings == numbers);
-            }
-        }
-
-        // DOM parsing with a callback
-        {
-            bool switched = false;
-            const auto cb = [&](int /*depth*/, json::parse_event_t event, json& /*parsed*/)
-            {
-                if (event == json::parse_event_t::array_start)
-                {
-                    switched = std::setlocale(LC_NUMERIC, transition.second) != nullptr;
-                }
-                return true;
-            };
-            const json j = json::parse(text, cb);
-            if (switched)
-            {
-                CHECK(j == expected);
-            }
-        }
-
-        // a long double goes through std::strtold unless std::from_chars supports it
-        {
-            bool switched = false;
-            const auto cb = [&](int /*depth*/, long_double_json::parse_event_t event, long_double_json& /*parsed*/)
-            {
-                if (event == long_double_json::parse_event_t::array_start)
-                {
-                    switched = std::setlocale(LC_NUMERIC, transition.second) != nullptr;
-                }
-                return true;
-            };
-            const long_double_json j = long_double_json::parse(text, cb);
-            if (switched)
-            {
-                CHECK(j == expected_ld);
-            }
-        }
-    }
-
-    std::setlocale(LC_NUMERIC, "C");
-}
-
-TEST_CASE("locale with a multi-byte decimal point")
-{
-    // Some locales use a decimal point that is not a single character, e.g.
-    // U+066B ARABIC DECIMAL SEPARATOR (two bytes in UTF-8). It cannot be
-    // substituted in place for '.', so the strtod fallback stops early. The
-    // conversion must still terminate rather than retry forever.
-    const std::array<const char*, 6> names = {{"ar_EG.UTF-8", "ar_SA.UTF-8", "fa_IR.UTF-8", "ps_AF.UTF-8", "ar_EG", "fa_IR"}};
-    bool tested = false;
-    for (const char* name : names)
-    {
-        if (std::setlocale(LC_NUMERIC, name) == nullptr)
-        {
-            continue;
-        }
-        const std::string decimal_point = std::localeconv()->decimal_point;
-        if (decimal_point.size() < 2)
-        {
-            continue;
-        }
-        CAPTURE(name);
-        tested = true;
-
-        // too many significant digits for Clinger's fast path, and an underflow
-        // that std::from_chars rejects: both reach the strtod fallback
-        json j;
-        CHECK_NOTHROW(j = json::parse("[3.14159265358979323846, 1.5e-400, -0.000123456789012345678]"));
-        CHECK(j.is_array());
-        CHECK(json::accept("3.14159265358979323846"));
-
-        // a value the locale-independent paths convert is not affected
-        CHECK(json::parse("12.5") == 12.5);
-    }
-    if (!tested)
-    {
-        MESSAGE("no locale with a multi-byte decimal point is usable");
-    }
-
-    std::setlocale(LC_NUMERIC, "C");
-}