Merge branch 'develop' into claude/binary-reader-narrow-numbers

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/from_bjdata.md

@@ -120,12 +120,3 @@ Linear in the size of the input.
 - Extended container support (1) to include types with lvalue-only ADL `begin`/`end` (matching `std::begin`/`std::end` semantics) in version 3.13.0.
 - Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
 - Added `error_handler` parameter in version 3.13.0.
-
-!!! warning "Deprecation"
-
-    - Overload (2) replaces calls to `from_bjdata` with a pointer and a length as first two parameters, which has been
-      deprecated in version 3.13.0. This overload will be removed in version 4.0.0. Please replace all calls like
-      `#!cpp from_bjdata(ptr, len, ...);` with `#!cpp from_bjdata(ptr, ptr+len, ...);`.
-
-    You should be warned by your compiler with a `-Wdeprecated-declarations` warning if you are using a deprecated
-    function.

docs/mkdocs/docs/api/basic_json/from_bon8.md

@@ -106,12 +106,3 @@ Linear in the size of the input.
 ## Version history
 
 - Added in version 3.13.0.
-
-!!! warning "Deprecation"
-
-    - Overload (2) replaces calls to `from_bon8` with a pointer and a length as first two parameters, which has been
-      deprecated in version 3.13.0. This overload will be removed in version 4.0.0. Please replace all calls like
-      `#!cpp from_bon8(ptr, len, ...);` with `#!cpp from_bon8(ptr, ptr+len, ...);`.
-
-    You should be warned by your compiler with a `-Wdeprecated-declarations` warning if you are using a deprecated
-    function.

docs/mkdocs/docs/api/basic_json/number_float_t.md

@@ -55,6 +55,10 @@ This implementation does exactly follow this approach, as it uses double precisi
 smaller than `-1.79769313486232e+308` and values greater than `1.79769313486232e+308` will be stored as NaN internally
 and be serialized to `null`.
 
+During deserialization (from JSON text or any of the binary formats), a finite number that does not fit into
+`number_float_t` is rejected with [`out_of_range.406`](../../home/exceptions.md#jsonexceptionout_of_range406), for
+example a double-precision number in a binary format when `number_float_t` is `#!cpp float`.
+
 ### Storage
 
 Floating-point number values are stored directly inside a `basic_json` type.

docs/mkdocs/docs/api/basic_json/number_integer_t.md

@@ -47,8 +47,9 @@ With the default values for `NumberIntegerType` (`std::int64_t`), the default va
 
 When the default type is used, the maximal integer number that can be stored is `9223372036854775807` (INT64_MAX) and
 the minimal integer number that can be stored is `-9223372036854775808` (INT64_MIN). Integer numbers that are out of
-range will yield over/underflow when used in a constructor. During deserialization, too large or small integer numbers
+range will yield over/underflow when used in a constructor. During deserialization (from JSON text or any of the binary
-will automatically be stored as [`number_unsigned_t`](number_unsigned_t.md) or [`number_float_t`](number_float_t.md).
+formats), too large or small integer numbers will automatically be stored as [`number_unsigned_t`](number_unsigned_t.md)
+or [`number_float_t`](number_float_t.md).
 
 [RFC 8259](https://tools.ietf.org/html/rfc8259) further states:
 > Note that when such software is used, numbers that are integers and are in the range [-2<sup>53</sup>+1, 2<sup>53</sup>-1] are

docs/mkdocs/docs/api/basic_json/number_unsigned_t.md

@@ -48,8 +48,9 @@ With the default values for `NumberUnsignedType` (`std::uint64_t`), the default
 
 When the default type is used, the maximal integer number that can be stored is `18446744073709551615` (UINT64_MAX) and
 the minimal integer number that can be stored is `0`. Integer numbers that are out of range will yield over/underflow
-when used in a constructor. During deserialization, too large or small integer numbers will automatically be stored
+when used in a constructor. During deserialization (from JSON text or any of the binary formats), too large or small
-as [`number_integer_t`](number_integer_t.md) or [`number_float_t`](number_float_t.md).
+integer numbers will automatically be stored as [`number_integer_t`](number_integer_t.md) or
+[`number_float_t`](number_float_t.md).
 
 [RFC 8259](https://tools.ietf.org/html/rfc8259) further states:
 > Note that when such software is used, numbers that are integers and are in the range [-2<sup>53</sup>+1, 2<sup>53</sup>-1] are

docs/mkdocs/docs/features/binary_formats/cbor.md

@@ -168,9 +168,9 @@ The library maps CBOR types to JSON value types as follows:
 !!! warning "Negative integer overflow"
 
     CBOR negative integers (major type 1) are decoded as `-1 - n`. If the encoded magnitude `n` is too large for the
-    result to fit into `number_integer_t` (`std::int64_t` by default), parsing fails with a
+    result to fit into `number_integer_t` (`std::int64_t` by default), the result is stored as `number_float_t`, like
-    [`parse_error.112`](../../home/exceptions.md#jsonexceptionparse_error112) exception rather than overflowing
+    a too small integer in JSON text. For example, `-18446744073709551616` (`0x3B` followed by eight `0xFF` bytes) is
-    silently.
+    stored as `-1.8446744073709552e+19`.
 
 !!! warning "Object keys"
 

docs/mkdocs/docs/home/exceptions.md

@@ -331,9 +331,6 @@ An unexpected byte was read in a [binary format](../features/binary_formats/inde
     [json.exception.parse_error.112] parse error at byte 15: syntax error while parsing BSON binary: byte array length cannot be negative, is -1
     ```
     ```
-    [json.exception.parse_error.112] parse error at byte 9: syntax error while parsing CBOR value: negative integer overflow
-    ```
-    ```
     [json.exception.parse_error.112] parse error at byte 5: syntax error while parsing BSON document: document size 6 does not match the number of bytes read (5)
     ```
 
@@ -863,13 +860,18 @@ The JSON Patch operations 'remove' and 'add' cannot be applied to the root eleme
 
 ### json.exception.out_of_range.406
 
-A parsed number could not be stored as without changing it to NaN or INF.
+A parsed number could not be stored without changing it to NaN or INF. For the binary formats, this happens when a
+finite floating-point number does not fit into [`number_float_t`](../api/basic_json/number_float_t.md), for example a
+double-precision number when `number_float_t` is `#!cpp float`.
 
-!!! failure "Example message"
+!!! failure "Example messages"
 
     ```
     number overflow parsing '10E1000'
     ```
+    ```
+    [json.exception.out_of_range.406] syntax error while parsing CBOR value: number overflow
+    ```
 
 ### json.exception.out_of_range.407
 

include/nlohmann/json.hpp

@@ -5936,16 +5936,6 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         return from_binary_impl(detail::input_adapter(std::move(first), std::move(last)), input_format_t::bjdata, strict, allow_exceptions, error_handler);
     }
 
-    template<typename T>
-    JSON_HEDLEY_WARN_UNUSED_RESULT
-    JSON_HEDLEY_DEPRECATED_FOR(3.13.0, from_bjdata(ptr, ptr + len))
-    static basic_json from_bjdata(const T* ptr, std::size_t len,
-                                  const bool strict = true,
-                                  const bool allow_exceptions = true)
-    {
-        return from_bjdata(ptr, ptr + len, strict, allow_exceptions);
-    }
-
     /// @brief create a JSON value from an input in BON8 format
     /// @sa https://json.nlohmann.me/api/basic_json/from_bon8/
     template<typename InputType>
@@ -5969,16 +5959,6 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         return from_binary_impl(detail::input_adapter(std::move(first), std::move(last)), input_format_t::bon8, strict, allow_exceptions);
     }
 
-    template<typename T>
-    JSON_HEDLEY_WARN_UNUSED_RESULT
-    JSON_HEDLEY_DEPRECATED_FOR(3.13.0, from_bon8(ptr, ptr + len))
-    static basic_json from_bon8(const T* ptr, std::size_t len,
-                                const bool strict = true,
-                                const bool allow_exceptions = true)
-    {
-        return from_bon8(ptr, ptr + len, strict, allow_exceptions);
-    }
-
     /// @brief create a JSON value from an input in BSON format
     /// @sa https://json.nlohmann.me/api/basic_json/from_bson/
     template<typename InputType>

tests/src/unit-binary_formats.cpp

@@ -11,7 +11,12 @@
 #include <nlohmann/json.hpp>
 using nlohmann::json;
 
+#include <cmath>
 #include <fstream>
+#include <limits>
+#include <map>
+#include <string>
+#include <vector>
 #include "make_test_data_available.hpp"
 
 TEST_CASE("Binary Formats" * doctest::skip())
@@ -224,3 +229,139 @@ TEST_CASE("Binary Formats" * doctest::skip())
         CHECK((100.0 * double(ubjson_3_size) / double(json_size)) == Approx(89.450));
     }
 }
+
+namespace
+{
+// the binary formats as function pointers for "Binary formats with narrow number types";
+// named functions rather than lambdas, because clang 3.5 cannot convert a lambda
+// to a function pointer in the braced initializer of the format table
+using narrow_json = nlohmann::basic_json<std::map, std::vector, std::string, bool, std::int32_t, std::uint32_t, float>;
+using bytes = std::vector<std::uint8_t>;
+
+bytes encode_cbor(const json& j)
+{
+    return json::to_cbor(j);
+}
+narrow_json decode_cbor(const bytes& v, bool allow_exceptions)
+{
+    return narrow_json::from_cbor(v, true, allow_exceptions);
+}
+
+bytes encode_msgpack(const json& j)
+{
+    return json::to_msgpack(j);
+}
+narrow_json decode_msgpack(const bytes& v, bool allow_exceptions)
+{
+    return narrow_json::from_msgpack(v, true, allow_exceptions);
+}
+
+bytes encode_ubjson(const json& j)
+{
+    return json::to_ubjson(j);
+}
+narrow_json decode_ubjson(const bytes& v, bool allow_exceptions)
+{
+    return narrow_json::from_ubjson(v, true, allow_exceptions);
+}
+
+bytes encode_bjdata(const json& j)
+{
+    return json::to_bjdata(j);
+}
+narrow_json decode_bjdata(const bytes& v, bool allow_exceptions)
+{
+    return narrow_json::from_bjdata(v, true, allow_exceptions);
+}
+
+// BSON can only store numbers as object members
+bytes encode_bson(const json& j)
+{
+    return json::to_bson(json{{"a", j}});
+}
+narrow_json decode_bson(const bytes& v, bool allow_exceptions)
+{
+    const auto result = narrow_json::from_bson(v, true, allow_exceptions);
+    return result.is_discarded() ? result : result.at("a");
+}
+
+bytes encode_bon8(const json& j)
+{
+    return json::to_bon8(j);
+}
+narrow_json decode_bon8(const bytes& v, bool allow_exceptions)
+{
+    return narrow_json::from_bon8(v, true, allow_exceptions);
+}
+
+} // namespace
+
+TEST_CASE("Binary formats with narrow number types")
+{
+    // Numbers that do not fit the number types are handled like the lexer
+    // handles them in JSON text: an integer that fits neither integer type is
+    // stored as a floating-point number, and a finite floating-point number
+    // that overflows number_float_t is rejected with out_of_range.406.
+    struct binary_format
+    {
+        const char* name;
+        bytes (*encode)(const json&);
+        narrow_json (*decode)(const bytes&, bool);
+    };
+
+    const std::vector<binary_format> formats =
+    {
+        {"CBOR", encode_cbor, decode_cbor},
+        {"MessagePack", encode_msgpack, decode_msgpack},
+        {"UBJSON", encode_ubjson, decode_ubjson},
+        {"BJData", encode_bjdata, decode_bjdata},
+        {"BSON", encode_bson, decode_bson},
+        {"BON8", encode_bon8, decode_bon8},
+    };
+
+    for (const auto& format : formats)
+    {
+        const std::string name = format.name;
+        INFO("format := ", name);
+        const auto roundtrip = [&format](const json & j)
+        {
+            return format.decode(format.encode(j), true);
+        };
+
+        // integers that fit keep their type
+        CHECK(roundtrip(json(-5)).is_number_integer());
+        CHECK(roundtrip(json(-5)).get<std::int32_t>() == -5);
+        CHECK(roundtrip(json(3000000000u)).is_number_unsigned());
+        CHECK(roundtrip(json(3000000000u)).get<std::uint32_t>() == 3000000000u);
+
+        // integers that fit neither integer type are stored as float
+        CHECK(roundtrip(json(5000000000u)).is_number_float());
+        CHECK(roundtrip(json(5000000000u)).get<float>() == 5000000000.0f);
+        if (name != "BON8") // BON8 cannot encode integers above INT64_MAX
+        {
+            CHECK(roundtrip(json(10000000000000000000u)).is_number_float());
+            CHECK(roundtrip(json(10000000000000000000u)).get<float>() == 10000000000000000000.0f);
+        }
+        CHECK(roundtrip(json(-3000000000LL)).is_number_float());
+        CHECK(roundtrip(json(-3000000000LL)).get<float>() == -3000000000.0f);
+        CHECK(roundtrip(json(-5000000000LL)).is_number_float());
+        CHECK(roundtrip(json(-5000000000LL)).get<float>() == -5000000000.0f);
+
+        // floating-point numbers that fit
+        CHECK(roundtrip(json(1.5)).get<float>() == 1.5f);
+        const auto just_above_max = std::nextafter(static_cast<double>((std::numeric_limits<float>::max)()),
+                                    std::numeric_limits<double>::infinity());
+        CHECK(roundtrip(json(just_above_max)).get<float>() == (std::numeric_limits<float>::max)());
+
+        // infinity and NaN are passed on
+        CHECK(std::isinf(roundtrip(json(std::numeric_limits<double>::infinity())).get<float>()));
+        CHECK(std::isnan(roundtrip(json(std::numeric_limits<double>::quiet_NaN())).get<float>()));
+
+        // finite floating-point numbers that overflow number_float_t are rejected
+        const std::string message = "[json.exception.out_of_range.406] syntax error while parsing " + name
+                                    + " value: number overflow";
+        CHECK_THROWS_WITH_AS(roundtrip(json(1e300)), message.c_str(), narrow_json::out_of_range&);
+        CHECK_THROWS_WITH_AS(roundtrip(json(-1e300)), message.c_str(), narrow_json::out_of_range&);
+        CHECK(format.decode(format.encode(json(1e300)), false).is_discarded());
+    }
+}

tests/src/unit-bjdata.cpp

@@ -4490,31 +4490,3 @@ TEST_CASE("BJData roundtrips" * doctest::skip())
         }
     }
 }
-
-TEST_CASE("issue #5648 - from_bjdata(ptr, len) must read len bytes, not treat ptr as a C string")
-{
-    // to_bjdata() encodes the integer 0 as the two bytes 'i' 0x00 (a BJData
-    // type marker followed by the value byte 0x00), so the packed data
-    // below contains a 0x00 byte before its end.
-    const json j = {{"a", 0}};
-    const std::vector<std::uint8_t> packed = json::to_bjdata(j);
-    bool contains_nul = false;
-    for (const auto byte : packed)
-    {
-        contains_nul |= (byte == 0x00);
-    }
-    REQUIRE(contains_nul);
-
-    // before the fix, from_bjdata had no (ptr, len) overload, so this call
-    // bound to from_bjdata(InputType&&, bool strict) instead: ptr was read
-    // as a NUL-terminated C string (stopping at the embedded 0x00 byte), and
-    // len was silently converted to the strict flag. The deprecated
-    // overload added for this issue forwards to from_bjdata(ptr, ptr + len,
-    // ...) instead, like from_ubjson's deprecated (ptr, len) overload does.
-    json result;
-    CHECK_NOTHROW(result = json::from_bjdata(packed.data(), packed.size()));
-    CHECK(result == j);
-
-    // len must not collapse into the strict flag either
-    CHECK(json::from_bjdata(packed.data(), packed.size(), false) == j);
-}

tests/src/unit-bon8.cpp

@@ -1012,34 +1012,6 @@ TEST_CASE("BON8 roundtrips" * doctest::skip())
     }
 }
 
-TEST_CASE("issue #5648 - from_bon8(ptr, len) must read len bytes, not treat ptr as a C string")
-{
-    // to_bon8() encodes the integer 40 as the two bytes 0xC2 0x00 (a BON8
-    // lead byte followed by a continuation byte of 0x00), so the packed data
-    // below contains a 0x00 byte before its end.
-    const json j = {{"a", 40}, {"b", "x"}};
-    const std::vector<std::uint8_t> packed = json::to_bon8(j);
-    bool contains_nul = false;
-    for (const auto byte : packed)
-    {
-        contains_nul |= (byte == 0x00);
-    }
-    REQUIRE(contains_nul);
-
-    // before the fix, from_bon8 had no (ptr, len) overload, so this call
-    // bound to from_bon8(InputType&&, bool strict) instead: ptr was read as
-    // a NUL-terminated C string (stopping at the embedded 0x00 byte), and
-    // len was silently converted to the strict flag. The deprecated
-    // overload added for this issue forwards to from_bon8(ptr, ptr + len,
-    // ...) instead, like from_cbor's deprecated (ptr, len) overload does.
-    json result;
-    CHECK_NOTHROW(result = json::from_bon8(packed.data(), packed.size()));
-    CHECK(result == j);
-
-    // len must not collapse into the strict flag either
-    CHECK(json::from_bon8(packed.data(), packed.size(), false) == j);
-}
-
 #ifdef JSON_HAS_CPP_17
 TEST_CASE("BON8 with std::byte")
 {

tests/src/unit-cbor.cpp

@@ -3183,7 +3183,8 @@ TEST_CASE("Tagged values")
         // CBOR encodes negative integers as: result = -1 - n
         // For type 0x3B, n is an 8-byte uint64_t. Valid range for n with
         // the default int64_t is [0, INT64_MAX], producing results in [INT64_MIN, -1].
-        // When n > INT64_MAX, the result exceeds int64_t range and is rejected.
+        // When n > INT64_MAX, the result exceeds int64_t range and is stored
+        // as a floating-point number, as the lexer does for JSON text.
 
         SECTION("n = 0 is valid (result = -1)")
         {
@@ -3204,33 +3205,34 @@ TEST_CASE("Tagged values")
             CHECK(result.get<int64_t>() == (std::numeric_limits<int64_t>::min)());
         }
 
-        SECTION("n = INT64_MAX + 1 is rejected (overflow)")
+        SECTION("n = INT64_MAX + 1 is stored as float")
         {
             // n = INT64_MAX + 1 (0x8000000000000000)
-            // result = -1 - n = -9223372036854775809, which exceeds int64_t range
+            // result = -1 - n = -9223372036854775809, which exceeds int64_t range;
+            // the nearest double is -9223372036854775808.0
             const std::vector<uint8_t> input = {0x3B, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
-            json _;
+            const auto result = json::from_cbor(input);
-            CHECK_THROWS_WITH_AS(_ = json::from_cbor(input),
+            CHECK(result.is_number_float());
-                                 "[json.exception.parse_error.112] parse error at byte 9: syntax error while parsing CBOR value: negative integer overflow",
+            CHECK(result.get<double>() == -9223372036854775808.0);
-                                 json::parse_error);
+            CHECK(result == json::parse("-9223372036854775809"));
         }
 
-        SECTION("n = UINT64_MAX is rejected (overflow)")
+        SECTION("n = UINT64_MAX is stored as float")
         {
             // n = UINT64_MAX (0xFFFFFFFFFFFFFFFF)
             // result = -1 - n = -18446744073709551616, which exceeds int64_t range
             const std::vector<uint8_t> input = {0x3B, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF};
-            json _;
+            const auto result = json::from_cbor(input);
-            CHECK_THROWS_WITH_AS(_ = json::from_cbor(input),
+            CHECK(result.is_number_float());
-                                 "[json.exception.parse_error.112] parse error at byte 9: syntax error while parsing CBOR value: negative integer overflow",
+            CHECK(result.get<double>() == -18446744073709551616.0);
-                                 json::parse_error);
+            CHECK(result == json::parse("-18446744073709551616"));
         }
 
-        SECTION("overflow with allow_exceptions=false returns discarded")
+        SECTION("overflow with allow_exceptions=false is not an error")
         {
             const std::vector<uint8_t> input = {0x3B, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
             const auto result = json::from_cbor(input, true, false);
-            CHECK(result.is_discarded());
+            CHECK(result.is_number_float());
         }
     }