Check the iterator range for binary values in basic_json(first, last)

basic_json(first, last) treated value_t::binary like the structured
types (array, object) in the range check, so it always copied the
whole binary value regardless of the iterators, even for an empty
range such as (b.end(), b.end()). The other primitive types (number,
boolean, string) already reject such a range with
invalid_iterator.204, and erase(first, last) already does the same
for binary values, so this made the constructor inconsistent with
both. Move case value_t::binary into the group of checked primitive
types.

Also update the two matching passages in basic_json.md that describe
overload 7, and add a version-history note.

Fixes #5670.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/basic_json.md

@@ -139,8 +139,8 @@ basic_json(basic_json&& other) noexcept;
 
     - In case of a `#!json null` type, [invalid_iterator.206](../../home/exceptions.md#jsonexceptioninvalid_iterator206)
       is thrown.
-    - In case of other primitive types (number, boolean, or string), `first` must be `begin()` and `last` must be
-      `end()`. In this case, the value is copied. Otherwise,
+    - In case of other primitive types (number, boolean, string, or binary), `first` must be `begin()` and `last`
+      must be `end()`. In this case, the value is copied. Otherwise,
       [`invalid_iterator.204`](../../home/exceptions.md#jsonexceptioninvalid_iterator204) is thrown.
     - In case of structured types (array, object), the constructor behaves as similar versions for `std::vector` or
       `std::map`; that is, a JSON array or object is constructed from the values in the range.
@@ -242,8 +242,8 @@ basic_json(basic_json&& other) noexcept;
       and `last` are not compatible (i.e., do not belong to the same JSON value). In this case, the range
       `[first, last)` is undefined.
     - Throws [`invalid_iterator.204`](../../home/exceptions.md#jsonexceptioninvalid_iterator204) if iterators `first`
-      and `last` belong to a primitive type (number, boolean, or string), but `first` does not point to the first
-      element anymore. In this case, the range `[first, last)` is undefined. See the example code below.
+      and `last` belong to a primitive type (number, boolean, string, or binary), but `first` does not point to the
+      first element anymore. In this case, the range `[first, last)` is undefined. See the example code below.
     - Throws [`invalid_iterator.206`](../../home/exceptions.md#jsonexceptioninvalid_iterator206) if iterators `first`
       and `last` belong to a `#!json null` value. In this case, the range `[first, last)` is undefined.
 8. (none)
@@ -423,6 +423,8 @@ basic_json(basic_json&& other) noexcept;
 4. Since version 3.2.0.
 5. Since version 1.0.0.
 6. Since version 1.0.0.
-7. Since version 1.0.0.
+7. Since version 1.0.0. Fixed in version 3.13.0 to also check the iterator range for binary values; before, a range
+   that did not cover the whole value (such as `(end(), end())`) was accepted and the whole binary value was copied,
+   unlike the other primitive types.
 8. Since version 1.0.0.
 9. Since version 1.0.0.

docs/mkdocs/docs/features/binary_formats/bjdata.md

@@ -132,14 +132,8 @@ The library uses the following mapping from JSON values types to BJData types ac
       parsed back as a regular array,
     - every entry of `"_ArraySize_"` is a positive integer, and their product is representable as a `std::size_t`,
     - `"_ArrayData_"` is an array holding exactly that many elements, and
-    - every element of `"_ArrayData_"` is a number of the kind named by `"_ArrayType_"`: for the integer types, a
-      value that fits the named width; for `double`, any value; for `single`, a value that survives narrowing to
-      `float` and back without change (for instance, `0.1` does not, since it is not exactly representable as
-      `float`).
-
-    An annotated object is always read back with its keys in the order shown above, `"_ArrayType_"`, `"_ArraySize_"`,
-    `"_ArrayData_"`, regardless of the order the ND-array's header stores them in on the wire. This matters for
-    `ordered_json`, whose comparison takes key order into account.
+    - every element of `"_ArrayData_"` is a number of the kind named by `"_ArrayType_"` (a floating-point number for
+      `single` and `double`, an integer otherwise).
 
     The current version of this library does not yet support automatic detection of and conversion from a nested JSON
     array input to a BJData ND-array.

include/nlohmann/detail/input/binary_reader.hpp

@@ -2728,15 +2728,10 @@ class binary_reader
                                is_ndarray can only return `true` when its initial value
                                is `false`
     @param[in] prefix  type marker if already read, otherwise set to 0
-    @param[in] ndarray_dtype  the element type marker of the enclosing bjdata ndarray if
-                              already known (it precedes the dimension vector read here),
-                              otherwise 0; used to emit the "_ArrayType_" annotation key
-                              before "_ArraySize_" if a dimension vector turns out to
-                              describe an ndarray
 
     @return whether size determination completed
     */
-    bool get_ubjson_size_value(std::size_t& result, bool& is_ndarray, char_int_type prefix = 0, char_int_type ndarray_dtype = 0)
+    bool get_ubjson_size_value(std::size_t& result, bool& is_ndarray, char_int_type prefix = 0)
     {
         if (prefix == 0)
         {
@@ -2906,37 +2901,8 @@ class binary_reader
                         }
                     }
 
-                    if (JSON_HEDLEY_UNLIKELY(!sax->start_object(3)))
-                    {
-                        return false;
-                    }
-
-                    // the element type precedes the dimension vector (see get_ubjson_size_type)
-                    // and is passed down as ndarray_dtype; emit it here so the annotation keys
-                    // follow the documented _ArrayType_, _ArraySize_, _ArrayData_ order
-                    if (ndarray_dtype != 0)
-                    {
-                        auto it = std::lower_bound(bjd_types_map.begin(), bjd_types_map.end(), ndarray_dtype, [](const bjd_type & p, char_int_type t)
-                        {
-                            return p.first < t;
-                        });
-                        if (JSON_HEDLEY_UNLIKELY(it == bjd_types_map.end() || it->first != ndarray_dtype))
-                        {
-                            auto last_token = get_token_string();
-                            return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read,
-                                                    exception_message(input_format, "invalid byte: 0x" + last_token, "type"), nullptr));
-                        }
-
-                        string_t type_key = "_ArrayType_";
-                        string_t type = it->second; // sax->string() takes a reference
-                        if (JSON_HEDLEY_UNLIKELY(!sax->key(type_key) || !sax->string(type)))
-                        {
-                            return false;
-                        }
-                    }
-
                     string_t key = "_ArraySize_";
-                    if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->start_array(dim.size())))
+                    if (JSON_HEDLEY_UNLIKELY(!sax->start_object(3) || !sax->key(key) || !sax->start_array(dim.size())))
                     {
                         return false;
                     }
@@ -3037,7 +3003,7 @@ class binary_reader
                                         exception_message(input_format, concat("expected '#' after type information; last byte: 0x", last_token), "size"), nullptr));
             }
 
-            const bool is_error = get_ubjson_size_value(result.first, is_ndarray, 0, result.second);
+            const bool is_error = get_ubjson_size_value(result.first, is_ndarray);
             // an ndarray was read here only if the flag flipped; when it was
             // seeded true, get_ubjson_size_value() already rejected the nested
             // dimension vector
@@ -3273,17 +3239,30 @@ class binary_reader
         if (input_format == input_format_t::bjdata && size_and_type.first != npos && (size_and_type.second & (1 << 8)) != 0)
         {
             size_and_type.second &= ~(static_cast<char_int_type>(1) << 8);  // use bit 8 to indicate ndarray, here we remove the bit to restore the type marker
+            auto it = std::lower_bound(bjd_types_map.begin(), bjd_types_map.end(), size_and_type.second, [](const bjd_type & p, char_int_type t)
+            {
+                return p.first < t;
+            });
+            string_t key = "_ArrayType_";
+            if (JSON_HEDLEY_UNLIKELY(it == bjd_types_map.end() || it->first != size_and_type.second))
+            {
+                auto last_token = get_token_string();
+                return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read,
+                                        exception_message(input_format, "invalid byte: 0x" + last_token, "type"), nullptr));
+            }
+
+            string_t type = it->second; // sax->string() takes a reference
+            if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->string(type)))
+            {
+                return false;
+            }
 
-            // the "_ArrayType_" and "_ArraySize_" annotation keys were already emitted by
-            // get_ubjson_size_value() (the type marker is known before the dimension vector
-            // that determines size_and_type.first is read, so it is emitted first there to
-            // match the documented _ArrayType_, _ArraySize_, _ArrayData_ key order)
             if (size_and_type.second == 'C' || size_and_type.second == 'B')
             {
                 size_and_type.second = 'U';
             }
 
-            string_t key = "_ArrayData_";
+            key = "_ArrayData_";
             if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->start_array(size_and_type.first) ))
             {
                 return false;

include/nlohmann/detail/output/binary_writer.hpp

@@ -1991,21 +1991,9 @@ class binary_writer
                 case 'd':
                 {
                     const auto dval = el.template get<double>();
-#ifdef __GNUC__
-                    JSON_HEDLEY_DIAGNOSTIC_PUSH
-                    JSON_HEDLEY_PRAGMA(GCC diagnostic ignored "-Wfloat-equal")
-#endif
-                    // a value that would be rounded (rather than exactly represented) by the
-                    // narrowing to float is treated like an out-of-range integer element above;
-                    // this is the same criterion write_compact_float() uses for CBOR/MessagePack
-                    in_range = std::isnan(dval) ||
+                    in_range = !std::isfinite(dval) ||
                                (dval >= static_cast<double>(std::numeric_limits<float>::lowest()) &&
-                                dval <= static_cast<double>((std::numeric_limits<float>::max)()) &&
-                                static_cast<double>(static_cast<float>(dval)) == dval) ||
-                               std::isinf(dval);
-#ifdef __GNUC__
-                    JSON_HEDLEY_DIAGNOSTIC_POP
-#endif
+                                dval <= static_cast<double>((std::numeric_limits<float>::max)()));
                     break;
                 }
                 default:

include/nlohmann/json.hpp

@@ -1829,6 +1829,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
             case value_t::number_integer:
             case value_t::number_unsigned:
             case value_t::string:
+            case value_t::binary:
             {
                 if (JSON_HEDLEY_UNLIKELY(!first.m_it.primitive_iterator.is_begin()
                                          || !last.m_it.primitive_iterator.is_end()))
@@ -1841,7 +1842,6 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
             case value_t::null:
             case value_t::object:
             case value_t::array:
-            case value_t::binary:
             case value_t::discarded:
             default:
                 break;

single_include/nlohmann/json.hpp

@@ -15495,15 +15495,10 @@ class binary_reader
                                is_ndarray can only return `true` when its initial value
                                is `false`
     @param[in] prefix  type marker if already read, otherwise set to 0
-    @param[in] ndarray_dtype  the element type marker of the enclosing bjdata ndarray if
-                              already known (it precedes the dimension vector read here),
-                              otherwise 0; used to emit the "_ArrayType_" annotation key
-                              before "_ArraySize_" if a dimension vector turns out to
-                              describe an ndarray
 
     @return whether size determination completed
     */
-    bool get_ubjson_size_value(std::size_t& result, bool& is_ndarray, char_int_type prefix = 0, char_int_type ndarray_dtype = 0)
+    bool get_ubjson_size_value(std::size_t& result, bool& is_ndarray, char_int_type prefix = 0)
     {
         if (prefix == 0)
         {
@@ -15673,37 +15668,8 @@ class binary_reader
                         }
                     }
 
-                    if (JSON_HEDLEY_UNLIKELY(!sax->start_object(3)))
-                    {
-                        return false;
-                    }
-
-                    // the element type precedes the dimension vector (see get_ubjson_size_type)
-                    // and is passed down as ndarray_dtype; emit it here so the annotation keys
-                    // follow the documented _ArrayType_, _ArraySize_, _ArrayData_ order
-                    if (ndarray_dtype != 0)
-                    {
-                        auto it = std::lower_bound(bjd_types_map.begin(), bjd_types_map.end(), ndarray_dtype, [](const bjd_type & p, char_int_type t)
-                        {
-                            return p.first < t;
-                        });
-                        if (JSON_HEDLEY_UNLIKELY(it == bjd_types_map.end() || it->first != ndarray_dtype))
-                        {
-                            auto last_token = get_token_string();
-                            return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read,
-                                                    exception_message(input_format, "invalid byte: 0x" + last_token, "type"), nullptr));
-                        }
-
-                        string_t type_key = "_ArrayType_";
-                        string_t type = it->second; // sax->string() takes a reference
-                        if (JSON_HEDLEY_UNLIKELY(!sax->key(type_key) || !sax->string(type)))
-                        {
-                            return false;
-                        }
-                    }
-
                     string_t key = "_ArraySize_";
-                    if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->start_array(dim.size())))
+                    if (JSON_HEDLEY_UNLIKELY(!sax->start_object(3) || !sax->key(key) || !sax->start_array(dim.size())))
                     {
                         return false;
                     }
@@ -15804,7 +15770,7 @@ class binary_reader
                                         exception_message(input_format, concat("expected '#' after type information; last byte: 0x", last_token), "size"), nullptr));
             }
 
-            const bool is_error = get_ubjson_size_value(result.first, is_ndarray, 0, result.second);
+            const bool is_error = get_ubjson_size_value(result.first, is_ndarray);
             // an ndarray was read here only if the flag flipped; when it was
             // seeded true, get_ubjson_size_value() already rejected the nested
             // dimension vector
@@ -16040,17 +16006,30 @@ class binary_reader
         if (input_format == input_format_t::bjdata && size_and_type.first != npos && (size_and_type.second & (1 << 8)) != 0)
         {
             size_and_type.second &= ~(static_cast<char_int_type>(1) << 8);  // use bit 8 to indicate ndarray, here we remove the bit to restore the type marker
+            auto it = std::lower_bound(bjd_types_map.begin(), bjd_types_map.end(), size_and_type.second, [](const bjd_type & p, char_int_type t)
+            {
+                return p.first < t;
+            });
+            string_t key = "_ArrayType_";
+            if (JSON_HEDLEY_UNLIKELY(it == bjd_types_map.end() || it->first != size_and_type.second))
+            {
+                auto last_token = get_token_string();
+                return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read,
+                                        exception_message(input_format, "invalid byte: 0x" + last_token, "type"), nullptr));
+            }
+
+            string_t type = it->second; // sax->string() takes a reference
+            if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->string(type)))
+            {
+                return false;
+            }
 
-            // the "_ArrayType_" and "_ArraySize_" annotation keys were already emitted by
-            // get_ubjson_size_value() (the type marker is known before the dimension vector
-            // that determines size_and_type.first is read, so it is emitted first there to
-            // match the documented _ArrayType_, _ArraySize_, _ArrayData_ key order)
             if (size_and_type.second == 'C' || size_and_type.second == 'B')
             {
                 size_and_type.second = 'U';
             }
 
-            string_t key = "_ArrayData_";
+            key = "_ArrayData_";
             if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->start_array(size_and_type.first) ))
             {
                 return false;
@@ -22342,21 +22321,9 @@ class binary_writer
                 case 'd':
                 {
                     const auto dval = el.template get<double>();
-#ifdef __GNUC__
-                    JSON_HEDLEY_DIAGNOSTIC_PUSH
-                    JSON_HEDLEY_PRAGMA(GCC diagnostic ignored "-Wfloat-equal")
-#endif
-                    // a value that would be rounded (rather than exactly represented) by the
-                    // narrowing to float is treated like an out-of-range integer element above;
-                    // this is the same criterion write_compact_float() uses for CBOR/MessagePack
-                    in_range = std::isnan(dval) ||
+                    in_range = !std::isfinite(dval) ||
                                (dval >= static_cast<double>(std::numeric_limits<float>::lowest()) &&
-                                dval <= static_cast<double>((std::numeric_limits<float>::max)()) &&
-                                static_cast<double>(static_cast<float>(dval)) == dval) ||
-                               std::isinf(dval);
-#ifdef __GNUC__
-                    JSON_HEDLEY_DIAGNOSTIC_POP
-#endif
+                                dval <= static_cast<double>((std::numeric_limits<float>::max)()));
                     break;
                 }
                 default:
@@ -27943,6 +27910,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
             case value_t::number_integer:
             case value_t::number_unsigned:
             case value_t::string:
+            case value_t::binary:
             {
                 if (JSON_HEDLEY_UNLIKELY(!first.m_it.primitive_iterator.is_begin()
                                          || !last.m_it.primitive_iterator.is_end()))
@@ -27955,7 +27923,6 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
             case value_t::null:
             case value_t::object:
             case value_t::array:
-            case value_t::binary:
             case value_t::discarded:
             default:
                 break;

tests/src/unit-bjdata.cpp

@@ -11,7 +11,6 @@
 #define JSON_TESTS_PRIVATE
 #include <nlohmann/json.hpp>
 using nlohmann::json;
-using ordered_json = nlohmann::ordered_json;
 
 #include <algorithm>
 #include <climits>
@@ -2295,33 +2294,29 @@ TEST_CASE("BJData")
 
         SECTION("start_array() in ndarray _ArraySize_")
         {
-            // _ArrayType_ (2 events: key + string) is now emitted before
-            // _ArraySize_ (see GitHub issue #5661), which shifts the events
-            // below later by the same 2 events
             std::vector<uint8_t> const v = {'[', '$', 'i', '#', '[', '$', 'i', '#', 'i', 2, 2, 1, 1, 2};
-            SaxCountdown scp(4);
+            SaxCountdown scp(2);
             CHECK_FALSE(json::sax_parse(v, &scp, json::input_format_t::bjdata));
         }
 
         SECTION("number_integer() in ndarray _ArraySize_")
         {
             std::vector<uint8_t> const v = {'[', '$', 'U', '#', '[', '$', 'i', '#', 'i', 2, 2, 1, 1, 2};
-            SaxCountdown scp(5);
+            SaxCountdown scp(3);
             CHECK_FALSE(json::sax_parse(v, &scp, json::input_format_t::bjdata));
         }
 
         SECTION("key() in ndarray _ArrayType_")
         {
-            // _ArrayType_ is emitted right after start_object(), before _ArraySize_
             std::vector<uint8_t> const v = {'[', '$', 'U', '#', '[', '$', 'U', '#', 'i', 2, 2, 2, 1, 2, 3, 4};
-            SaxCountdown scp(1);
+            SaxCountdown scp(6);
             CHECK_FALSE(json::sax_parse(v, &scp, json::input_format_t::bjdata));
         }
 
         SECTION("string() in ndarray _ArrayType_")
         {
             std::vector<uint8_t> const v = {'[', '$', 'U', '#', '[', '$', 'U', '#', 'i', 2, 2, 2, 1, 2, 3, 4};
-            SaxCountdown scp(2);
+            SaxCountdown scp(7);
             CHECK_FALSE(json::sax_parse(v, &scp, json::input_format_t::bjdata));
         }
 
@@ -2924,22 +2919,6 @@ TEST_CASE("BJData")
                 CHECK(out_single.at(0) == '{');
                 CHECK(json::from_bjdata(out_single) == j_single);
 
-                // a double element that is finite and within the range of "single"
-                // but is not exactly representable as a float, so narrowing it would
-                // silently round it (0.1 is read back as 0.10000000149011612); this,
-                // like the overflow case above, falls back to a plain object (see
-                // GitHub issue #5661)
-                json const j_single_rounded = json({{"_ArrayType_", "single"}, {"_ArraySize_", {2, 1}}, {"_ArrayData_", {1.5, 0.1}}});
-                const auto out_single_rounded = json::to_bjdata(j_single_rounded);
-                CHECK(out_single_rounded.at(0) == '{');
-                CHECK(json::from_bjdata(out_single_rounded) == j_single_rounded);
-
-                // a double element that underflows to 0 when narrowed to "single"
-                json const j_single_underflow = json({{"_ArrayType_", "single"}, {"_ArraySize_", {2, 1}}, {"_ArrayData_", {1.5, 1e-300}}});
-                const auto out_single_underflow = json::to_bjdata(j_single_underflow);
-                CHECK(out_single_underflow.at(0) == '{');
-                CHECK(json::from_bjdata(out_single_underflow) == j_single_underflow);
-
                 // in-range boundary values still use the compact ndarray encoding
                 json const j_uint8_ok = json({{"_ArrayType_", "uint8"}, {"_ArraySize_", {2, 1}}, {"_ArrayData_", {0, 255}}});
                 CHECK(json::to_bjdata(j_uint8_ok) == std::vector<uint8_t>({'[', '$', 'U', '#', '[', 'i', 2, 'i', 1, ']', 0, 255}));
@@ -2953,23 +2932,6 @@ TEST_CASE("BJData")
                 CHECK(json::from_bjdata(out_single_ok) == json({{"_ArrayType_", "single"}, {"_ArraySize_", {2, 1}}, {"_ArrayData_", {1.5f, -1.5f}}}));
             }
 
-            SECTION("ndarray annotation keys are read back in the documented order")
-            {
-                // from_bjdata() must emit the annotation object's keys in the order
-                // used throughout the documentation, _ArrayType_, _ArraySize_,
-                // _ArrayData_: the type marker precedes the dimension vector on the
-                // wire (see get_ubjson_size_type()), so it is known, and emitted,
-                // before _ArraySize_. For a plain json this key order is invisible
-                // (its comparison ignores it), but for an ordered_json it is not (see
-                // GitHub issue #5661).
-                const ordered_json o = ordered_json::parse(R"({"_ArrayType_":"uint8","_ArraySize_":[2,2],"_ArrayData_":[1,2,3,4]})");
-                const auto packed = ordered_json::to_bjdata(o);
-                CHECK(packed.at(0) == '[');
-                const ordered_json o_back = ordered_json::from_bjdata(packed);
-                CHECK(o_back == o);
-                CHECK(o_back.dump() == o.dump());
-            }
-
             SECTION("ndarray that would not be read back as an annotated object stays as object")
             {
                 // the reader only restores an annotated object from an ND-array

tests/src/unit-constructor1.cpp

@@ -1648,6 +1648,20 @@ TEST_CASE("constructors")
                         CHECK_THROWS_WITH_AS(json(j.cbegin(), j.cbegin()), "[json.exception.invalid_iterator.204] iterators out of range", json::invalid_iterator&);
                     }
                 }
+
+                SECTION("binary")
+                {
+                    {
+                        json j = json::binary({1, 2, 3});
+                        CHECK_THROWS_WITH_AS(json(j.end(), j.end()), "[json.exception.invalid_iterator.204] iterators out of range", json::invalid_iterator&);
+                        CHECK_THROWS_WITH_AS(json(j.begin(), j.begin()), "[json.exception.invalid_iterator.204] iterators out of range", json::invalid_iterator&);
+                    }
+                    {
+                        json const j = json::binary({1, 2, 3});
+                        CHECK_THROWS_WITH_AS(json(j.cend(), j.cend()), "[json.exception.invalid_iterator.204] iterators out of range", json::invalid_iterator&);
+                        CHECK_THROWS_WITH_AS(json(j.cbegin(), j.cbegin()), "[json.exception.invalid_iterator.204] iterators out of range", json::invalid_iterator&);
+                    }
+                }
             }
         }
     }