Merge branch 'json-view/23-zmij' into json-view/08-view-builder

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

.github/workflows/check_amalgamation.yml

@@ -152,6 +152,13 @@ jobs:
           VERSION_MAJOR=$(grep -m1 'define NLOHMANN_JSON_VERSION_MAJOR' $ABI_MACROS | grep -o '[0-9]\+')
           VERSION_MINOR=$(grep -m1 'define NLOHMANN_JSON_VERSION_MINOR' $ABI_MACROS | grep -o '[0-9]\+')
           VERSION_PATCH=$(grep -m1 'define NLOHMANN_JSON_VERSION_PATCH' $ABI_MACROS | grep -o '[0-9]\+')
+          # The template is what this check verifies the committed natvis
+          # file against, so take it from this PR rather than from the
+          # develop checkout: generate_natvis.py loads the template from its
+          # own directory, and with develop's copy a PR that changes the
+          # template could never pass. Like the macro_builder step above,
+          # this only renders PR content in a job without credentials.
+          cp tools/generate_natvis/nlohmann_json.natvis.j2 $NATVIS_TOOL_DIR/
           python3 $NATVIS_TOOL_DIR/generate_natvis.py --version "$VERSION_MAJOR.$VERSION_MINOR.$VERSION_PATCH" $MAIN_DIR
 
       - name: Build patch and check for differences

README.md

@@ -361,7 +361,7 @@ std::cout << j_string << " == " << serialized_string << std::endl;
 
 [`.dump()`](https://json.nlohmann.me/api/basic_json/dump/) returns the originally stored string value.
 
-Note the library only supports UTF-8. When you store strings with different encodings in the library, calling [`dump()`](https://json.nlohmann.me/api/basic_json/dump/) may throw an exception unless `json::error_handler_t::replace` or `json::error_handler_t::ignore` are used as error handlers.
+Note the library only supports UTF-8. When you store strings with different encodings in the library, calling [`dump()`](https://json.nlohmann.me/api/basic_json/dump/) may throw an exception unless `json::error_handler_t::replace`, `json::error_handler_t::ignore`, or `json::error_handler_t::keep` are used as error handlers.
 
 #### To/from streams (e.g., files, string streams)
 
@@ -1927,7 +1927,7 @@ The library supports **Unicode input** as follows:
 - [Unicode noncharacters](https://www.unicode.org/faq/private_use.html#nonchar1) will not be replaced by the library.
 - Invalid surrogates (e.g., incomplete pairs such as `\uDEAD`) will yield parse errors.
 - The strings stored in the library are UTF-8 encoded. When using the default string type (`std::string`), note that its length/size functions return the number of stored bytes rather than the number of characters or glyphs.
-- When you store strings with different encodings in the library, calling [`dump()`](https://json.nlohmann.me/api/basic_json/dump/) may throw an exception unless `json::error_handler_t::replace` or `json::error_handler_t::ignore` are used as error handlers.
+- When you store strings with different encodings in the library, calling [`dump()`](https://json.nlohmann.me/api/basic_json/dump/) may throw an exception unless `json::error_handler_t::replace`, `json::error_handler_t::ignore`, or `json::error_handler_t::keep` are used as error handlers.
 - To store wide strings (e.g., `std::wstring`), you need to convert them to a UTF-8 encoded `std::string` before, see [an example](https://json.nlohmann.me/home/faq/#wide-string-handling).
 
 ### Comments in JSON

docs/mkdocs/docs/api/basic_json/std_hash.md

@@ -7,8 +7,14 @@ namespace std {
 ```
 
 Return a hash value for a JSON object. The hash function tries to rely on `std::hash` where possible. Furthermore, the
-type of the JSON value is taken into account to have different hash values for `#!json null`, `#!cpp 0`, `#!cpp 0U`, and
+type of the JSON value is taken into account, so `#!json null`, `#!cpp false`, and numbers may hash differently from
-`#!cpp false`, etc.
+each other. Numbers that compare equal under [`operator==`](operator_eq.md) always hash equally, regardless of
+whether they are stored as signed integer, unsigned integer, or floating-point number.
+
+Numbers are hashed by their value converted to `number_float_t`. Converting an integer to `number_float_t` therefore
+keeps its hash, but converting a floating-point number to an integer type is lossy and can change it: `#!cpp 0.5`
+converts to `#!cpp 0`, which need not have the same hash. Unequal numbers can also share a hash value, for example two
+large integers that convert to the same `number_float_t`.
 
 ## Examples
 
@@ -26,7 +32,8 @@ type of the JSON value is taken into account to have different hash values for `
     --8<-- "examples/std_hash.output"
     ```
 
-    Note the output is platform-dependent.
+    The hash values shown are examples only. They depend on the platform, the compiler, and the compiler version, and
+    they can change between versions of this library. Do not persist them or rely on specific values.
 
 ## See also
 
@@ -36,3 +43,5 @@ type of the JSON value is taken into account to have different hash values for `
 
 - Added in version 1.0.0.
 - Extended for arbitrary basic_json types in version 3.10.5.
+- Numbers that compare equal hash equally since version 3.13.0; before, `#!cpp 0`, `#!cpp 0U`, and `#!cpp 0.0` had
+  different hash values.

docs/mkdocs/docs/examples/std_hash.cpp

@@ -11,6 +11,7 @@ int main()
               << "hash(false) = " << std::hash<json> {}(json(false)) << '\n'
               << "hash(0) = " << std::hash<json> {}(json(0)) << '\n'
               << "hash(0U) = " << std::hash<json> {}(json(0U)) << '\n'
+              << "hash(0.0) = " << std::hash<json> {}(json(0.0)) << '\n'
               << "hash(\"\") = " << std::hash<json> {}(json("")) << '\n'
               << "hash({}) = " << std::hash<json> {}(json::object()) << '\n'
               << "hash([]) = " << std::hash<json> {}(json::array()) << '\n'

docs/mkdocs/docs/examples/std_hash.output

@@ -1,8 +1,9 @@
 hash(null) = 2654435769
 hash(false) = 2654436030
-hash(0) = 2654436095
+hash(0) = 2654436221
-hash(0U) = 2654436156
+hash(0U) = 2654436221
-hash("") = 6142509191626859748
+hash(0.0) = 2654436221
+hash("") = 11160318156688833227
 hash({}) = 2654435832
 hash([]) = 2654435899
-hash({"hello": "world"}) = 4469488738203676328
+hash({"hello": "world"}) = 3701319991624763853

docs/mkdocs/docs/features/binary_formats/bjdata.md

@@ -141,8 +141,14 @@ The library uses the following mapping from JSON values types to BJData types ac
       parsed back as a regular array,
     - every entry of `"_ArraySize_"` is a positive integer, and their product is representable as a `std::size_t`,
     - `"_ArrayData_"` is an array holding exactly that many elements, and
-    - every element of `"_ArrayData_"` is a number of the kind named by `"_ArrayType_"` (a floating-point number for
+    - every element of `"_ArrayData_"` is a number of the kind named by `"_ArrayType_"`: for the integer types, a
-      `single` and `double`, an integer otherwise).
+      value that fits the named width; for `double`, any value; for `single`, a value that survives narrowing to
+      `float` and back without change (for instance, `0.1` does not, since it is not exactly representable as
+      `float`).
+
+    An annotated object is always read back with its keys in the order shown above, `"_ArrayType_"`, `"_ArraySize_"`,
+    `"_ArrayData_"`, regardless of the order the ND-array's header stores them in on the wire. This matters for
+    `ordered_json`, whose comparison takes key order into account.
 
     The current version of this library does not yet support automatic detection of and conversion from a nested JSON
     array input to a BJData ND-array.

docs/mkdocs/docs/features/serialization.md

@@ -64,6 +64,7 @@ serialization fails by default. The fourth argument of `dump` selects an
 - `strict` (default) — throw a [`type_error.316`](../home/exceptions.md#jsonexceptiontype_error316) exception.
 - `replace` — replace invalid bytes with the Unicode replacement character U+FFFD (`�`).
 - `ignore` — silently drop invalid bytes.
+- `keep` — copy invalid bytes to the output unchanged; the result is not valid UTF-8.
 
 ??? example "Example: serialize invalid UTF-8 with different error handlers"
 

docs/mkdocs/docs/home/exceptions.md

@@ -771,6 +771,7 @@ as well for a string value or object key that is not valid UTF-8 if their `error
     - Pass an error handler as last parameter to the `dump()` function to avoid this exception:
         - `json::error_handler_t::replace` will replace invalid bytes sequences with `U+FFFD` 
         - `json::error_handler_t::ignore` will silently ignore invalid byte sequences
+        - `json::error_handler_t::keep` will copy invalid byte sequences to the output unchanged
 
 ### json.exception.type_error.317
 

docs/mkdocs/docs/home/faq.md

@@ -85,7 +85,7 @@ The library supports **Unicode input** as follows:
 - The library will not replace [Unicode noncharacters](http://www.unicode.org/faq/private_use.html#nonchar1).
 - Invalid surrogates (e.g., incomplete pairs such as `\uDEAD`) will yield parse errors.
 - The strings stored in the library are UTF-8 encoded. When using the default string type (`std::string`), note that its length/size functions return the number of stored bytes rather than the number of characters or glyphs.
-- When you store strings with different encodings in the library, calling [`dump()`](../api/basic_json/dump.md) may throw an exception unless `json::error_handler_t::replace` or `json::error_handler_t::ignore` are used as error handlers.
+- When you store strings with different encodings in the library, calling [`dump()`](../api/basic_json/dump.md) may throw an exception unless `json::error_handler_t::replace`, `json::error_handler_t::ignore`, or `json::error_handler_t::keep` are used as error handlers.
 
 In most cases, the parser is right to complain, because the input is not UTF-8 encoded. This is especially true for Microsoft Windows, where Latin-1 or ISO 8859-1 is often the standard encoding.
 

include/nlohmann/detail/hash.hpp

@@ -35,8 +35,10 @@ std::size_t hash_iteratively(const BasicJsonType& j);
 @brief hash a JSON value
 
 The hash function tries to rely on std::hash where possible. Furthermore, the
-type of the JSON value is taken into account to have different hash values for
+type of the JSON value is taken into account, so null, false, and numbers may
-null, 0, 0U, and false, etc.
+hash differently from each other, but any two numbers that compare equal
+under operator== hash equally regardless of which of number_integer,
+number_unsigned, or number_float actually holds the value.
 
 Hashing an array or an object hashes its elements, which used to call this
 function again once per nesting level, so a value nested deeply enough
@@ -55,8 +57,6 @@ template<typename BasicJsonType>
 std::size_t hash(const BasicJsonType& j, const std::size_t depth = 0)
 {
     using string_t = typename BasicJsonType::string_t;
-    using number_integer_t = typename BasicJsonType::number_integer_t;
-    using number_unsigned_t = typename BasicJsonType::number_unsigned_t;
     using number_float_t = typename BasicJsonType::number_float_t;
 
     const auto type = static_cast<std::size_t>(j.type());
@@ -113,21 +113,24 @@ std::size_t hash(const BasicJsonType& j, const std::size_t depth = 0)
         }
 
         case BasicJsonType::value_t::number_integer:
-        {
-            const auto h = std::hash<number_integer_t> {}(j.template get<number_integer_t>());
-            return combine(type, h);
-        }
-
         case BasicJsonType::value_t::number_unsigned:
-        {
-            const auto h = std::hash<number_unsigned_t> {}(j.template get<number_unsigned_t>());
-            return combine(type, h);
-        }
-
         case BasicJsonType::value_t::number_float:
         {
-            const auto h = std::hash<number_float_t> {}(j.template get<number_float_t>());
+            // operator== compares numbers by their mathematical value across
-            return combine(type, h);
+            // number_integer, number_unsigned, and number_float, so equal
+            // numbers of different internal types (0, 0U, 0.0) must hash the
+            // same. Two equal numbers have the same value, which converts to
+            // the same number_float_t, so all numbers share one type tag and
+            // hash that converted value. Adding zero turns -0.0 (equal to 0)
+            // into 0.0, as std::hash need not map both to the same hash.
+            // The converse does not hold: converting a number_float_t value
+            // to an integer type is lossy, so the result can hash
+            // differently, and unequal numbers that convert to the same
+            // number_float_t (e.g., 2^53 and 2^53 + 1) share a hash.
+            const auto number_type = static_cast<std::size_t>(BasicJsonType::value_t::number_float);
+            const auto value = j.template get<number_float_t>() + static_cast<number_float_t>(0);
+            const auto h = std::hash<number_float_t> {}(value);
+            return combine(number_type, h);
         }
 
         case BasicJsonType::value_t::binary:

include/nlohmann/detail/input/binary_reader.hpp

@@ -2716,10 +2716,15 @@ class binary_reader
                                is_ndarray can only return `true` when its initial value
                                is `false`
     @param[in] prefix  type marker if already read, otherwise set to 0
+    @param[in] ndarray_dtype  the element type marker of the enclosing bjdata ndarray if
+                              already known (it precedes the dimension vector read here),
+                              otherwise 0; used to emit the "_ArrayType_" annotation key
+                              before "_ArraySize_" if a dimension vector turns out to
+                              describe an ndarray
 
     @return whether size determination completed
     */
-    bool get_ubjson_size_value(std::size_t& result, bool& is_ndarray, char_int_type prefix = 0)
+    bool get_ubjson_size_value(std::size_t& result, bool& is_ndarray, char_int_type prefix = 0, char_int_type ndarray_dtype = 0)
     {
         if (prefix == 0)
         {
@@ -2832,8 +2837,34 @@ class binary_reader
                         }
                     }
 
+                    if (JSON_HEDLEY_UNLIKELY(!sax->start_object(3)))
+                    {
+                        return false;
+                    }
+
+                    // the element type precedes the dimension vector (see get_ubjson_size_type)
+                    // and is passed down as ndarray_dtype; emit it here so the annotation keys
+                    // follow the documented _ArrayType_, _ArraySize_, _ArrayData_ order
+                    if (ndarray_dtype != 0)
+                    {
+                        const char* type_name = bjd_type_name(ndarray_dtype);
+                        if (JSON_HEDLEY_UNLIKELY(type_name == nullptr))
+                        {
+                            auto last_token = get_token_string();
+                            return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read,
+                                                    exception_message("invalid byte: 0x" + last_token, "type"), nullptr));
+                        }
+
+                        string_t type_key = "_ArrayType_";
+                        string_t type = type_name; // sax->string() takes a reference
+                        if (JSON_HEDLEY_UNLIKELY(!sax->key(type_key) || !sax->string(type)))
+                        {
+                            return false;
+                        }
+                    }
+
                     string_t key = "_ArraySize_";
-                    if (JSON_HEDLEY_UNLIKELY(!sax->start_object(3) || !sax->key(key) || !sax->start_array(dim.size())))
+                    if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->start_array(dim.size())))
                     {
                         return false;
                     }
@@ -2941,7 +2972,7 @@ class binary_reader
                                         exception_message(concat("expected '#' after type information; last byte: 0x", last_token), "size"), nullptr));
             }
 
-            const bool is_error = get_ubjson_size_value(result.first, is_ndarray);
+            const bool is_error = get_ubjson_size_value(result.first, is_ndarray, 0, result.second);
             // an ndarray was read here only if the flag flipped; when it was
             // seeded true, get_ubjson_size_value() already rejected the nested
             // dimension vector
@@ -3134,27 +3165,17 @@ class binary_reader
         if (input_format == input_format_t::bjdata && size_and_type.first != npos && (size_and_type.second & (1 << 8)) != 0)
         {
             size_and_type.second &= ~(static_cast<char_int_type>(1) << 8);  // use bit 8 to indicate ndarray, here we remove the bit to restore the type marker
-            const char* type_name = bjd_type_name(size_and_type.second);
-            string_t key = "_ArrayType_";
-            if (JSON_HEDLEY_UNLIKELY(type_name == nullptr))
-            {
-                auto last_token = get_token_string();
-                return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read,
-                                        exception_message("invalid byte: 0x" + last_token, "type"), nullptr));
-            }
-
-            string_t type = type_name; // sax->string() takes a reference
-            if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->string(type)))
-            {
-                return false;
-            }
 
+            // the "_ArrayType_" and "_ArraySize_" annotation keys were already emitted by
+            // get_ubjson_size_value() (the type marker is known before the dimension vector
+            // that determines size_and_type.first is read, so it is emitted first there to
+            // match the documented _ArrayType_, _ArraySize_, _ArrayData_ key order)
             if (size_and_type.second == 'C' || size_and_type.second == 'B')
             {
                 size_and_type.second = 'U';
             }
 
-            key = "_ArrayData_";
+            string_t key = "_ArrayData_";
             if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->start_array(size_and_type.first) ))
             {
                 return false;

include/nlohmann/detail/input/number_parse.hpp

@@ -988,10 +988,10 @@ void convert_float_locale_aware(StringType& token, std::size_t decimal_point_pos
         }
         else
         {
-            std::string copy(token.data(), token.size());
+            std::string buffer(token.data(), token.size());
-            copy.replace(decimal_point_position, 1, decimal_point);
+            buffer.replace(decimal_point_position, 1, decimal_point);
-            strtof_by_type(value, copy.c_str(), &endptr);
+            strtof_by_type(value, buffer.c_str(), &endptr);
-            complete = endptr == copy.c_str() + copy.size();
+            complete = endptr == buffer.c_str() + buffer.size();
         }
 
         if (JSON_HEDLEY_LIKELY(complete))

include/nlohmann/detail/input/pow5_table.hpp

@@ -22,6 +22,10 @@ namespace detail
 constexpr std::int64_t pow5_128_smallest_power = -342;
 constexpr std::int64_t pow5_128_largest_power = 308;
 
+// every entry of pow5_128() holds two 64-bit halves of 5^q, one per covered power of 5
+static_assert((pow5_128_largest_power - pow5_128_smallest_power + 1) * 2 == 1302,
+              "pow5_128_smallest_power/pow5_128_largest_power must match the size of the pow5_128() table");
+
 /*!
 @brief 128-bit approximations of 5^q for q in [-342, 308]
 

include/nlohmann/detail/output/binary_writer.hpp

@@ -28,6 +28,7 @@
 #include <nlohmann/detail/macro_scope.hpp>
 #include <nlohmann/detail/output/error_handler.hpp>
 #include <nlohmann/detail/output/output_adapters.hpp>
+#include <nlohmann/detail/recursion_depth_limit.hpp>
 #include <nlohmann/detail/string_concat.hpp>
 #include <nlohmann/detail/string_utils.hpp>
 
@@ -156,13 +157,30 @@ class binary_writer
     }
 
     /*!
-    @param[in] j  JSON value to serialize
+    @param[in] j      JSON value to serialize
+    @param[in] depth  nesting level of @a j, counted from the top-level value
+                      passed to @ref basic_json::to_cbor
     @throw type_error.316 if a string value or an object key is not valid
            UTF-8
     @throw type_error.321 if @a j or a value nested in it is discarded
+
+    Serializing a container descends into its elements, so a value nested deeply
+    enough used to exhaust the call stack and terminate the process with no
+    exception to catch. The descent is bounded here: once @ref recursion_depth_limit
+    levels have been entered, @ref write_cbor_iterative writes out what is left
+    without the call stack. A value nested less deeply than that - all but a
+    vanishing minority - is written by exactly the code that always wrote it.
+
+    @sa https://github.com/nlohmann/json/issues/5392
     */
-    void write_cbor(const BasicJsonType& j)
+    void write_cbor(const BasicJsonType& j, const std::size_t depth = 0)
     {
+        if (JSON_HEDLEY_UNLIKELY(depth >= recursion_depth_limit()) && (j.is_array() || j.is_object()))
+        {
+            write_cbor_iterative(j);
+            return;
+        }
+
         switch (j.type())
         {
             case value_t::null:
@@ -244,10 +262,9 @@ class binary_writer
                 // step 1: write control byte and the array size
                 write_cbor_head(0x80, j.m_data.m_value.array->size());
 
-                // step 2: write each element
                 for (const auto& el : *j.m_data.m_value.array)
                 {
-                    write_cbor(el);
+                    write_cbor(el, depth + 1);
                 }
                 break;
             }
@@ -302,7 +319,6 @@ class binary_writer
                 // step 1: write control byte and the object size
                 write_cbor_head(0xA0, j.m_data.m_value.object->size());
 
-                // step 2: write each element
                 for (const auto& el : *j.m_data.m_value.object)
                 {
                     // el.first is checked here, against the object as
@@ -317,7 +333,7 @@ class binary_writer
                         check_utf8(el.first, j);
                     }
                     write_cbor(el.first);
-                    write_cbor(el.second);
+                    write_cbor(el.second, depth + 1);
                 }
                 break;
             }
@@ -383,11 +399,22 @@ class binary_writer
     }
 
     /*!
-    @param[in] j  JSON value to serialize
+    @param[in] j      JSON value to serialize
+    @param[in] depth  nesting level of @a j, counted from the top-level value
+                      passed to @ref basic_json::to_msgpack
     @throw type_error.321 if @a j or a value nested in it is discarded
+
+    @sa @ref write_cbor
+    @sa https://github.com/nlohmann/json/issues/5392
     */
-    void write_msgpack(const BasicJsonType& j)
+    void write_msgpack(const BasicJsonType& j, const std::size_t depth = 0)
     {
+        if (JSON_HEDLEY_UNLIKELY(depth >= recursion_depth_limit()) && (j.is_array() || j.is_object()))
+        {
+            write_msgpack_iterative(j);
+            return;
+        }
+
         switch (j.type())
         {
             case value_t::null: // nil
@@ -503,29 +530,11 @@ class binary_writer
             case value_t::array:
             {
                 // step 1: write control byte and the array size
-                const auto N = to_msgpack_length(j.m_data.m_value.array->size(), j);
+                write_msgpack_array_prefix(j.m_data.m_value.array->size(), j);
-                if (N <= 15)
-                {
-                    // fixarray
-                    write_number(static_cast<std::uint8_t>(0x90 | N));
-                }
-                else if (N <= (std::numeric_limits<std::uint16_t>::max)())
-                {
-                    // array 16
-                    oa.write_character(to_char_type(0xDC));
-                    write_number(static_cast<std::uint16_t>(N));
-                }
-                else
-                {
-                    // array 32
-                    oa.write_character(to_char_type(0xDD));
-                    write_number(static_cast<std::uint32_t>(N));
-                }
 
-                // step 2: write each element
                 for (const auto& el : *j.m_data.m_value.array)
                 {
-                    write_msgpack(el);
+                    write_msgpack(el, depth + 1);
                 }
                 break;
             }
@@ -621,26 +630,8 @@ class binary_writer
             case value_t::object:
             {
                 // step 1: write control byte and the object size
-                const auto N = to_msgpack_length(j.m_data.m_value.object->size(), j);
+                write_msgpack_object_prefix(j.m_data.m_value.object->size(), j);
-                if (N <= 15)
-                {
-                    // fixmap
-                    write_number(static_cast<std::uint8_t>(0x80 | (N & 0xF)));
-                }
-                else if (N <= (std::numeric_limits<std::uint16_t>::max)())
-                {
-                    // map 16
-                    oa.write_character(to_char_type(0xDE));
-                    write_number(static_cast<std::uint16_t>(N));
-                }
-                else
-                {
-                    // map 32
-                    oa.write_character(to_char_type(0xDF));
-                    write_number(static_cast<std::uint32_t>(N));
-                }
 
-                // step 2: write each element
                 for (const auto& el : *j.m_data.m_value.object)
                 {
                     // as in write_cbor, el.first is checked here against the
@@ -651,7 +642,7 @@ class binary_writer
                         check_utf8(el.first, j);
                     }
                     write_msgpack(el.first);
-                    write_msgpack(el.second);
+                    write_msgpack(el.second, depth + 1);
                 }
                 break;
             }
@@ -669,14 +660,26 @@ class binary_writer
     @param[in] add_prefix  whether prefixes need to be used for this value
     @param[in] use_bjdata  whether write in BJData format, default is false
     @param[in] bjdata_version  which BJData version to use, default is draft2
+    @param[in] depth  nesting level of @a j, counted from the top-level value
+                      passed to @ref basic_json::to_ubjson or @ref basic_json::to_bjdata
     @throw type_error.316 if a string value or an object key is not valid
            UTF-8
     @throw type_error.321 if @a j or a value nested in it is discarded
+
+    @sa @ref write_cbor
+    @sa https://github.com/nlohmann/json/issues/5392
     */
     void write_ubjson(const BasicJsonType& j, const bool use_count,
                       const bool use_type, const bool add_prefix = true,
-                      const bool use_bjdata = false, const bjdata_version_t bjdata_version = bjdata_version_t::draft2)
+                      const bool use_bjdata = false, const bjdata_version_t bjdata_version = bjdata_version_t::draft2,
+                      const std::size_t depth = 0)
     {
+        if (JSON_HEDLEY_UNLIKELY(depth >= recursion_depth_limit()) && (j.is_array() || j.is_object()))
+        {
+            write_ubjson_iterative(j, use_count, use_type, add_prefix, use_bjdata, bjdata_version);
+            return;
+        }
+
         const bool bjdata_draft3 = use_bjdata && bjdata_version == bjdata_version_t::draft3;
 
         switch (j.type())
@@ -737,55 +740,15 @@ class binary_writer
 
             case value_t::array:
             {
-                if (add_prefix)
-                {
-                    oa.write_character(to_char_type('['));
-                }
-
                 bool prefix_required = true;
-                if (use_type && !j.m_data.m_value.array->empty())
+                const bool write_closer = write_ubjson_start_array(j, use_count, use_type, add_prefix, use_bjdata, prefix_required);
-                {
-                    if (!use_count)
-                    {
-                        JSON_THROW(other_error::create(502, "use_type requires use_size = true", &j));
-                    }
-                    const CharType first_prefix = ubjson_prefix(j.front(), use_bjdata);
-                    const bool same_prefix = std::all_of(j.begin() + 1, j.end(),
-                                                         [this, first_prefix, use_bjdata](const BasicJsonType & v)
-                    {
-                        return ubjson_prefix(v, use_bjdata) == first_prefix;
-                    });
-
-                    // an optimized array of a valueless type carries no payload, so a
-                    // reader has nothing but the declared count to bound the allocation
-                    // by and refuses an excessive one. Write the unoptimized form for
-                    // those, at one byte per element, so the result can be read back.
-                    // Objects are not affected: every element is preceded by its key.
-                    const bool valueless_type = (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');
-                    const bool excessive_valueless = valueless_type
-                                                     && j.m_data.m_value.array->size() > detail::max_valueless_container_size;
-
-                    if (same_prefix && !excessive_valueless
-                            && !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
-                    {
-                        prefix_required = false;
-                        oa.write_character(to_char_type('$'));
-                        oa.write_character(first_prefix);
-                    }
-                }
-
-                if (use_count)
-                {
-                    oa.write_character(to_char_type('#'));
-                    write_number_with_ubjson_prefix(j.m_data.m_value.array->size(), true, use_bjdata);
-                }
 
                 for (const auto& el : *j.m_data.m_value.array)
                 {
-                    write_ubjson(el, use_count, use_type, prefix_required, use_bjdata, bjdata_version);
+                    write_ubjson(el, use_count, use_type, prefix_required, use_bjdata, bjdata_version, depth + 1);
                 }
 
-                if (!use_count)
+                if (write_closer)
                 {
                     oa.write_character(to_char_type(']'));
                 }
@@ -843,7 +806,7 @@ class binary_writer
 
             case value_t::object:
             {
-                if (use_bjdata && j.m_data.m_value.object->size() == 3 && j.m_data.m_value.object->find("_ArrayType_") != j.m_data.m_value.object->end() && j.m_data.m_value.object->find("_ArraySize_") != j.m_data.m_value.object->end() && j.m_data.m_value.object->find("_ArrayData_") != j.m_data.m_value.object->end())
+                if (use_bjdata && is_bjdata_ndarray(j))
                 {
                     if (!write_bjdata_ndarray(*j.m_data.m_value.object, use_count, use_type, bjdata_version))  // decode bjdata ndarray in the JData format (https://github.com/NeuroJSON/jdata)
                     {
@@ -851,38 +814,8 @@ class binary_writer
                     }
                 }
 
-                if (add_prefix)
-                {
-                    oa.write_character(to_char_type('{'));
-                }
-
                 bool prefix_required = true;
-                if (use_type && !j.m_data.m_value.object->empty())
+                const bool write_closer = write_ubjson_start_object(j, use_count, use_type, add_prefix, use_bjdata, prefix_required);
-                {
-                    if (!use_count)
-                    {
-                        JSON_THROW(other_error::create(502, "use_type requires use_size = true", &j));
-                    }
-                    const CharType first_prefix = ubjson_prefix(j.front(), use_bjdata);
-                    const bool same_prefix = std::all_of(j.begin(), j.end(),
-                                                         [this, first_prefix, use_bjdata](const BasicJsonType & v)
-                    {
-                        return ubjson_prefix(v, use_bjdata) == first_prefix;
-                    });
-
-                    if (same_prefix && !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
-                    {
-                        prefix_required = false;
-                        oa.write_character(to_char_type('$'));
-                        oa.write_character(first_prefix);
-                    }
-                }
-
-                if (use_count)
-                {
-                    oa.write_character(to_char_type('#'));
-                    write_number_with_ubjson_prefix(j.m_data.m_value.object->size(), true, use_bjdata);
-                }
 
                 for (const auto& el : *j.m_data.m_value.object)
                 {
@@ -892,10 +825,10 @@ class binary_writer
                     oa.write_characters(
                           reinterpret_cast<const CharType*>(key.data()),
                           key.size());
-                    write_ubjson(el.second, use_count, use_type, prefix_required, use_bjdata, bjdata_version);
+                    write_ubjson(el.second, use_count, use_type, prefix_required, use_bjdata, bjdata_version, depth + 1);
                 }
 
-                if (!use_count)
+                if (write_closer)
                 {
                     oa.write_character(to_char_type('}'));
                 }
@@ -936,6 +869,517 @@ class binary_writer
         JSON_THROW(type_error::create(321, concat("cannot serialize discarded value to ", format_name), &j));
     }
 
+    void write_msgpack_array_prefix(const std::size_t N, const BasicJsonType& j)
+    {
+        const auto n = to_msgpack_length(N, j);
+        if (n <= 15)
+        {
+            // fixarray
+            write_number(static_cast<std::uint8_t>(0x90 | n));
+        }
+        else if (n <= (std::numeric_limits<std::uint16_t>::max)())
+        {
+            // array 16
+            oa.write_character(to_char_type(0xDC));
+            write_number(static_cast<std::uint16_t>(n));
+        }
+        else
+        {
+            // array 32
+            oa.write_character(to_char_type(0xDD));
+            write_number(static_cast<std::uint32_t>(n));
+        }
+    }
+
+    void write_msgpack_object_prefix(const std::size_t N, const BasicJsonType& j)
+    {
+        const auto n = to_msgpack_length(N, j);
+        if (n <= 15)
+        {
+            // fixmap
+            write_number(static_cast<std::uint8_t>(0x80 | (n & 0xF)));
+        }
+        else if (n <= (std::numeric_limits<std::uint16_t>::max)())
+        {
+            // map 16
+            oa.write_character(to_char_type(0xDE));
+            write_number(static_cast<std::uint16_t>(n));
+        }
+        else
+        {
+            // map 32
+            oa.write_character(to_char_type(0xDF));
+            write_number(static_cast<std::uint32_t>(n));
+        }
+    }
+
+    /// @brief a CBOR or MessagePack array or object whose elements
+    ///        @ref write_cbor_iterative or @ref write_msgpack_iterative is
+    ///        still writing
+    struct binary_container_frame
+    {
+        explicit binary_container_frame(const BasicJsonType* value_) noexcept
+            : value(value_)
+        {
+            if (value->is_object())
+            {
+                object_it = value->m_data.m_value.object->cbegin();
+            }
+            else
+            {
+                array_it = value->m_data.m_value.array->cbegin();
+            }
+        }
+
+        // declared for GCC's -Weffc++, which asks for them in a class with
+        // pointer members and a non-trivial destructor; the exception
+        // specifications are left implicit, as GCC 4.8 rejects explicit ones
+        // that differ from them
+        binary_container_frame(const binary_container_frame&) = default;
+        binary_container_frame(binary_container_frame&&) = default;
+        binary_container_frame& operator=(const binary_container_frame&) = default;
+        binary_container_frame& operator=(binary_container_frame&&) = default;
+        ~binary_container_frame() = default;
+
+        /// the array or object being written
+        const BasicJsonType* value;
+        /// value's elements still to write; which of the two is live follows
+        /// from the type of value. They are kept side by side rather than in
+        /// a union, which would need its special members written out by
+        /// hand, see detail/iterators/internal_iterator.hpp
+        typename BasicJsonType::object_t::const_iterator object_it{};
+        typename BasicJsonType::array_t::const_iterator array_it{};
+    };
+
+    /*!
+    @brief write @a j with @ref write_cbor, or write its header and push a
+           frame for @ref write_cbor_iterative to continue with its elements
+
+    A scalar, and an empty array or object, are written out in full: there is
+    nothing below them for @ref write_cbor_iterative to come back to, so
+    nothing is pushed for them.
+    */
+    void write_cbor_value_or_push(const BasicJsonType& j, std::vector<binary_container_frame>& stack)
+    {
+        if (j.is_array())
+        {
+            write_cbor_head(0x80, j.m_data.m_value.array->size());
+            if (!j.m_data.m_value.array->empty())
+            {
+                stack.emplace_back(&j);
+            }
+            return;
+        }
+
+        if (j.is_object())
+        {
+            write_cbor_head(0xA0, j.m_data.m_value.object->size());
+            if (!j.m_data.m_value.object->empty())
+            {
+                stack.emplace_back(&j);
+            }
+            return;
+        }
+
+        write_cbor(j);
+    }
+
+    /*!
+    @brief write out @a root and everything below it without the call stack
+
+    Emits the same bytes as @ref write_cbor, keeping the containers it has
+    entered on an explicit stack instead of descending into them. Only reached
+    for values nested deeper than @ref recursion_depth_limit, which is why it
+    is not written for speed.
+    */
+    void write_cbor_iterative(const BasicJsonType& root)
+    {
+        // only a container with elements is ever pushed; see write_cbor_value_or_push
+        std::vector<binary_container_frame> stack;
+        write_cbor_value_or_push(root, stack);
+
+        while (!stack.empty())
+        {
+            const binary_container_frame current = stack.back();
+
+            if (current.value->is_array())
+            {
+                const auto& array = *current.value->m_data.m_value.array;
+                if (current.array_it == array.cend())
+                {
+                    stack.pop_back();
+                    continue;
+                }
+
+                // read the child before pushing: entering it can move every frame
+                const BasicJsonType* child = &(*current.array_it);
+                ++stack.back().array_it;
+                write_cbor_value_or_push(*child, stack);
+            }
+            else
+            {
+                const auto& object = *current.value->m_data.m_value.object;
+                if (current.object_it == object.cend())
+                {
+                    stack.pop_back();
+                    continue;
+                }
+
+                // el.first is checked here, against the object as diagnostics
+                // context, like the matching check in write_cbor's object case
+                if (error_handler == error_handler_t::strict)
+                {
+                    check_utf8(current.object_it->first, *current.value);
+                }
+                write_cbor(current.object_it->first);
+                const BasicJsonType* child = &(current.object_it->second);
+                ++stack.back().object_it;
+                write_cbor_value_or_push(*child, stack);
+            }
+        }
+    }
+
+    /*!
+    @brief write @a j with @ref write_msgpack, or write its header and push a
+           frame for @ref write_msgpack_iterative to continue with its elements
+
+    @sa @ref write_cbor_value_or_push
+    */
+    void write_msgpack_value_or_push(const BasicJsonType& j, std::vector<binary_container_frame>& stack)
+    {
+        if (j.is_array())
+        {
+            write_msgpack_array_prefix(j.m_data.m_value.array->size(), j);
+            if (!j.m_data.m_value.array->empty())
+            {
+                stack.emplace_back(&j);
+            }
+            return;
+        }
+
+        if (j.is_object())
+        {
+            write_msgpack_object_prefix(j.m_data.m_value.object->size(), j);
+            if (!j.m_data.m_value.object->empty())
+            {
+                stack.emplace_back(&j);
+            }
+            return;
+        }
+
+        write_msgpack(j);
+    }
+
+    /*!
+    @brief write out @a root and everything below it without the call stack
+
+    @sa @ref write_cbor_iterative
+    */
+    void write_msgpack_iterative(const BasicJsonType& root)
+    {
+        std::vector<binary_container_frame> stack;
+        write_msgpack_value_or_push(root, stack);
+
+        while (!stack.empty())
+        {
+            const binary_container_frame current = stack.back();
+
+            if (current.value->is_array())
+            {
+                const auto& array = *current.value->m_data.m_value.array;
+                if (current.array_it == array.cend())
+                {
+                    stack.pop_back();
+                    continue;
+                }
+
+                const BasicJsonType* child = &(*current.array_it);
+                ++stack.back().array_it;
+                write_msgpack_value_or_push(*child, stack);
+            }
+            else
+            {
+                const auto& object = *current.value->m_data.m_value.object;
+                if (current.object_it == object.cend())
+                {
+                    stack.pop_back();
+                    continue;
+                }
+
+                if (error_handler == error_handler_t::strict)
+                {
+                    check_utf8(current.object_it->first, *current.value);
+                }
+                write_msgpack(current.object_it->first);
+                const BasicJsonType* child = &(current.object_it->second);
+                ++stack.back().object_it;
+                write_msgpack_value_or_push(*child, stack);
+            }
+        }
+    }
+
+    /// @return true when a closing ']' still has to be written after the elements
+    bool write_ubjson_start_array(const BasicJsonType& j, const bool use_count, const bool use_type,
+                                  const bool add_prefix, const bool use_bjdata, bool& prefix_required)
+    {
+        prefix_required = true;
+        if (add_prefix)
+        {
+            oa.write_character(to_char_type('['));
+        }
+
+        if (use_type && !j.m_data.m_value.array->empty())
+        {
+            if (!use_count)
+            {
+                JSON_THROW(other_error::create(502, "use_type requires use_size = true", &j));
+            }
+            const CharType first_prefix = ubjson_prefix(j.front(), use_bjdata);
+            const bool same_prefix = std::all_of(j.begin() + 1, j.end(),
+                                                 [this, first_prefix, use_bjdata](const BasicJsonType & v)
+            {
+                return ubjson_prefix(v, use_bjdata) == first_prefix;
+            });
+
+            // an optimized array of a valueless type carries no payload, so a
+            // reader has nothing but the declared count to bound the allocation
+            // by and refuses an excessive one. Write the unoptimized form for
+            // those, at one byte per element, so the result can be read back.
+            // Objects are not affected: every element is preceded by its key.
+            const bool valueless_type = (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');
+            const bool excessive_valueless = valueless_type
+                                             && j.m_data.m_value.array->size() > detail::max_valueless_container_size;
+
+            if (same_prefix && !excessive_valueless
+                    && !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
+            {
+                prefix_required = false;
+                oa.write_character(to_char_type('$'));
+                oa.write_character(first_prefix);
+            }
+        }
+
+        if (use_count)
+        {
+            oa.write_character(to_char_type('#'));
+            write_number_with_ubjson_prefix(j.m_data.m_value.array->size(), true, use_bjdata);
+        }
+
+        return !use_count;
+    }
+
+    /// @return true when a closing '}' still has to be written after the elements
+    bool write_ubjson_start_object(const BasicJsonType& j, const bool use_count, const bool use_type,
+                                   const bool add_prefix, const bool use_bjdata, bool& prefix_required)
+    {
+        prefix_required = true;
+        if (add_prefix)
+        {
+            oa.write_character(to_char_type('{'));
+        }
+
+        if (use_type && !j.m_data.m_value.object->empty())
+        {
+            if (!use_count)
+            {
+                JSON_THROW(other_error::create(502, "use_type requires use_size = true", &j));
+            }
+            const CharType first_prefix = ubjson_prefix(j.front(), use_bjdata);
+            const bool same_prefix = std::all_of(j.begin(), j.end(),
+                                                 [this, first_prefix, use_bjdata](const BasicJsonType & v)
+            {
+                return ubjson_prefix(v, use_bjdata) == first_prefix;
+            });
+
+            if (same_prefix && !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
+            {
+                prefix_required = false;
+                oa.write_character(to_char_type('$'));
+                oa.write_character(first_prefix);
+            }
+        }
+
+        if (use_count)
+        {
+            oa.write_character(to_char_type('#'));
+            write_number_with_ubjson_prefix(j.m_data.m_value.object->size(), true, use_bjdata);
+        }
+
+        return !use_count;
+    }
+
+    /*!
+    @brief whether @a j is a BJData ND-array annotation object
+           (https://github.com/NeuroJSON/jdata)
+
+    Used by both the recursive object case of @ref write_ubjson and
+    @ref write_ubjson_value_or_push, which must agree on what counts as an
+    ND-array: @a j is only actually written as one once @ref
+    write_bjdata_ndarray has also accepted its contents.
+
+    @pre @a j.is_object()
+    */
+    static bool is_bjdata_ndarray(const BasicJsonType& j)
+    {
+        const auto& object = *j.m_data.m_value.object;
+        return object.size() == 3
+               && object.find("_ArrayType_") != object.end()
+               && object.find("_ArraySize_") != object.end()
+               && object.find("_ArrayData_") != object.end();
+    }
+
+    /// @brief an object or array @ref write_ubjson_iterative is still writing
+    ///        the elements of
+    struct ubjson_frame
+    {
+        ubjson_frame(const BasicJsonType* value_, const bool prefix_required_) noexcept
+            : value(value_)
+            , prefix_required(prefix_required_)
+        {
+            if (value->is_object())
+            {
+                object_it = value->m_data.m_value.object->cbegin();
+            }
+            else
+            {
+                array_it = value->m_data.m_value.array->cbegin();
+            }
+        }
+
+        // declared for GCC's -Weffc++, which asks for them in a class with
+        // pointer members and a non-trivial destructor; the exception
+        // specifications are left implicit, as GCC 4.8 rejects explicit ones
+        // that differ from them
+        ubjson_frame(const ubjson_frame&) = default;
+        ubjson_frame(ubjson_frame&&) = default;
+        ubjson_frame& operator=(const ubjson_frame&) = default;
+        ubjson_frame& operator=(ubjson_frame&&) = default;
+        ~ubjson_frame() = default;
+
+        /// the array or object being written
+        const BasicJsonType* value;
+        /// whether value's elements each carry their own type marker; an
+        /// optimized ($type) container writes it once for all of them instead
+        bool prefix_required;
+        typename BasicJsonType::object_t::const_iterator object_it{};
+        typename BasicJsonType::array_t::const_iterator array_it{};
+    };
+
+    /*!
+    @brief write @a j with @ref write_ubjson, or write its header and push a
+           frame for @ref write_ubjson_iterative to continue with its elements
+
+    @param[in] add_prefix  whether @a j's own type marker is written now (the
+               elements of an optimized container, and everything below the
+               top level, never repeat it)
+
+    @sa @ref write_cbor_value_or_push
+    */
+    void write_ubjson_value_or_push(const BasicJsonType& j, const bool add_prefix, const bool use_count,
+                                    const bool use_type, const bool use_bjdata, const bjdata_version_t bjdata_version,
+                                    std::vector<ubjson_frame>& stack)
+    {
+        if (!j.is_array() && !j.is_object())
+        {
+            write_ubjson(j, use_count, use_type, add_prefix, use_bjdata, bjdata_version);
+            return;
+        }
+
+        if (use_bjdata && j.is_object() && is_bjdata_ndarray(j)
+                && !write_bjdata_ndarray(*j.m_data.m_value.object, use_count, use_type, bjdata_version))
+        {
+            // fully written as an ND-array: nothing below it to come back to
+            return;
+        }
+
+        const bool is_array = j.is_array();
+        bool prefix_required = true;
+        if (is_array)
+        {
+            write_ubjson_start_array(j, use_count, use_type, add_prefix, use_bjdata, prefix_required);
+        }
+        else
+        {
+            write_ubjson_start_object(j, use_count, use_type, add_prefix, use_bjdata, prefix_required);
+        }
+
+        const bool empty = is_array ? j.m_data.m_value.array->empty() : j.m_data.m_value.object->empty();
+        if (!empty)
+        {
+            stack.emplace_back(&j, prefix_required);
+            return;
+        }
+
+        // write_ubjson_start_array/_object return !use_count, i.e. whether a
+        // closer still has to be written; use_count is constant for the whole
+        // document, so that is recomputed here instead of being carried along
+        if (!use_count)
+        {
+            oa.write_character(to_char_type(is_array ? ']' : '}'));
+        }
+    }
+
+    /*!
+    @brief write out @a root and everything below it without the call stack
+
+    @sa @ref write_cbor_iterative
+    */
+    void write_ubjson_iterative(const BasicJsonType& root, const bool use_count, const bool use_type,
+                                const bool add_prefix, const bool use_bjdata, const bjdata_version_t bjdata_version)
+    {
+        std::vector<ubjson_frame> stack;
+        write_ubjson_value_or_push(root, add_prefix, use_count, use_type, use_bjdata, bjdata_version, stack);
+
+        while (!stack.empty())
+        {
+            const ubjson_frame current = stack.back();
+            const BasicJsonType& j = *current.value;
+
+            if (j.is_array())
+            {
+                const auto& array = *j.m_data.m_value.array;
+                if (current.array_it == array.cend())
+                {
+                    if (!use_count)
+                    {
+                        oa.write_character(to_char_type(']'));
+                    }
+                    stack.pop_back();
+                    continue;
+                }
+
+                const BasicJsonType* child = &(*current.array_it);
+                const bool child_prefix = current.prefix_required;
+                ++stack.back().array_it;
+                write_ubjson_value_or_push(*child, child_prefix, use_count, use_type, use_bjdata, bjdata_version, stack);
+            }
+            else
+            {
+                const auto& object = *j.m_data.m_value.object;
+                if (current.object_it == object.cend())
+                {
+                    if (!use_count)
+                    {
+                        oa.write_character(to_char_type('}'));
+                    }
+                    stack.pop_back();
+                    continue;
+                }
+
+                string_t storage;
+                const string_t& key = sanitize_utf8_for_write(current.object_it->first, j, storage);
+                write_number_with_ubjson_prefix(key.size(), true, use_bjdata);
+                oa.write_characters(
+                      reinterpret_cast<const CharType*>(key.data()),
+                      key.size());
+                const BasicJsonType* child = &(current.object_it->second);
+                const bool child_prefix = current.prefix_required;
+                ++stack.back().object_it;
+                write_ubjson_value_or_push(*child, child_prefix, use_count, use_type, use_bjdata, bjdata_version, stack);
+            }
+        }
+    }
+
     //////////
     // BSON //
     //////////
@@ -1839,16 +2283,31 @@ class binary_writer
 
     /*!
     @brief validate (dry_run) or write one BJData ND-array element of dtype 'd' (single precision)
-    @return whether @a el's value fits a float without overflow; always true when @a dry_run is false
+    @return whether @a el's value survives narrowing to float and back without any change
+            (so the ND-array round-trips exactly), or is infinite or NaN; always true when
+            @a dry_run is false
     */
     bool write_bjdata_ndarray_float_element(const BasicJsonType& el, const bool dry_run)
     {
         const auto dval = el.template get<double>();
         if (dry_run)
         {
-            return !std::isfinite(dval) ||
+#ifdef __GNUC__
-                   (dval >= static_cast<double>(std::numeric_limits<float>::lowest()) &&
+            JSON_HEDLEY_DIAGNOSTIC_PUSH
-                    dval <= static_cast<double>((std::numeric_limits<float>::max)()));
+            JSON_HEDLEY_PRAGMA(GCC diagnostic ignored "-Wfloat-equal")
+#endif
+            // a value that would be rounded (rather than exactly represented) by the
+            // narrowing to float is treated like an out-of-range integer element; this
+            // is the same criterion write_compact_float() uses for CBOR/MessagePack
+            const bool in_range = std::isnan(dval) ||
+                                  (dval >= static_cast<double>(std::numeric_limits<float>::lowest()) &&
+                                   dval <= static_cast<double>((std::numeric_limits<float>::max)()) &&
+                                   static_cast<double>(static_cast<float>(dval)) == dval) ||
+                                  std::isinf(dval);
+#ifdef __GNUC__
+            JSON_HEDLEY_DIAGNOSTIC_POP
+#endif
+            return in_range;
         }
         write_number(static_cast<float>(dval), true);
         return true;

tests/src/unit-bjdata.cpp

@@ -19,6 +19,7 @@
 
 #include <nlohmann/json.hpp>
 using nlohmann::json;
+using ordered_json = nlohmann::ordered_json;
 
 #include <algorithm>
 #include <climits>
@@ -2258,29 +2259,33 @@ TEST_CASE("BJData")
 
         SECTION("start_array() in ndarray _ArraySize_")
         {
+            // _ArrayType_ (2 events: key + string) is now emitted before
+            // _ArraySize_ (see GitHub issue #5661), which shifts the events
+            // below later by the same 2 events
             std::vector<uint8_t> const v = {'[', '$', 'i', '#', '[', '$', 'i', '#', 'i', 2, 2, 1, 1, 2};
-            SaxCountdown scp(2);
+            SaxCountdown scp(4);
             CHECK_FALSE(json::sax_parse(v, &scp, json::input_format_t::bjdata));
         }
 
         SECTION("number_integer() in ndarray _ArraySize_")
         {
             std::vector<uint8_t> const v = {'[', '$', 'U', '#', '[', '$', 'i', '#', 'i', 2, 2, 1, 1, 2};
-            SaxCountdown scp(3);
+            SaxCountdown scp(5);
             CHECK_FALSE(json::sax_parse(v, &scp, json::input_format_t::bjdata));
         }
 
         SECTION("key() in ndarray _ArrayType_")
         {
+            // _ArrayType_ is emitted right after start_object(), before _ArraySize_
             std::vector<uint8_t> const v = {'[', '$', 'U', '#', '[', '$', 'U', '#', 'i', 2, 2, 2, 1, 2, 3, 4};
-            SaxCountdown scp(6);
+            SaxCountdown scp(1);
             CHECK_FALSE(json::sax_parse(v, &scp, json::input_format_t::bjdata));
         }
 
         SECTION("string() in ndarray _ArrayType_")
         {
             std::vector<uint8_t> const v = {'[', '$', 'U', '#', '[', '$', 'U', '#', 'i', 2, 2, 2, 1, 2, 3, 4};
-            SaxCountdown scp(7);
+            SaxCountdown scp(2);
             CHECK_FALSE(json::sax_parse(v, &scp, json::input_format_t::bjdata));
         }
 
@@ -2883,6 +2888,22 @@ TEST_CASE("BJData")
                 CHECK(out_single.at(0) == '{');
                 CHECK(json::from_bjdata(out_single) == j_single);
 
+                // a double element that is finite and within the range of "single"
+                // but is not exactly representable as a float, so narrowing it would
+                // silently round it (0.1 is read back as 0.10000000149011612); this,
+                // like the overflow case above, falls back to a plain object (see
+                // GitHub issue #5661)
+                json const j_single_rounded = json({{"_ArrayType_", "single"}, {"_ArraySize_", {2, 1}}, {"_ArrayData_", {1.5, 0.1}}});
+                const auto out_single_rounded = json::to_bjdata(j_single_rounded);
+                CHECK(out_single_rounded.at(0) == '{');
+                CHECK(json::from_bjdata(out_single_rounded) == j_single_rounded);
+
+                // a double element that underflows to 0 when narrowed to "single"
+                json const j_single_underflow = json({{"_ArrayType_", "single"}, {"_ArraySize_", {2, 1}}, {"_ArrayData_", {1.5, 1e-300}}});
+                const auto out_single_underflow = json::to_bjdata(j_single_underflow);
+                CHECK(out_single_underflow.at(0) == '{');
+                CHECK(json::from_bjdata(out_single_underflow) == j_single_underflow);
+
                 // in-range boundary values still use the compact ndarray encoding
                 json const j_uint8_ok = json({{"_ArrayType_", "uint8"}, {"_ArraySize_", {2, 1}}, {"_ArrayData_", {0, 255}}});
                 CHECK(json::to_bjdata(j_uint8_ok) == std::vector<uint8_t>({'[', '$', 'U', '#', '[', 'i', 2, 'i', 1, ']', 0, 255}));
@@ -2896,6 +2917,23 @@ TEST_CASE("BJData")
                 CHECK(json::from_bjdata(out_single_ok) == json({{"_ArrayType_", "single"}, {"_ArraySize_", {2, 1}}, {"_ArrayData_", {1.5f, -1.5f}}}));
             }
 
+            SECTION("ndarray annotation keys are read back in the documented order")
+            {
+                // from_bjdata() must emit the annotation object's keys in the order
+                // used throughout the documentation, _ArrayType_, _ArraySize_,
+                // _ArrayData_: the type marker precedes the dimension vector on the
+                // wire (see get_ubjson_size_type()), so it is known, and emitted,
+                // before _ArraySize_. For a plain json this key order is invisible
+                // (its comparison ignores it), but for an ordered_json it is not (see
+                // GitHub issue #5661).
+                const ordered_json o = ordered_json::parse(R"({"_ArrayType_":"uint8","_ArraySize_":[2,2],"_ArrayData_":[1,2,3,4]})");
+                const auto packed = ordered_json::to_bjdata(o);
+                CHECK(packed.at(0) == '[');
+                const ordered_json o_back = ordered_json::from_bjdata(packed);
+                CHECK(o_back == o);
+                CHECK(o_back.dump() == o.dump());
+            }
+
             SECTION("ndarray that would not be read back as an annotated object stays as object")
             {
                 // the reader only restores an annotated object from an ND-array

tests/src/unit-class_lexer.cpp

@@ -801,7 +801,7 @@ TEST_CASE("lexer escape fast path")
         // digits and non-hex bytes, at varying distances from the start of
         // the string, to compare the two scanners on many more shapes than
         // are practical to enumerate by hand.
-        std::mt19937 gen(7654321); // NOLINT(cert-msc32-c,cert-msc51-cpp)
+        std::mt19937 gen(7654321); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed)
         const std::string hex_alphabet = "0123456789AaBbCcDdEeFf";
         std::uniform_int_distribution<std::size_t> pick_hex(0, hex_alphabet.size() - 1);
         std::uniform_int_distribution<int> pick_byte(1, 255); // never NUL
@@ -1599,7 +1599,7 @@ template<typename Json, typename Bits>
 void check_parse(const std::string& token, Bits expected, Bits infinity)
 {
     std::stringstream stream(token);
-    if ((expected & ~(Bits{1} << (8 * sizeof(Bits) - 1))) == infinity)
+    if ((expected & ~(Bits{1} << ((8 * sizeof(Bits)) - 1))) == infinity)
     {
         Json _;
         CHECK_THROWS_WITH_AS(_ = Json::parse(token), ("[json.exception.out_of_range.406] number overflow parsing '" + token + "'").c_str(), typename Json::out_of_range&);

tests/src/unit-hash.cpp

@@ -12,8 +12,10 @@
 using json = nlohmann::json;
 using ordered_json = nlohmann::ordered_json;
 
+#include <limits>
 #include <set>
 #include <string>
+#include <unordered_set>
 
 namespace
 {
@@ -91,6 +93,9 @@ TEST_CASE("hash<nlohmann::json>")
     // Collect hashes for different JSON values and make sure that they are distinct
     // We cannot compare against fixed values, because the implementation of
     // std::hash may differ between compilers.
+    //
+    // numbers that compare equal under operator== (0 == 0U == 0.0) must hash
+    // equally, so they are only inserted once below and checked separately.
 
     std::set<std::size_t> hashes;
 
@@ -107,10 +112,7 @@ TEST_CASE("hash<nlohmann::json>")
 
     // number
     hashes.insert(std::hash<json> {}(json(0)));
-    hashes.insert(std::hash<json> {}(json(static_cast<unsigned>(0))));
-
     hashes.insert(std::hash<json> {}(json(-1)));
-    hashes.insert(std::hash<json> {}(json(0.0)));
     hashes.insert(std::hash<json> {}(json(42.23)));
 
     // array
@@ -132,7 +134,36 @@ TEST_CASE("hash<nlohmann::json>")
     // discarded
     hashes.insert(std::hash<json> {}(json(json::value_t::discarded)));
 
-    CHECK(hashes.size() == 21);
+    CHECK(hashes.size() == 19);
+
+    // numbers that compare equal under operator== must hash equally,
+    // regardless of which of number_integer, number_unsigned, or
+    // number_float actually holds the value
+    CHECK(json(0) == json(static_cast<unsigned>(0)));
+    CHECK(json(0) == json(0.0));
+    CHECK(std::hash<json> {}(json(0)) == std::hash<json> {}(json(static_cast<unsigned>(0))));
+    CHECK(std::hash<json> {}(json(0)) == std::hash<json> {}(json(0.0)));
+    CHECK(std::hash<json> {}(json(-1)) == std::hash<json> {}(json(-1.0)));
+
+    // a std::unordered_set relies on this same consistency between == and hash
+    const std::unordered_set<json> numbers {json(0), json(static_cast<unsigned>(0)), json(0.0)};
+    CHECK(numbers.size() == 1);
+
+    // -0.0 compares equal to 0 and 0.0
+    CHECK(json(-0.0) == json(0));
+    CHECK(std::hash<json> {}(json(-0.0)) == std::hash<json> {}(json(0)));
+    CHECK(std::hash<json> {}(json(-0.0)) == std::hash<json> {}(json(0.0)));
+
+    // the ends of the integer ranges, which equal floats exactly
+    const auto int_min = (std::numeric_limits<json::number_integer_t>::min)();
+    const auto int_max = (std::numeric_limits<json::number_integer_t>::max)();
+    const auto two_63 = json::number_unsigned_t(1) << 63U;
+    CHECK(json(int_min) == json(-9223372036854775808.0));
+    CHECK(std::hash<json> {}(json(int_min)) == std::hash<json> {}(json(-9223372036854775808.0)));
+    CHECK(json(two_63) == json(9223372036854775808.0));
+    CHECK(std::hash<json> {}(json(two_63)) == std::hash<json> {}(json(9223372036854775808.0)));
+    CHECK(json(json::number_unsigned_t(int_max)) == json(int_max));
+    CHECK(std::hash<json> {}(json(json::number_unsigned_t(int_max))) == std::hash<json> {}(json(int_max)));
 }
 
 TEST_CASE("hash<nlohmann::ordered_json>")
@@ -156,10 +187,7 @@ TEST_CASE("hash<nlohmann::ordered_json>")
 
     // number
     hashes.insert(std::hash<ordered_json> {}(ordered_json(0)));
-    hashes.insert(std::hash<ordered_json> {}(ordered_json(static_cast<unsigned>(0))));
-
     hashes.insert(std::hash<ordered_json> {}(ordered_json(-1)));
-    hashes.insert(std::hash<ordered_json> {}(ordered_json(0.0)));
     hashes.insert(std::hash<ordered_json> {}(ordered_json(42.23)));
 
     // array
@@ -181,7 +209,10 @@ TEST_CASE("hash<nlohmann::ordered_json>")
     // discarded
     hashes.insert(std::hash<ordered_json> {}(ordered_json(ordered_json::value_t::discarded)));
 
-    CHECK(hashes.size() == 21);
+    CHECK(hashes.size() == 19);
+
+    CHECK(std::hash<ordered_json> {}(ordered_json(0)) == std::hash<ordered_json> {}(ordered_json(static_cast<unsigned>(0))));
+    CHECK(std::hash<ordered_json> {}(ordered_json(0)) == std::hash<ordered_json> {}(ordered_json(0.0)));
 }
 
 TEST_CASE("hash of deeply nested values")

tests/src/unit-large_json.cpp

@@ -13,6 +13,7 @@ using nlohmann::json;
 
 #include <algorithm>
 #include <string>
+#include <utility>
 #include <vector>
 
 TEST_CASE("tests on very large JSONs")
@@ -354,3 +355,199 @@ TEST_CASE("tests on deeply nested JSONs")
     }
 }
 
+namespace
+{
+json nested_array(const std::size_t depth, json leaf)
+{
+    json j = std::move(leaf);
+    for (std::size_t i = 0; i < depth; ++i)
+    {
+        json a = json::array();
+        a.push_back(std::move(j));
+        j = std::move(a);
+    }
+    return j;
+}
+
+json nested_object(const std::size_t depth, json leaf)
+{
+    json j = std::move(leaf);
+    for (std::size_t i = 0; i < depth; ++i)
+    {
+        json o = json::object();
+        o["k"] = std::move(j);
+        j = std::move(o);
+    }
+    return j;
+}
+} // namespace
+
+TEST_CASE("issue #5392 - binary writers on deeply nested values")
+{
+    // 200 is past the point where the writers stop recursing, and still
+    // shallow enough that from_* and operator== (which still recurse) are fine.
+    const json deep_array = nested_array(200, json(0));
+    const json deep_object = nested_object(200, json("x"));
+    const json empty_array = nested_array(200, json::array());
+    const json empty_object = nested_object(200, json::object());
+    const json mixed = nested_object(80, nested_array(80, json(true)));
+
+    SECTION("roundtrip past the recursion bound")
+    {
+        CHECK(json::from_cbor(json::to_cbor(deep_array)) == deep_array);
+        CHECK(json::from_msgpack(json::to_msgpack(deep_array)) == deep_array);
+        CHECK(json::from_ubjson(json::to_ubjson(deep_array)) == deep_array);
+        CHECK(json::from_ubjson(json::to_ubjson(deep_array, true, false)) == deep_array);
+        CHECK(json::from_ubjson(json::to_ubjson(deep_array, true, true)) == deep_array);
+        CHECK(json::from_bjdata(json::to_bjdata(deep_array)) == deep_array);
+
+        CHECK(json::from_cbor(json::to_cbor(deep_object)) == deep_object);
+        CHECK(json::from_msgpack(json::to_msgpack(deep_object)) == deep_object);
+        CHECK(json::from_ubjson(json::to_ubjson(deep_object)) == deep_object);
+        CHECK(json::from_ubjson(json::to_ubjson(deep_object, true, true)) == deep_object);
+        CHECK(json::from_bjdata(json::to_bjdata(deep_object)) == deep_object);
+
+        CHECK(json::from_cbor(json::to_cbor(empty_array)) == empty_array);
+        CHECK(json::from_msgpack(json::to_msgpack(empty_array)) == empty_array);
+        CHECK(json::from_ubjson(json::to_ubjson(empty_array)) == empty_array);
+        CHECK(json::from_ubjson(json::to_ubjson(empty_array, true, true)) == empty_array);
+
+        CHECK(json::from_cbor(json::to_cbor(empty_object)) == empty_object);
+        CHECK(json::from_msgpack(json::to_msgpack(empty_object)) == empty_object);
+        CHECK(json::from_ubjson(json::to_ubjson(empty_object)) == empty_object);
+
+        CHECK(json::from_cbor(json::to_cbor(mixed)) == mixed);
+        CHECK(json::from_msgpack(json::to_msgpack(mixed)) == mixed);
+        CHECK(json::from_ubjson(json::to_ubjson(mixed)) == mixed);
+        CHECK(json::from_bjdata(json::to_bjdata(mixed)) == mixed);
+    }
+
+    SECTION("the two ways of writing a value meet at the bound")
+    {
+        for (std::size_t depth = 120; depth <= 140; ++depth)
+        {
+            CAPTURE(depth);
+
+            const json array = nested_array(depth, json(7));
+            CHECK(json::from_cbor(json::to_cbor(array)) == array);
+            CHECK(json::from_msgpack(json::to_msgpack(array)) == array);
+            CHECK(json::from_ubjson(json::to_ubjson(array, true, true)) == array);
+
+            const json object = nested_object(depth, json(7));
+            CHECK(json::from_cbor(json::to_cbor(object)) == object);
+            CHECK(json::from_msgpack(json::to_msgpack(object)) == object);
+            CHECK(json::from_bjdata(json::to_bjdata(object)) == object);
+        }
+    }
+
+    SECTION("a BJData ndarray below the bound is still an ndarray")
+    {
+        const json ndarray = json({{"_ArrayType_", "uint8"}, {"_ArraySize_", {2, 3}}, {"_ArrayData_", {1, 2, 3, 4, 5, 6}}});
+        const json invalid = json({{"_ArrayType_", "nope"}, {"_ArraySize_", {1}}, {"_ArrayData_", {1}}});
+
+        const json deep_ndarray = nested_array(140, ndarray);
+        const json deep_invalid = nested_array(140, invalid);
+
+        CHECK(json::from_bjdata(json::to_bjdata(deep_ndarray)) == deep_ndarray);
+        CHECK(json::from_bjdata(json::to_bjdata(deep_invalid)) == deep_invalid);
+        CHECK(json::from_bjdata(json::to_bjdata(ndarray)) == ndarray);
+    }
+
+    SECTION("byte-exact across the switch-over")
+    {
+        // nested one-element arrays around the recursion bound: the exact
+        // bytes a writer produces do not depend on whether it stayed on the
+        // call stack or moved to the heap one partway through
+        for (const std::size_t depth :
+                {
+                    nlohmann::detail::recursion_depth_limit() - 1, nlohmann::detail::recursion_depth_limit(),
+                    nlohmann::detail::recursion_depth_limit() + 1, nlohmann::detail::recursion_depth_limit() + 2
+                })
+        {
+            CAPTURE(depth);
+            const json array = nested_array(depth, json(0));
+
+            std::vector<std::uint8_t> expected_cbor(depth, 0x81);
+            expected_cbor.push_back(0x00);
+            CHECK(json::to_cbor(array) == expected_cbor);
+
+            std::vector<std::uint8_t> expected_msgpack(depth, 0x91);
+            expected_msgpack.push_back(0x00);
+            CHECK(json::to_msgpack(array) == expected_msgpack);
+
+            std::string expected_ubjson(depth, '[');
+            expected_ubjson += "i";
+            expected_ubjson += '\0';
+            expected_ubjson.append(depth, ']');
+            const auto packed_ubjson = json::to_ubjson(array);
+            CHECK(std::string(packed_ubjson.begin(), packed_ubjson.end()) == expected_ubjson);
+        }
+    }
+
+    SECTION("a deep object, and a BJData ndarray, past the recursion bound")
+    {
+        const std::size_t depth = nlohmann::detail::recursion_depth_limit() + 50;
+
+        const json object = nested_object(depth, json(42));
+        CHECK(json::from_cbor(json::to_cbor(object)) == object);
+        CHECK(json::from_msgpack(json::to_msgpack(object)) == object);
+        CHECK(json::from_ubjson(json::to_ubjson(object, true, true)) == object);
+        CHECK(json::from_bjdata(json::to_bjdata(object)) == object);
+
+        const json ndarray = json({{"_ArrayType_", "uint8"}, {"_ArraySize_", {2, 3}}, {"_ArrayData_", {1, 2, 3, 4, 5, 6}}});
+        const json deep_ndarray = nested_array(depth, ndarray);
+        CHECK(json::from_bjdata(json::to_bjdata(deep_ndarray)) == deep_ndarray);
+    }
+
+    SECTION("a discarded value past the recursion bound still throws type_error.321")
+    {
+        const std::size_t depth = nlohmann::detail::recursion_depth_limit() + 50;
+        const json discarded_leaf(json::value_t::discarded);
+        const json deep_discarded = nested_array(depth, discarded_leaf);
+
+        CHECK_THROWS_WITH_AS(json::to_cbor(deep_discarded), "[json.exception.type_error.321] cannot serialize discarded value to CBOR", json::type_error);
+        CHECK_THROWS_WITH_AS(json::to_msgpack(deep_discarded), "[json.exception.type_error.321] cannot serialize discarded value to MessagePack", json::type_error);
+        CHECK_THROWS_WITH_AS(json::to_ubjson(deep_discarded), "[json.exception.type_error.321] cannot serialize discarded value to UBJSON", json::type_error);
+        CHECK_THROWS_WITH_AS(json::to_bjdata(deep_discarded), "[json.exception.type_error.321] cannot serialize discarded value to BJData", json::type_error);
+    }
+
+    SECTION("does not overflow the C++ stack")
+    {
+        const std::size_t depth = 100000;
+        const json j = json::parse(std::string(depth, '[') + "0" + std::string(depth, ']'));
+
+        std::vector<std::uint8_t> packed;
+        CHECK_NOTHROW(packed = json::to_cbor(j));
+        CHECK(json::from_cbor(packed) == j);
+
+        CHECK_NOTHROW(packed = json::to_msgpack(j));
+        CHECK(json::from_msgpack(packed) == j);
+
+        CHECK_NOTHROW(packed = json::to_ubjson(j));
+        CHECK(json::from_ubjson(packed) == j);
+
+        CHECK_NOTHROW(packed = json::to_ubjson(j, true, false));
+        CHECK(json::from_ubjson(packed) == j);
+
+        CHECK_NOTHROW(packed = json::to_bjdata(j));
+        CHECK(json::from_bjdata(packed) == j);
+    }
+
+    SECTION("regression test for https://issues.oss-fuzz.com/issues/566583014")
+    {
+        // 200000 nested one-element CBOR arrays, the innermost holding null;
+        // round-tripping this used to recurse once per level on the way back
+        // out through to_cbor(), deep enough to overflow the stack
+        std::vector<std::uint8_t> v(200000, 0x81);
+        v.push_back(0xf6);
+        const json j = json::from_cbor(v);
+        CHECK(json::to_cbor(j) == v);
+
+        // the MessagePack analogue: fixarray of 1 nesting down to nil
+        std::vector<std::uint8_t> v_msgpack(200000, 0x91);
+        v_msgpack.push_back(0xc0);
+        const json j_msgpack = json::from_msgpack(v_msgpack);
+        CHECK(json::to_msgpack(j_msgpack) == v_msgpack);
+    }
+}
+

tests/src/unit-regression2.cpp

@@ -888,6 +888,15 @@ TEST_CASE("regression tests 2")
         CHECK(j == k);
     }
 
+    SECTION("issue #4552 - UTF-8 invalid characters are not always ignored when dumping with error_handler_t::ignore")
+    {
+        json node;
+        node["test"] = "test\334\005";
+        CHECK(node.dump(-1, ' ', false, json::error_handler_t::ignore) == "{\"test\":\"test\\u0005\"}");
+        CHECK(node.dump(-1, ' ', false, json::error_handler_t::keep) == "{\"test\":\"test\334\\u0005\"}");
+        CHECK(node.dump(-1, ' ', true, json::error_handler_t::keep) == "{\"test\":\"test\334\\u0005\"}");
+    }
+
 #ifdef JSON_HAS_CPP_17
     SECTION("issue #5066 - MSVC converts json to std::variant<json> via the conversion operator")
     {

tests/src/unit-serialization.cpp

@@ -102,6 +102,8 @@ TEST_CASE("serialization")
             CHECK(j.dump(-1, ' ', false, json::error_handler_t::ignore) == "\"äü\"");
             CHECK(j.dump(-1, ' ', false, json::error_handler_t::replace) == "\"ä\xEF\xBF\xBDü\"");
             CHECK(j.dump(-1, ' ', true, json::error_handler_t::replace) == "\"\\u00e4\\ufffd\\u00fc\"");
+            CHECK(j.dump(-1, ' ', false, json::error_handler_t::keep) == "\"ä\xA9ü\"");
+            CHECK(j.dump(-1, ' ', true, json::error_handler_t::keep) == "\"\\u00e4\xA9\\u00fc\"");
         }
 
         SECTION("invalid character (regression guard for shared UTF-8 decoder, see #5529)")
@@ -124,6 +126,8 @@ TEST_CASE("serialization")
             CHECK(j.dump(-1, ' ', false, json::error_handler_t::ignore) == "\"123\"");
             CHECK(j.dump(-1, ' ', false, json::error_handler_t::replace) == "\"123\xEF\xBF\xBD\"");
             CHECK(j.dump(-1, ' ', true, json::error_handler_t::replace) == "\"123\\ufffd\"");
+            CHECK(j.dump(-1, ' ', false, json::error_handler_t::keep) == "\"123\xC2\"");
+            CHECK(j.dump(-1, ' ', true, json::error_handler_t::keep) == "\"123\xC2\"");
         }
 
         SECTION("unexpected character")
@@ -136,6 +140,39 @@ TEST_CASE("serialization")
             CHECK(j.dump(-1, ' ', false, json::error_handler_t::ignore) == "\"123456\"");
             CHECK(j.dump(-1, ' ', false, json::error_handler_t::replace) == "\"123\xEF\xBF\xBD\x34\x35\x36\"");
             CHECK(j.dump(-1, ' ', true, json::error_handler_t::replace) == "\"123\\ufffd456\"");
+            CHECK(j.dump(-1, ' ', false, json::error_handler_t::keep) == "\"123\xF1\xB0\x34\x35\x36\"");
+            CHECK(j.dump(-1, ' ', true, json::error_handler_t::keep) == "\"123\xF1\xB0\x34\x35\x36\"");
+        }
+
+        SECTION("keep: valid characters are still escaped")
+        {
+            // an invalid byte followed by characters that must be escaped
+            const json j = "\xC2\"\\\n\xFF\x05";
+            CHECK(j.dump(-1, ' ', false, json::error_handler_t::keep) == "\"\xC2\\\"\\\\\\n\xFF\\u0005\"");
+            CHECK(j.dump(-1, ' ', true, json::error_handler_t::keep) == "\"\xC2\\\"\\\\\\n\xFF\\u0005\"");
+        }
+
+        SECTION("keep: truncated multibyte sequences")
+        {
+            CHECK(json("\xF0\x9F\x98").dump(-1, ' ', false, json::error_handler_t::keep) == "\"\xF0\x9F\x98\"");
+            CHECK(json("\xF0\x9F\x98").dump(-1, ' ', true, json::error_handler_t::keep) == "\"\xF0\x9F\x98\"");
+            CHECK(json("\xF0\x9F\x98" "a").dump(-1, ' ', false, json::error_handler_t::keep) == "\"\xF0\x9F\x98" "a\"");
+            CHECK(json("\xF0\x9F\x98" "a").dump(-1, ' ', true, json::error_handler_t::keep) == "\"\xF0\x9F\x98" "a\"");
+        }
+
+        SECTION("keep: long string with many invalid bytes")
+        {
+            // exceeds the internal string buffer several times
+            std::string input;
+            std::string expected = "\"";
+            for (int i = 0; i < 2000; ++i)
+            {
+                input += "\xFF\xE2\x82\n\xC3\xA4";
+                expected += "\xFF\xE2\x82\\n\xC3\xA4";
+            }
+            expected += "\"";
+            const json j = input;
+            CHECK(j.dump(-1, ' ', false, json::error_handler_t::keep) == expected);
         }
 
         SECTION("U+FFFD Substitution of Maximal Subparts")

tests/src/unit-unicode.cpp

@@ -11,6 +11,7 @@
 #include <nlohmann/json.hpp>
 using nlohmann::json;
 
+#include <algorithm>
 #include <fstream>
 #include <string>
 #include "make_test_data_available.hpp"
@@ -67,8 +68,11 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
     static std::string s_replaced2;
     static std::string s_replaced_ascii;
     static std::string s_replaced2_ascii;
+    static std::string s_kept;
+    static std::string s_kept2;
+    static std::string s_kept_ascii;
 
-    // dumping with ignore/replace must not throw in any case
+    // dumping with ignore/replace/keep must not throw in any case
     s_ignored = j.dump(-1, ' ', false, json::error_handler_t::ignore);
     s_ignored2 = j2.dump(-1, ' ', false, json::error_handler_t::ignore);
     s_ignored_ascii = j.dump(-1, ' ', true, json::error_handler_t::ignore);
@@ -77,6 +81,9 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
     s_replaced2 = j2.dump(-1, ' ', false, json::error_handler_t::replace);
     s_replaced_ascii = j.dump(-1, ' ', true, json::error_handler_t::replace);
     s_replaced2_ascii = j2.dump(-1, ' ', true, json::error_handler_t::replace);
+    s_kept = j.dump(-1, ' ', false, json::error_handler_t::keep);
+    s_kept2 = j2.dump(-1, ' ', false, json::error_handler_t::keep);
+    s_kept_ascii = j.dump(-1, ' ', true, json::error_handler_t::keep);
 
     if (success_expected)
     {
@@ -86,6 +93,7 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
         // all dumps should agree on the string
         CHECK(s_strict == s_ignored);
         CHECK(s_strict == s_replaced);
+        CHECK(s_strict == s_kept);
     }
     else
     {
@@ -97,6 +105,20 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
 
         // check that replace string contains a replacement character
         CHECK(s_replaced.find("\xEF\xBF\xBD") != std::string::npos);
+
+        // ignore drops the invalid bytes, keep copies them
+        CHECK(s_ignored != s_kept);
+        CHECK(s_ignored_ascii != s_kept_ascii);
+
+        // unless a byte needs escaping, keep copies the input unchanged
+        const bool needs_escaping = std::any_of(json_string.begin(), json_string.end(), [](char c)
+        {
+            return static_cast<unsigned char>(c) < 0x20 || c == '"' || c == '\\';
+        });
+        if (!needs_escaping)
+        {
+            CHECK(s_kept == "\"" + json_string + "\"");
+        }
     }
 
     // check that prefix and suffix are preserved
@@ -108,6 +130,8 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
     CHECK(s_replaced2.substr(s_replaced2.size() - 4, 3) == "xyz");
     CHECK(s_replaced2_ascii.substr(1, 3) == "abc");
     CHECK(s_replaced2_ascii.substr(s_replaced2_ascii.size() - 4, 3) == "xyz");
+    CHECK(s_kept2.substr(1, 3) == "abc");
+    CHECK(s_kept2.substr(s_kept2.size() - 4, 3) == "xyz");
 }
 
 void check_utf8string(bool success_expected, int byte1, int byte2, int byte3, int byte4);

tools/generate_natvis/nlohmann_json.natvis.j2

@@ -36,5 +36,31 @@
         </Expand>
     </Type>
 
+    <!-- Fallback for when the basic_json entry above does not match: json_default_base is the (empty) default
+         base class of basic_json, and base class visualizers are inherited by derived types and evaluated
+         against the derived object, so m_data is accessible here. The class lives in {{ ns }} after
+         3.12.0 (#5238) and in {{ ns }}::detail up to 3.12.0, so both names are listed. -->
+{% for default_base in ['json_default_base', 'detail::json_default_base'] %}
+    <Type Name="{{ ns }}::{{ default_base }}">
+        <DisplayString Condition="m_data.m_type == {{ ns }}::detail::value_t::null">null</DisplayString>
+        <DisplayString Condition="m_data.m_type == {{ ns }}::detail::value_t::object">{*(m_data.m_value.object)}</DisplayString>
+        <DisplayString Condition="m_data.m_type == {{ ns }}::detail::value_t::array">{*(m_data.m_value.array)}</DisplayString>
+        <DisplayString Condition="m_data.m_type == {{ ns }}::detail::value_t::string">{*(m_data.m_value.string)}</DisplayString>
+        <DisplayString Condition="m_data.m_type == {{ ns }}::detail::value_t::boolean">{m_data.m_value.boolean}</DisplayString>
+        <DisplayString Condition="m_data.m_type == {{ ns }}::detail::value_t::number_integer">{m_data.m_value.number_integer}</DisplayString>
+        <DisplayString Condition="m_data.m_type == {{ ns }}::detail::value_t::number_unsigned">{m_data.m_value.number_unsigned}</DisplayString>
+        <DisplayString Condition="m_data.m_type == {{ ns }}::detail::value_t::number_float">{m_data.m_value.number_float}</DisplayString>
+        <DisplayString Condition="m_data.m_type == {{ ns }}::detail::value_t::discarded">discarded</DisplayString>
+        <Expand>
+            <ExpandedItem Condition="m_data.m_type == {{ ns }}::detail::value_t::object">
+                *(m_data.m_value.object),view(simple)
+            </ExpandedItem>
+            <ExpandedItem Condition="m_data.m_type == {{ ns }}::detail::value_t::array">
+                *(m_data.m_value.array),view(simple)
+            </ExpandedItem>
+        </Expand>
+    </Type>
+
+{% endfor %}
 {% endfor %}
 </AutoVisualizer>