Read MessagePack containers without recursing per nesting level

get_msgpack_array() and get_msgpack_object() read their elements by calling
back into parse_msgpack_internal(), which calls them again for a nested
container. The native call stack therefore grew with the nesting depth of the
input, and each level costs only one byte to encode: 0x91 is a one-element
array, so a few hundred thousand of them crash the process before any of the
input is rejected (#5104).

Keep the open containers on a heap stack instead, the way
parser::sax_parse_internal() has always done for JSON text. A frame records
how many elements are left and whether to close with end_object() or
end_array(); parse_msgpack_value() reads a single value and, for a container,
only opens it; and parse_msgpack_internal() loops, resuming the innermost
container after each element and closing it when its count runs out. Whether
the value that was begun is complete is answered by the stack being empty, so
no separate bookkeeping is needed.

The switch that decodes a value is untouched apart from the six container
cases, which now call enter_container() rather than a reader that loops. That
keeps this diff to the control flow and leaves the decoding of every other
type byte-identical.

enter_container() is the only place a binary reader emits start_object() or
start_array(), so a check that rejects a container can be added there once and
is guaranteed to run before the start event. The frame type and the stack are
shared, ready for the other three formats.

Verified against develop over empty, nested, counted (array 16/32, map 16/32)
and truncated inputs: identical values, error codes, messages and byte
offsets. 300,000 levels now report parse_error.110 instead of crashing, and a
well-formed 300,000-level value is read to completion through the SAX
interface, where develop crashes.

Reading such a value into a basic_json needs the return-by-move change as
well, without which the recursive copy constructor overflows on the way out;
that is the parent commit, and the test for the value path covers the two
together. Timing is unchanged: parsing 60,000 small objects and one array of
a million integers is within run-to-run noise of develop either way.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

.github/CONTRIBUTING.md

@@ -108,9 +108,9 @@ The tests are located in [`tests/src/unit-*.cpp`](https://github.com/nlohmann/js
 are structured along the features of the library or the nature of the tests. Usually, it should be clear from the
 context which existing file needs to be extended, and only very few cases require creating new test files.
 
-When fixing a bug, edit `unit-regression3.cpp` and add a test case referencing the fixed issue. Its predecessors
-`unit-regression1.cpp` and `unit-regression2.cpp` stay as they are: the MinGW linker fails on the object a file this
-size produces, which is why the tests are spread over several files in the first place.
+When fixing a bug, edit `unit-regression3.cpp` and add a section referencing the fixed issue.
+`unit-regression2.cpp` holds the older tests; the two files exist because a single one grew large enough for the
+MinGW linker to fail relocating it, so please keep adding to the smaller file rather than growing the larger one.
 
 #### Exceptions
 

.github/workflows/ubuntu.yml

@@ -100,7 +100,7 @@ jobs:
     container: ubuntu:focal
     strategy:
       matrix:
-        target: [ci_cmake_flags, ci_test_diagnostics, ci_test_diagnostic_positions, ci_test_noexceptions, ci_test_noimplicitconversions, ci_test_legacycomparison, ci_test_noglobaludls, ci_test_no_thread_local]
+        target: [ci_cmake_flags, ci_test_diagnostics, ci_test_diagnostic_positions, ci_test_noexceptions, ci_test_noimplicitconversions, ci_test_legacycomparison, ci_test_noglobaludls]
     steps:
       - name: Install build-essential
         run: apt-get update ; apt-get install -y build-essential unzip wget git libssl-dev

.github/workflows/windows.yml

@@ -158,10 +158,6 @@ jobs:
       # to fit: IMAGE_REL_AMD64_SECREL against `.debug_line'" because the
       # MinGW linker cannot relocate the debug sections this test produces.
       # The tests are only built and run here, so the debug info is not used.
-      # Do not add -O1 here to shrink the objects further: it does make them
-      # link, but the binaries clang 11.0.1 and clang 18.1.8 then produce crash
-      # before doctest prints its first line - 39 of 102 tests on clang 18.
-      # Keep the objects small by splitting the test files instead.
       - name: Run CMake
         run: cmake -S . -B build ^
              -DCMAKE_CXX_COMPILER="C:/Program Files/LLVM/bin/clang++.exe" ^

cmake/ci.cmake

@@ -242,25 +242,6 @@ add_custom_target(ci_test_noglobaludls
     COMMENT "Compile and test with global UDLs disabled"
 )
 
-###############################################################################
-# Disable thread-local storage.
-###############################################################################
-
-# Without thread-local storage, copying and comparing cannot bound their
-# descent and handle every object and array without the call stack. Those paths
-# are otherwise only reached by values nested deeper than the bound, so this
-# target is what runs the whole test suite through them.
-add_custom_target(ci_test_no_thread_local
-    COMMAND ${CMAKE_COMMAND}
-    -DCMAKE_BUILD_TYPE=Debug -GNinja
-    -DJSON_BuildTests=ON
-    -DCMAKE_CXX_FLAGS=-DJSON_NO_THREAD_LOCAL
-    -S${PROJECT_SOURCE_DIR} -B${PROJECT_BINARY_DIR}/build_no_thread_local
-    COMMAND ${CMAKE_COMMAND} --build ${PROJECT_BINARY_DIR}/build_no_thread_local
-    COMMAND cd ${PROJECT_BINARY_DIR}/build_no_thread_local && ${CMAKE_CTEST_COMMAND} --parallel ${N} --output-on-failure
-    COMMENT "Compile and test without thread-local storage"
-)
-
 ###############################################################################
 # Coverage.
 ###############################################################################

docs/mkdocs/docs/api/macros/index.md

@@ -22,7 +22,6 @@ header. See also the [macro overview page](../../features/macros.md).
 - [**JSON_HAS_STD_FORMAT**](json_has_std_format.md) - control `std::format`/`std::formatter` support
 - [**JSON_HAS_THREE_WAY_COMPARISON**](json_has_three_way_comparison.md) - control 3-way comparison support
 - [**JSON_NO_IO**](json_no_io.md) - switch off functions relying on certain C++ I/O headers
-- [**JSON_NO_THREAD_LOCAL**](json_no_thread_local.md) - switch off the use of `thread_local` storage
 - [**JSON_SKIP_UNSUPPORTED_COMPILER_CHECK**](json_skip_unsupported_compiler_check.md) - do not warn about unsupported compilers
 - [**JSON_USE_GLOBAL_UDLS**](json_use_global_udls.md) - place user-defined string literals (UDLs) into the global namespace
 

docs/mkdocs/docs/api/macros/json_no_thread_local.md

@@ -1,48 +0,0 @@
-# JSON_NO_THREAD_LOCAL
-
-```cpp
-#define JSON_NO_THREAD_LOCAL
-```
-
-When defined, the library does not use `#!cpp thread_local` storage. This is relevant for the few environments whose
-toolchain does not support it.
-
-Copying a value and comparing two values both descend into the first levels by letting the containers copy or compare
-themselves, and finish whatever is nested deeper than that without the call stack, so that neither can exhaust the stack
-however deeply the values are nested. Each counts the levels it has descended into in a `#!cpp thread_local` variable, as
-a counter shared between threads would be raced.
-
-Without those counters, no descent can be bounded safely, so objects and arrays are copied and compared without the call
-stack right away. Both keep working exactly as they do otherwise - the same values come out, the same comparisons hold,
-and deeply nested values are handled just as safely - but both are slower, because the containers no longer copy or
-compare themselves. Copying the benchmark documents takes 9% (`canada.json`) to 34% (`twitter.json`) longer, and
-comparing two equal ones 10% (`citm_catalog.json`) to 90% (`canada.json`) longer.
-
-## Default definition
-
-By default, `#!cpp JSON_NO_THREAD_LOCAL` is not defined.
-
-```cpp
-#undef JSON_NO_THREAD_LOCAL
-```
-
-The library defines it by itself for Clang targeting MinGW, which does not survive the `#!cpp thread_local` storage:
-copying a value segfaults there, with both old and current Clang versions, while GCC targeting MinGW is unaffected.
-Copying and comparing fall back to working without the call stack there, as they do whenever the macro is defined.
-
-## Examples
-
-??? example
-
-    The code below forces the library not to use `#!cpp thread_local` storage.
-
-    ```cpp
-    #define JSON_NO_THREAD_LOCAL 1
-    #include <nlohmann/json.hpp>
-
-    ...
-    ```
-
-## Version history
-
-- Added in version 3.12.1.

docs/mkdocs/docs/features/binary_formats/ubjson.md

@@ -69,6 +69,13 @@ The library uses the following mapping from JSON values types to UBJSON types ac
     Note that `use_size = true` alone may result in larger representations - the benefit of this parameter is that the
     receiving side is immediately informed on the number of elements of the container.
 
+    An array whose type marker is `Z` (null), `T` (true) or `F` (false) stores no payload at all, because the marker
+    already is the value. Its declared count is therefore the only thing that decides how much memory the receiving side
+    allocates, and a handful of bytes can describe billions of elements. `from_ubjson` rejects such an array with
+    [`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408) when the count exceeds 1,048,576
+    (`1 << 20`), and `to_ubjson` writes longer arrays of these types without the annotation, so any value it produces
+    can be read back.
+
 !!! info "Binary values"
 
     If the JSON data contains the binary type, the value stored is a list of integers, as suggested by the UBJSON

docs/mkdocs/docs/features/macros.md

@@ -91,14 +91,6 @@ security reasons (e.g., Intel Software Guard Extensions (SGX)).
 
 See [full documentation of `JSON_NO_IO`](../api/macros/json_no_io.md).
 
-## `JSON_NO_THREAD_LOCAL`
-
-When defined, the library does not use `#!cpp thread_local` storage. Copying a value and comparing two values then
-always avoid the call stack rather than descending into a bounded number of levels first, which is slower but yields the
-same values and the same comparisons.
-
-See [full documentation of `JSON_NO_THREAD_LOCAL`](../api/macros/json_no_thread_local.md).
-
 ## `JSON_SKIP_LIBRARY_VERSION_CHECK`
 
 When defined, the library will not create a compiler warning when a different version of the library was already

docs/mkdocs/docs/home/exceptions.md

@@ -868,6 +868,12 @@ The size of an array or object in a [binary format](../features/binary_formats/i
 the size following `#` for [UBJSON](../features/binary_formats/ubjson.md)/[BJData](../features/binary_formats/bjdata.md),
 or the encoded length for [CBOR](../features/binary_formats/cbor.md).
 
+The exception is also thrown for a [UBJSON](../features/binary_formats/ubjson.md) array of a type that is encoded by its
+marker alone (`Z`, `T` or `F`) whose declared count exceeds 1,048,576 (`1 << 20`). Such an array has no payload, so its
+count alone decides how much memory is allocated, and a handful of bytes would otherwise describe billions of values.
+[`to_ubjson`](../api/basic_json/to_ubjson.md) writes longer arrays of these types without the size and type annotation,
+so any value it produces can still be read back.
+
 !!! failure "Example messages"
 
     ```
@@ -879,6 +885,9 @@ or the encoded length for [CBOR](../features/binary_formats/cbor.md).
     ```
     [json.exception.out_of_range.408] syntax error while parsing CBOR size: excessive map size
     ```
+    ```
+    [json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive array size
+    ```
 
 ### json.exception.out_of_range.409
 

docs/mkdocs/mkdocs.yml

@@ -291,7 +291,6 @@ nav:
           - 'JSON_HAS_THREE_WAY_COMPARISON': api/macros/json_has_three_way_comparison.md
           - 'JSON_NOEXCEPTION': api/macros/json_noexception.md
           - 'JSON_NO_IO': api/macros/json_no_io.md
-          - 'JSON_NO_THREAD_LOCAL': api/macros/json_no_thread_local.md
           - 'JSON_SKIP_LIBRARY_VERSION_CHECK': api/macros/json_skip_library_version_check.md
           - 'JSON_SKIP_UNSUPPORTED_COMPILER_CHECK': api/macros/json_skip_unsupported_compiler_check.md
           - 'JSON_USE_GLOBAL_UDLS': api/macros/json_use_global_udls.md

include/nlohmann/detail/input/binary_reader.hpp

@@ -58,6 +58,26 @@ inline bool little_endianness(int num = 1) noexcept
     return *reinterpret_cast<char*>(&num) == 1;
 }
 
+/*!
+@brief largest element count accepted for a UBJSON container of a valueless type
+
+An element of type 'Z' (null), 'T' (true) or 'F' (false) is encoded by its
+type marker alone, so an optimized container of one of those types has no
+payload at all and its declared count is the only thing that decides how much
+is allocated: `[$Z#L` followed by a large count turns some ten bytes of input
+into that many values (see #2793, which reports 35 GB and 150 seconds). Every
+other type costs at least one byte per element and is bounded by the end of
+the input.
+
+This is a sanity bound rather than a security boundary, and it is far above
+any container met in practice. @ref binary_writer falls back to the
+unoptimized encoding for longer containers, so that a value serialized by
+this library can always be read back.
+
+@sa https://github.com/nlohmann/json/issues/2793
+*/
+JSON_INLINE_VARIABLE constexpr std::size_t max_valueless_container_size = 1 << 20;
+
 ///////////////////
 // binary reader //
 ///////////////////
@@ -110,6 +130,7 @@ class binary_reader
                    const cbor_tag_handler_t tag_handler = cbor_tag_handler_t::error)
     {
         sax = sax_;
+        container_stack.clear();
         bool result = false;
 
         switch (format)
@@ -159,6 +180,69 @@ class binary_reader
     }
 
   private:
+    ////////////////////////
+    // nested containers  //
+    ////////////////////////
+
+    /*!
+    @brief a container that has been opened and not closed yet
+
+    The binary readers do not call themselves once per nesting level. Like
+    @ref parser::sax_parse_internal, which does the same for JSON text, they
+    keep the containers they are inside of on a heap-allocated stack, so that
+    the native call stack does not grow with the nesting depth of the input
+    and a deeply nested value is bounded by memory rather than by the stack
+    (see #5104).
+
+    The members are ordered by decreasing alignment, which is the ordering that
+    keeps a struct from growing as members are added to it.
+    */
+    struct container_frame
+    {
+        container_frame(const std::size_t remaining_, const bool is_object_) noexcept
+            : remaining(remaining_), is_object(is_object_) {}
+
+        /// number of elements that have not been read yet
+        std::size_t remaining;
+        /// whether to close this container with end_object() or end_array()
+        bool is_object;
+    };
+
+    /*!
+    @brief open a nested array or object
+
+    Emits the SAX start event and records the container. This is the only
+    place the binary readers start a container, so a check that rejects one
+    can be made here and is then guaranteed to run before the start event.
+
+    @param[in] is_object  whether an object (true) or an array (false) begins
+    @param[in] len        number of elements the container declares
+
+    @return whether the SAX parser accepted the start event
+    */
+    bool enter_container(const bool is_object, const std::size_t len)
+    {
+        if (JSON_HEDLEY_UNLIKELY(is_object ? !sax->start_object(len) : !sax->start_array(len)))
+        {
+            return false;
+        }
+
+        container_stack.emplace_back(len, is_object);
+        return true;
+    }
+
+    /// @copydoc enter_container
+    bool enter_array(const std::size_t len)
+    {
+        return enter_container(/*is_object*/false, len);
+    }
+
+    /// @copydoc enter_container
+    bool enter_object(const std::size_t len)
+    {
+        return enter_container(/*is_object*/true, len);
+    }
+
     //////////
     // BSON //
     //////////
@@ -996,23 +1080,21 @@ class binary_reader
     }
 
     /*!
-    @brief reads a CBOR string
+    @brief reads a definite-length CBOR string
 
-    This function first reads starting bytes to determine the expected
-    string length and then copies this number of bytes into a string.
-    Additionally, CBOR's strings with indefinite lengths are supported.
+    Reads everything @ref get_cbor_string accepts except the indefinite-length
+    form, which that function handles itself. The bytes are appended to @a
+    result, so consecutive chunks of an indefinite-length string can be read
+    into the same string.
 
-    @param[out] result  created string
+    @param[out] result  string the bytes are appended to
 
     @return whether string creation completed
-    */
-    bool get_cbor_string(string_t& result)
-    {
-        if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::cbor, "string")))
-        {
-            return false;
-        }
 
+    @pre @a current is not EOF
+    */
+    bool get_cbor_string_chunk(string_t& result)
+    {
         switch (current)
         {
             // UTF-8 string (0x00..0x17 bytes follow)
@@ -1068,20 +1150,6 @@ class binary_reader
                 return get_number(input_format_t::cbor, len) && get_string(input_format_t::cbor, len, result);
             }
 
-            case 0x7F: // UTF-8 string (indefinite length)
-            {
-                while (get() != 0xFF)
-                {
-                    string_t chunk;
-                    if (!get_cbor_string(chunk))
-                    {
-                        return false;
-                    }
-                    result.append(chunk);
-                }
-                return true;
-            }
-
             default:
             {
                 auto last_token = get_token_string();
@@ -1092,23 +1160,82 @@ class binary_reader
     }
 
     /*!
-    @brief reads a CBOR byte array
+    @brief reads a CBOR string
 
     This function first reads starting bytes to determine the expected
-    byte array length and then copies this number of bytes into the byte array.
-    Additionally, CBOR's byte arrays with indefinite lengths are supported.
+    string length and then copies this number of bytes into a string.
+    Additionally, CBOR's strings with indefinite lengths are supported.
 
-    @param[out] result  created byte array
+    @param[out] result  created string
+
+    @return whether string creation completed
+    */
+    bool get_cbor_string(string_t& result)
+    {
+        // number of indefinite-length strings that have been opened and not
+        // closed yet. RFC 8949, Section 3.2.3 does not permit nesting them,
+        // but this reader has always accepted it, so the open levels are
+        // counted instead of recursed through, which overflowed the stack for
+        // an input of repeated 0x7F bytes (see #5104). Every chunk is appended
+        // to the same result, so no per-level state is needed.
+        std::size_t open = 0;
+
+        while (true)
+        {
+            if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::cbor, "string")))
+            {
+                return false;
+            }
+
+            if (current == 0x7F) // UTF-8 string (indefinite length)
+            {
+                ++open;
+                get();
+                continue;
+            }
+
+            // a break marker closes the innermost indefinite-length string;
+            // outside of one it is not a string and falls through to the error
+            if (open != 0 && current == 0xFF)
+            {
+                if (--open == 0)
+                {
+                    return true;
+                }
+                get();
+                continue;
+            }
+
+            if (JSON_HEDLEY_UNLIKELY(!get_cbor_string_chunk(result)))
+            {
+                return false;
+            }
+
+            if (open == 0)
+            {
+                return true;
+            }
+
+            get();
+        }
+    }
+
+    /*!
+    @brief reads a definite-length CBOR byte array
+
+    Reads everything @ref get_cbor_binary accepts except the indefinite-length
+    form, which that function handles itself. The bytes are appended to @a
+    result, so consecutive chunks of an indefinite-length byte array can be
+    read into the same byte array.
+
+    @param[out] result  byte array the bytes are appended to
 
     @return whether byte array creation completed
-    */
-    bool get_cbor_binary(binary_t& result)
-    {
-        if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::cbor, "binary")))
-        {
-            return false;
-        }
 
+    @pre @a current is not EOF
+    */
+    bool get_cbor_binary_chunk(binary_t& result)
+    {
         switch (current)
         {
             // Binary data (0x00..0x17 bytes follow)
@@ -1168,20 +1295,6 @@ class binary_reader
                        get_binary(input_format_t::cbor, len, result);
             }
 
-            case 0x5F: // Binary data (indefinite length)
-            {
-                while (get() != 0xFF)
-                {
-                    binary_t chunk;
-                    if (!get_cbor_binary(chunk))
-                    {
-                        return false;
-                    }
-                    result.insert(result.end(), chunk.begin(), chunk.end());
-                }
-                return true;
-            }
-
             default:
             {
                 auto last_token = get_token_string();
@@ -1191,6 +1304,63 @@ class binary_reader
         }
     }
 
+    /*!
+    @brief reads a CBOR byte array
+
+    This function first reads starting bytes to determine the expected
+    byte array length and then copies this number of bytes into the byte array.
+    Additionally, CBOR's byte arrays with indefinite lengths are supported.
+
+    @param[out] result  created byte array
+
+    @return whether byte array creation completed
+    */
+    bool get_cbor_binary(binary_t& result)
+    {
+        // the open indefinite-length byte arrays are counted rather than
+        // recursed through, for the reason given in @ref get_cbor_string
+        std::size_t open = 0;
+
+        while (true)
+        {
+            if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::cbor, "binary")))
+            {
+                return false;
+            }
+
+            if (current == 0x5F) // Binary data (indefinite length)
+            {
+                ++open;
+                get();
+                continue;
+            }
+
+            // a break marker closes the innermost indefinite-length byte
+            // array; outside of one it falls through to the error below
+            if (open != 0 && current == 0xFF)
+            {
+                if (--open == 0)
+                {
+                    return true;
+                }
+                get();
+                continue;
+            }
+
+            if (JSON_HEDLEY_UNLIKELY(!get_cbor_binary_chunk(result)))
+            {
+                return false;
+            }
+
+            if (open == 0)
+            {
+                return true;
+            }
+
+            get();
+        }
+    }
+
     /*!
     @brief narrow a definite CBOR array/map length to std::size_t
 
@@ -1316,7 +1486,17 @@ class binary_reader
     /*!
     @return whether a valid MessagePack value was passed to the SAX parser
     */
-    bool parse_msgpack_internal()
+    /*!
+    @brief read one MessagePack value
+
+    Reads a single value and passes it to the SAX parser. A value that begins
+    a container is not read to its end: the container is opened with
+    @ref enter_container and its elements are read by
+    @ref parse_msgpack_internal, so that nesting does not consume native stack.
+
+    @return whether reading the value succeeded
+    */
+    bool parse_msgpack_value()
     {
         switch (get())
         {
@@ -1472,7 +1652,7 @@ class binary_reader
             case 0x8D:
             case 0x8E:
             case 0x8F:
-                return get_msgpack_object(conditional_static_cast<std::size_t>(static_cast<unsigned int>(current) & 0x0Fu));
+                return enter_object(conditional_static_cast<std::size_t>(static_cast<unsigned int>(current) & 0x0Fu));
 
             // fixarray
             case 0x90:
@@ -1491,7 +1671,7 @@ class binary_reader
             case 0x9D:
             case 0x9E:
             case 0x9F:
-                return get_msgpack_array(conditional_static_cast<std::size_t>(static_cast<unsigned int>(current) & 0x0Fu));
+                return enter_array(conditional_static_cast<std::size_t>(static_cast<unsigned int>(current) & 0x0Fu));
 
             // fixstr
             case 0xA0:
@@ -1622,25 +1802,25 @@ class binary_reader
             case 0xDC: // array 16
             {
                 std::uint16_t len{};
-                return get_number(input_format_t::msgpack, len) && get_msgpack_array(static_cast<std::size_t>(len));
+                return get_number(input_format_t::msgpack, len) && enter_array(static_cast<std::size_t>(len));
             }
 
             case 0xDD: // array 32
             {
                 std::uint32_t len{};
-                return get_number(input_format_t::msgpack, len) && get_msgpack_array(conditional_static_cast<std::size_t>(len));
+                return get_number(input_format_t::msgpack, len) && enter_array(conditional_static_cast<std::size_t>(len));
             }
 
             case 0xDE: // map 16
             {
                 std::uint16_t len{};
-                return get_number(input_format_t::msgpack, len) && get_msgpack_object(static_cast<std::size_t>(len));
+                return get_number(input_format_t::msgpack, len) && enter_object(static_cast<std::size_t>(len));
             }
 
             case 0xDF: // map 32
             {
                 std::uint32_t len{};
-                return get_number(input_format_t::msgpack, len) && get_msgpack_object(conditional_static_cast<std::size_t>(len));
+                return get_number(input_format_t::msgpack, len) && enter_object(conditional_static_cast<std::size_t>(len));
             }
 
             // negative fixint
@@ -1888,55 +2068,69 @@ class binary_reader
     }
 
     /*!
-    @param[in] len  the length of the array
-    @return whether array creation completed
+    @brief read a MessagePack value and everything nested inside it
+
+    Reads values until the one that was begun here is complete, resuming the
+    enclosing container each time an element ends, so that the nesting depth
+    of the input costs heap rather than native stack (see #5104).
+
+    @return whether reading the value succeeded
     */
-    bool get_msgpack_array(const std::size_t len)
+    bool parse_msgpack_internal()
     {
-        if (JSON_HEDLEY_UNLIKELY(!sax->start_array(len)))
-        {
-            return false;
-        }
-
-        for (std::size_t i = 0; i < len; ++i)
-        {
-            if (JSON_HEDLEY_UNLIKELY(!parse_msgpack_internal()))
-            {
-                return false;
-            }
-        }
-
-        return sax->end_array();
-    }
-
-    /*!
-    @param[in] len  the length of the object
-    @return whether object creation completed
-    */
-    bool get_msgpack_object(const std::size_t len)
-    {
-        if (JSON_HEDLEY_UNLIKELY(!sax->start_object(len)))
-        {
-            return false;
-        }
-
+        // the key currently being read; hoisted out of the loop so that its
+        // capacity is reused across elements and across nesting levels
         string_t key;
-        for (std::size_t i = 0; i < len; ++i)
+
+        while (true)
         {
-            get();
-            if (JSON_HEDLEY_UNLIKELY(!get_msgpack_string(key) || !sax->key(key)))
+            if (!container_stack.empty())
+            {
+                // copied out before anything can push onto the stack and
+                // invalidate a reference into it
+                const bool is_object = container_stack.back().is_object;
+
+                if (container_stack.back().remaining == 0)
+                {
+                    container_stack.pop_back();
+                    if (JSON_HEDLEY_UNLIKELY(is_object ? !sax->end_object() : !sax->end_array()))
+                    {
+                        return false;
+                    }
+                    // the value begun here is complete once its container is
+                    if (container_stack.empty())
+                    {
+                        return true;
+                    }
+                    continue;
+                }
+
+                // claim the element about to be read
+                --container_stack.back().remaining;
+
+                if (is_object)
+                {
+                    get();
+                    key.clear();
+                    if (JSON_HEDLEY_UNLIKELY(!get_msgpack_string(key) || !sax->key(key)))
+                    {
+                        return false;
+                    }
+                }
+            }
+
+            if (JSON_HEDLEY_UNLIKELY(!parse_msgpack_value()))
             {
                 return false;
             }
 
-            if (JSON_HEDLEY_UNLIKELY(!parse_msgpack_internal()))
+            // a value that opened a container left it on the stack; one that
+            // did not, and that was not inside a container, was the whole value
+            if (container_stack.empty())
             {
-                return false;
+                return true;
             }
-            key.clear();
         }
-
-        return sax->end_object();
     }
 
     ////////////
@@ -2391,7 +2585,12 @@ class binary_reader
     {
         result.first = npos; // size
         result.second = 0; // type
-        bool is_ndarray = false;
+        // seed the flag with the caller's context: inside an ndarray dimension
+        // vector another ndarray is not allowed, and get_ubjson_size_value()
+        // rejects it up front instead of reading it and reporting afterwards.
+        // Seeding it with `false` made every '#' of a "[#[#[..." chain descend
+        // another level, which overflowed the stack (see #5104).
+        bool is_ndarray = inside_ndarray;
 
         get_ignore_noop();
 
@@ -2424,13 +2623,11 @@ class binary_reader
             }
 
             const bool is_error = get_ubjson_size_value(result.first, is_ndarray);
-            if (input_format == input_format_t::bjdata && is_ndarray)
+            // an ndarray was read here only if the flag flipped; when it was
+            // seeded true, get_ubjson_size_value() already rejected the nested
+            // dimension vector
+            if (input_format == input_format_t::bjdata && is_ndarray && !inside_ndarray)
             {
-                if (inside_ndarray)
-                {
-                    return sax->parse_error(chars_read, get_token_string(), parse_error::create(112, chars_read,
-                                            exception_message(input_format, "ndarray can not be recursive", "size"), nullptr));
-                }
                 result.second |= (1 << 8); // use bit 8 to indicate ndarray, all UBJSON and BJData markers should be ASCII letters
             }
             return is_error;
@@ -2439,7 +2636,7 @@ class binary_reader
         if (current == '#')
         {
             const bool is_error = get_ubjson_size_value(result.first, is_ndarray);
-            if (input_format == input_format_t::bjdata && is_ndarray)
+            if (input_format == input_format_t::bjdata && is_ndarray && !inside_ndarray)
             {
                 return sax->parse_error(chars_read, get_token_string(), parse_error::create(112, chars_read,
                                         exception_message(input_format, "ndarray requires both type and size", "size"), nullptr));
@@ -2710,6 +2907,17 @@ class binary_reader
 
         if (size_and_type.first != npos)
         {
+            // reading an element of a valueless type consumes no input, so the
+            // declared count alone decides how much is allocated; the check is
+            // made before the start event so that no container is opened that
+            // is then abandoned. See @ref max_valueless_container_size.
+            if (JSON_HEDLEY_UNLIKELY((size_and_type.second == 'Z' || size_and_type.second == 'T' || size_and_type.second == 'F')
+                                     && size_and_type.first > max_valueless_container_size))
+            {
+                return sax->parse_error(chars_read, get_token_string(), out_of_range::create(408,
+                                        exception_message(input_format, "excessive array size", "size"), nullptr));
+            }
+
             if (JSON_HEDLEY_UNLIKELY(!sax->start_array(size_and_type.first)))
             {
                 return false;
@@ -3227,6 +3435,9 @@ class binary_reader
     /// the SAX parser
     json_sax_t* sax = nullptr;
 
+    /// the containers that have been opened and not closed yet; see @ref container_frame
+    std::vector<container_frame> container_stack{};
+
     // excluded markers in bjdata optimized type
 #define JSON_BINARY_READER_MAKE_BJD_OPTIMIZED_TYPE_MARKERS_ \
     make_array<char_int_type>('F', 'H', 'N', 'S', 'T', 'Z', '[', '{')

include/nlohmann/detail/macro_scope.hpp

@@ -186,15 +186,6 @@
     #define JSON_NO_UNIQUE_ADDRESS
 #endif
 
-// Clang targeting MinGW does not survive the thread_local storage the copy
-// constructor uses to bound its descent: every test that copies a value
-// segfaults with clang 11.0.1 and clang 18.1.8, while the same tests pass with
-// GCC targeting MinGW and with every other toolchain the library is tested on.
-// Copying works the same way without the counter, only more slowly.
-#if !defined(JSON_NO_THREAD_LOCAL) && defined(__clang__) && defined(__MINGW32__)
-    #define JSON_NO_THREAD_LOCAL 1
-#endif
-
 // disable documentation warnings on clang
 #if defined(__clang__)
     #pragma clang diagnostic push

include/nlohmann/detail/output/binary_writer.hpp

@@ -826,7 +826,17 @@ class binary_writer
 
                     std::vector<CharType> bjdx = {'[', '{', 'S', 'H', 'T', 'F', 'N', 'Z'}; // excluded markers in bjdata optimized type
 
-                    if (same_prefix && !(use_bjdata && std::find(bjdx.begin(), bjdx.end(), first_prefix) != bjdx.end()))
+                    // an optimized array of a valueless type carries no payload, so a
+                    // reader has nothing but the declared count to bound the allocation
+                    // by and refuses an excessive one. Write the unoptimized form for
+                    // those, at one byte per element, so the result can be read back.
+                    // Objects are not affected: every element is preceded by its key.
+                    const bool valueless_type = (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');
+                    const bool excessive_valueless = valueless_type
+                                                     && j.m_data.m_value.array->size() > detail::max_valueless_container_size;
+
+                    if (same_prefix && !excessive_valueless
+                            && !(use_bjdata && std::find(bjdx.begin(), bjdx.end(), first_prefix) != bjdx.end()))
                     {
                         prefix_required = false;
                         oa->write_character(to_char_type('$'));

include/nlohmann/json.hpp

@@ -28,14 +28,14 @@
     #pragma GCC diagnostic ignored "-Wignored-attributes"
 #endif
 
-#include <algorithm> // all_of, find, for_each, none_of
+#include <algorithm> // all_of, find, for_each
 #include <cstddef> // nullptr_t, ptrdiff_t, size_t
 #include <functional> // hash, less
 #include <initializer_list> // initializer_list
 #ifndef JSON_NO_IO
     #include <iosfwd> // istream, ostream
 #endif  // JSON_NO_IO
-#include <iterator> // make_move_iterator, random_access_iterator_tag
+#include <iterator> // random_access_iterator_tag
 #include <memory> // unique_ptr
 #include <string> // string, stoi, to_string
 #include <utility> // declval, forward, move, pair, swap
@@ -822,623 +822,6 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         return j;
     }
 
-#ifndef JSON_NO_THREAD_LOCAL
-    /// the number of levels an operation descends into before it finishes the
-    /// value below it without the call stack
-    static constexpr std::uint8_t nesting_depth_limit()
-    {
-        return 128;
-    }
-
-    /*!
-    @brief how many levels the operation going on in this thread has descended into
-
-    Copying a value and comparing two values share this count. The library never
-    nests one inside the other - copying a value does not compare one, and
-    comparing two values does not copy them - and where user code nests them
-    anyway, sharing the count only ends a descent sooner than it had to, which
-    costs a little speed and is never wrong.
-
-    A byte is enough: the count never exceeds the limit by more than the single
-    level that notices the limit has been reached.
-    */
-    static std::uint8_t& nesting_depth() noexcept
-    {
-        static thread_local std::uint8_t depth = 0; // NOLINT(misc-use-internal-linkage)
-        return depth;
-    }
-#endif
-
-    /*!
-    @brief whether a descent must stop here and finish without the call stack
-
-    @a may_descend says whether the operator descends at all; it is a constant
-    at every call site, and is passed rather than tested by the caller so that
-    the test does not become a constant condition there, which MSVC reports as
-    C4127.
-
-    The comparison operators use this rather than @ref nesting_depth_guard::okay,
-    because they are written as a macro and a macro cannot use the preprocessor
-    the way the guard's constructor does; @ref copy_structured, which can, asks
-    the guard instead and never calls this.
-    */
-    static bool nesting_depth_exhausted(bool may_descend = true) noexcept
-    {
-#ifdef JSON_NO_THREAD_LOCAL
-        // without a count of its own per thread, a descent cannot be bounded
-        // without racing another one, so none is made
-        static_cast<void>(may_descend);
-        return true;
-#else
-        return !may_descend || nesting_depth() >= nesting_depth_limit();
-#endif
-    }
-
-    /*!
-    @brief counts one level of a bounded descent for as long as it runs, and
-           reports whether the descent was still within the limit when it began
-
-    Looks the count up and tests it against the limit itself, rather than
-    leaving that to the caller: either way it is reached exactly once, so
-    there is nothing to be gained by making the caller do it.
-
-    Does nothing and is never @ref okay without thread-local storage, where no
-    descent can be bounded at all: a caller that only descends while this says
-    it may always ends up finishing without the call stack, exactly as if every
-    value were nested past the limit.
-    */
-    class nesting_depth_guard
-    {
-      public:
-        nesting_depth_guard() noexcept
-#ifdef JSON_NO_THREAD_LOCAL
-            : m_okay(false)
-#else
-            : m_okay(nesting_depth() < nesting_depth_limit())
-#endif
-        {
-#ifndef JSON_NO_THREAD_LOCAL
-            ++nesting_depth();
-#endif
-        }
-
-        ~nesting_depth_guard()
-        {
-#ifndef JSON_NO_THREAD_LOCAL
-            --nesting_depth();
-#endif
-        }
-
-        nesting_depth_guard(const nesting_depth_guard&) = delete;
-        nesting_depth_guard& operator=(const nesting_depth_guard&) = delete;
-        nesting_depth_guard(nesting_depth_guard&&) = delete;
-        nesting_depth_guard& operator=(nesting_depth_guard&&) = delete;
-
-        bool okay() const noexcept
-        {
-            return m_okay;
-        }
-
-      private:
-        bool m_okay;
-    };
-
-    /// an entry of the iterative deep copy's worklist: a structured value and
-    /// the value that is to become its copy
-    using copy_worklist_t = std::vector<std::pair<const basic_json*, basic_json*>>;
-
-    /// scratch space to build the key skeleton of an object copy in one go
-    using copy_scratch_t = std::vector<std::pair<typename object_t::key_type, basic_json>>;
-
-    /// @brief copy everything of @a src into @a dst but its type and value
-    static void copy_metadata(const basic_json& src, basic_json& dst)
-    {
-        // a custom base class is only required to be copy-constructible and
-        // move-assignable, so the copy has to go through a temporary
-        static_cast<json_base_class_t&>(dst) = json_base_class_t(static_cast<const json_base_class_t&>(src));
-
-#if JSON_DIAGNOSTIC_POSITIONS
-        dst.start_position = src.start_position;
-        dst.end_position = src.end_position;
-#else
-        static_cast<void>(src);
-        static_cast<void>(dst);
-#endif
-    }
-
-    /*!
-    @brief copy the value of @a src into @a dst, which must not be structured
-
-    Objects and arrays are left alone: creating those is the one thing the copy
-    constructor and @ref copy_shallow do differently from one another, and it is
-    the reason copying a value can descend at all.
-    */
-    /// @note inlined on purpose: both callers have already told an object or an
-    ///       array apart from the rest, and letting the compiler fold that test
-    ///       into this switch is worth a few percent when copying a value made
-    ///       mostly of numbers
-    JSON_HEDLEY_ALWAYS_INLINE
-    static void copy_leaf_value(const basic_json& src, basic_json& dst)
-    {
-        switch (src.m_data.m_type)
-        {
-            case value_t::string:
-            {
-                dst.m_data.m_value = *src.m_data.m_value.string;
-                break;
-            }
-
-            case value_t::binary:
-            {
-                dst.m_data.m_value = *src.m_data.m_value.binary;
-                break;
-            }
-
-            case value_t::boolean:
-            {
-                dst.m_data.m_value = src.m_data.m_value.boolean;
-                break;
-            }
-
-            case value_t::number_integer:
-            {
-                dst.m_data.m_value = src.m_data.m_value.number_integer;
-                break;
-            }
-
-            case value_t::number_unsigned:
-            {
-                dst.m_data.m_value = src.m_data.m_value.number_unsigned;
-                break;
-            }
-
-            case value_t::number_float:
-            {
-                dst.m_data.m_value = src.m_data.m_value.number_float;
-                break;
-            }
-
-            case value_t::object:
-            case value_t::array:
-            case value_t::null:
-            case value_t::discarded:
-            default:
-                break;
-        }
-    }
-
-    /*!
-    @brief copy everything of @a src into the null value @a dst but the children
-
-    Objects and arrays are not copied here; they are appended to @a worklist to
-    be created later by @ref copy_iteratively. Until that happens, @a dst remains
-    a null value, so that a partially built copy can be destroyed at any point
-    without ever violating the class invariants.
-    */
-    static void copy_shallow(const basic_json& src, basic_json& dst, copy_worklist_t& worklist)
-    {
-        copy_metadata(src, dst);
-
-        if (src.m_data.m_type == value_t::object || src.m_data.m_type == value_t::array)
-        {
-            // defer: dst stays a null value until its container exists
-            worklist.emplace_back(&src, &dst);
-            return;
-        }
-
-        copy_leaf_value(src, dst);
-
-        // only now that the value exists may the type be set: had the creation
-        // of the value thrown, dst would have been left as a valid null value
-        dst.m_data.m_type = src.m_data.m_type;
-    }
-
-    /// @brief create the copy of the array @a src in @a dst
-    /// @note structured elements are appended to @a worklist instead
-    static void copy_array_level(const basic_json& src, basic_json& dst, copy_worklist_t& worklist)
-    {
-        const array_t& src_array = *src.m_data.m_value.array;
-
-        // create all elements up front: growing the array afterwards could
-        // invalidate the pointers that are handed to the worklist
-        dst.m_data.m_value.array = create<array_t>(src_array.size(), basic_json());
-
-        auto dst_it = dst.m_data.m_value.array->begin();
-        for (auto src_it = src_array.cbegin(); src_it != src_array.cend(); ++src_it, ++dst_it)
-        {
-            copy_shallow(*src_it, *dst_it, worklist);
-        }
-    }
-
-    /// @brief create the copy of the object @a src in @a dst
-    /// @note structured values are appended to @a worklist instead
-    static void copy_object_level(const basic_json& src, basic_json& dst,
-                                  copy_worklist_t& worklist, copy_scratch_t& scratch)
-    {
-        const object_t& src_object = *src.m_data.m_value.object;
-
-        // build the complete key skeleton and hand it to the object's range
-        // constructor: adding the keys one by one would be quadratic for object
-        // types that are backed by a vector, such as nlohmann::ordered_map
-        scratch.clear();
-        scratch.reserve(src_object.size());
-        for (const auto& element : src_object)
-        {
-            scratch.emplace_back(element.first, basic_json());
-        }
-
-        dst.m_data.m_value.object = create<object_t>(std::make_move_iterator(scratch.begin()),
-                                    std::make_move_iterator(scratch.end()));
-        scratch.clear();
-
-        // pair every value of the copy with its counterpart in the original;
-        // both are enumerated in the same order for every object type with a
-        // deterministic order, so the lookup is only needed for exotic ones
-        auto src_it = src_object.cbegin();
-        for (auto& element : *dst.m_data.m_value.object)
-        {
-            if (JSON_HEDLEY_LIKELY(src_it != src_object.cend() && src_it->first == element.first))
-            {
-                copy_shallow(src_it->second, element.second, worklist);
-                ++src_it;
-            }
-            else
-            {
-                const auto found = src_object.find(element.first);
-                JSON_ASSERT(found != src_object.cend());
-                copy_shallow(found->second, element.second, worklist);
-            }
-        }
-    }
-
-    /*!
-    @brief deep-copy the object or array @a src into this value without recursing
-
-    The values whose copy has not been created yet are kept on an explicit
-    worklist rather than on the call stack. This is only reached for values
-    nested deeper than @ref nesting_depth_limit levels, which is why it copies
-    every container by hand instead of letting the container do it: the fast
-    ways of doing so would descend into the elements and defeat the purpose.
-    */
-    void copy_iteratively(const basic_json& src)
-    {
-        copy_worklist_t worklist;
-        copy_scratch_t scratch;
-
-        const basic_json* src_value = &src;
-        basic_json* dst_value = this;
-
-        for (;;)
-        {
-            if (src_value->m_data.m_type == value_t::array)
-            {
-                copy_array_level(*src_value, *dst_value, worklist);
-            }
-            else
-            {
-                copy_object_level(*src_value, *dst_value, worklist, scratch);
-            }
-
-            // the container is complete and will not be modified again
-            dst_value->set_parents();
-
-            if (worklist.empty())
-            {
-                break;
-            }
-
-            const auto& next = worklist.back();
-            src_value = next.first;
-            dst_value = next.second;
-            worklist.pop_back();
-
-            // the value stops being a null value exactly here
-            dst_value->m_data.m_type = src_value->m_data.m_type;
-        }
-    }
-
-    /*!
-    @brief copy one level of the object or array @a src into this value
-
-    The container copies its own elements, which is the fastest way to fill it.
-    Every element that is structured itself comes back to @ref copy_structured.
-    */
-    void copy_level(const basic_json& src)
-    {
-        if (m_data.m_type == value_t::object)
-        {
-            m_data.m_value = *src.m_data.m_value.object;
-        }
-        else
-        {
-            m_data.m_value = *src.m_data.m_value.array;
-        }
-
-        set_parents();
-    }
-
-    /*!
-    @brief deep-copy the object or array @a src into this value
-
-    Copying a container copies its elements, so a value nested deeply enough
-    used to exhaust the call stack. The descent is bounded here: the first
-    @ref nesting_depth_limit levels are copied by the containers themselves, just
-    as they always were, and anything below that is copied without the call
-    stack by @ref copy_iteratively. Copying a value can therefore no longer
-    exhaust the stack, however deeply it is nested, just like destroying one
-    cannot since #1436.
-
-    Nothing has to be scanned or built by hand to reach that: a value that is
-    not nested deeper than the limit - all but a vanishing minority - is copied
-    exactly as it was before, and this whole detour costs it one counter.
-
-    @sa https://github.com/nlohmann/json/issues/5387
-    */
-    void copy_structured(const basic_json& src)
-    {
-        const nesting_depth_guard guard;
-
-        if (JSON_HEDLEY_LIKELY(guard.okay()))
-        {
-            copy_level(src);
-            return;
-        }
-
-        // Finish this value without descending any further. It is completed
-        // before this returns, so a copy made by a custom base class - or by
-        // anything else that runs while a copy is going on - is unaffected by
-        // the copy it is nested in.
-        copy_iteratively(src);
-    }
-
-
-    /// the result of comparing two values, including values that cannot be
-    /// ordered at all, such as a discarded value or a NaN
-    enum class compare_result { less, equal, greater, unordered };
-
-#if JSON_HAS_THREE_WAY_COMPARISON
-    /// @brief the ordering that @a result stands for
-    static std::partial_ordering to_partial_ordering(compare_result result) noexcept // *NOPAD*
-    {
-        switch (result)
-        {
-            case compare_result::less:
-                return std::partial_ordering::less;
-            case compare_result::greater:
-                return std::partial_ordering::greater;
-            case compare_result::equal:
-                return std::partial_ordering::equivalent;
-            case compare_result::unordered:
-            default:
-                return std::partial_ordering::unordered;
-        }
-    }
-#endif
-
-    /*!
-    @brief compare two values that are not both an array or both an object
-
-    Such a pair is compared by the operators themselves, which cannot descend
-    into it and therefore cannot recurse.
-
-    That holds for a pair whose types differ as much as for a pair of leaves: an
-    array and an object are told apart by their types alone, because an operator
-    only ever descends into two values of the same type. So `==` reports them as
-    unequal without looking inside either, and an ordering falls back to the
-    order of the types - an object sorts before an array - exactly as it does
-    for a value that is not nested deeply enough to get here.
-    */
-    template<bool Ordered>
-    static compare_result compare_leaves(const_reference lhs, const_reference rhs) noexcept
-    {
-        if (lhs == rhs)
-        {
-            return compare_result::equal;
-        }
-
-        return order_leaves(lhs, rhs, std::integral_constant<bool, Ordered> {});
-    }
-
-    /*!
-    @brief compare two object keys
-
-    An object compares its entries as pairs of a key and a value, so its keys
-    are compared exactly as std::pair compares them: with < where the objects
-    are being ordered, and with == where they are only checked for equality.
-    Note that this is not the object's own comparator, which for a vector-backed
-    object type such as nlohmann::ordered_map tells equality rather than order.
-    */
-    static compare_result compare_keys(const typename object_t::key_type& lhs,
-                                       const typename object_t::key_type& rhs,
-                                       std::true_type /*ordered*/)
-    {
-        if (lhs < rhs)
-        {
-            return compare_result::less;
-        }
-
-        if (rhs < lhs)
-        {
-            return compare_result::greater;
-        }
-
-        return compare_result::equal;
-    }
-
-    /// @brief check two object keys for equality
-    static compare_result compare_keys(const typename object_t::key_type& lhs,
-                                       const typename object_t::key_type& rhs,
-                                       std::false_type /*ordered*/)
-    {
-        return lhs == rhs ? compare_result::equal : compare_result::unordered;
-    }
-
-    /// @brief tell apart two values that are not equal
-    /// @note only instantiated where the values are being ordered, as a key or
-    ///       string type is not required to be ordered to be compared for equality
-    static compare_result order_leaves(const_reference lhs, const_reference rhs, std::true_type /*ordered*/) noexcept
-    {
-        if (lhs < rhs)
-        {
-            return compare_result::less;
-        }
-
-        if (rhs < lhs)
-        {
-            return compare_result::greater;
-        }
-
-        return compare_result::unordered;
-    }
-
-    /// @brief report two values as not equal without ordering them
-    static compare_result order_leaves(const_reference /*lhs*/, const_reference /*rhs*/, std::false_type /*ordered*/) noexcept
-    {
-        return compare_result::unordered;
-    }
-
-    /*!
-    @brief compare @a lhs and @a rhs without descending into them
-
-    Reached once a comparison has descended @ref nesting_depth_limit levels, so
-    that comparing values cannot exhaust the call stack however deeply they are
-    nested. The two values are walked in lockstep on an explicit stack and
-    compared lexicographically, element by element in the order the containers
-    enumerate them - which is how the container types this library ships compare
-    themselves: a std::map enumerates its entries in key order, and
-    nlohmann::ordered_map in insertion order. An object type that enumerates its
-    entries in an unspecified order, such as std::unordered_map, compares them
-    pairwise instead; the difference could only ever show below the bound.
-
-    Note that the stack this walks with is allocated, while the comparison
-    operators are noexcept and the container comparison this replaces allocated
-    nothing. Failing that allocation therefore ends the process rather than
-    throwing. It only arises for values nested past the bound, and only when
-    memory has run out - where the same comparison used to exhaust the call
-    stack instead - but it is a way to fail that the operators did not have.
-    */
-    template<bool Ordered>
-    static compare_result compare_iteratively(const_reference lhs, const_reference rhs,
-            const bool unordered_compares_equal) noexcept
-    {
-        /// a pair of containers being compared in lockstep
-        struct frame
-        {
-            const basic_json* lhs_value{nullptr};
-            const basic_json* rhs_value{nullptr};
-            typename array_t::const_iterator lhs_array_it{};
-            typename array_t::const_iterator rhs_array_it{};
-            typename object_t::const_iterator lhs_object_it{};
-            typename object_t::const_iterator rhs_object_it{};
-        };
-
-        std::vector<frame> stack;
-        const basic_json* left = &lhs;
-        const basic_json* right = &rhs;
-
-        for (;;)
-        {
-            const auto type = left->m_data.m_type;
-
-            if (type == right->m_data.m_type && (type == value_t::array || type == value_t::object))
-            {
-                // descend: the elements decide, and are compared further down
-                stack.emplace_back();
-                frame& pushed = stack.back();
-                pushed.lhs_value = left;
-                pushed.rhs_value = right;
-
-                if (type == value_t::array)
-                {
-                    pushed.lhs_array_it = left->m_data.m_value.array->cbegin();
-                    pushed.rhs_array_it = right->m_data.m_value.array->cbegin();
-                }
-                else
-                {
-                    pushed.lhs_object_it = left->m_data.m_value.object->cbegin();
-                    pushed.rhs_object_it = right->m_data.m_value.object->cbegin();
-                }
-            }
-            else
-            {
-                const compare_result result = compare_leaves<Ordered>(*left, *right);
-
-                // Values that cannot be ordered - a NaN, say - end an ordered
-                // comparison for std::lexicographical_compare_three_way, but
-                // std::lexicographical_compare treats them as equivalent and
-                // carries on with the next element. Both are reproduced here,
-                // so that a value nested too deeply to descend into compares
-                // exactly as one that is not.
-                if (result != compare_result::equal &&
-                        !(unordered_compares_equal && result == compare_result::unordered))
-                {
-                    return result;
-                }
-            }
-
-            // walk back up past the containers that are exhausted, then take the
-            // next pair of elements from the innermost one that is not
-            for (;;)
-            {
-                if (stack.empty())
-                {
-                    return compare_result::equal;
-                }
-
-                frame& current = stack.back();
-                const bool is_object = current.lhs_value->m_data.m_type == value_t::object;
-
-                const bool lhs_done = is_object
-                                      ? current.lhs_object_it == current.lhs_value->m_data.m_value.object->cend()
-                                      : current.lhs_array_it == current.lhs_value->m_data.m_value.array->cend();
-                const bool rhs_done = is_object
-                                      ? current.rhs_object_it == current.rhs_value->m_data.m_value.object->cend()
-                                      : current.rhs_array_it == current.rhs_value->m_data.m_value.array->cend();
-
-                if (lhs_done || rhs_done)
-                {
-                    // whichever ran out first holds the smaller container; if
-                    // both did, they are equal and the container above decides
-                    if (lhs_done != rhs_done)
-                    {
-                        return lhs_done ? compare_result::less : compare_result::greater;
-                    }
-
-                    stack.pop_back();
-                    continue;
-                }
-
-                if (is_object)
-                {
-                    // an entry is a key and a value, and the key decides first
-                    const compare_result key_result =
-                        compare_keys(current.lhs_object_it->first, current.rhs_object_it->first,
-                                     std::integral_constant<bool, Ordered> {});
-
-                    if (key_result != compare_result::equal)
-                    {
-                        return key_result;
-                    }
-
-                    left = &(current.lhs_object_it->second);
-                    right = &(current.rhs_object_it->second);
-                    ++current.lhs_object_it;
-                    ++current.rhs_object_it;
-                }
-                else
-                {
-                    left = &(*current.lhs_array_it);
-                    right = &(*current.rhs_array_it);
-                    ++current.lhs_array_it;
-                    ++current.rhs_array_it;
-                }
-
-                break;
-            }
-        }
-    }
-
-
   public:
     //////////////////////////
     // JSON parser callback //
@@ -1818,15 +1201,60 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         // check of passed value is valid
         other.assert_invariant();
 
-        if (m_data.m_type == value_t::object || m_data.m_type == value_t::array)
+        switch (m_data.m_type)
         {
-            // copying the container directly would call this constructor again
-            // for every element, once per nesting level
-            copy_structured(other);
-        }
-        else
-        {
-            copy_leaf_value(other, *this);
+            case value_t::object:
+            {
+                m_data.m_value = *other.m_data.m_value.object;
+                break;
+            }
+
+            case value_t::array:
+            {
+                m_data.m_value = *other.m_data.m_value.array;
+                break;
+            }
+
+            case value_t::string:
+            {
+                m_data.m_value = *other.m_data.m_value.string;
+                break;
+            }
+
+            case value_t::boolean:
+            {
+                m_data.m_value = other.m_data.m_value.boolean;
+                break;
+            }
+
+            case value_t::number_integer:
+            {
+                m_data.m_value = other.m_data.m_value.number_integer;
+                break;
+            }
+
+            case value_t::number_unsigned:
+            {
+                m_data.m_value = other.m_data.m_value.number_unsigned;
+                break;
+            }
+
+            case value_t::number_float:
+            {
+                m_data.m_value = other.m_data.m_value.number_float;
+                break;
+            }
+
+            case value_t::binary:
+            {
+                m_data.m_value = *other.m_data.m_value.binary;
+                break;
+            }
+
+            case value_t::null:
+            case value_t::discarded:
+            default:
+                break;
         }
 
         set_parents();
@@ -4258,7 +3686,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
     // because any negative signed value is smaller than any unsigned value.
     // Otherwise, the non-negative signed value is cast to unsigned before the
     // comparison to avoid wraparound.
-#define JSON_IMPLEMENT_OPERATOR(op, null_result, unordered_result, default_result, deep_result, may_descend) \
+#define JSON_IMPLEMENT_OPERATOR(op, null_result, unordered_result, default_result)                       \
     const auto lhs_type = lhs.type();                                                                    \
     const auto rhs_type = rhs.type();                                                                    \
     \
@@ -4267,25 +3695,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         switch (lhs_type)                                                                                \
         {                                                                                                \
             case value_t::array:                                                                         \
-            {                                                                                            \
-                if (JSON_HEDLEY_UNLIKELY(nesting_depth_exhausted(may_descend)))                        \
-                {                                                                                        \
-                    return (deep_result);                                                                \
-                }                                                                                        \
-                const nesting_depth_guard guard;                                                         \
                 return (*lhs.m_data.m_value.array) op (*rhs.m_data.m_value.array);                                     \
-            }                                                                                            \
-            \
+                \
             case value_t::object:                                                                        \
-            {                                                                                            \
-                if (JSON_HEDLEY_UNLIKELY(nesting_depth_exhausted(may_descend)))                        \
-                {                                                                                        \
-                    return (deep_result);                                                                \
-                }                                                                                        \
-                const nesting_depth_guard guard;                                                         \
                 return (*lhs.m_data.m_value.object) op (*rhs.m_data.m_value.object);                                   \
-            }                                                                                            \
-            \
+                \
             case value_t::null:                                                                          \
                 return (null_result);                                                                    \
                 \
@@ -4385,8 +3799,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         JSON_HEDLEY_PRAGMA(GCC diagnostic ignored "-Wfloat-equal")
 #endif
         const_reference lhs = *this;
-        JSON_IMPLEMENT_OPERATOR( ==, true, false, false,
-                                 compare_iteratively<false>(lhs, rhs, false) == compare_result::equal, true)
+        JSON_IMPLEMENT_OPERATOR( ==, true, false, false)
 #ifdef __GNUC__
         JSON_HEDLEY_DIAGNOSTIC_POP
 #endif
@@ -4411,8 +3824,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         JSON_IMPLEMENT_OPERATOR(<=>, // *NOPAD*
                                 std::partial_ordering::equivalent,
                                 std::partial_ordering::unordered,
-                                lhs_type <=> rhs_type, // *NOPAD*
-                                to_partial_ordering(compare_iteratively<true>(lhs, rhs, false)), true)
+                                lhs_type <=> rhs_type) // *NOPAD*
     }
 
     /// @brief comparison: 3-way
@@ -4479,8 +3891,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         JSON_HEDLEY_DIAGNOSTIC_PUSH
         JSON_HEDLEY_PRAGMA(GCC diagnostic ignored "-Wfloat-equal")
 #endif
-        JSON_IMPLEMENT_OPERATOR( ==, true, false, false,
-                                 compare_iteratively<false>(lhs, rhs, false) == compare_result::equal, true)
+        JSON_IMPLEMENT_OPERATOR( ==, true, false, false)
 #ifdef __GNUC__
         JSON_HEDLEY_DIAGNOSTIC_POP
 #endif
@@ -4536,8 +3947,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         // default_result is used if we cannot compare values. In that case,
         // we compare types. Note we have to call the operator explicitly,
         // because MSVC has problems otherwise.
-        JSON_IMPLEMENT_OPERATOR( <, false, false, operator<(lhs_type, rhs_type),
-                                 compare_iteratively<true>(lhs, rhs, true) == compare_result::less, false)
+        JSON_IMPLEMENT_OPERATOR( <, false, false, operator<(lhs_type, rhs_type))
     }
 
     /// @brief comparison: less than
@@ -5091,8 +4501,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         basic_json result;
         auto ia = detail::input_adapter(std::forward<InputType>(i));
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
-        const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler)) // cppcheck-suppress[accessMoved]
+        {
+            result = value_t::discarded;
+        }
+        return result;
     }
 
     /// @brief create a JSON value from an input in CBOR format (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -5108,8 +4521,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         basic_json result;
         auto ia = detail::input_adapter(std::move(first), std::move(last));
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
-        const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler)) // cppcheck-suppress[accessMoved]
+        {
+            result = value_t::discarded;
+        }
+        return result;
     }
 
     template<typename T>
@@ -5134,8 +4550,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         auto ia = i.get();
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
         // NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg)
-        const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler)) // cppcheck-suppress[accessMoved]
+        {
+            result = value_t::discarded;
+        }
+        return result;
     }
 
     /// @brief create a JSON value from an input in MessagePack format
@@ -5149,8 +4568,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         basic_json result;
         auto ia = detail::input_adapter(std::forward<InputType>(i));
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
-        const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict)) // cppcheck-suppress[accessMoved]
+        {
+            result = value_t::discarded;
+        }
+        return result;
     }
 
     /// @brief create a JSON value from an input in MessagePack format (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -5165,8 +4587,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         basic_json result;
         auto ia = detail::input_adapter(std::move(first), std::move(last));
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
-        const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict)) // cppcheck-suppress[accessMoved]
+        {
+            result = value_t::discarded;
+        }
+        return result;
     }
 
     template<typename T>
@@ -5189,8 +4614,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         auto ia = i.get();
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
         // NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg)
-        const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict)) // cppcheck-suppress[accessMoved]
+        {
+            result = value_t::discarded;
+        }
+        return result;
     }
 
     /// @brief create a JSON value from an input in UBJSON format
@@ -5204,8 +4632,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         basic_json result;
         auto ia = detail::input_adapter(std::forward<InputType>(i));
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
-        const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict)) // cppcheck-suppress[accessMoved]
+        {
+            result = value_t::discarded;
+        }
+        return result;
     }
 
     /// @brief create a JSON value from an input in UBJSON format (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -5220,8 +4651,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         basic_json result;
         auto ia = detail::input_adapter(std::move(first), std::move(last));
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
-        const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict)) // cppcheck-suppress[accessMoved]
+        {
+            result = value_t::discarded;
+        }
+        return result;
     }
 
     template<typename T>
@@ -5244,8 +4678,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         auto ia = i.get();
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
         // NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg)
-        const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict)) // cppcheck-suppress[accessMoved]
+        {
+            result = value_t::discarded;
+        }
+        return result;
     }
 
     /// @brief create a JSON value from an input in BJData format
@@ -5259,8 +4696,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         basic_json result;
         auto ia = detail::input_adapter(std::forward<InputType>(i));
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
-        const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::bjdata).sax_parse(input_format_t::bjdata, &sdp, strict); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::bjdata).sax_parse(input_format_t::bjdata, &sdp, strict)) // cppcheck-suppress[accessMoved]
+        {
+            result = value_t::discarded;
+        }
+        return result;
     }
 
     /// @brief create a JSON value from an input in BJData format (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -5275,8 +4715,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         basic_json result;
         auto ia = detail::input_adapter(std::move(first), std::move(last));
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
-        const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::bjdata).sax_parse(input_format_t::bjdata, &sdp, strict); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::bjdata).sax_parse(input_format_t::bjdata, &sdp, strict)) // cppcheck-suppress[accessMoved]
+        {
+            result = value_t::discarded;
+        }
+        return result;
     }
 
     /// @brief create a JSON value from an input in BSON format
@@ -5290,8 +4733,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         basic_json result;
         auto ia = detail::input_adapter(std::forward<InputType>(i));
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
-        const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict)) // cppcheck-suppress[accessMoved]
+        {
+            result = value_t::discarded;
+        }
+        return result;
     }
 
     /// @brief create a JSON value from an input in BSON format (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -5306,8 +4752,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         basic_json result;
         auto ia = detail::input_adapter(std::move(first), std::move(last));
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
-        const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict)) // cppcheck-suppress[accessMoved]
+        {
+            result = value_t::discarded;
+        }
+        return result;
     }
 
     template<typename T>
@@ -5330,8 +4779,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         auto ia = i.get();
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
         // NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg)
-        const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict)) // cppcheck-suppress[accessMoved]
+        {
+            result = value_t::discarded;
+        }
+        return result;
     }
     /// @}
 

tests/CMakeLists.txt

@@ -72,12 +72,7 @@ target_compile_options(test_main PUBLIC
     #                              is annotated JSON_HEDLEY_NO_RETURN (it always throws), which
     #                              makes MSVC flag the code following its call in binary_reader.hpp
     #                              as unreachable for that instantiation, in both Debug and Release
-    #       Disable warning C4503: decorated name length exceeded, name was truncated; the deep
-    #                              copy support added for #5387 pushes the mangled name of
-    #                              std::allocator_traits<...>::construct for the custom-base-class
-    #                              test's map type past VS2015's limit. The name is only used for
-    #                              debug info, so truncation does not affect the build.
-    $<$<CXX_COMPILER_ID:MSVC>:/W4;/wd4566;/wd4996;/wd4702;/wd4503>
+    $<$<CXX_COMPILER_ID:MSVC>:/W4;/wd4566;/wd4996;/wd4702>
     # https://github.com/nlohmann/json/issues/1114
     $<$<CXX_COMPILER_ID:MSVC>:/bigobj> $<$<BOOL:${MINGW}>:-Wa,-mbig-obj>
 

tests/src/unit-allocator.cpp

@@ -216,57 +216,6 @@ TEST_CASE("controlled bad_alloc")
             CHECK_THROWS_AS(my_json(s), std::bad_alloc&);
             next_construct_fails = false;
         }
-
-        SECTION("basic_json(const basic_json&) of a deeply nested value (#5387)")
-        {
-            // Copying a value nested deeper than the descent bound builds the
-            // copy from the top down: every value whose own copy has not been
-            // made yet stays a null value until it is. Failing an allocation
-            // part-way through is what proves such a half-built copy can still
-            // be destroyed.
-            //
-            // Which path the failure lands in depends on the build: the first
-            // allocation of a copy belongs to the outermost level, so here it
-            // is the descending one. Built with JSON_NO_THREAD_LOCAL - as the
-            // ci_test_no_thread_local target builds the whole suite - no
-            // descent is made at all and the very same failure lands in the
-            // iterative path instead, part-way through its worklist.
-            const auto check_deep_copy = [](bool objects)
-            {
-                CAPTURE(objects);
-
-                next_construct_fails = false;
-
-                // deeper than the 128 levels the copy constructor descends into
-                const std::size_t depth = 300;
-
-                my_json j = 1;
-                for (std::size_t i = 0; i < depth; ++i)
-                {
-                    if (objects)
-                    {
-                        my_json wrapper = my_json::object();
-                        wrapper["a"] = std::move(j);
-                        j = std::move(wrapper);
-                    }
-                    else
-                    {
-                        j = my_json::array({std::move(j)});
-                    }
-                }
-
-                // NOLINTNEXTLINE(performance-unnecessary-copy-initialization): the copy is what is tested
-                CHECK_NOTHROW(my_json(j));
-
-                next_construct_fails = true;
-                // NOLINTNEXTLINE(performance-unnecessary-copy-initialization): the copy is what is tested
-                CHECK_THROWS_AS(my_json(j), std::bad_alloc&);
-                next_construct_fails = false;
-            };
-
-            check_deep_copy(false);
-            check_deep_copy(true);
-        }
     }
 }
 

tests/src/unit-bjdata.cpp

@@ -3288,8 +3288,10 @@ TEST_CASE("BJData")
             CHECK_THROWS_WITH_AS(_ = json::from_bjdata(vR1), "[json.exception.parse_error.113] parse error at byte 6: syntax error while parsing BJData size: ndarray dimensional vector is not allowed", json::parse_error&);
             CHECK(json::from_bjdata(vR1, true, false).is_discarded());
 
+            // a dimension vector that opens another one is rejected where the
+            // nested '[' is read, rather than after it has been descended into
             std::vector<uint8_t> const vR2 = {'[', '$', 'i', '#', '[', '#', '[', 'i', 1, ']', ']', 1};
-            CHECK_THROWS_WITH_AS(_ = json::from_bjdata(vR2), "[json.exception.parse_error.113] parse error at byte 11: syntax error while parsing BJData size: expected length type specification (U, i, u, I, m, l, M, L) after '#'; last byte: 0x5D", json::parse_error&);
+            CHECK_THROWS_WITH_AS(_ = json::from_bjdata(vR2), "[json.exception.parse_error.113] parse error at byte 7: syntax error while parsing BJData size: ndarray dimensional vector is not allowed", json::parse_error&);
             CHECK(json::from_bjdata(vR2, true, false).is_discarded());
 
             std::vector<uint8_t> const vR3 = {'[', '#', '[', 'i', '2', 'i', 2, ']'};
@@ -3297,7 +3299,7 @@ TEST_CASE("BJData")
             CHECK(json::from_bjdata(vR3, true, false).is_discarded());
 
             std::vector<uint8_t> const vR4 = {'[', '$', 'i', '#', '[', '$', 'i', '#', '[', 'i', 1, ']', 1};
-            CHECK_THROWS_WITH_AS(_ = json::from_bjdata(vR4), "[json.exception.parse_error.110] parse error at byte 14: syntax error while parsing BJData number: unexpected end of input", json::parse_error&);
+            CHECK_THROWS_WITH_AS(_ = json::from_bjdata(vR4), "[json.exception.parse_error.113] parse error at byte 9: syntax error while parsing BJData size: ndarray dimensional vector is not allowed", json::parse_error&);
             CHECK(json::from_bjdata(vR4, true, false).is_discarded());
 
             std::vector<uint8_t> const vR5 = {'[', '$', 'i', '#', '[', '[', '[', ']', ']', ']'};
@@ -3305,12 +3307,25 @@ TEST_CASE("BJData")
             CHECK(json::from_bjdata(vR5, true, false).is_discarded());
 
             std::vector<uint8_t> const vR6 = {'[', '$', 'i', '#', '[', '$', 'i', '#', '[', 'i', '2', 'i', 2, ']'};
-            CHECK_THROWS_WITH_AS(_ = json::from_bjdata(vR6), "[json.exception.parse_error.112] parse error at byte 14: syntax error while parsing BJData size: ndarray can not be recursive", json::parse_error&);
+            CHECK_THROWS_WITH_AS(_ = json::from_bjdata(vR6), "[json.exception.parse_error.113] parse error at byte 9: syntax error while parsing BJData size: ndarray dimensional vector is not allowed", json::parse_error&);
             CHECK(json::from_bjdata(vR6, true, false).is_discarded());
 
             std::vector<uint8_t> const vH = {'[', 'H', '[', '#', '[', '$', 'i', '#', '[', 'i', '2', 'i', 2, ']'};
             CHECK_THROWS_WITH_AS(_ = json::from_bjdata(vH), "[json.exception.parse_error.113] parse error at byte 3: syntax error while parsing BJData size: ndarray dimensional vector is not allowed", json::parse_error&);
             CHECK(json::from_bjdata(vH, true, false).is_discarded());
+
+            // Every "#[" of this chain used to open another dimension vector
+            // and cost several stack frames before anything was rejected, so a
+            // long enough chain crashed the process (see #5104). The nested
+            // vector is refused where it is read, so the length is irrelevant.
+            std::vector<uint8_t> vRdeep = {'['};
+            for (std::size_t i = 0; i < 100000; ++i)
+            {
+                vRdeep.push_back('#');
+                vRdeep.push_back('[');
+            }
+            CHECK_THROWS_WITH_AS(_ = json::from_bjdata(vRdeep), "[json.exception.parse_error.113] parse error at byte 5: syntax error while parsing BJData size: ndarray dimensional vector is not allowed", json::parse_error&);
+            CHECK(json::from_bjdata(vRdeep, true, false).is_discarded());
         }
 
         SECTION("objects")

tests/src/unit-cbor.cpp

@@ -2035,6 +2035,58 @@ TEST_CASE("CBOR definite length equal to the indefinite-length sentinel")
     }
 }
 
+TEST_CASE("CBOR indefinite-length strings do not recurse per chunk")
+{
+    // Reading an indefinite-length string or byte array used to call itself
+    // once per chunk, so a payload of repeated 0x7F (or 0x5F) bytes exhausted
+    // the call stack before any of the input was rejected. The open levels are
+    // counted now, and the levels below prove the reader still reads the same
+    // values and reports the same errors at the same byte offsets.
+    json _;
+
+    SECTION("many open levels are reported, not crashed on")
+    {
+        const std::vector<uint8_t> input(200000, 0x7F);
+        CHECK_THROWS_WITH_AS(_ = json::from_cbor(input), "[json.exception.parse_error.110] parse error at byte 200001: syntax error while parsing CBOR string: unexpected end of input", json::parse_error&);
+        CHECK(json::from_cbor(input, true, false).is_discarded());
+    }
+
+    SECTION("many open levels are reported, not crashed on (binary)")
+    {
+        const std::vector<uint8_t> input(200000, 0x5F);
+        CHECK_THROWS_WITH_AS(_ = json::from_cbor(input), "[json.exception.parse_error.110] parse error at byte 200001: syntax error while parsing CBOR binary: unexpected end of input", json::parse_error&);
+        CHECK(json::from_cbor(input, true, false).is_discarded());
+    }
+
+    SECTION("chunks are still concatenated")
+    {
+        CHECK(json::from_cbor(std::vector<uint8_t>({0x7F, 0xFF})) == json(""));
+        CHECK(json::from_cbor(std::vector<uint8_t>({0x7F, 0x61, 0x61, 0xFF})) == json("a"));
+        // nested indefinite-length strings are concatenated across levels
+        CHECK(json::from_cbor(std::vector<uint8_t>({0x7F, 0x7F, 0x61, 0x61, 0xFF, 0x61, 0x62, 0xFF})) == json("ab"));
+        CHECK(json::from_cbor(std::vector<uint8_t>({0x7F, 0x7F, 0x7F, 0x61, 0x7A, 0xFF, 0xFF, 0xFF})) == json("z"));
+        CHECK(json::from_cbor(std::vector<uint8_t>({0xA1, 0x7F, 0x61, 0x61, 0xFF, 0x01})) == json({{"a", 1}}));
+    }
+
+    SECTION("chunks are still concatenated (binary)")
+    {
+        CHECK(json::from_cbor(std::vector<uint8_t>({0x5F, 0x41, 0x61, 0xFF})) == json::binary({0x61}));
+        CHECK(json::from_cbor(std::vector<uint8_t>({0x5F, 0x5F, 0x41, 0x61, 0xFF, 0x41, 0x62, 0xFF})) == json::binary({0x61, 0x62}));
+    }
+
+    SECTION("a chunk that is not a string is still rejected")
+    {
+        CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0x7F, 0x7F, 0x00})), "[json.exception.parse_error.113] parse error at byte 3: syntax error while parsing CBOR string: expected length specification (0x60-0x7B) or indefinite string type (0x7F); last byte: 0x00", json::parse_error&);
+        CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0x5F, 0x5F, 0x00})), "[json.exception.parse_error.113] parse error at byte 3: syntax error while parsing CBOR binary: expected length specification (0x40-0x5B) or indefinite binary array type (0x5F); last byte: 0x00", json::parse_error&);
+    }
+
+    SECTION("a break marker outside an indefinite-length string is not a string")
+    {
+        // 0xFF only closes a string that was opened; on its own it is not one
+        CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0xA1, 0xFF, 0x01})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR string: expected length specification (0x60-0x7B) or indefinite string type (0x7F); last byte: 0xFF", json::parse_error&);
+    }
+}
+
 TEST_CASE("CBOR roundtrips" * doctest::skip())
 {
     SECTION("input from flynn")

tests/src/unit-diagnostic-positions.cpp

@@ -68,72 +68,6 @@ TEST_CASE("Better diagnostics with positions")
         CHECK(j.end_pos() == root.size());
     }
 
-    SECTION("copying keeps the positions of nested values (#5387)")
-    {
-        // Values nested deeper than the copy constructor's descent bound are
-        // copied without the call stack, on a path that has to carry the
-        // positions over itself; shallower ones copy their containers, which
-        // bring the positions along. Both sides of the bound are checked here.
-        const auto check_copy = [](std::size_t depth, bool objects)
-        {
-            CAPTURE(depth)
-            CAPTURE(objects)
-
-            const std::string opening = objects ? R"({"a":)" : "[";
-            const std::string closing = objects ? "}" : "]";
-
-            std::string text;
-            for (std::size_t i = 0; i < depth; ++i)
-            {
-                text += opening;
-            }
-            text += "12";
-            for (std::size_t i = 0; i < depth; ++i)
-            {
-                text += closing;
-            }
-
-            const json original = json::parse(text);
-            const json copy(original); // NOLINT(performance-unnecessary-copy-initialization)
-
-            const json* o = &original;
-            const json* c = &copy;
-            for (std::size_t level = 0; level <= depth; ++level)
-            {
-                CAPTURE(level)
-                REQUIRE(c->start_pos() == o->start_pos());
-                REQUIRE(c->end_pos() == o->end_pos());
-
-                if (level < depth)
-                {
-                    o = objects ? &o->at("a") : &o->at(0);
-                    c = objects ? &c->at("a") : &c->at(0);
-                }
-            }
-        };
-
-        const auto check_arrays = [&check_copy](std::size_t depth)
-        {
-            check_copy(depth, false);
-        };
-        const auto check_objects = [&check_copy](std::size_t depth)
-        {
-            check_copy(depth, true);
-        };
-
-        check_arrays(1);
-        check_arrays(127);
-        check_arrays(128);
-        check_arrays(129);
-        check_arrays(300);
-
-        check_objects(1);
-        check_objects(127);
-        check_objects(128);
-        check_objects(129);
-        check_objects(300);
-    }
-
     SECTION("JSON patch add to primitive parent (#4292)")
     {
         // the JSON Patch "add" target /foo/bar/baz has a string parent

tests/src/unit-diagnostics.cpp

@@ -273,62 +273,5 @@ TEST_CASE("Regression tests for extended diagnostics")
         CHECK(j1["numbers"]["two"] == 2);
         CHECK(j1["string"] == "t");
     }
-
-    SECTION("Regression test for issue #5387 - copying keeps the parents of nested values")
-    {
-        // A value nested deeper than the copy constructor's descent bound is
-        // copied without the call stack. Every container that path creates has
-        // to have the parents of its children set, or the JSON Pointer in the
-        // diagnostic is cut short.
-        const std::size_t depth = 300;
-
-        SECTION("objects")
-        {
-            json j = "not a number";
-            std::string pointer;
-            for (std::size_t i = 0; i < depth; ++i)
-            {
-                j = json{{"a", j}};
-                pointer += "/a";
-            }
-
-            json const copy(j); // NOLINT(performance-unnecessary-copy-initialization)
-
-            const json* inner = &copy;
-            for (std::size_t i = 0; i < depth; ++i)
-            {
-                inner = &inner->at("a");
-            }
-
-            std::string const expected = "[json.exception.type_error.302] (" + pointer + ") type must be number, but is string";
-            int i = 0;
-            CHECK_THROWS_WITH_AS(i = inner->get<int>(), expected.c_str(), json::type_error);
-            CHECK(i == 0);
-        }
-
-        SECTION("arrays")
-        {
-            json j = "not a number";
-            std::string pointer;
-            for (std::size_t i = 0; i < depth; ++i)
-            {
-                j = json::array({j});
-                pointer += "/0";
-            }
-
-            json const copy(j); // NOLINT(performance-unnecessary-copy-initialization)
-
-            const json* inner = &copy;
-            for (std::size_t i = 0; i < depth; ++i)
-            {
-                inner = &inner->at(0);
-            }
-
-            std::string const expected = "[json.exception.type_error.302] (" + pointer + ") type must be number, but is string";
-            int i = 0;
-            CHECK_THROWS_WITH_AS(i = inner->get<int>(), expected.c_str(), json::type_error);
-            CHECK(i == 0);
-        }
-    }
 }
 

tests/src/unit-large_json.cpp

@@ -12,7 +12,6 @@
 using nlohmann::json;
 
 #include <algorithm>
-#include <string>
 
 TEST_CASE("tests on very large JSONs")
 {
@@ -28,201 +27,3 @@ TEST_CASE("tests on very large JSONs")
     }
 }
 
-namespace
-{
-
-// Descend a chain of single-element containers and return the value at its end,
-// reporting the number of levels traversed in @a depth.
-//
-// The values in the test case below are nested far deeper than the call stack
-// can follow, so they must not be inspected with operator== or dump(): both are
-// still recursive and would overflow the stack themselves.
-const json* innermost_value(const json& j, std::size_t& depth)
-{
-    const json* current = &j;
-    depth = 0;
-
-    while ((current->is_array() || current->is_object()) && !current->empty())
-    {
-        current = current->is_array()
-                  ? &current->front()
-                  : &current->begin().value();
-        ++depth;
-    }
-
-    return current;
-}
-
-} // namespace
-
-TEST_CASE("tests on deeply nested JSONs")
-{
-    // deep enough to exhaust the call stack, but small enough to stay cheap:
-    // parsing is iterative, so building the values below costs little
-    const std::size_t depth = 100000;
-
-    SECTION("issue #5387 - stack overflow in the copy constructor")
-    {
-        SECTION("array")
-        {
-            const json j = json::parse(std::string(depth, '[') + '0' + std::string(depth, ']'));
-
-            const json copy(j); // NOLINT(performance-unnecessary-copy-initialization): the copy is what is tested
-
-            std::size_t copy_depth = 0;
-            CHECK(*innermost_value(copy, copy_depth) == 0);
-            CHECK(copy_depth == depth);
-        }
-
-        SECTION("object")
-        {
-            std::string s;
-            s.reserve((6 * depth) + 1);
-            for (std::size_t i = 0; i < depth; ++i)
-            {
-                s += "{\"a\":";
-            }
-            s += '1';
-            s.append(depth, '}');
-
-            const json j = json::parse(s);
-
-            const json copy(j); // NOLINT(performance-unnecessary-copy-initialization): the copy is what is tested
-
-            std::size_t copy_depth = 0;
-            CHECK(*innermost_value(copy, copy_depth) == 1);
-            CHECK(copy_depth == depth);
-        }
-
-        SECTION("copy assignment")
-        {
-            // operator=(basic_json) takes its argument by value, so the deep
-            // copy happens in the copy constructor
-            const json j = json::parse(std::string(depth, '[') + '0' + std::string(depth, ']'));
-
-            json target;
-            target = j;
-
-            std::size_t target_depth = 0;
-            CHECK(*innermost_value(target, target_depth) == 0);
-            CHECK(target_depth == depth);
-        }
-
-        SECTION("depths around the bound of the recursive descent")
-        {
-            // The copy constructor descends into a bounded number of levels and
-            // completes whatever is below that without the call stack. Cover
-            // every depth around that bound, so that the two ways of copying
-            // are known to meet cleanly - wherever the bound is set.
-            for (std::size_t d = 1; d <= 300; ++d)
-            {
-                CAPTURE(d);
-
-                const json array = json::parse(std::string(d, '[') + '0' + std::string(d, ']'));
-                const json array_copy(array); // NOLINT(performance-unnecessary-copy-initialization): the copy is what is tested
-                std::size_t array_depth = 0;
-                CHECK(*innermost_value(array_copy, array_depth) == 0);
-                CHECK(array_depth == d);
-
-                std::string object_text;
-                for (std::size_t i = 0; i < d; ++i)
-                {
-                    object_text += "{\"a\":";
-                }
-                object_text += '1';
-                object_text.append(d, '}');
-
-                const json object = json::parse(object_text);
-                const json object_copy(object); // NOLINT(performance-unnecessary-copy-initialization): the copy is what is tested
-                std::size_t object_depth = 0;
-                CHECK(*innermost_value(object_copy, object_depth) == 1);
-                CHECK(object_depth == d);
-            }
-        }
-
-        SECTION("a value that is deep in one place only")
-        {
-            json j = json::object();
-            j["shallow"] = 1;
-            j["deep"] = json::parse(std::string(depth, '[') + '0' + std::string(depth, ']'));
-            j["also_shallow"] = json::array({1, 2, 3});
-
-            const json copy(j);
-
-            CHECK(copy["shallow"] == 1);
-            CHECK(copy["also_shallow"] == json::array({1, 2, 3}));
-
-            std::size_t deep_depth = 0;
-            CHECK(*innermost_value(copy["deep"], deep_depth) == 0);
-            CHECK(deep_depth == depth);
-        }
-
-        SECTION("comparing")
-        {
-            // Comparing used to descend once per level, and an ordered
-            // comparison used to compare every pair of elements twice, once in
-            // each direction, which took exponentially long in the nesting
-            // depth. Both are gone: these finish in milliseconds, where the
-            // second used to take longer than anyone would wait even for a
-            // value nested only a few dozen levels deep.
-            const std::string text = std::string(depth, '[') + '0' + std::string(depth, ']');
-            const json j = json::parse(text);
-            const json same = json::parse(text);
-            const json larger = json::parse(std::string(depth, '[') + '1' + std::string(depth, ']'));
-
-            CHECK(j == same);
-            CHECK_FALSE(j == larger);
-            CHECK(j != larger);
-
-            CHECK(j < larger);
-            CHECK_FALSE(larger < j);
-            CHECK(larger > j);
-            CHECK(j <= same);
-            CHECK(j >= same);
-
-            // a value that ends earlier is the smaller one
-            const json shorter = json::parse(std::string(depth - 1, '[') + '0' + std::string(depth - 1, ']'));
-            CHECK_FALSE(j == shorter);
-        }
-
-        SECTION("comparing objects")
-        {
-            std::string text;
-            text.reserve((6 * depth) + 1);
-            for (std::size_t i = 0; i < depth; ++i)
-            {
-                text += "{\"a\":";
-            }
-            text += '1';
-            text.append(depth, '}');
-
-            const json j = json::parse(text);
-            const json same = json::parse(text);
-
-            CHECK(j == same);
-            CHECK_FALSE(j != same);
-            CHECK(j <= same);
-            CHECK(j >= same);
-        }
-
-        SECTION("the copy is independent of the original")
-        {
-            const json j = json::parse(std::string(depth, '[') + '0' + std::string(depth, ']'));
-
-            json copy(j);
-
-            // reach the innermost value without recursing and replace it
-            json* current = &copy;
-            while (current->is_array() && !current->empty())
-            {
-                current = &current->front();
-            }
-            *current = 42;
-
-            std::size_t unused = 0;
-            CHECK(*innermost_value(copy, unused) == 42);
-            CHECK(*innermost_value(j, unused) == 0);
-        }
-    }
-}
-

tests/src/unit-msgpack.cpp

@@ -1598,6 +1598,67 @@ TEST_CASE("MessagePack")
 }
 
 // use this testcase outside [hide] to run it with Valgrind
+TEST_CASE("MessagePack nesting does not consume the call stack")
+{
+    // Reading a container used to call back into the value reader once per
+    // element, so the native call stack grew with the nesting depth of the
+    // input: one frame per byte for repeated 0x91 (a one-element array), which
+    // crashes the process long before the input is exhausted (#5104). The
+    // containers are kept on a heap stack now.
+    //
+    // Note that deeply nested values must not be compared, copied or dumped
+    // here: those operations are still recursive, and would reintroduce the
+    // very crash this checks for. Depth is measured by descending instead.
+
+    SECTION("an unterminated chain is reported, not crashed on")
+    {
+        json _;
+        const std::vector<uint8_t> input(300000, 0x91);
+        CHECK_THROWS_WITH_AS(_ = json::from_msgpack(input), "[json.exception.parse_error.110] parse error at byte 300001: syntax error while parsing MessagePack value: unexpected end of input", json::parse_error&);
+        CHECK(json::from_msgpack(input, true, false).is_discarded());
+    }
+
+    SECTION("a well-formed deep value is read through the SAX interface")
+    {
+        std::vector<uint8_t> input(300000, 0x91);
+        input.push_back(0x01); // innermost value
+
+        SaxCountdown accept_all(600001);
+        CHECK(json::sax_parse(input, &accept_all, json::input_format_t::msgpack));
+    }
+
+    SECTION("a well-formed deep value is read into a value")
+    {
+        const std::size_t depth = 10000;
+        std::vector<uint8_t> input(depth, 0x91);
+        input.push_back(0x01);
+
+        json j = json::from_msgpack(input);
+
+        std::size_t measured = 0;
+        const json* p = &j;
+        while (p->is_array() && !p->empty())
+        {
+            p = &p->front();
+            ++measured;
+        }
+        CHECK(measured == depth);
+        CHECK(p->is_number());
+    }
+
+    SECTION("containers are still read the same way")
+    {
+        CHECK(json::from_msgpack(std::vector<uint8_t>({0x90})) == json::array());
+        CHECK(json::from_msgpack(std::vector<uint8_t>({0x80})) == json::object());
+        CHECK(json::from_msgpack(std::vector<uint8_t>({0x92, 0x90, 0x80})) == json({json::array(), json::object()}));
+        CHECK(json::from_msgpack(std::vector<uint8_t>({0x91, 0x91, 0x91, 0x90})) == json({{{json::array()}}}));
+        CHECK(json::from_msgpack(std::vector<uint8_t>({0x81, 0xA1, 'a', 0x81, 0xA1, 'b', 0x92, 0x01, 0x02})) == json({{"a", {{"b", {1, 2}}}}}));
+        // array 16 and map 32, i.e. the counted forms
+        CHECK(json::from_msgpack(std::vector<uint8_t>({0xDC, 0x00, 0x02, 0x01, 0x02})) == json({1, 2}));
+        CHECK(json::from_msgpack(std::vector<uint8_t>({0xDF, 0x00, 0x00, 0x00, 0x01, 0xA1, 'k', 0xC3})) == json({{"k", true}}));
+    }
+}
+
 TEST_CASE("single MessagePack roundtrip")
 {
     SECTION("sample.json")

tests/src/unit-ordered_json.cpp

@@ -81,37 +81,3 @@ TEST_CASE("regression test for issue #3732 - iteration_proxy_value<iter_impl<ord
     };
     static_cast<void>(fn);
 }
-
-TEST_CASE("copying an ordered_json with nested values")
-{
-    // ordered_map is backed by a vector, so copying an object that has
-    // structured values takes a different route than copying a std::map-backed
-    // one; see https://github.com/nlohmann/json/issues/5387
-    ordered_json oj;
-    oj["z"] = 1;
-    oj["a"]["y"] = 2;
-    oj["a"]["b"]["x"] = 3;
-    oj["m"] = {1, 2, {{"w", 4}}};
-
-    const ordered_json copy(oj);
-
-    SECTION("the copy is equal to the original")
-    {
-        CHECK(copy == oj);
-        CHECK(copy.dump() == oj.dump());
-    }
-
-    SECTION("the key order is preserved at every level")
-    {
-        CHECK(copy.dump() == R"({"z":1,"a":{"y":2,"b":{"x":3}},"m":[1,2,{"w":4}]})");
-    }
-
-    SECTION("the copy is independent of the original")
-    {
-        ordered_json mutated(oj);
-        mutated["a"]["b"]["x"] = 99;
-
-        CHECK(oj["a"]["b"]["x"] == 3);
-        CHECK(mutated["a"]["b"]["x"] == 99);
-    }
-}

tests/src/unit-regression2.cpp

@@ -44,6 +44,22 @@ using ordered_json = nlohmann::ordered_json;
     #elif __has_include(<experimental/optional>)
         #include <experimental/optional>
     #endif
+
+    /////////////////////////////////////////////////////////////////////
+    // for #4804
+    /////////////////////////////////////////////////////////////////////
+    using json_4804 = nlohmann::basic_json<std::map,        // ObjectType
+    std::vector,     // ArrayType
+    std::string,     // StringType
+    bool,            // BooleanType
+    std::int64_t,    // NumberIntegerType
+    std::uint64_t,   // NumberUnsignedType
+    double,          // NumberFloatType
+    std::allocator,  // AllocatorType
+    nlohmann::adl_serializer,  // JSONSerializer
+    std::vector<std::byte>,    // BinaryType
+    void                       // CustomBaseClass
+    >;
 #endif
 
 #ifdef JSON_HAS_CPP_20
@@ -159,6 +175,71 @@ struct NotSerializableData
     float myfloat;
 };
 
+/////////////////////////////////////////////////////////////////////
+// for #2574
+/////////////////////////////////////////////////////////////////////
+
+struct NonDefaultConstructible
+{
+    explicit NonDefaultConstructible(int a)
+        : x(a)
+    {}
+    int x;
+};
+
+namespace nlohmann
+{
+template<>
+struct adl_serializer<NonDefaultConstructible>
+{
+    static NonDefaultConstructible from_json(json const& j)
+    {
+        return NonDefaultConstructible(j.get<int>());
+    }
+};
+}  // namespace nlohmann
+
+/////////////////////////////////////////////////////////////////////
+// for #2824
+/////////////////////////////////////////////////////////////////////
+
+class sax_no_exception : public nlohmann::detail::json_sax_dom_parser<json, nlohmann::detail::string_input_adapter_type>
+{
+  public:
+    explicit sax_no_exception(json& j)
+        : nlohmann::detail::json_sax_dom_parser<json, nlohmann::detail::string_input_adapter_type>(j, false)
+    {}
+
+    static bool parse_error(std::size_t /*position*/, const std::string& /*last_token*/, const json::exception& ex)
+    {
+        error_string = new std::string(ex.what());  // NOLINT(cppcoreguidelines-owning-memory)
+        return false;
+    }
+
+    static std::string* error_string;
+};
+
+std::string* sax_no_exception::error_string = nullptr;
+
+/////////////////////////////////////////////////////////////////////
+// for #2982
+/////////////////////////////////////////////////////////////////////
+
+template<class T>
+class my_allocator : public std::allocator<T>
+{
+  public:
+    using std::allocator<T>::allocator;
+
+    my_allocator() = default;
+    template<class U> my_allocator(const my_allocator<U>& /*unused*/) { }
+
+    template <class U>
+    struct rebind
+    {
+        using other = my_allocator<U>;
+    };
+};
 
 TEST_CASE("regression tests 2")
 {
@@ -446,6 +527,240 @@ TEST_CASE("regression tests 2")
         CHECK(result.dump() == R"([{"op":"add","path":"/foo/-","value":"3"}])");
     }
 
+    SECTION("issue #2067 - cannot serialize binary data to text JSON")
+    {
+        const std::array<unsigned char, 23> data = {{0x81, 0xA4, 0x64, 0x61, 0x74, 0x61, 0xC4, 0x0F, 0x33, 0x30, 0x30, 0x32, 0x33, 0x34, 0x30, 0x31, 0x30, 0x37, 0x30, 0x35, 0x30, 0x31, 0x30}};
+        const json j = json::from_msgpack(data.data(), data.size());
+        // dump() is nodiscard; this only checks that dumping does not throw
+        CHECK_NOTHROW(
+            utils::ignore_return_value(
+                j.dump(4,                             // Indent
+                       ' ',                           // Indent char
+                       false,                         // Ensure ascii
+                       json::error_handler_t::strict  // Error
+                      )));
+    }
+
+    SECTION("PR #2181 - regression bug with lvalue")
+    {
+        // see https://github.com/nlohmann/json/pull/2181#issuecomment-653326060
+        const json j{{"x", "test"}};
+        const std::string defval = "default value";
+        auto val = j.value("x", defval); // NOLINT(bugprone-unused-local-non-trivial-variable)
+        auto val2 = j.value("y", defval); // NOLINT(bugprone-unused-local-non-trivial-variable)
+    }
+
+    SECTION("issue #2293 - eof doesn't cause parsing to stop")
+    {
+        const std::vector<uint8_t> data =
+        {
+            0x7B,
+            0x6F,
+            0x62,
+            0x6A,
+            0x65,
+            0x63,
+            0x74,
+            0x20,
+            0x4F,
+            0x42
+        };
+        const json result = json::from_cbor(data, true, false);
+        CHECK(result.is_discarded());
+    }
+
+    SECTION("issue #2315 - json.update and vector<pair>does not work with ordered_json")
+    {
+        nlohmann::ordered_json jsonAnimals = {{"animal", "dog"}};
+        const nlohmann::ordered_json jsonCat = {{"animal", "cat"}};
+        jsonAnimals.update(jsonCat);
+        CHECK(jsonAnimals["animal"] == "cat");
+
+        auto jsonAnimals_parsed = nlohmann::ordered_json::parse(jsonAnimals.dump());
+        CHECK(jsonAnimals == jsonAnimals_parsed);
+
+        const std::vector<std::pair<std::string, int64_t>> intData = {std::make_pair("aaaa", 11),
+                                                                      std::make_pair("bbb", 222)
+                                                                     };
+        nlohmann::ordered_json jsonObj;
+        for (const auto& data : intData)
+        {
+            jsonObj[data.first] = data.second;
+        }
+        CHECK(jsonObj["aaaa"] == 11);
+        CHECK(jsonObj["bbb"] == 222);
+    }
+
+    SECTION("issue #2330 - ignore_comment=true fails on multiple consecutive lines starting with comments")
+    {
+        const std::string ss = "//\n//\n{\n}\n";
+        const json j = json::parse(ss, nullptr, true, true);
+        CHECK(j.dump() == "{}");
+    }
+
+#ifdef JSON_HAS_CPP_20
+#ifndef _LIBCPP_VERSION // see https://github.com/nlohmann/json/issues/4490
+    // classic Intel ICC reports <span> as includable but cannot actually compile
+    // std::span/std::as_bytes usage below
+#if __has_include(<span>) && !defined(__ICC) && !defined(__INTEL_COMPILER)
+    SECTION("issue #2546 - parsing containers of std::byte")
+    {
+        const char DATA[] = R"("Hello, world!")"; // NOLINT(misc-const-correctness,cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
+        const auto s = std::as_bytes(std::span(DATA));
+        const json j = json::parse(s);
+        CHECK(j.dump() == "\"Hello, world!\"");
+    }
+#endif
+#endif
+#endif
+
+    SECTION("issue #2574 - Deserialization to std::array, std::pair, and std::tuple with non-default constructable types fails")
+    {
+        SECTION("std::array")
+        {
+            {
+                const json j = {7, 4};
+                auto arr = j.get<std::array<NonDefaultConstructible, 2>>();
+                CHECK(arr[0].x == 7);
+                CHECK(arr[1].x == 4);
+            }
+
+            {
+                const json j = 7;
+                CHECK_THROWS_AS((j.get<std::array<NonDefaultConstructible, 1>>()), json::type_error);
+            }
+        }
+
+        SECTION("std::pair")
+        {
+            {
+                const json j = {3, 8};
+                auto p = j.get<std::pair<NonDefaultConstructible, NonDefaultConstructible>>();
+                CHECK(p.first.x == 3);
+                CHECK(p.second.x == 8);
+            }
+
+            {
+                const json j = {4, 1};
+                auto p = j.get<std::pair<int, NonDefaultConstructible>>();
+                CHECK(p.first == 4);
+                CHECK(p.second.x == 1);
+            }
+
+            {
+                const json j = {6, 7};
+                auto p = j.get<std::pair<NonDefaultConstructible, int>>();
+                CHECK(p.first.x == 6);
+                CHECK(p.second == 7);
+            }
+
+            {
+                const json j = 7;
+                CHECK_THROWS_AS((j.get<std::pair<NonDefaultConstructible, int>>()), json::type_error);
+            }
+        }
+
+        SECTION("std::tuple")
+        {
+            {
+                const json j = {9};
+                auto t = j.get<std::tuple<NonDefaultConstructible>>();
+                CHECK(std::get<0>(t).x == 9);
+            }
+
+            {
+                const json j = {9, 8, 7};
+                auto t = j.get<std::tuple<NonDefaultConstructible, int, NonDefaultConstructible>>();
+                CHECK(std::get<0>(t).x == 9);
+                CHECK(std::get<1>(t) == 8);
+                CHECK(std::get<2>(t).x == 7);
+            }
+
+            {
+                const json j = 7;
+                CHECK_THROWS_AS((j.get<std::tuple<NonDefaultConstructible>>()), json::type_error);
+            }
+        }
+    }
+
+    SECTION("issue #4530 - Serialization of empty tuple")
+    {
+        const auto source_tuple = std::tuple<>();
+        const nlohmann::json j = source_tuple;
+
+        CHECK(j.get<decltype(source_tuple)>() == source_tuple);
+        CHECK("[]" == j.dump());
+    }
+
+    SECTION("issue #2865 - ASAN detects memory leaks")
+    {
+        // the code below is expected to not leak memory
+        {
+            nlohmann::json o;
+            const std::string s = "bar";
+
+            nlohmann::to_json(o["foo"], s);
+
+            nlohmann::json p = o;
+
+            // call to_json with a non-null JSON value
+            nlohmann::to_json(p["foo"], s);
+        }
+
+        {
+            nlohmann::json o;
+            const std::string s = "bar";
+
+            nlohmann::to_json(o["foo"], s);
+
+            // call to_json with a non-null JSON value
+            nlohmann::to_json(o["foo"], s);
+        }
+    }
+
+    SECTION("issue #2824 - encoding of json::exception::what()")
+    {
+        json j;
+        sax_no_exception sax(j);
+
+        CHECK(!json::sax_parse("xyz", &sax));
+        CHECK(*sax_no_exception::error_string == "[json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - invalid literal; last read: 'x'");
+        delete sax_no_exception::error_string;  // NOLINT(cppcoreguidelines-owning-memory)
+    }
+
+    SECTION("issue #2825 - Properly constrain the basic_json conversion operator")
+    {
+        static_assert(std::is_copy_assignable<nlohmann::ordered_json>::value, "ordered_json must be copy assignable");
+    }
+
+    SECTION("issue #2958 - Inserting in unordered json using a pointer retains the leading slash")
+    {
+        const std::string p = "/root";
+
+        json test1;
+        test1[json::json_pointer(p)] = json::object();
+        CHECK(test1.dump() == "{\"root\":{}}");
+
+        ordered_json test2;
+        test2[ordered_json::json_pointer(p)] = json::object();
+        CHECK(test2.dump() == "{\"root\":{}}");
+
+        // json::json_pointer and ordered_json::json_pointer are the same type; behave as above
+        ordered_json test3;
+        test3[json::json_pointer(p)] = json::object();
+        CHECK(std::is_same<json::json_pointer::string_t, ordered_json::json_pointer::string_t>::value);
+        CHECK(test3.dump() == "{\"root\":{}}");
+    }
+
+    SECTION("issue #2982 - to_{binary format} does not provide a mechanism for specifying a custom allocator for the returned type")
+    {
+        std::vector<std::uint8_t, my_allocator<std::uint8_t>> my_vector;
+        const json j = {1, 2, 3, 4};
+        json::to_cbor(j, my_vector);
+        json k = json::from_cbor(my_vector);
+        CHECK(j == k);
+    }
+
 }
 
 DOCTEST_CLANG_SUPPRESS_WARNING_POP

tests/src/unit-ubjson.cpp

@@ -2149,6 +2149,67 @@ TEST_CASE("UBJSON")
     }
 }
 
+TEST_CASE("UBJSON optimized arrays of a valueless type are bounded")
+{
+    // An element of type 'Z', 'T' or 'F' is encoded by its marker alone, so an
+    // optimized array of one of those has no payload and the declared count is
+    // the only thing deciding how much is allocated. Ten bytes used to produce
+    // billions of values (#2793); every other type costs at least one byte per
+    // element and is bounded by the end of the input.
+    json _;
+
+    SECTION("an excessive count is rejected")
+    {
+        // 'l' is a big-endian int32: 0x7FFFFFFF elements, about 34 GB of value
+        for (const auto marker :
+                {'Z', 'T', 'F'
+                })
+        {
+            const std::vector<uint8_t> input = {'[', '$', static_cast<uint8_t>(marker), '#', 'l', 0x7F, 0xFF, 0xFF, 0xFF};
+            CHECK_THROWS_WITH_AS(_ = json::from_ubjson(input), "[json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive array size", json::out_of_range&);
+            CHECK(json::from_ubjson(input, true, false).is_discarded());
+        }
+    }
+
+    SECTION("ordinary counts are unaffected")
+    {
+        CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'Z', '#', 'i', 3})) == json({nullptr, nullptr, nullptr}));
+        CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'T', '#', 'i', 2})) == json({true, true}));
+        CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'F', '#', 'i', 2})) == json({false, false}));
+        // 'N' is a no-op rather than a value, and still yields an empty array
+        CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'N', '#', 'i', 2})) == json::array());
+    }
+
+    SECTION("a type with a payload is unaffected")
+    {
+        // A count past the limit is not rejected for 'U', which costs a byte
+        // per element and is bounded by the end of the input instead. The
+        // count is kept just past the limit rather than made huge, because a
+        // count that also exceeds the array's max_size() is reported as
+        // out_of_range before the input runs out, and max_size() depends on
+        // the width of std::size_t.
+        const std::vector<uint8_t> input = {'[', '$', 'U', '#', 'l', 0x00, 0x10, 0x00, 0x01};
+        CHECK_THROWS_WITH_AS(_ = json::from_ubjson(input), "[json.exception.parse_error.110] parse error at byte 10: syntax error while parsing UBJSON number: unexpected end of input", json::parse_error&);
+        CHECK(json::from_ubjson(input, true, false).is_discarded());
+    }
+
+    SECTION("the writer stays within what the reader accepts")
+    {
+        // below the limit the optimized form is used and is tiny; above it the
+        // writer falls back so that the result can still be read back
+        json const at_limit(1048576, nullptr);
+        const auto v_at_limit = json::to_ubjson(at_limit, true, true);
+        CHECK(v_at_limit.size() == 9);
+        CHECK(v_at_limit.at(1) == '$');
+        CHECK(json::from_ubjson(v_at_limit) == at_limit);
+
+        json const above_limit(1048577, nullptr);
+        const auto v_above_limit = json::to_ubjson(above_limit, true, true);
+        CHECK(v_above_limit.at(1) != '$');
+        CHECK(json::from_ubjson(v_above_limit) == above_limit);
+    }
+}
+
 TEST_CASE("Universal Binary JSON Specification Examples 1")
 {
     SECTION("Null Value")