Read UBJSON and BJData containers without recursing per nesting level

get_ubjson_array() and get_ubjson_object() read their elements by calling
back into the value reader, which called them again for a nested container,
so the native call stack grew with the nesting depth of the input. '[' alone
opens a container, so half a million of them crashes the process before the
input runs out (#5104). The optimized forms reach the same path through a
size or type annotation, and in plain UBJSON '[' and '{' are permitted as the
type of an optimized container, so "[$[#i\x01" repeated nests just as deeply
at six bytes a level.

Both readers now only open their container, and parse_ubjson_internal() loops:
it closes the containers that have ended, claims the next element of the
innermost one, reads its key when it is an object, and works out the marker
of the value to read next. That last part is where the formats differ, and
the loop follows what the four element loops used to do:

  - a sized, typed container gives its elements no marker of their own
  - a sized, untyped container reads one for each element
  - a container that ends at a marker has the byte already, from the test
    against ']' or '}'; for an object it is the first byte of the key

The ND-array wrapper and the 'B' binary shortcut stay as they are. Both read
a complete value rather than opening a container, and their elements are
always scalars: BJData does not permit '[' or '{' as an optimized type, which
is also why only plain UBJSON needed the type-marker case above.

A container of no-ops keeps its behaviour of holding no elements while still
announcing its declared size to the SAX parser, by opening it and then
setting its count to zero.

unit-ubjson and unit-bjdata pass unchanged, 1.39 million assertions between
them, and a behaviour comparison against the previous commit over every
container form -- sized, unsized, typed, untyped, empty, no-op, ND-array,
binary, and the forms nested inside one another -- gives identical values,
error codes, messages and byte offsets. 500,000 levels of each vector now
report a parse error instead of crashing, and a well-formed 100,000-level
value is read to completion.

The driver costs about 3 % on parsing 60,000 small objects and one array of a
million integers, for the reason given in the previous commit; reading the
frame once per element rather than per branch halved what it cost before.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/features/binary_formats/ubjson.md

@@ -69,6 +69,12 @@ The library uses the following mapping from JSON values types to UBJSON types ac
     Note that `use_size = true` alone may result in larger representations - the benefit of this parameter is that the
     receiving side is immediately informed on the number of elements of the container.
 
+    An array whose type marker is `Z` (null), `T` (true) or `F` (false) stores no payload at all, because the marker
+    already is the value. Its declared count is therefore the only thing that decides how much memory the receiving side
+    allocates, and a handful of bytes can describe billions of elements. `from_ubjson` rejects such an array with
+    [`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408) when the count exceeds 1,048,576, and
+    `to_ubjson` writes longer arrays of these types without the annotation, so any value it produces can be read back.
+
 !!! info "Binary values"
 
     If the JSON data contains the binary type, the value stored is a list of integers, as suggested by the UBJSON

docs/mkdocs/docs/home/exceptions.md

@@ -868,6 +868,12 @@ The size of an array or object in a [binary format](../features/binary_formats/i
 the size following `#` for [UBJSON](../features/binary_formats/ubjson.md)/[BJData](../features/binary_formats/bjdata.md),
 or the encoded length for [CBOR](../features/binary_formats/cbor.md).
 
+The exception is also thrown for a [UBJSON](../features/binary_formats/ubjson.md) array of a type that is encoded by its
+marker alone (`Z`, `T` or `F`) whose declared count exceeds 1,048,576. Such an array has no payload, so its count alone
+decides how much memory is allocated, and a handful of bytes would otherwise describe billions of values.
+[`to_ubjson`](../api/basic_json/to_ubjson.md) writes longer arrays of these types without the size and type annotation,
+so any value it produces can still be read back.
+
 !!! failure "Example messages"
 
     ```
@@ -879,6 +885,9 @@ or the encoded length for [CBOR](../features/binary_formats/cbor.md).
     ```
     [json.exception.out_of_range.408] syntax error while parsing CBOR size: excessive map size
     ```
+    ```
+    [json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive array size
+    ```
 
 ### json.exception.out_of_range.409
 

include/nlohmann/detail/input/lexer.hpp

@@ -149,11 +149,10 @@ class lexer : public lexer_base<BasicJsonType>
   public:
     using token_type = typename lexer_base<BasicJsonType>::token_type;
 
-    explicit lexer(InputAdapterType&& adapter, bool ignore_comments_ = false, bool discard_number_values_ = false) noexcept
+    explicit lexer(InputAdapterType&& adapter, bool ignore_comments_ = false) noexcept
         : ia(std::move(adapter))
         , ignore_comments(ignore_comments_)
         , decimal_point_char(static_cast<char_int_type>(get_decimal_point()))
-        , discard_number_values(discard_number_values_)
     {}
 
     // deleted because of pointer members
@@ -1280,58 +1279,6 @@ scan_number_done:
         // we are done scanning a number)
         unget();
 
-        // If the caller does not need the converted value (only whether the
-        // input is syntactically valid; see json_sax_acceptor/accept()), an
-        // unsigned/integer token can be reported without calling
-        // strtoull()/strtoll() at all, *provided* we can already tell from
-        // the digit count alone that the conversion cannot overflow 64 bits.
-        // Such tokens are always finite and are accepted unconditionally by
-        // the parser regardless of their actual value (parser::sax_parse_internal()
-        // never checks finiteness for value_unsigned/value_integer), so the
-        // classification below is all that is needed.
-        //
-        // A decimal number with up to 18 digits is always representable in
-        // both std::uint64_t and std::int64_t (18 nines is ~1e18, well below
-        // both UINT64_MAX ~1.8e19 and INT64_MAX ~9.2e18), so strtoull()/strtoll()
-        // could not have set errno to ERANGE for it. Numbers with more digits
-        // (rare in practice) fall through to the exact code below, unchanged,
-        // so their handling -- including reclassification to value_float when
-        // the value overflows 64 bits, and rejection when it is not even
-        // finite as a double -- is bit-for-bit identical to before this
-        // optimization.
-        //
-        // Note this reasons about std::uint64_t/std::int64_t, not about
-        // number_unsigned_t/number_integer_t (BasicJsonType's own, possibly
-        // narrower, template parameters -- e.g. std::uint32_t). That is fine
-        // *only* because discard_number_values is exclusively set by
-        // accept() (see json.hpp), and accept() always parses through the
-        // library's own json_sax_acceptor -- never a user-supplied SAX
-        // consumer -- whose number_unsigned()/number_integer()/number_float()
-        // callbacks unconditionally discard their argument and return true.
-        // So for every caller that can reach this branch, neither the token
-        // classification below nor the eventual (possibly narrowed, and on
-        // this fast path left stale/unset) value_unsigned/value_integer is
-        // ever consulted -- an unsigned/integer token is accepted outright,
-        // and even a >18-digit token that this fast path deliberately falls
-        // through for is, once reclassified to value_float, still finite
-        // (and thus accepted) for any digit count that fits in number_unsigned_t
-        // or number_integer_t regardless of that type's width. If this
-        // function is ever taught to run with discard_number_values true for
-        // a caller that *does* read the converted value, this reasoning (and
-        // the fast path below) would need to be revisited.
-        if (discard_number_values)
-        {
-            constexpr std::size_t safe_digit_count = 18;
-            if (number_type == token_type::value_unsigned && token_buffer.size() <= safe_digit_count)
-            {
-                return token_type::value_unsigned;
-            }
-            if (number_type == token_type::value_integer && token_buffer.size() - 1 <= safe_digit_count)
-            {
-                return token_type::value_integer;
-            }
-        }
-
         char* endptr = nullptr; // NOLINT(misc-const-correctness,cppcoreguidelines-pro-type-vararg,hicpp-vararg)
         errno = 0;
 
@@ -1446,7 +1393,8 @@ scan_number_done:
     */
     char_int_type get()
     {
-        advance_position();
+        ++position.chars_read_total;
+        ++position.chars_read_current_line;
 
         if (next_unget)
         {
@@ -1458,23 +1406,6 @@ scan_number_done:
             current = ia.get_character();
         }
 
-        return track_after_read();
-    }
-
-    /// shared head of get() / get_ignoring_pending_unget(): bump the
-    /// per-character position counters (line-count-on-'\n' bookkeeping is
-    /// handled afterwards, in track_after_read(), once `current` is known)
-    void advance_position() noexcept
-    {
-        ++position.chars_read_total;
-        ++position.chars_read_current_line;
-    }
-
-    /// shared tail of get() / get_ignoring_pending_unget(): capture the
-    /// character for error messages (if needed) and update line/column
-    /// bookkeeping for the character now in `current`
-    char_int_type track_after_read()
-    {
         // seekable adapters reconstruct the token lazily on error (see
         // get_token_string), so the eager per-character copy is skipped
         capture_char(std::integral_constant<bool, lazy_token_string> {});
@@ -1488,29 +1419,6 @@ scan_number_done:
         return current;
     }
 
-    /*!
-    @brief like get(), but for call sites that can prove no unget() is pending
-
-    get() has to check the `next_unget` flag on every call, because a
-    previous token may have ended with unget() (e.g. scan_number() always
-    ungets the character that terminated the number, so the next call to
-    scan() can see it again). skip_whitespace() reads that first,
-    possibly-ungotten character via a plain get(), but every further
-    character it reads is guaranteed to be a fresh read: nothing between
-    those calls invokes unget(). This variant skips the (otherwise always
-    false) next_unget branch for those calls; it is not a general
-    replacement for get().
-    */
-    char_int_type get_ignoring_pending_unget()
-    {
-        JSON_ASSERT(!next_unget);
-
-        advance_position();
-        current = ia.get_character();
-
-        return track_after_read();
-    }
-
     /// seekable adapter: nothing to capture, the token is rebuilt on error
     void capture_char(std::true_type /*lazy*/) const noexcept {}
 
@@ -1704,37 +1612,13 @@ scan_number_done:
         return true;
     }
 
-    /// whether `current` is one of the four JSON whitespace characters
-    bool current_is_whitespace() const noexcept
-    {
-        return current == ' ' || current == '\t' || current == '\n' || current == '\r';
-    }
-
     void skip_whitespace()
     {
-        // the first character may be a pending unget() left over from the
-        // previous token (see get_ignoring_pending_unget()); every
-        // subsequent character read by this loop is guaranteed fresh, since
-        // nothing below calls unget()
-        get();
-
-        if (!current_is_whitespace())
-        {
-            return;
-        }
-
-        // this is written as an if-guarded do-while (rather than a plain
-        // while loop) because that shape is what lets both GCC and Clang
-        // keep the input adapter's read pointer in a register across
-        // iterations; the equivalent while-loop measurably defeated that
-        // optimization in testing, turning long whitespace runs (e.g. the
-        // indentation of pretty-printed JSON) from a register-only loop
-        // into one that reloads the pointer from memory every character
         do
         {
-            get_ignoring_pending_unget();
+            get();
         }
-        while (current_is_whitespace());
+        while (current == ' ' || current == '\t' || current == '\n' || current == '\r');
     }
 
     token_type scan()
@@ -1870,13 +1754,6 @@ scan_number_done:
     const char_int_type decimal_point_char = '.';
     /// the position of the decimal point in the input
     std::size_t decimal_point_position = std::string::npos;
-
-    /// whether the caller (e.g. accept()/json_sax_acceptor) only needs the
-    /// token classification and never looks at the converted numeric value;
-    /// when set, scan_number() may skip strtoull()/strtoll() for
-    /// value_unsigned/value_integer tokens whose digit count guarantees they
-    /// fit into 64 bits (see scan_number())
-    const bool discard_number_values = false;
 };
 
 }  // namespace detail

include/nlohmann/detail/input/parser.hpp

@@ -72,10 +72,9 @@ class parser
                     parser_callback_t<BasicJsonType> cb = nullptr,
                     const bool allow_exceptions_ = true,
                     const bool ignore_comments = false,
-                    const bool ignore_trailing_commas_ = false,
-                    const bool discard_number_values_ = false)
+                    const bool ignore_trailing_commas_ = false)
         : callback(std::move(cb))
-        , m_lexer(std::move(adapter), ignore_comments, discard_number_values_)
+        , m_lexer(std::move(adapter), ignore_comments)
         , allow_exceptions(allow_exceptions_)
         , ignore_trailing_commas(ignore_trailing_commas_)
     {

include/nlohmann/detail/output/binary_writer.hpp

@@ -826,7 +826,17 @@ class binary_writer
 
                     std::vector<CharType> bjdx = {'[', '{', 'S', 'H', 'T', 'F', 'N', 'Z'}; // excluded markers in bjdata optimized type
 
-                    if (same_prefix && !(use_bjdata && std::find(bjdx.begin(), bjdx.end(), first_prefix) != bjdx.end()))
+                    // an optimized array of a valueless type carries no payload, so a
+                    // reader has nothing but the declared count to bound the allocation
+                    // by and refuses an excessive one. Write the unoptimized form for
+                    // those, at one byte per element, so the result can be read back.
+                    // Objects are not affected: every element is preceded by its key.
+                    const bool valueless_type = (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');
+                    const bool excessive_valueless = valueless_type
+                                                     && j.m_data.m_value.array->size() > detail::max_valueless_container_size;
+
+                    if (same_prefix && !excessive_valueless
+                            && !(use_bjdata && std::find(bjdx.begin(), bjdx.end(), first_prefix) != bjdx.end()))
                     {
                         prefix_required = false;
                         oa->write_character(to_char_type('$'));

include/nlohmann/json.hpp

@@ -164,12 +164,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         detail::parser_callback_t<basic_json>cb = nullptr,
         const bool allow_exceptions = true,
         const bool ignore_comments = false,
-        const bool ignore_trailing_commas = false,
-        const bool discard_number_values = false
+        const bool ignore_trailing_commas = false
     )
     {
         return ::nlohmann::detail::parser<basic_json, InputAdapterType>(std::move(adapter),
-               std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas, discard_number_values);
+               std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas);
     }
 
   private:
@@ -4134,7 +4133,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
                        const bool ignore_comments = false,
                        const bool ignore_trailing_commas = false)
     {
-        return parser(detail::input_adapter(std::forward<InputType>(i)), nullptr, false, ignore_comments, ignore_trailing_commas, true).accept(true);
+        return parser(detail::input_adapter(std::forward<InputType>(i)), nullptr, false, ignore_comments, ignore_trailing_commas).accept(true);
     }
 
     /// @brief check if the input is valid JSON (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -4145,7 +4144,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
                        const bool ignore_comments = false,
                        const bool ignore_trailing_commas = false)
     {
-        return parser(detail::input_adapter(std::move(first), std::move(last)), nullptr, false, ignore_comments, ignore_trailing_commas, true).accept(true);
+        return parser(detail::input_adapter(std::move(first), std::move(last)), nullptr, false, ignore_comments, ignore_trailing_commas).accept(true);
     }
 
     JSON_HEDLEY_WARN_UNUSED_RESULT
@@ -4154,7 +4153,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
                        const bool ignore_comments = false,
                        const bool ignore_trailing_commas = false)
     {
-        return parser(i.get(), nullptr, false, ignore_comments, ignore_trailing_commas, true).accept(true);
+        return parser(i.get(), nullptr, false, ignore_comments, ignore_trailing_commas).accept(true);
     }
 
     /// @brief generate SAX events
@@ -4475,7 +4474,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         auto ia = detail::input_adapter(std::forward<InputType>(i));
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
         const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        return res ? std::move(result) : basic_json(value_t::discarded);
     }
 
     /// @brief create a JSON value from an input in CBOR format (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -4492,7 +4491,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         auto ia = detail::input_adapter(std::move(first), std::move(last));
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
         const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        return res ? std::move(result) : basic_json(value_t::discarded);
     }
 
     template<typename T>
@@ -4518,7 +4517,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
         // NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg)
         const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        return res ? std::move(result) : basic_json(value_t::discarded);
     }
 
     /// @brief create a JSON value from an input in MessagePack format
@@ -4533,7 +4532,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         auto ia = detail::input_adapter(std::forward<InputType>(i));
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
         const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        return res ? std::move(result) : basic_json(value_t::discarded);
     }
 
     /// @brief create a JSON value from an input in MessagePack format (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -4549,7 +4548,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         auto ia = detail::input_adapter(std::move(first), std::move(last));
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
         const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        return res ? std::move(result) : basic_json(value_t::discarded);
     }
 
     template<typename T>
@@ -4573,7 +4572,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
         // NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg)
         const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        return res ? std::move(result) : basic_json(value_t::discarded);
     }
 
     /// @brief create a JSON value from an input in UBJSON format
@@ -4588,7 +4587,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         auto ia = detail::input_adapter(std::forward<InputType>(i));
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
         const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        return res ? std::move(result) : basic_json(value_t::discarded);
     }
 
     /// @brief create a JSON value from an input in UBJSON format (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -4604,7 +4603,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         auto ia = detail::input_adapter(std::move(first), std::move(last));
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
         const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        return res ? std::move(result) : basic_json(value_t::discarded);
     }
 
     template<typename T>
@@ -4628,7 +4627,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
         // NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg)
         const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        return res ? std::move(result) : basic_json(value_t::discarded);
     }
 
     /// @brief create a JSON value from an input in BJData format
@@ -4643,7 +4642,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         auto ia = detail::input_adapter(std::forward<InputType>(i));
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
         const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::bjdata).sax_parse(input_format_t::bjdata, &sdp, strict); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        return res ? std::move(result) : basic_json(value_t::discarded);
     }
 
     /// @brief create a JSON value from an input in BJData format (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -4659,7 +4658,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         auto ia = detail::input_adapter(std::move(first), std::move(last));
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
         const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::bjdata).sax_parse(input_format_t::bjdata, &sdp, strict); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        return res ? std::move(result) : basic_json(value_t::discarded);
     }
 
     /// @brief create a JSON value from an input in BSON format
@@ -4674,7 +4673,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         auto ia = detail::input_adapter(std::forward<InputType>(i));
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
         const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        return res ? std::move(result) : basic_json(value_t::discarded);
     }
 
     /// @brief create a JSON value from an input in BSON format (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -4690,7 +4689,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         auto ia = detail::input_adapter(std::move(first), std::move(last));
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
         const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        return res ? std::move(result) : basic_json(value_t::discarded);
     }
 
     template<typename T>
@@ -4714,7 +4713,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
         // NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg)
         const bool res = binary_reader<decltype(ia)>(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict); // cppcheck-suppress[accessMoved]
-        return res ? result : basic_json(value_t::discarded);
+        return res ? std::move(result) : basic_json(value_t::discarded);
     }
     /// @}
 

tests/src/unit-bjdata.cpp

@@ -3288,8 +3288,10 @@ TEST_CASE("BJData")
             CHECK_THROWS_WITH_AS(_ = json::from_bjdata(vR1), "[json.exception.parse_error.113] parse error at byte 6: syntax error while parsing BJData size: ndarray dimensional vector is not allowed", json::parse_error&);
             CHECK(json::from_bjdata(vR1, true, false).is_discarded());
 
+            // a dimension vector that opens another one is rejected where the
+            // nested '[' is read, rather than after it has been descended into
             std::vector<uint8_t> const vR2 = {'[', '$', 'i', '#', '[', '#', '[', 'i', 1, ']', ']', 1};
-            CHECK_THROWS_WITH_AS(_ = json::from_bjdata(vR2), "[json.exception.parse_error.113] parse error at byte 11: syntax error while parsing BJData size: expected length type specification (U, i, u, I, m, l, M, L) after '#'; last byte: 0x5D", json::parse_error&);
+            CHECK_THROWS_WITH_AS(_ = json::from_bjdata(vR2), "[json.exception.parse_error.113] parse error at byte 7: syntax error while parsing BJData size: ndarray dimensional vector is not allowed", json::parse_error&);
             CHECK(json::from_bjdata(vR2, true, false).is_discarded());
 
             std::vector<uint8_t> const vR3 = {'[', '#', '[', 'i', '2', 'i', 2, ']'};
@@ -3297,7 +3299,7 @@ TEST_CASE("BJData")
             CHECK(json::from_bjdata(vR3, true, false).is_discarded());
 
             std::vector<uint8_t> const vR4 = {'[', '$', 'i', '#', '[', '$', 'i', '#', '[', 'i', 1, ']', 1};
-            CHECK_THROWS_WITH_AS(_ = json::from_bjdata(vR4), "[json.exception.parse_error.110] parse error at byte 14: syntax error while parsing BJData number: unexpected end of input", json::parse_error&);
+            CHECK_THROWS_WITH_AS(_ = json::from_bjdata(vR4), "[json.exception.parse_error.113] parse error at byte 9: syntax error while parsing BJData size: ndarray dimensional vector is not allowed", json::parse_error&);
             CHECK(json::from_bjdata(vR4, true, false).is_discarded());
 
             std::vector<uint8_t> const vR5 = {'[', '$', 'i', '#', '[', '[', '[', ']', ']', ']'};
@@ -3305,12 +3307,25 @@ TEST_CASE("BJData")
             CHECK(json::from_bjdata(vR5, true, false).is_discarded());
 
             std::vector<uint8_t> const vR6 = {'[', '$', 'i', '#', '[', '$', 'i', '#', '[', 'i', '2', 'i', 2, ']'};
-            CHECK_THROWS_WITH_AS(_ = json::from_bjdata(vR6), "[json.exception.parse_error.112] parse error at byte 14: syntax error while parsing BJData size: ndarray can not be recursive", json::parse_error&);
+            CHECK_THROWS_WITH_AS(_ = json::from_bjdata(vR6), "[json.exception.parse_error.113] parse error at byte 9: syntax error while parsing BJData size: ndarray dimensional vector is not allowed", json::parse_error&);
             CHECK(json::from_bjdata(vR6, true, false).is_discarded());
 
             std::vector<uint8_t> const vH = {'[', 'H', '[', '#', '[', '$', 'i', '#', '[', 'i', '2', 'i', 2, ']'};
             CHECK_THROWS_WITH_AS(_ = json::from_bjdata(vH), "[json.exception.parse_error.113] parse error at byte 3: syntax error while parsing BJData size: ndarray dimensional vector is not allowed", json::parse_error&);
             CHECK(json::from_bjdata(vH, true, false).is_discarded());
+
+            // Every "#[" of this chain used to open another dimension vector
+            // and cost several stack frames before anything was rejected, so a
+            // long enough chain crashed the process (see #5104). The nested
+            // vector is refused where it is read, so the length is irrelevant.
+            std::vector<uint8_t> vRdeep = {'['};
+            for (std::size_t i = 0; i < 100000; ++i)
+            {
+                vRdeep.push_back('#');
+                vRdeep.push_back('[');
+            }
+            CHECK_THROWS_WITH_AS(_ = json::from_bjdata(vRdeep), "[json.exception.parse_error.113] parse error at byte 5: syntax error while parsing BJData size: ndarray dimensional vector is not allowed", json::parse_error&);
+            CHECK(json::from_bjdata(vRdeep, true, false).is_discarded());
         }
 
         SECTION("objects")

tests/src/unit-cbor.cpp

@@ -2035,6 +2035,145 @@ TEST_CASE("CBOR definite length equal to the indefinite-length sentinel")
     }
 }
 
+TEST_CASE("CBOR nesting does not consume the call stack")
+{
+    // Containers used to be read by calling back into the value reader once
+    // per element, and a tag by calling it for the tagged value, so the native
+    // call stack grew with the nesting depth of the input. Each of the three
+    // costs a single byte to encode -- 0x9F, 0x81 and 0xC2 -- so a payload of
+    // repeated bytes crashed the process (#5104). The containers are kept on a
+    // heap stack now, and a tag is read in a loop.
+    //
+    // Deeply nested values must not be compared, copied or dumped here: those
+    // operations are still recursive and would reintroduce the crash.
+    json _;
+
+    SECTION("indefinite-length containers")
+    {
+        const std::vector<uint8_t> input(500000, 0x9F);
+        CHECK_THROWS_WITH_AS(_ = json::from_cbor(input), "[json.exception.parse_error.110] parse error at byte 500001: syntax error while parsing CBOR value: unexpected end of input", json::parse_error&);
+        CHECK(json::from_cbor(input, true, false).is_discarded());
+    }
+
+    SECTION("definite-length containers")
+    {
+        const std::vector<uint8_t> input(500000, 0x81);
+        CHECK_THROWS_WITH_AS(_ = json::from_cbor(input), "[json.exception.parse_error.110] parse error at byte 500001: syntax error while parsing CBOR value: unexpected end of input", json::parse_error&);
+        CHECK(json::from_cbor(input, true, false).is_discarded());
+    }
+
+    SECTION("tags")
+    {
+        // a tag is not a value of its own, so a chain of them used to recurse
+        const std::vector<uint8_t> input(500000, 0xC2);
+        CHECK_THROWS_WITH_AS(_ = json::from_cbor(input, true, true, json::cbor_tag_handler_t::ignore), "[json.exception.parse_error.110] parse error at byte 500001: syntax error while parsing CBOR value: unexpected end of input", json::parse_error&);
+        CHECK(json::from_cbor(input, true, false, json::cbor_tag_handler_t::ignore).is_discarded());
+    }
+
+    SECTION("a well-formed deep value is read through the SAX interface")
+    {
+        std::vector<uint8_t> input(200000, 0x9F);
+        input.insert(input.end(), 200000, 0xFF);
+
+        SaxCountdown accept_all(1000000);
+        CHECK(json::sax_parse(input, &accept_all, json::input_format_t::cbor));
+    }
+
+    SECTION("a well-formed deep value is read into a value")
+    {
+        const std::size_t depth = 10000;
+        std::vector<uint8_t> input(depth, 0x81);
+        input.push_back(0x00);
+
+        json j = json::from_cbor(input);
+
+        std::size_t measured = 0;
+        const json* p = &j;
+        while (p->is_array() && !p->empty())
+        {
+            p = &p->front();
+            ++measured;
+        }
+        CHECK(measured == depth);
+        CHECK(p->is_number());
+    }
+
+    SECTION("containers are still read the same way")
+    {
+        CHECK(json::from_cbor(std::vector<uint8_t>({0x80})) == json::array());
+        CHECK(json::from_cbor(std::vector<uint8_t>({0xA0})) == json::object());
+        CHECK(json::from_cbor(std::vector<uint8_t>({0x9F, 0xFF})) == json::array());
+        CHECK(json::from_cbor(std::vector<uint8_t>({0xBF, 0xFF})) == json::object());
+        CHECK(json::from_cbor(std::vector<uint8_t>({0x9F, 0x01, 0x02, 0xFF})) == json({1, 2}));
+        CHECK(json::from_cbor(std::vector<uint8_t>({0xBF, 0x61, 'a', 0x01, 0xFF})) == json({{"a", 1}}));
+        // definite and indefinite forms nested inside each other
+        CHECK(json::from_cbor(std::vector<uint8_t>({0x9F, 0x82, 0x01, 0x02, 0xA1, 0x61, 'k', 0xBF, 0xFF, 0xFF})) == json({{1, 2}, {{"k", json::object()}}}));
+    }
+
+    SECTION("tagged values are still read the same way")
+    {
+        const auto ignore = json::cbor_tag_handler_t::ignore;
+        CHECK(json::from_cbor(std::vector<uint8_t>({0xC2, 0x01}), true, true, ignore) == json(1));
+        // a chain of tags resolves to the value that follows it
+        CHECK(json::from_cbor(std::vector<uint8_t>({0xC2, 0xC2, 0xC2, 0x01}), true, true, ignore) == json(1));
+        // a tag inside a container, and one in front of a container
+        CHECK(json::from_cbor(std::vector<uint8_t>({0x82, 0xC2, 0x01, 0x02}), true, true, ignore) == json({1, 2}));
+        CHECK(json::from_cbor(std::vector<uint8_t>({0xC2, 0x82, 0x01, 0x02}), true, true, ignore) == json({1, 2}));
+    }
+}
+
+TEST_CASE("CBOR indefinite-length strings do not recurse per chunk")
+{
+    // Reading an indefinite-length string or byte array used to call itself
+    // once per chunk, so a payload of repeated 0x7F (or 0x5F) bytes exhausted
+    // the call stack before any of the input was rejected. The open levels are
+    // counted now, and the levels below prove the reader still reads the same
+    // values and reports the same errors at the same byte offsets.
+    json _;
+
+    SECTION("many open levels are reported, not crashed on")
+    {
+        const std::vector<uint8_t> input(200000, 0x7F);
+        CHECK_THROWS_WITH_AS(_ = json::from_cbor(input), "[json.exception.parse_error.110] parse error at byte 200001: syntax error while parsing CBOR string: unexpected end of input", json::parse_error&);
+        CHECK(json::from_cbor(input, true, false).is_discarded());
+    }
+
+    SECTION("many open levels are reported, not crashed on (binary)")
+    {
+        const std::vector<uint8_t> input(200000, 0x5F);
+        CHECK_THROWS_WITH_AS(_ = json::from_cbor(input), "[json.exception.parse_error.110] parse error at byte 200001: syntax error while parsing CBOR binary: unexpected end of input", json::parse_error&);
+        CHECK(json::from_cbor(input, true, false).is_discarded());
+    }
+
+    SECTION("chunks are still concatenated")
+    {
+        CHECK(json::from_cbor(std::vector<uint8_t>({0x7F, 0xFF})) == json(""));
+        CHECK(json::from_cbor(std::vector<uint8_t>({0x7F, 0x61, 0x61, 0xFF})) == json("a"));
+        // nested indefinite-length strings are concatenated across levels
+        CHECK(json::from_cbor(std::vector<uint8_t>({0x7F, 0x7F, 0x61, 0x61, 0xFF, 0x61, 0x62, 0xFF})) == json("ab"));
+        CHECK(json::from_cbor(std::vector<uint8_t>({0x7F, 0x7F, 0x7F, 0x61, 0x7A, 0xFF, 0xFF, 0xFF})) == json("z"));
+        CHECK(json::from_cbor(std::vector<uint8_t>({0xA1, 0x7F, 0x61, 0x61, 0xFF, 0x01})) == json({{"a", 1}}));
+    }
+
+    SECTION("chunks are still concatenated (binary)")
+    {
+        CHECK(json::from_cbor(std::vector<uint8_t>({0x5F, 0x41, 0x61, 0xFF})) == json::binary({0x61}));
+        CHECK(json::from_cbor(std::vector<uint8_t>({0x5F, 0x5F, 0x41, 0x61, 0xFF, 0x41, 0x62, 0xFF})) == json::binary({0x61, 0x62}));
+    }
+
+    SECTION("a chunk that is not a string is still rejected")
+    {
+        CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0x7F, 0x7F, 0x00})), "[json.exception.parse_error.113] parse error at byte 3: syntax error while parsing CBOR string: expected length specification (0x60-0x7B) or indefinite string type (0x7F); last byte: 0x00", json::parse_error&);
+        CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0x5F, 0x5F, 0x00})), "[json.exception.parse_error.113] parse error at byte 3: syntax error while parsing CBOR binary: expected length specification (0x40-0x5B) or indefinite binary array type (0x5F); last byte: 0x00", json::parse_error&);
+    }
+
+    SECTION("a break marker outside an indefinite-length string is not a string")
+    {
+        // 0xFF only closes a string that was opened; on its own it is not one
+        CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0xA1, 0xFF, 0x01})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR string: expected length specification (0x60-0x7B) or indefinite string type (0x7F); last byte: 0xFF", json::parse_error&);
+    }
+}
+
 TEST_CASE("CBOR roundtrips" * doctest::skip())
 {
     SECTION("input from flynn")

tests/src/unit-class_parser.cpp

@@ -930,98 +930,6 @@ TEST_CASE("parser class")
                 CHECK(accept_helper("+1") == false);
                 CHECK(accept_helper("+0") == false);
             }
-
-            SECTION("issue #5411 - skip conversion when accept() does not need the numeric value")
-            {
-                // lexer::scan_number() may skip strtoull()/strtoll() for
-                // value_unsigned/value_integer tokens when the caller (e.g.
-                // json::accept()) does not need the converted value, as long
-                // as the digit count alone guarantees no 64-bit overflow (see
-                // the "safe_digit_count" fast path in scan_number()). This
-                // differential test checks that json::accept() (which enables
-                // the fast path) and json::parse() (which never does) always
-                // agree, over a corpus that exercises both the fast path
-                // (<=18 digits) and the untouched, exact fallback path (>=19
-                // digits) -- including reclassification of huge digit-only
-                // integers to a (possibly non-finite) floating-point value.
-                const std::vector<std::pair<std::string, bool>> cases =
-                {
-                    // normal small/large integers, both signs
-                    {"0", true}, {"1", true}, {"-1", true}, {"42", true}, {"-42", true},
-                    {"123456789", true}, {"-123456789", true},
-
-                    // digit-count boundary around the 18-digit safe cutoff (both signs)
-                    {std::string(17, '9'), true},
-                    {std::string(18, '9'), true},
-                    {std::string(19, '9'), true},
-                    {std::string(20, '9'), true},
-                    {"-" + std::string(17, '9'), true},
-                    {"-" + std::string(18, '9'), true},
-                    {"-" + std::string(19, '9'), true},
-                    {"-" + std::string(20, '9'), true},
-
-                    // 64-bit boundaries
-                    {"9223372036854775807", true},    // INT64_MAX
-                    {"-9223372036854775808", true},   // INT64_MIN
-                    {"18446744073709551615", true},   // UINT64_MAX
-                    {"18446744073709551616", true},   // UINT64_MAX + 1 (overflows uint64_t, finite double)
-
-                    // the 28-digit example from the issue: overflows uint64_t
-                    // but is finite as a double, so the scanner reclassifies
-                    // it to value_float and it is accepted
-                    {"9999999999999999999999999999", true},
-
-                    // huge digit-only integers that overflow even a double -> rejected
-                    {std::string(309, '9'), false},
-                    {std::string(400, '9'), false},
-                    {"1" + std::string(400, '0'), false},
-
-                    // 1e999 / 1e400 style overflow -> rejected
-                    {"1e999", false},
-                    {"1e400", false},
-                    {"-1e999", false},
-                    {"1E999", false},
-
-                    // values straddling DBL_MAX
-                    {"1.7976931348623157e308", true},   // <= DBL_MAX, finite
-                    {"1.7976931348623159e308", false},  // > DBL_MAX, overflows to inf
-
-                    // a mix of other valid/invalid numeric syntax
-                    {"3.14159", true},
-                    {"-0.0", true},
-                    {"1.0e10", true},
-                    {"01", false},
-                    {"-", false},
-                    {"1.", false},
-                    {"1e", false},
-                    {"+1", false},
-                };
-
-                for (const auto& c : cases)
-                {
-                    const std::string& number = c.first;
-                    const bool expected = c.second;
-                    CAPTURE(number)
-                    CAPTURE(expected)
-
-                    // accept() takes the fast path (skips conversion when possible)
-                    CHECK(json::accept(number) == expected);
-
-                    // parse() always performs the full conversion; it must agree
-                    json j;
-                    CHECK_NOTHROW(json::parser(nlohmann::detail::input_adapter(number), nullptr, false).parse(true, j));
-                    CHECK(!j.is_discarded() == expected);
-
-                    // wrap in an array so get_token() is exercised beyond the
-                    // very first (constructor-time) scan as well
-                    std::string wrapped = "[";
-                    wrapped += number;
-                    wrapped += ",";
-                    wrapped += number;
-                    wrapped += "]";
-                    CHECK(json::accept(wrapped) == expected);
-                }
-            }
         }
     }
 
@@ -1486,69 +1394,6 @@ TEST_CASE("parser class")
         CHECK(accept_helper("\"\\uD80C\\uFFFF\"") == false);
     }
 
-    SECTION("issue #5412 - whitespace skipping bookkeeping (compact vs. pretty-printed)")
-    {
-        // lexer::skip_whitespace() reads its first character with get() (to
-        // honor a possibly pending unget() from the previous token) and every
-        // further whitespace character with get_ignoring_pending_unget() (a
-        // get() variant that skips the then-always-false next_unget check).
-        // This must not change the reported byte offset, line, or column of
-        // a syntax error, even when a long run of whitespace containing
-        // multiple newlines is skipped beforehand (as with pretty-printed
-        // input). The expected values below were captured from the
-        // unmodified do-while(get()) loop, so any regression that miscounts
-        // characters or newlines while skipping whitespace changes them.
-        const auto check_error = [](const std::string & input, std::size_t expected_byte,
-                                    const std::string & expected_what)
-        {
-            CAPTURE(input)
-            try
-            {
-                json _ = json::parse(input);
-                FAIL_CHECK("expected a parse_error, but parsing succeeded");
-            }
-            catch (const json::parse_error& e)
-            {
-                CHECK(e.byte == expected_byte);
-                CHECK(std::string(e.what()) == expected_what);
-            }
-        };
-
-        // a nested document, serialized both compactly and pretty-printed
-        // (dump(4)), each truncated right before the final closing '}' so
-        // that the parser hits EOF after skipping all of the (in the
-        // pretty-printed case, substantial) indentation whitespace
-        const json doc =
-        {
-            {"a", 1},
-            {"b", json::array({true, false, nullptr, "x"})},
-            {"c", json::object({{"d", 3.14}, {"e", json::array({1, 2, 3})}})}
-        };
-
-        const std::string compact = doc.dump();
-        const std::string pretty = doc.dump(4);
-
-        check_error(compact.substr(0, compact.size() - 1), 60,
-                    "[json.exception.parse_error.101] parse error at line 1, column 60: syntax error while parsing object - unexpected end of input; expected '}'");
-        check_error(pretty.substr(0, pretty.size() - 1), 193,
-                    "[json.exception.parse_error.101] parse error at line 17, column 1: syntax error while parsing object - unexpected end of input; expected '}'");
-
-        // an invalid token appearing after several indented, multi-line
-        // whitespace runs vs. the same document without any of that
-        // whitespace
-        check_error(R"({
-    "a": 1,
-    "b": [
-        true,
-        false
-    ],
-    "c": @
-})", 70,
-                    "[json.exception.parse_error.101] parse error at line 7, column 10: syntax error while parsing value - invalid literal; last read: '\"c\": @'");
-        check_error("{\"a\":1,\"b\":[true,false],\"c\":@}", 29,
-                    "[json.exception.parse_error.101] parse error at line 1, column 29: syntax error while parsing value - invalid literal; last read: '\"c\":@'");
-    }
-
     SECTION("tests found by mutate++")
     {
         // test case to make sure no comma precedes the first key

tests/src/unit-msgpack.cpp

@@ -1598,6 +1598,67 @@ TEST_CASE("MessagePack")
 }
 
 // use this testcase outside [hide] to run it with Valgrind
+TEST_CASE("MessagePack nesting does not consume the call stack")
+{
+    // Reading a container used to call back into the value reader once per
+    // element, so the native call stack grew with the nesting depth of the
+    // input: one frame per byte for repeated 0x91 (a one-element array), which
+    // crashes the process long before the input is exhausted (#5104). The
+    // containers are kept on a heap stack now.
+    //
+    // Note that deeply nested values must not be compared, copied or dumped
+    // here: those operations are still recursive, and would reintroduce the
+    // very crash this checks for. Depth is measured by descending instead.
+
+    SECTION("an unterminated chain is reported, not crashed on")
+    {
+        json _;
+        const std::vector<uint8_t> input(300000, 0x91);
+        CHECK_THROWS_WITH_AS(_ = json::from_msgpack(input), "[json.exception.parse_error.110] parse error at byte 300001: syntax error while parsing MessagePack value: unexpected end of input", json::parse_error&);
+        CHECK(json::from_msgpack(input, true, false).is_discarded());
+    }
+
+    SECTION("a well-formed deep value is read through the SAX interface")
+    {
+        std::vector<uint8_t> input(300000, 0x91);
+        input.push_back(0x01); // innermost value
+
+        SaxCountdown accept_all(600001);
+        CHECK(json::sax_parse(input, &accept_all, json::input_format_t::msgpack));
+    }
+
+    SECTION("a well-formed deep value is read into a value")
+    {
+        const std::size_t depth = 10000;
+        std::vector<uint8_t> input(depth, 0x91);
+        input.push_back(0x01);
+
+        json j = json::from_msgpack(input);
+
+        std::size_t measured = 0;
+        const json* p = &j;
+        while (p->is_array() && !p->empty())
+        {
+            p = &p->front();
+            ++measured;
+        }
+        CHECK(measured == depth);
+        CHECK(p->is_number());
+    }
+
+    SECTION("containers are still read the same way")
+    {
+        CHECK(json::from_msgpack(std::vector<uint8_t>({0x90})) == json::array());
+        CHECK(json::from_msgpack(std::vector<uint8_t>({0x80})) == json::object());
+        CHECK(json::from_msgpack(std::vector<uint8_t>({0x92, 0x90, 0x80})) == json({json::array(), json::object()}));
+        CHECK(json::from_msgpack(std::vector<uint8_t>({0x91, 0x91, 0x91, 0x90})) == json({{{json::array()}}}));
+        CHECK(json::from_msgpack(std::vector<uint8_t>({0x81, 0xA1, 'a', 0x81, 0xA1, 'b', 0x92, 0x01, 0x02})) == json({{"a", {{"b", {1, 2}}}}}));
+        // array 16 and map 32, i.e. the counted forms
+        CHECK(json::from_msgpack(std::vector<uint8_t>({0xDC, 0x00, 0x02, 0x01, 0x02})) == json({1, 2}));
+        CHECK(json::from_msgpack(std::vector<uint8_t>({0xDF, 0x00, 0x00, 0x00, 0x01, 0xA1, 'k', 0xC3})) == json({{"k", true}}));
+    }
+}
+
 TEST_CASE("single MessagePack roundtrip")
 {
     SECTION("sample.json")

tests/src/unit-ubjson.cpp

@@ -2149,6 +2149,166 @@ TEST_CASE("UBJSON")
     }
 }
 
+TEST_CASE("UBJSON nesting does not consume the call stack")
+{
+    // Containers used to be read by calling back into the value reader once
+    // per element, so the native call stack grew with the nesting depth of the
+    // input. '[' alone opens a container, so a payload of repeated '[' crashed
+    // the process (#5104), as did the optimized forms, which reach the same
+    // path through a type or size annotation. The containers are kept on a
+    // heap stack now.
+    //
+    // Deeply nested values must not be compared, copied or dumped here: those
+    // operations are still recursive and would reintroduce the crash.
+    json _;
+
+    SECTION("containers that end at a marker")
+    {
+        const std::vector<uint8_t> input(500000, '[');
+        CHECK_THROWS_WITH_AS(_ = json::from_ubjson(input), "[json.exception.parse_error.110] parse error at byte 500001: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
+        CHECK(json::from_ubjson(input, true, false).is_discarded());
+    }
+
+    SECTION("containers with a size")
+    {
+        std::vector<uint8_t> input;
+        for (std::size_t i = 0; i < 100000; ++i)
+        {
+            input.push_back('[');
+            input.push_back('#');
+            input.push_back('i');
+            input.push_back(1);
+        }
+        CHECK_THROWS_AS(_ = json::from_ubjson(input), json::parse_error&);
+        CHECK(json::from_ubjson(input, true, false).is_discarded());
+    }
+
+    SECTION("containers with a type and a size")
+    {
+        // '[' is a permitted optimized type in UBJSON, so each element of such
+        // a container is itself a container, read without a marker of its own
+        std::vector<uint8_t> input;
+        for (std::size_t i = 0; i < 100000; ++i)
+        {
+            const std::vector<uint8_t> level = {'[', '$', '[', '#', 'i', 1};
+            input.insert(input.end(), level.begin(), level.end());
+        }
+        CHECK_THROWS_AS(_ = json::from_ubjson(input), json::parse_error&);
+        CHECK(json::from_ubjson(input, true, false).is_discarded());
+    }
+
+    SECTION("a well-formed deep value is read through the SAX interface")
+    {
+        std::vector<uint8_t> input(100000, '[');
+        input.insert(input.end(), 100000, ']');
+
+        SaxCountdown accept_all(1000000);
+        CHECK(json::sax_parse(input, &accept_all, json::input_format_t::ubjson));
+    }
+
+    SECTION("a well-formed deep value is read into a value")
+    {
+        const std::size_t depth = 10000;
+        std::vector<uint8_t> input(depth, '[');
+        input.insert(input.end(), depth, ']');
+
+        json j = json::from_ubjson(input);
+
+        std::size_t measured = 0;
+        const json* p = &j;
+        while (p->is_array() && !p->empty())
+        {
+            p = &p->front();
+            ++measured;
+        }
+        // the innermost array is empty, so the descent stops one level short
+        CHECK(measured == depth - 1);
+    }
+
+    SECTION("containers are still read the same way")
+    {
+        CHECK(json::from_ubjson(std::vector<uint8_t>({'[', ']'})) == json::array());
+        CHECK(json::from_ubjson(std::vector<uint8_t>({'{', '}'})) == json::object());
+        CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '#', 'i', 0})) == json::array());
+        CHECK(json::from_ubjson(std::vector<uint8_t>({'{', '#', 'i', 0})) == json::object());
+        CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'i', '#', 'i', 2, 1, 2})) == json({1, 2}));
+        CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '#', 'i', 2, 'i', 1, 'i', 2})) == json({1, 2}));
+        CHECK(json::from_ubjson(std::vector<uint8_t>({'{', '$', 'i', '#', 'i', 1, 'i', 1, 'a', 1})) == json({{"a", 1}}));
+        // a no-op is not a value, so a container of them holds none
+        CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'N', '#', 'i', 2})) == json::array());
+        // sized and unsized forms nested inside one another
+        CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '[', '#', 'i', 2, 'i', 1, 'i', 2, ']'})) == json({{1, 2}}));
+        CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '#', 'i', 1, '[', 'i', 1, ']'})) == json({{1}}));
+        // an optimized container of containers
+        CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', '[', '#', 'i', 2, 'i', 1, ']', 'i', 2, ']'})) == json({{1}, {2}}));
+    }
+
+    SECTION("BJData containers are still read the same way")
+    {
+        // the ND-array wrapper and the binary shortcut are complete values,
+        // not containers the reader descends into
+        CHECK(json::from_bjdata(std::vector<uint8_t>({'[', '$', 'U', '#', '[', '$', 'i', '#', 'i', 2, 2, 3, 1, 2, 3, 4, 5, 6})) ==
+        json({{"_ArrayType_", "uint8"}, {"_ArraySize_", {2, 3}}, {"_ArrayData_", {1, 2, 3, 4, 5, 6}}}));
+        CHECK(json::from_bjdata(std::vector<uint8_t>({'[', '$', 'i', '#', 'i', 2, 1, 2})) == json({1, 2}));
+        CHECK(json::from_bjdata(std::vector<uint8_t>({'[', '[', 'i', 1, ']', ']'})) == json({{1}}));
+    }
+}
+
+TEST_CASE("UBJSON optimized arrays of a valueless type are bounded")
+{
+    // An element of type 'Z', 'T' or 'F' is encoded by its marker alone, so an
+    // optimized array of one of those has no payload and the declared count is
+    // the only thing deciding how much is allocated. Ten bytes used to produce
+    // billions of values (#2793); every other type costs at least one byte per
+    // element and is bounded by the end of the input.
+    json _;
+
+    SECTION("an excessive count is rejected")
+    {
+        // 'l' is a big-endian int32: 0x7FFFFFFF elements, about 34 GB of value
+        for (const auto marker :
+                {'Z', 'T', 'F'
+                })
+        {
+            const std::vector<uint8_t> input = {'[', '$', static_cast<uint8_t>(marker), '#', 'l', 0x7F, 0xFF, 0xFF, 0xFF};
+            CHECK_THROWS_WITH_AS(_ = json::from_ubjson(input), "[json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive array size", json::out_of_range&);
+            CHECK(json::from_ubjson(input, true, false).is_discarded());
+        }
+    }
+
+    SECTION("ordinary counts are unaffected")
+    {
+        CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'Z', '#', 'i', 3})) == json({nullptr, nullptr, nullptr}));
+        CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'T', '#', 'i', 2})) == json({true, true}));
+        CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'F', '#', 'i', 2})) == json({false, false}));
+        // 'N' is a no-op rather than a value, and still yields an empty array
+        CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'N', '#', 'i', 2})) == json::array());
+    }
+
+    SECTION("a type with a payload is unaffected")
+    {
+        // the same count for 'U' is bounded by the end of the input instead
+        const std::vector<uint8_t> input = {'[', '$', 'U', '#', 'l', 0x7F, 0xFF, 0xFF, 0xFF};
+        CHECK_THROWS_AS(_ = json::from_ubjson(input), json::parse_error&);
+    }
+
+    SECTION("the writer stays within what the reader accepts")
+    {
+        // below the limit the optimized form is used and is tiny; above it the
+        // writer falls back so that the result can still be read back
+        json const at_limit(1048576, nullptr);
+        const auto v_at_limit = json::to_ubjson(at_limit, true, true);
+        CHECK(v_at_limit.size() == 9);
+        CHECK(v_at_limit.at(1) == '$');
+        CHECK(json::from_ubjson(v_at_limit) == at_limit);
+
+        json const above_limit(1048577, nullptr);
+        const auto v_above_limit = json::to_ubjson(above_limit, true, true);
+        CHECK(v_above_limit.at(1) != '$');
+        CHECK(json::from_ubjson(v_above_limit) == above_limit);
+    }
+}
+
 TEST_CASE("Universal Binary JSON Specification Examples 1")
 {
     SECTION("Null Value")