Declare the partner iterator in the loop initializer in diff_object_keys()

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/to_bjdata.md

@@ -123,6 +123,4 @@ Linear in the size of the JSON value `j`.
   that is not valid UTF-8 unchanged, as before; `strict` (the default if
   [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled) throws `type_error.316`.
 - Throws `type_error.321` for a discarded value since version 3.13.0; previously, a discarded value nested in an
-  array or object was silently skipped, producing invalid BJData.
-- Writes unsigned integers wider than 64 bits as high-precision numbers since version 3.13.0; previously, they were
-  silently truncated to 64 bits.
+  array or object was silently skipped, producing invalid BJData.

docs/mkdocs/docs/api/basic_json/to_bon8.md

@@ -37,9 +37,8 @@ With (2), the bytes written before the exception remain in the output adapter.
 
 ## Exceptions
 
-- Throws [out_of_range.407](../../home/exceptions.md#jsonexceptionout_of_range407) if `j` contains an integer outside
-  the range of int64 (an unsigned integer above 9223372036854775807, or, with a number type wider than 64 bits, any
-  integer beyond int64), which BON8 cannot represent
+- Throws [out_of_range.407](../../home/exceptions.md#jsonexceptionout_of_range407) if `j` contains an unsigned integer
+  above 9223372036854775807, which BON8 cannot represent
 - Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if `j` contains a string that is not
   valid UTF-8
 - Throws [type_error.321](../../home/exceptions.md#jsonexceptiontype_error321) if `j` or a value nested in it is

docs/mkdocs/docs/api/basic_json/to_bson.md

@@ -47,10 +47,6 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
 
 - Throws [`type_error.317`](../../home/exceptions.md#jsonexceptiontype_error317) if the top-level type of the JSON value
   is not an object; example: `"to serialize to BSON, top-level type must be object, but is string"`
-- Throws [`out_of_range.407`](../../home/exceptions.md#jsonexceptionout_of_range407) if `j` contains a signed integer
-  outside the range of int64 or an unsigned integer outside the range of uint64, which is only possible with a number
-  type wider than 64 bits; example:
-  `"integer number 9223372036854775808 cannot be represented by BSON as it does not fit int64"`
 - Throws [`out_of_range.409`](../../home/exceptions.md#jsonexceptionout_of_range409) if a key in the JSON object contains
   a null byte (code point U+0000); example: `"BSON key cannot contain code point U+0000 (at byte 2)"`
 - Throws [`out_of_range.412`](../../home/exceptions.md#jsonexceptionout_of_range412) if the length of a document, array,
@@ -123,5 +119,3 @@ pass before anything is written.
   that is not valid UTF-8 unchanged, as before; `strict` (the default if
   [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled) throws `type_error.316` before anything
   is written.
-- Throws `out_of_range.407` for integers that do not fit 64 bits since version 3.13.0; previously, integers of a
-  number type wider than 64 bits were silently truncated.

docs/mkdocs/docs/api/basic_json/to_cbor.md

@@ -46,9 +46,6 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
 
 ## Exceptions
 
-- Throws [`out_of_range.407`](../../home/exceptions.md#jsonexceptionout_of_range407) if `j` contains an integer
-  outside [-2^64, 2^64-1], which is only possible with a number type wider than 64 bits; example:
-  `"integer number 18446744073709551616 cannot be represented by CBOR as it does not fit [-2^64, 2^64-1]"`
 - Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if a string or object key in `j` is
   not valid UTF-8 and `error_handler` is `strict` (the default only if
   [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled)
@@ -93,5 +90,3 @@ Linear in the size of the JSON value `j`.
   [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled) throws `type_error.316`.
 - Throws `type_error.321` for a discarded value since version 3.13.0; previously, a discarded value nested in an
   array or object was silently skipped, producing invalid CBOR.
-- Throws `out_of_range.407` for integers that do not fit 64 bits since version 3.13.0; previously, integers of a
-  number type wider than 64 bits were silently truncated.

docs/mkdocs/docs/api/basic_json/to_msgpack.md

@@ -46,9 +46,6 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
 
 ## Exceptions
 
-- Throws [`out_of_range.407`](../../home/exceptions.md#jsonexceptionout_of_range407) if `j` contains an integer
-  outside [-2^63, 2^64-1], which is only possible with a number type wider than 64 bits; example:
-  `"integer number 18446744073709551616 cannot be represented by MessagePack as it does not fit [-2^63, 2^64-1]"`
 - Throws [`out_of_range.412`](../../home/exceptions.md#jsonexceptionout_of_range412) if the length of a string, binary
   value, array, or object exceeds 4294967295, the maximum MessagePack can store; example:
   `"MessagePack length 4294967296 exceeds maximum of 4294967295"`
@@ -115,5 +112,3 @@ Linear in the size of the JSON value `j`.
   `number_unsigned_t`.
 - Throws `type_error.321` for a discarded value since version 3.13.0; previously, a discarded value nested in an
   array or object was silently skipped, producing invalid MessagePack.
-- Throws `out_of_range.407` for integers that do not fit 64 bits since version 3.13.0; previously, integers of a
-  number type wider than 64 bits were silently truncated.

docs/mkdocs/docs/home/exceptions.md

@@ -916,19 +916,9 @@ double-precision number when `number_float_t` is `#!cpp float`.
 
 ### json.exception.out_of_range.407
 
-An integer number cannot be represented by the binary format it is serialized to:
-
-- [BON8](../features/binary_formats/bon8.md) only stores integers that fit into int64, so
-  [`to_bon8`](../api/basic_json/to_bon8.md) throws for unsigned integers greater than 9223372036854775807.
-- [CBOR](../features/binary_formats/cbor.md), [MessagePack](../features/binary_formats/messagepack.md), and
-  [BSON](../features/binary_formats/bson.md) store integers in at most 64 bits. With the default number types, every
-  integer fits, but a [`number_integer_t`](../api/basic_json/number_integer_t.md) or
-  [`number_unsigned_t`](../api/basic_json/number_unsigned_t.md) wider than 64 bits (e.g., `__int128`) can hold values
-  outside the range of the format: [-2^64, 2^64-1] for CBOR, [-2^63, 2^64-1] for MessagePack, and the range of int64
-  (signed integers) or uint64 (unsigned integers) for BSON.
-
-[UBJSON](../features/binary_formats/ubjson.md) and [BJData](../features/binary_formats/bjdata.md) never throw this
-exception, because they serialize integers beyond 64 bits as high-precision numbers.
+An unsigned integer number greater than 9223372036854775807 cannot be represented in the binary format. This exception
+is only thrown by [`to_bon8`](../api/basic_json/to_bon8.md), as [BON8](../features/binary_formats/bon8.md) has no
+unsigned 64-bit integer type.
 
 !!! note "Former occurrences"
 
@@ -937,17 +927,11 @@ exception, because they serialize integers beyond 64 bits as high-precision numb
     - Since version 3.9.0, integer numbers beyond int64 are serialized as high-precision UBJSON numbers.
     - Since version 3.12.0, integer numbers beyond int64 are serialized as uint64 BSON numbers.
 
-    Before version 3.13.0, CBOR, MessagePack, and BSON silently truncated integers wider than 64 bits, and BJData
-    truncated unsigned integers wider than 64 bits.
-
-!!! failure "Example messages"
+!!! failure "Example message"
 
     ```
     integer number 18446744073709551615 cannot be represented by BON8 as it does not fit int64
     ```
-    ```
-    integer number 1267650600228229401496703205376 cannot be represented by CBOR as it does not fit [-2^64, 2^64-1]
-    ```
 
 ### json.exception.out_of_range.408
 

include/nlohmann/detail/json_pointer.hpp

@@ -9,10 +9,13 @@
 #pragma once
 
 #include <algorithm> // all_of
+#include <array> // array
 #include <cctype> // isdigit
 #include <cerrno> // errno, ERANGE
 #include <cstddef> // size_t
+#include <cstdint> // uint64_t
 #include <cstdlib> // strtoull
+#include <cstring> // memcmp, memcpy
 #ifndef JSON_NO_IO
     #include <iosfwd> // ostream
 #endif  // JSON_NO_IO
@@ -492,50 +495,111 @@ class json_pointer
 
   private:
     /*!
-    @brief the pointer prefixes of a flattened object, and which of them denote arrays
+    @brief the pointer prefixes of a flattened object that denote arrays
 
-    @ref unflatten collects the pointer prefixes that have a reference token 0
-    among their children; @ref get_and_create creates arrays exactly below
-    those prefixes and objects everywhere else. Deciding this up front keeps
-    the result independent of the order in which the flattened object is
-    iterated, which is unspecified for some object types.
+    @ref unflatten creates arrays exactly below the prefixes that have a
+    reference token 0 among their children, and objects everywhere else.
+    Deciding this up front keeps the result independent of the order in which
+    the flattened object is iterated, which is unspecified for some object
+    types.
 
-    The prefixes form a tree and are numbered (number 0 is the empty prefix),
-    so each of them is stored only once (as a node) rather than as a copy of
-    all of its reference tokens. The tree does not own the reference tokens:
-    it refers to the tokens of the parsed keys, which must outlive it.
+    A prefix is identified by the text of the key it is the beginning of:
+    escaping is unambiguous, so equal prefixes of valid keys are equal texts.
+    The set refers to the keys, which must outlive it.
     */
-    struct prefix_tree
+    class array_prefix_set
     {
-        // open-addressing hash table from (parent, last reference token) to
-        // the number of the node; empty slots have token == nullptr
+      public:
+        void insert(const char* data, std::size_t size)
+        {
+            // flattened objects are usually sorted, so most keys that have a
+            // 0 yield the prefix of the previous one
+            if (last_data != nullptr && last_size == size && std::memcmp(last_data, data, size) == 0)
+            {
+                return;
+            }
+            last_data = data;
+            last_size = size;
+
+            if (slots.empty())
+            {
+                slots.resize(16);
+            }
+            const std::size_t hash = hash_of(data, size);
+            const std::size_t mask = slots.size() - 1;
+            std::size_t i = hash & mask;
+            while (slots[i].data != nullptr)
+            {
+                if (slots[i].hash == hash && slots[i].size == size && std::memcmp(slots[i].data, data, size) == 0)
+                {
+                    return;
+                }
+                i = (i + 1) & mask;
+            }
+            slots[i] = slot(hash, data, size);
+
+            // keep the load factor below 1/2
+            if (++used * 2 > slots.size())
+            {
+                grow();
+            }
+        }
+
+        bool empty() const noexcept
+        {
+            return used == 0;
+        }
+
+        bool contains(const char* data, std::size_t size) const noexcept
+        {
+            if (used == 0)
+            {
+                return false;
+            }
+            const std::size_t hash = hash_of(data, size);
+            const std::size_t mask = slots.size() - 1;
+            for (std::size_t i = hash & mask; slots[i].data != nullptr; i = (i + 1) & mask)
+            {
+                if (slots[i].hash == hash && slots[i].size == size && std::memcmp(slots[i].data, data, size) == 0)
+                {
+                    return true;
+                }
+            }
+            return false;
+        }
+
+      private:
         struct slot
         {
-            std::size_t hash;
-            std::size_t parent;
-            const string_t* token;
-            std::size_t id;
+            slot() noexcept = default;
 
-            slot() noexcept
-                : hash(0), parent(0), token(nullptr), id(0)
+            slot(std::size_t h, const char* d, std::size_t s) noexcept
+                : hash(h), data(d), size(s)
             {}
 
-            slot(std::size_t h, std::size_t p, const string_t* t, std::size_t i) noexcept
-                : hash(h), parent(p), token(t), id(i)
-            {}
+            std::size_t hash = 0;
+            const char* data = nullptr; // nullptr: the slot is empty
+            std::size_t size = 0;
         };
 
-        std::vector<slot> slots;
-        std::size_t used = 0;
-
-        static std::size_t hash_of(std::size_t parent, const string_t& token) noexcept
+        static std::size_t hash_of(const char* data, std::size_t size) noexcept
         {
-            std::size_t h = parent * static_cast<std::size_t>(0x9E3779B97F4A7C15ULL & (std::numeric_limits<std::size_t>::max)());
-            for (std::size_t i = 0; i < token.size(); ++i)
+            std::uint64_t h = 0x9E3779B97F4A7C15ULL ^ size;
+            std::size_t i = 0;
+            for (; i + 8 <= size; i += 8)
             {
-                h = (h ^ static_cast<unsigned char>(token[i])) * static_cast<std::size_t>(0x100000001B3ULL & (std::numeric_limits<std::size_t>::max)());
+                std::uint64_t w = 0;
+                std::memcpy(&w, data + i, 8);
+                h = (h ^ w) * 0x100000001B3ULL;
+                h ^= h >> 32u;
             }
-            return h ^ (h >> 29u);
+            std::uint64_t w = 0;
+            if (i < size)
+            {
+                std::memcpy(&w, data + i, size - i);
+            }
+            h = (h ^ w) * 0x9E3779B97F4A7C15ULL;
+            return static_cast<std::size_t>(h ^ (h >> 29u));
         }
 
         void grow()
@@ -545,10 +609,10 @@ class json_pointer
             const std::size_t mask = slots.size() - 1;
             for (const auto& s : old)
             {
-                if (s.token != nullptr)
+                if (s.data != nullptr)
                 {
                     std::size_t i = s.hash & mask;
-                    while (slots[i].token != nullptr)
+                    while (slots[i].data != nullptr)
                     {
                         i = (i + 1) & mask;
                     }
@@ -557,117 +621,160 @@ class json_pointer
             }
         }
 
-        // parent[id] is the number of the prefix without its last token
-        std::vector<std::size_t> parent;
-        // is_array[id] is true iff some flattened key has the reference token
-        // 0 directly below the prefix with number id
-        std::vector<bool> is_array;
-
-        // start with the empty prefix only
-        prefix_tree()
-            : slots(16)
-            , parent(1, 0)
-            , is_array(1, false)
-        {}
-
-        // return the number of the prefix with number id extended by
-        // reference_token, adding it if it is new
-        std::size_t add_child(std::size_t id, const string_t& reference_token)
-        {
-            const std::size_t hash = hash_of(id, reference_token);
-            const std::size_t mask = slots.size() - 1;
-            std::size_t i = hash & mask;
-            while (slots[i].token != nullptr)
-            {
-                if (slots[i].hash == hash && slots[i].parent == id && *slots[i].token == reference_token)
-                {
-                    return slots[i].id;
-                }
-                i = (i + 1) & mask;
-            }
-
-            // new prefix
-            const std::size_t child = parent.size();
-            slots[i] = slot(hash, id, &reference_token, child);
-            if (reference_token == "0")
-            {
-                is_array[id] = true;
-            }
-            parent.push_back(id);
-            is_array.push_back(false);
-
-            // keep the load factor below 1/2
-            if (++used * 2 > slots.size())
-            {
-                grow();
-            }
-            return child;
-        }
+        std::vector<slot> slots{};
+        std::size_t used = 0;
+        const char* last_data = nullptr;
+        std::size_t last_size = 0;
     };
 
+    /// @brief the length of the longest common prefix of the first @a limit bytes of @a a and @a b
+    static std::size_t common_prefix_length(const char* a, const char* b, const std::size_t limit) noexcept
+    {
+        std::size_t i = 0;
+        for (; i + 8 <= limit; i += 8)
+        {
+            std::uint64_t x = 0;
+            std::uint64_t y = 0;
+            std::memcpy(&x, a + i, 8);
+            std::memcpy(&y, b + i, 8);
+            if (x != y)
+            {
+                break;
+            }
+        }
+        while (i < limit && a[i] == b[i])
+        {
+            ++i;
+        }
+        return i;
+    }
+
     /*!
-    @brief create and return a reference to the pointed to value
+    @brief check that @a key is a valid JSON pointer, and add the prefixes of
+           it that denote arrays to @a arrays
 
-    @param[in] j  the value to create the pointed to value in
-    @param[in] tokens  the reference tokens of the pointer
-    @param[in] prefixes  prefixes[i] is the number in @a tree of the prefix
-                         consisting of the first i reference tokens
+    @throw parse_error.107  if the key is not empty or begins with '/'
+    @throw parse_error.108  if character '~' is not followed by '0' or '1'
 
-    Complexity: Linear in the number of reference tokens.
+    The same checks as @ref split, which throws the same exceptions, but
+    without building the reference tokens.
+    */
+    static void scan_key(const string_t& key, const string_t* previous, array_prefix_set& arrays)
+    {
+        const std::size_t size = key.size();
+        if (size == 0)
+        {
+            return;
+        }
 
-    @throw parse_error.106 if an array index begins with '0'
-    @throw parse_error.109 if array index is not a number
-    @throw type_error.313 if value cannot be unflattened
+        const char* const data = key.data();
+        if (JSON_HEDLEY_UNLIKELY(data[0] != '/'))
+        {
+            JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_missing_slash, 1, detail::concat("JSON pointer must be empty or begin with '/' - was: '", key, "'"), nullptr));
+        }
+
+        // The beginning of the key that it has in common with the previous
+        // one, which was checked already, is skipped: the scan starts with
+        // the token after the last slash in it.
+        std::size_t begin = 1;
+        if (previous != nullptr)
+        {
+            std::size_t same = common_prefix_length(data, previous->data(), (std::min)(size, previous->size()));
+            while (same > 1 && data[same - 1] != '/')
+            {
+                --same;
+            }
+            if (same > 1)
+            {
+                begin = same;
+            }
+        }
+
+        for (std::size_t i = begin; i <= size; ++i)
+        {
+            if (i == size || data[i] == '/')
+            {
+                // a token 0 makes the pointer before it an array
+                if (i - begin == 1 && data[begin] == '0')
+                {
+                    arrays.insert(data, begin - 1);
+                }
+                begin = i + 1;
+            }
+            else if (data[i] == '~')
+            {
+                // ~ must be followed by 0 or 1
+                if (JSON_HEDLEY_UNLIKELY(i + 1 == size || (data[i + 1] != '0' && data[i + 1] != '1')))
+                {
+                    JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_invalid_escape, 0, "escape character '~' must be followed with '0' or '1'", nullptr));
+                }
+            }
+        }
+    }
+
+    /*!
+    @brief resolve a plain JSON pointer without building its reference tokens
+
+    @param[in] ptr     the value the pointer starts at
+    @param[in] path    a JSON pointer string
+    @param[in] length  the number of leading bytes of @a path to resolve; 0 or
+                       the position of a slash
+    @param[in,out] token  a string to hold the reference tokens, to reuse its storage
+
+    @return the pointed to value, or `nullptr` if the pointer does not denote
+            an existing value, or does so in a way that is not plain: a
+            reference token with an escape, an array index that is malformed or
+            out of range, or a "-". No exception is thrown for these; callers
+            use @ref json_pointer instead, which handles them.
     */
     template<typename BasicJsonType>
-    static BasicJsonType& get_and_create(BasicJsonType& j, const std::vector<string_t>& tokens,
-                                         const std::vector<std::size_t>& prefixes, const prefix_tree& tree)
+    static BasicJsonType* resolve_plain(BasicJsonType* ptr, const string_t& path, const std::size_t length, string_t& token)
     {
-        auto* result = &j;
-
-        // in case no reference tokens exist, return a reference to the JSON value
-        // j which will be overwritten by a primitive value
-        for (std::size_t i = 0; i < tokens.size(); ++i)
+        for (std::size_t pos = 0; pos < length;)
         {
-            const auto& reference_token = tokens[i];
-            switch (result->type())
+            if (JSON_HEDLEY_UNLIKELY(path[pos] != '/'))
             {
-                case detail::value_t::null:
-                {
-                    if (tree.is_array[prefixes[i]])
-                    {
-                        // some reference token below this position is 0, so the
-                        // value is an array
-                        result = &result->operator[](array_index<BasicJsonType>(reference_token));
-                    }
-                    else
-                    {
-                        // start a new object otherwise
-                        result = &result->operator[](reference_token);
-                    }
-                    break;
-                }
+                return nullptr;
+            }
 
+            // the reference token between the slash at pos and the next one
+            std::size_t end = path.find_first_of('/', pos + 1);
+            if (end == string_t::npos || end > length)
+            {
+                end = length;
+            }
+            token.clear();
+            token.append(path.data() + pos + 1, end - pos - 1);
+            if (JSON_HEDLEY_UNLIKELY(token.find_first_of('~') != string_t::npos))
+            {
+                return nullptr;
+            }
+
+            switch (ptr->type())
+            {
                 case detail::value_t::object:
                 {
-                    // create an entry in the object
-                    result = &object_member(*result, reference_token);
+                    const auto it = ptr->m_data.m_value.object->find(token);
+                    if (JSON_HEDLEY_UNLIKELY(it == ptr->m_data.m_value.object->end()))
+                    {
+                        return nullptr;
+                    }
+                    ptr = &it->second;
                     break;
                 }
 
                 case detail::value_t::array:
                 {
-                    // create an entry in the array
-                    result = &result->operator[](array_index<BasicJsonType>(reference_token));
+                    typename BasicJsonType::size_type idx{};
+                    if (JSON_HEDLEY_UNLIKELY(parse_array_index<BasicJsonType>(token, idx) != array_index_status::ok || idx >= ptr->m_data.m_value.array->size()))
+                    {
+                        return nullptr;
+                    }
+                    ptr = &(*ptr->m_data.m_value.array)[idx];
                     break;
                 }
 
-                /*
-                The following code is only reached if there exists a reference
-                token _and_ the current value is primitive. In this case, we have
-                an error situation, because primitive values may only occur as
-                a single value; that is, with an empty list of reference tokens.
-                */
+                case detail::value_t::null:
                 case detail::value_t::string:
                 case detail::value_t::boolean:
                 case detail::value_t::number_integer:
@@ -676,11 +783,12 @@ class json_pointer
                 case detail::value_t::binary:
                 case detail::value_t::discarded:
                 default:
-                    JSON_THROW(detail::type_error::create(detail::exception_id::unflatten_invalid_value, "invalid value to unflatten", &j));
+                    return nullptr;
             }
+            pos = end;
         }
 
-        return *result;
+        return ptr;
     }
 
     /*!
@@ -1135,20 +1243,45 @@ class json_pointer
     // Add the entry (key, v) to the object `obj`. Flattened paths are pairwise
     // distinct, so objects that can append a key known to be absent
     // (ordered_map) skip the lookup, which is linear and would make flatten
-    // quadratic; all other object types use operator[].
+    // quadratic. Objects with a positional insert (std::map) insert next to
+    // the entry added last: paths come out in nearly sorted order, so the
+    // position is usually right and no search from the root is needed.
+    // All other object types use operator[].
     template<typename ObjectType, typename ValueType>
-    static auto add_flattened(ObjectType& obj, const string_t& key, ValueType&& v, detail::priority_tag<1> /*unused*/)
+    static auto add_flattened(ObjectType& obj, typename ObjectType::iterator& /*hint*/, const string_t& key, ValueType&& v, detail::priority_tag<2> /*unused*/)
     -> decltype(obj.emplace_new_key(key, std::forward<ValueType>(v)), void())
     {
         obj.emplace_new_key(key, std::forward<ValueType>(v));
     }
 
     template<typename ObjectType, typename ValueType>
-    static void add_flattened(ObjectType& obj, const string_t& key, ValueType&& v, detail::priority_tag<0> /*unused*/)
+    static auto add_flattened(ObjectType& obj, typename ObjectType::iterator& hint, const string_t& key, ValueType&& v, detail::priority_tag<1> /*unused*/)
+    -> decltype(hint = obj.emplace_hint(hint, key, std::forward<ValueType>(v)), void())
+    {
+        hint = obj.emplace_hint(hint, key, std::forward<ValueType>(v));
+        ++hint;
+    }
+
+    template<typename ObjectType, typename ValueType>
+    static void add_flattened(ObjectType& obj, typename ObjectType::iterator& /*hint*/, const string_t& key, ValueType&& v, detail::priority_tag<0> /*unused*/)
     {
         obj[key] = std::forward<ValueType>(v);
     }
 
+    /// append the decimal representation of @a i to @a out
+    static void append_index(string_t& out, std::size_t i)
+    {
+        std::array<char, 24> buf{};
+        std::size_t pos = buf.size();
+        do
+        {
+            buf[--pos] = static_cast<char>('0' + i % 10);
+            i /= 10;
+        }
+        while (i != 0);
+        out.append(buf.data() + pos, buf.size() - pos);
+    }
+
     /*!
     @param[in] reference_string  the reference string to the current value
     @param[in] value             the value to consider
@@ -1194,7 +1327,8 @@ class json_pointer
         // flatten `v`, whose path is `path`: primitives and empty containers
         // are added to the result right away; other containers get a frame
         auto& object = *result.m_data.m_value.object;
-        const auto enter = [&stack, &path, &object](const BasicJsonType & v)
+        typename BasicJsonType::object_t::iterator hint = object.end();
+        const auto enter = [&stack, &path, &object, &hint](const BasicJsonType & v)
         {
             switch (v.type())
             {
@@ -1203,7 +1337,7 @@ class json_pointer
                     if (v.m_data.m_value.array->empty())
                     {
                         // flatten empty array as null
-                        add_flattened(object, path, nullptr, detail::priority_tag<1> {});
+                        add_flattened(object, hint, path, nullptr, detail::priority_tag<2> {});
                     }
                     else
                     {
@@ -1217,7 +1351,7 @@ class json_pointer
                     if (v.m_data.m_value.object->empty())
                     {
                         // flatten empty object as null
-                        add_flattened(object, path, nullptr, detail::priority_tag<1> {});
+                        add_flattened(object, hint, path, nullptr, detail::priority_tag<2> {});
                     }
                     else
                     {
@@ -1237,7 +1371,7 @@ class json_pointer
                 default:
                 {
                     // add a primitive value with its reference string
-                    add_flattened(object, path, v, detail::priority_tag<1> {});
+                    add_flattened(object, hint, path, v, detail::priority_tag<2> {});
                     return;
                 }
             }
@@ -1265,7 +1399,8 @@ class json_pointer
 
                 // iterate array and use index as a reference string
                 ++stack.back().index;
-                detail::concat_into(path, '/', detail::to_string<string_t>(i));
+                path.push_back('/');
+                append_index(path, i);
                 enter(array[i]);
             }
             else
@@ -1279,7 +1414,8 @@ class json_pointer
 
                 // iterate object and use keys as reference string
                 ++stack.back().member;
-                detail::concat_into(path, '/', detail::escape(it->first));
+                path.push_back('/');
+                detail::escape_into(path, it->first);
                 enter(it->second);
             }
         }
@@ -1310,53 +1446,30 @@ class json_pointer
 
         BasicJsonType result;
 
-        // parse every key once; the reference tokens are kept for the second
-        // pass (the vectors are never resized, so the tree may refer to them)
-        std::vector<std::vector<string_t>> keys;
-        keys.reserve(value.m_data.m_value.object->size());
+        // check every key, and collect the prefixes that have a reference
+        // token 0 among their children; the values below them are arrays,
+        // all others are objects (see array_prefix_set)
+        array_prefix_set arrays;
+        const string_t* previous_key = nullptr;
         for (const auto& element : *value.m_data.m_value.object)
         {
-            json_pointer ptr(element.first);
-            keys.push_back(std::move(ptr.reference_tokens));
+            scan_key(element.first, previous_key, arrays);
+            previous_key = &element.first;
         }
 
-        // collect the pointer prefixes that have a reference token 0 among
-        // their children; the values below them are arrays, all others are
-        // objects (see prefix_tree)
-        prefix_tree tree;
-        // the number of the prefix with i + 1 tokens of the previous key; keys
-        // are often sorted, so most of the prefix is the same as for the
-        // previous key and need not be looked up again
-        std::vector<std::size_t> path;
-        const std::vector<string_t>* previous = nullptr;
-        // the number of the complete prefix (i.e., the key) of each key
-        std::vector<std::size_t> leaves;
-        leaves.reserve(keys.size());
-        for (const auto& tokens : keys)
-        {
-            std::size_t common = 0;
-            if (previous != nullptr)
-            {
-                const std::size_t limit = (std::min)(tokens.size(), previous->size());
-                while (common < limit && tokens[common] == (*previous)[common])
-                {
-                    ++common;
-                }
-            }
-            path.resize(common);
-            std::size_t id = common == 0 ? 0 : path[common - 1];
-            for (std::size_t i = common; i < tokens.size(); ++i)
-            {
-                id = tree.add_child(id, tokens[i]);
-                path.push_back(id);
-            }
-            leaves.push_back(id);
-            previous = &tokens;
-        }
+        // values[i] is the value after the first i reference tokens of the
+        // previous key, ends[i] the length of that prefix of the key. Keys are
+        // often sorted, so most of the path to the value of a key is the
+        // same as for the previous key and need not be looked up again. Only
+        // the values below the common prefix are dropped, and only they can
+        // move when a value is added: the ones above are not inside the
+        // container that grows.
+        std::vector<BasicJsonType*> values(1, &result);
+        std::vector<std::size_t> ends(1, 0);
+        const string_t* previous = nullptr;
+        string_t token;
 
         // iterate the JSON object values
-        std::vector<std::size_t> prefixes;
-        std::size_t key_index = 0;
         for (const auto& element : *value.m_data.m_value.object)
         {
             if (JSON_HEDLEY_UNLIKELY(!element.second.is_primitive()))
@@ -1364,21 +1477,100 @@ class json_pointer
                 JSON_THROW(detail::type_error::create(detail::exception_id::unflatten_value_not_primitive, "values in object must be primitive", &element.second));
             }
 
-            const auto& tokens = keys[key_index];
-            // number the prefixes of this key from its last one to the first
-            prefixes.resize(tokens.size());
-            for (std::size_t i = tokens.size(), id = leaves[key_index]; i > 0; --i)
+            const string_t& key = element.first;
+            const char* const data = key.data();
+
+            // the number of reference tokens the key has in common with the
+            // previous one
+            std::size_t depth = 0;
+            if (previous != nullptr)
             {
-                id = tree.parent[id];
-                prefixes[i - 1] = id;
+                const std::size_t same = common_prefix_length(data, previous->data(), (std::min)(key.size(), previous->size()));
+                depth = ends.size() - 1;
+                while (depth > 0 && (ends[depth] > same || (ends[depth] == same && same < key.size() && data[same] != '/')))
+                {
+                    --depth;
+                }
             }
-            ++key_index;
+            values.resize(depth + 1);
+            ends.resize(depth + 1);
+            previous = &key;
 
             // Assign the value to the reference pointed to by JSON pointer. Note
             // that if the JSON pointer is "" (i.e., points to the whole value),
-            // function get_and_create returns a reference to the result itself.
-            // An assignment will then create a primitive value.
-            get_and_create(result, tokens, prefixes, tree) = element.second;
+            // the value is the result itself. An assignment will then create a
+            // primitive value.
+            BasicJsonType* current = values.back();
+            for (std::size_t pos = ends.back(); pos < key.size();)
+            {
+                // the reference token between the slash at pos and the next
+                std::size_t end = key.find_first_of('/', pos + 1);
+                if (end == string_t::npos)
+                {
+                    end = key.size();
+                }
+                token.clear();
+                token.append(data + pos + 1, end - pos - 1);
+                if (token.find_first_of('~') != string_t::npos)
+                {
+                    detail::unescape(token);
+                }
+
+                switch (current->type())
+                {
+                    case detail::value_t::null:
+                    {
+                        if (arrays.contains(data, pos))
+                        {
+                            // some reference token below this position is 0, so the
+                            // value is an array
+                            current = &current->operator[](array_index<BasicJsonType>(token));
+                        }
+                        else
+                        {
+                            // start a new object otherwise
+                            current = &current->operator[](token);
+                        }
+                        break;
+                    }
+
+                    case detail::value_t::object:
+                    {
+                        // create an entry in the object
+                        current = &object_member(*current, token);
+                        break;
+                    }
+
+                    case detail::value_t::array:
+                    {
+                        // create an entry in the array
+                        current = &current->operator[](array_index<BasicJsonType>(token));
+                        break;
+                    }
+
+                    /*
+                    The following code is only reached if there exists a reference
+                    token _and_ the current value is primitive. In this case, we have
+                    an error situation, because primitive values may only occur as
+                    a single value; that is, with an empty list of reference tokens.
+                    */
+                    case detail::value_t::string:
+                    case detail::value_t::boolean:
+                    case detail::value_t::number_integer:
+                    case detail::value_t::number_unsigned:
+                    case detail::value_t::number_float:
+                    case detail::value_t::binary:
+                    case detail::value_t::discarded:
+                    default:
+                        JSON_THROW(detail::type_error::create(detail::exception_id::unflatten_invalid_value, "invalid value to unflatten", &result));
+                }
+
+                values.push_back(current);
+                ends.push_back(end);
+                pos = end;
+            }
+
+            *current = element.second;
         }
 
         return result;

include/nlohmann/detail/output/binary_writer.hpp

@@ -207,10 +207,6 @@ class binary_writer
             {
                 if (j.m_data.m_value.number_integer >= 0)
                 {
-                    if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::uint64_t>(j.m_data.m_value.number_integer)))
-                    {
-                        throw_integer_out_of_range(j, "CBOR", "[-2^64, 2^64-1]");
-                    }
                     // CBOR does not differentiate between positive signed
                     // integers and unsigned integers
                     write_cbor_head(0x00, static_cast<std::uint64_t>(j.m_data.m_value.number_integer));
@@ -218,10 +214,6 @@ class binary_writer
                 else
                 {
                     // a negative integer n is encoded as -1 - n
-                    if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::uint64_t>(-1 - j.m_data.m_value.number_integer)))
-                    {
-                        throw_integer_out_of_range(j, "CBOR", "[-2^64, 2^64-1]");
-                    }
                     write_cbor_head(0x20, static_cast<std::uint64_t>(-1 - j.m_data.m_value.number_integer));
                 }
                 break;
@@ -229,10 +221,6 @@ class binary_writer
 
             case value_t::number_unsigned:
             {
-                if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::uint64_t>(j.m_data.m_value.number_unsigned)))
-                {
-                    throw_integer_out_of_range(j, "CBOR", "uint64");
-                }
                 write_cbor_head(0x00, conditional_static_cast<std::uint64_t>(j.m_data.m_value.number_unsigned));
                 break;
             }
@@ -435,20 +423,12 @@ class binary_writer
             {
                 if (j.m_data.m_value.number_integer >= 0)
                 {
-                    if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::uint64_t>(j.m_data.m_value.number_integer)))
-                    {
-                        throw_integer_out_of_range(j, "MessagePack", "[-2^63, 2^64-1]");
-                    }
                     // MessagePack does not differentiate between positive
                     // signed integers and unsigned integers.
                     write_msgpack_unsigned(static_cast<std::uint64_t>(j.m_data.m_value.number_integer));
                 }
                 else
                 {
-                    if (JSON_HEDLEY_UNLIKELY(!integer_fits_min<std::int64_t>(j.m_data.m_value.number_integer)))
-                    {
-                        throw_integer_out_of_range(j, "MessagePack", "[-2^63, 2^64-1]");
-                    }
                     if (j.m_data.m_value.number_integer >= -32)
                     {
                         // negative fixnum
@@ -487,10 +467,6 @@ class binary_writer
 
             case value_t::number_unsigned:
             {
-                if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::uint64_t>(j.m_data.m_value.number_unsigned)))
-                {
-                    throw_integer_out_of_range(j, "MessagePack", "uint64");
-                }
                 write_msgpack_unsigned(static_cast<std::uint64_t>(j.m_data.m_value.number_unsigned));
                 break;
             }
@@ -848,133 +824,6 @@ class binary_writer
         JSON_THROW(type_error::create(exception_id::discarded_value_used, concat("cannot serialize discarded value to ", format_name), &j));
     }
 
-    /*!
-    @brief whether integer @a n does not exceed the maximum of @a TargetType
-
-    The binary formats store integers in at most 64 bits, but number_integer_t
-    and number_unsigned_t may be wider (e.g., __int128). The range of the
-    integer is taken from std::numeric_limits, so a number type whose range
-    fits into @a TargetType is never checked: the function is then constant
-    true, and the default int64_t/uint64_t types pay nothing.
-    */
-    template<typename TargetType, typename NumberType>
-    static constexpr bool integer_fits_max(const NumberType n) noexcept
-    {
-        return integer_fits_max<TargetType>(n, std::integral_constant < bool,
-                                            (std::numeric_limits<NumberType>::digits > std::numeric_limits<TargetType>::digits) > ());
-    }
-
-    template<typename TargetType, typename NumberType>
-    static constexpr bool integer_fits_max(const NumberType /*unused*/, std::false_type /*may_exceed*/) noexcept
-    {
-        return true;
-    }
-
-    template<typename TargetType, typename NumberType>
-    static constexpr bool integer_fits_max(const NumberType n, std::true_type /*may_exceed*/) noexcept
-    {
-        // NumberType has more digits than TargetType, so it can hold its maximum
-        return n <= static_cast<NumberType>((std::numeric_limits<TargetType>::max)());
-    }
-
-    /*!
-    @brief whether integer @a n is not below the minimum of @a TargetType;
-           constant true if NumberType cannot hold such a value
-    */
-    template<typename TargetType, typename NumberType>
-    static constexpr bool integer_fits_min(const NumberType n) noexcept
-    {
-        return integer_fits_min<TargetType>(n, std::integral_constant < bool, std::numeric_limits<NumberType>::is_signed &&
-                                            (!std::numeric_limits<TargetType>::is_signed || std::numeric_limits<NumberType>::digits > std::numeric_limits<TargetType>::digits) > ());
-    }
-
-    template<typename TargetType, typename NumberType>
-    static constexpr bool integer_fits_min(const NumberType /*unused*/, std::false_type /*may_fall_below*/) noexcept
-    {
-        return true;
-    }
-
-    template<typename TargetType, typename NumberType>
-    static constexpr bool integer_fits_min(const NumberType n, std::true_type /*may_fall_below*/) noexcept
-    {
-        // NumberType is signed and either TargetType is unsigned (minimum 0)
-        // or NumberType has more digits, so it can hold the minimum
-        return n >= static_cast<NumberType>((std::numeric_limits<TargetType>::min)());
-    }
-
-    /*!
-    @brief whether integer @a n lies in the range of @a TargetType
-    */
-    template<typename TargetType, typename NumberType>
-    static constexpr bool integer_fits(const NumberType n) noexcept
-    {
-        return integer_fits_min<TargetType>(n) && integer_fits_max<TargetType>(n);
-    }
-
-    /*!
-    @brief the decimal digits of the non-negative integer @a n
-    */
-    template<typename NumberType>
-    static std::string integer_to_string(NumberType n, std::false_type /*is_signed*/)
-    {
-        std::string result;
-        do
-        {
-            result.push_back(static_cast<char>('0' + (n % 10)));
-            n = static_cast<NumberType>(n / 10);
-        }
-        while (n != 0);
-        std::reverse(result.begin(), result.end());
-        return result;
-    }
-
-    /*!
-    @brief the decimal digits of the integer @a n, with a leading '-' if it
-           is negative
-    */
-    template<typename NumberType>
-    static std::string integer_to_string(NumberType n, std::true_type /*is_signed*/)
-    {
-        // the remainder of a negative n is not positive; each digit is
-        // negated on its own, because negating the minimum of NumberType
-        // as a whole would overflow
-        const bool negative = n < 0;
-        std::string result;
-        do
-        {
-            const auto remainder = n % 10;
-            result.push_back(static_cast<char>('0' + (remainder < 0 ? -remainder : remainder)));
-            n = static_cast<NumberType>(n / 10);
-        }
-        while (n != 0);
-        if (negative)
-        {
-            result.push_back('-');
-        }
-        std::reverse(result.begin(), result.end());
-        return result;
-    }
-
-    /*!
-    @brief throws because the integer @a j is too large for @a format_name
-    @throw out_of_range.407 always
-    */
-    JSON_HEDLEY_NO_RETURN static void throw_integer_out_of_range(const BasicJsonType& j, const char* format_name, const char* range)
-    {
-        // neither std::to_string(), which has no overload for integer types
-        // wider than 64 bits, nor dump(), which would also instantiate the
-        // serializer for objects and fail for key types it cannot write
-        using integer_limits = std::numeric_limits<typename BasicJsonType::number_integer_t>;
-        using unsigned_limits = std::numeric_limits<typename BasicJsonType::number_unsigned_t>;
-        const std::string number = j.is_number_unsigned()
-                                   ? integer_to_string(j.m_data.m_value.number_unsigned, std::integral_constant<bool, unsigned_limits::is_signed>())
-                                   : integer_to_string(j.m_data.m_value.number_integer, std::integral_constant<bool, integer_limits::is_signed>());
-        static_cast<void>(number); // unused when JSON_NOEXCEPTION is defined
-        static_cast<void>(format_name);
-        static_cast<void>(range);
-        JSON_THROW(out_of_range::create(exception_id::integer_too_large, concat("integer number ", number, " cannot be represented by ", format_name, " as it does not fit ", range), &j));
-    }
-
     void write_msgpack_array_prefix(const std::size_t N, const BasicJsonType& j)
     {
         const auto n = to_msgpack_length(N, j);
@@ -1731,8 +1580,6 @@ class binary_writer
             is neither an object nor an array
     @throw out_of_range.415 if @a j is binary with a subtype that does not fit
            into a byte, before anything is written
-    @throw out_of_range.407 if @a j is an integer that does not fit the 64 bits
-           of a BSON integer, before anything is written
     @throw type_error.316 if @a j is a string that is not valid UTF-8, before
            anything is written
     @throw type_error.321 if @a j is discarded
@@ -1751,17 +1598,9 @@ class binary_writer
                 return 8ul;
 
             case value_t::number_integer:
-                if (JSON_HEDLEY_UNLIKELY(!integer_fits<std::int64_t>(j.m_data.m_value.number_integer)))
-                {
-                    throw_integer_out_of_range(j, "BSON", "int64");
-                }
-                return calc_bson_integer_size(static_cast<std::int64_t>(j.m_data.m_value.number_integer));
+                return calc_bson_integer_size(j.m_data.m_value.number_integer);
 
             case value_t::number_unsigned:
-                if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::uint64_t>(j.m_data.m_value.number_unsigned)))
-                {
-                    throw_integer_out_of_range(j, "BSON", "uint64");
-                }
                 return calc_bson_unsigned_size(conditional_static_cast<std::uint64_t>(j.m_data.m_value.number_unsigned));
 
             case value_t::string:
@@ -1800,9 +1639,8 @@ class binary_writer
             case value_t::number_float:
                 return write_bson_double(name, j.m_data.m_value.number_float);
 
-            // calc_bson_value_size() checked that integers fit 64 bits
             case value_t::number_integer:
-                return write_bson_integer(name, static_cast<std::int64_t>(j.m_data.m_value.number_integer));
+                return write_bson_integer(name, j.m_data.m_value.number_integer);
 
             case value_t::number_unsigned:
                 return write_bson_unsigned(name, conditional_static_cast<std::uint64_t>(j.m_data.m_value.number_unsigned));
@@ -2309,7 +2147,7 @@ class binary_writer
         {
             return 'L';
         }
-        if (use_bjdata && std::is_unsigned<NumberType>::value && integer_fits_max<std::uint64_t>(n))
+        if (use_bjdata && std::is_unsigned<NumberType>::value)
         {
             return 'M';
         }
@@ -2434,16 +2272,14 @@ class binary_writer
     @brief checks whether a JSON number fits into @a TargetType
     @param[in] el a JSON number of either the signed or unsigned integer kind
     @return whether @a el's value can be represented by @a TargetType without
-            wrapping, regardless of which of the two kinds it is stored as;
-            false for a value that does not even fit the 64-bit type it is
-            read as (possible for number types wider than 64 bits)
+            wrapping, regardless of which of the two kinds it is stored as
     */
     template<typename TargetType>
     static bool bjdata_ndarray_value_in_range(const BasicJsonType& el)
     {
         return el.is_number_unsigned()
-               ? integer_fits_max<std::uint64_t>(el.m_data.m_value.number_unsigned) && value_in_range_of<TargetType>(el.template get<std::uint64_t>())
-               : integer_fits<std::int64_t>(el.m_data.m_value.number_integer) && value_in_range_of<TargetType>(el.template get<std::int64_t>());
+               ? value_in_range_of<TargetType>(el.template get<std::uint64_t>())
+               : value_in_range_of<TargetType>(el.template get<std::int64_t>());
     }
 
     /*!
@@ -2675,11 +2511,10 @@ class binary_writer
         for (const auto& el : dims)
         {
             // a dimension is read as an unsigned value below, so anything that
-            // is not a non-negative integer in the range of std::uint64_t is
-            // rejected: a non-integer entry would pun unrelated bytes as the
-            // dimension, and a negative or wider one would wrap into a
-            // nonsensical length
-            if (!el.is_number_integer() || !bjdata_ndarray_value_in_range<std::uint64_t>(el))
+            // is not a non-negative integer is rejected: a non-integer entry
+            // would pun unrelated bytes as the dimension, and a negative one
+            // would wrap into a nonsensical length
+            if (!el.is_number_integer() || (!el.is_number_unsigned() && el.template get<std::int64_t>() < 0))
             {
                 return true;
             }
@@ -2811,9 +2646,9 @@ class binary_writer
 
             case value_t::number_unsigned:
             {
-                if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::int64_t>(j.m_data.m_value.number_unsigned)))
+                if (j.m_data.m_value.number_unsigned > static_cast<typename BasicJsonType::number_unsigned_t>((std::numeric_limits<std::int64_t>::max)()))
                 {
-                    throw_integer_out_of_range(j, "BON8", "int64");
+                    JSON_THROW(out_of_range::create(exception_id::integer_too_large, concat("integer number ", std::to_string(j.m_data.m_value.number_unsigned), " cannot be represented by BON8 as it does not fit int64"), &j));
                 }
                 write_bon8_integer(static_cast<std::int64_t>(j.m_data.m_value.number_unsigned));
                 string_open = false;
@@ -2822,10 +2657,6 @@ class binary_writer
 
             case value_t::number_integer:
             {
-                if (JSON_HEDLEY_UNLIKELY(!integer_fits<std::int64_t>(j.m_data.m_value.number_integer)))
-                {
-                    throw_integer_out_of_range(j, "BON8", "int64");
-                }
                 write_bon8_integer(static_cast<std::int64_t>(j.m_data.m_value.number_integer));
                 string_open = false;
                 break;

include/nlohmann/detail/output/serializer.hpp

@@ -1525,7 +1525,7 @@ class serializer
         // without leading zeros
         if (abs_value < 10000000000000000u)
         {
-            const auto value = static_cast<std::uint64_t>(abs_value); // fits: abs_value < 10^16
+            const std::uint64_t value = abs_value;
             const std::uint64_t upper = value / 100000000u;
             const std::uint64_t first = dtoa_impl::eight_digit_bytes(upper != 0 ? upper : value);
             const auto leading = static_cast<unsigned>(count_leading_zeros(first) / 8); // (first is not 0)

include/nlohmann/detail/value_t.hpp

@@ -14,6 +14,7 @@
 #include <cstdint> // uint8_t
 #include <limits> // numeric_limits
 #include <string> // string
+#include <type_traits> // is_signed
 
 #include <nlohmann/detail/macro_scope.hpp>
 #if JSON_HAS_THREE_WAY_COMPARISON
@@ -174,16 +175,13 @@ FloatType compare_integer_with_float(const IntegerType i, const FloatType f) noe
 
     // values of IntegerType lie in [-bound, bound) when signed and in
     // [0, bound) when unsigned; digits excludes the sign bit, so bound is a
-    // power of two that the float represents exactly; the signedness comes
-    // from numeric_limits as well, because std::is_signed is false for class
-    // types such as 128-bit or multiprecision integers
-    using limits = std::numeric_limits<IntegerType>;
-    const FloatType bound = std::ldexp(static_cast<FloatType>(1), limits::digits);
+    // power of two that the float represents exactly
+    const FloatType bound = std::ldexp(static_cast<FloatType>(1), std::numeric_limits<IntegerType>::digits);
     if (f >= bound)
     {
         return ordered(-1);
     }
-    if (limits::is_signed ? (f < -bound) : (f < static_cast<FloatType>(0)))
+    if (std::is_signed<IntegerType>::value ? (f < -bound) : (f < static_cast<FloatType>(0)))
     {
         return ordered(1);
     }

include/nlohmann/json.hpp

@@ -7066,6 +7066,125 @@ public:
             detail::throw_patch_not_an_array(json_patch);
         }
 
+        // the usual patch operations "add", "remove", "replace" and "test" on
+        // a pointer without escapes that denotes existing array elements and
+        // object members (as "add" does for the parent of the new value), as
+        // a diff creates them: apply the operation and return true; return
+        // false, with nothing changed, for all others
+        string_t token;
+        const string_t value_key = "value";
+        const auto try_plain = [&result, &token, &value_key](patch_operations operation, const string_t& path, const basic_json & val)
+        {
+            if (path.empty() || path.find_first_of('~') != string_t::npos)
+            {
+                return false;
+            }
+
+            // a missing "value" is reported below
+            const auto value_it = val.m_data.m_value.object->find(value_key);
+            const basic_json& value = value_it != val.m_data.m_value.object->end() ? value_it->second : val;
+            if (operation != patch_operations::remove && value_it == val.m_data.m_value.object->end())
+            {
+                return false;
+            }
+
+            switch (operation)
+            {
+                case patch_operations::replace:
+                {
+                    basic_json* const target = json_pointer::resolve_plain(&result, path, path.size(), token);
+                    if (target == nullptr)
+                    {
+                        return false;
+                    }
+                    *target = value;
+                    return true;
+                }
+
+                case patch_operations::test:
+                {
+                    const basic_json* const target = json_pointer::resolve_plain(&result, path, path.size(), token);
+                    // a failing test is reported below
+                    return target != nullptr && *target == value;
+                }
+
+                case patch_operations::add:
+                case patch_operations::remove:
+                {
+                    // the parent of the pointer and its last reference token
+                    std::size_t slash = path.size();
+                    while (slash > 0 && path[slash - 1] != '/')
+                    {
+                        --slash;
+                    }
+                    if (slash == 0)
+                    {
+                        return false;
+                    }
+                    --slash;
+
+                    basic_json* const parent = json_pointer::resolve_plain(&result, path, slash, token);
+                    if (parent == nullptr)
+                    {
+                        return false;
+                    }
+                    token.clear();
+                    token.append(path.data() + slash + 1, path.size() - slash - 1);
+
+                    if (operation == patch_operations::add)
+                    {
+                        if (parent->is_object())
+                        {
+                            (*parent)[token] = value;
+                            return true;
+                        }
+                        if (parent->is_array())
+                        {
+                            typename basic_json::size_type idx{};
+                            if (token == "-")
+                            {
+                                parent->push_back(value);
+                                return true;
+                            }
+                            if (json_pointer::template parse_array_index<basic_json_t>(token, idx) == json_pointer::array_index_status::ok && idx <= parent->size())
+                            {
+                                parent->insert(parent->begin() + static_cast<difference_type>(idx), value);
+                                return true;
+                            }
+                        }
+                        return false;
+                    }
+
+                    if (parent->is_object())
+                    {
+                        const auto it = parent->find(token);
+                        if (it == parent->end())
+                        {
+                            return false;
+                        }
+                        parent->erase(it);
+                        return true;
+                    }
+                    if (parent->is_array())
+                    {
+                        typename basic_json::size_type idx{};
+                        if (json_pointer::template parse_array_index<basic_json_t>(token, idx) == json_pointer::array_index_status::ok && idx < parent->size())
+                        {
+                            parent->erase(idx);
+                            return true;
+                        }
+                    }
+                    return false;
+                }
+
+                case patch_operations::move:
+                case patch_operations::copy:
+                case patch_operations::invalid:
+                default:
+                    return false;
+            }
+        };
+
         // iterate and apply the operations
         for (const auto& val : json_patch)
         {
@@ -7102,9 +7221,19 @@ public:
             // collect mandatory members
             const string_t& op = *get_value("op", "op", true).m_data.m_value.string;
             const string_t& path = *get_value(op, "path", true).m_data.m_value.string;
+            const patch_operations operation = get_op(op);
+
+            // the usual operations on a plain pointer are applied without
+            // building the pointer's reference tokens; anything else (and
+            // every error) is handled below
+            if (try_plain(operation, path, val))
+            {
+                continue;
+            }
+
             json_pointer ptr(path);
 
-            switch (get_op(op))
+            switch (operation)
             {
                 case patch_operations::add:
                 {
@@ -7212,7 +7341,8 @@ public:
     {
         // the patch
         basic_json result(value_t::array);
-        diff_recursively(result, source, target, path, 0);
+        string_t buffer = path;
+        diff_recursively(result, source, target, buffer, 0);
         return result;
     }
 
@@ -7256,31 +7386,69 @@ public:
     // function it appears in. In diff_recursively, which is on the call stack
     // once per nesting level, that made every level cost kilobytes of stack.
 
+    /// @brief append an operation @a op for @a path to @a result, with @a value if it is not null
+    static void diff_operation(basic_json& result, const char* op, const string_t& path, const basic_json* value)
+    {
+        basic_json entry(value_t::object);
+        entry["op"] = op;
+        entry["path"] = path;
+        if (value != nullptr)
+        {
+            entry["value"] = *value;
+        }
+        result.push_back(std::move(entry));
+    }
+
     /// @brief append a "replace" operation for @a path with @a value to @a result
     static void diff_replace(basic_json& result, const string_t& path, const basic_json& value)
     {
-        result.push_back(
-        {
-            {"op", "replace"}, {"path", path}, {"value", value}
-        });
+        diff_operation(result, "replace", path, &value);
     }
 
     /// @brief append a "remove" operation for @a path to @a result
     static void diff_remove(basic_json& result, const string_t& path)
     {
-        result.push_back(object(
-        {
-            {"op", "remove"}, {"path", path}
-        }));
+        diff_operation(result, "remove", path, nullptr);
     }
 
     /// @brief append an "add" operation for @a path with @a value to @a result
     static void diff_add(basic_json& result, const string_t& path, const basic_json& value)
     {
-        result.push_back(
-        {
-            {"op", "add"}, {"path", path}, {"value", value}
-        });
+        diff_operation(result, "add", path, &value);
+    }
+
+    /// @brief append "/" and the decimal representation of @a index to @a path
+    static void diff_append_index(string_t& path, std::size_t index)
+    {
+        path.push_back('/');
+        json_pointer::append_index(path, index);
+    }
+
+    /// @brief append "/" and the escaped @a key to @a path
+    static void diff_append_key(string_t& path, const string_t& key)
+    {
+        path.push_back('/');
+        detail::escape_into(path, key);
+    }
+
+    /// @brief the path of the member @a key of the value at @a path
+    static string_t diff_key_path(const string_t& path, const string_t& key)
+    {
+        string_t result;
+        result.reserve(path.size() + key.size() + 1);
+        result.append(path.data(), path.size());
+        diff_append_key(result, key);
+        return result;
+    }
+
+    /// @brief the path of the element @a index of the value at @a path
+    static string_t diff_index_path(const string_t& path, const std::size_t index)
+    {
+        string_t result;
+        result.reserve(path.size() + 8);
+        result.append(path.data(), path.size());
+        diff_append_index(result, index);
+        return result;
     }
 
     /// @brief append the "remove" operations for the elements of array
@@ -7293,13 +7461,18 @@ public:
         // in that order avoids the quadratic reinsertion done before
         for (std::size_t j = source.size(); j > index; --j)
         {
-            diff_remove(result, detail::concat<string_t>(path, '/', detail::to_string<string_t>(j - 1)));
+            diff_remove(result, diff_index_path(path, j - 1));
         }
 
         // add other remaining elements
-        for (std::size_t i = source.size(); i < target.size(); ++i)
+        if (source.size() < target.size())
         {
-            diff_add(result, detail::concat<string_t>(path, "/-"), target[i]);
+            string_t tail_path = path;
+            tail_path.append("/-", 2);
+            for (std::size_t i = source.size(); i < target.size(); ++i)
+            {
+                diff_add(result, tail_path, target[i]);
+            }
         }
     }
 
@@ -7308,7 +7481,7 @@ public:
     {
         for (const auto& it : added)
         {
-            diff_add(result, detail::concat<string_t>(path, '/', detail::escape(it.key())), it.value());
+            diff_add(result, diff_key_path(path, it.key()), it.value());
         }
     }
 
@@ -7329,10 +7502,25 @@ public:
                                  std::vector<const_iterator>& added)
     {
         // first pass: one lookup per source key
+        // Objects of the same size usually have the same keys, in the same
+        // order: walk them together as long as that holds, which takes no lookups.
         target_values.reserve(source.size());
         std::size_t common = 0;
-        for (auto it = source.cbegin(); it != source.cend(); ++it)
+        bool together = source.size() == target.size();
+        for (auto it = source.cbegin(), partner = target.cbegin(); it != source.cend(); ++it)
         {
+            if (together)
+            {
+                if (it.key() == partner.key())
+                {
+                    target_values.push_back(&partner.value());
+                    ++common;
+                    ++partner;
+                    continue;
+                }
+                together = false;
+            }
+
             const auto found = target.find(it.key());
             if (found != target.cend())
             {
@@ -7345,6 +7533,12 @@ public:
             }
         }
 
+        // the same keys in the same order
+        if (together)
+        {
+            return true;
+        }
+
         // second pass: find the keys only target has. An object type that
         // keeps its members in insertion order, such as the one backing
         // ordered_json, also needs the common keys to be in the same order
@@ -7410,6 +7604,13 @@ public:
             return true;
         }
 
+        // equal objects, whose members are in a different order, need no
+        // operations
+        if (source == target)
+        {
+            return false;
+        }
+
         // slow path: the common keys are in a different relative
         // order in source and target (only possible for a
         // reorderable object_t like ordered_map). Building a
@@ -7426,7 +7627,7 @@ public:
         // it moves it to the end, fixing its position.
         for (auto it = source.cbegin(); it != source.cend(); ++it)
         {
-            diff_remove(result, detail::concat<string_t>(path, '/', detail::escape(it.key())));
+            diff_remove(result, diff_key_path(path, it.key()));
         }
 
         // add every key that is either common (just removed
@@ -7435,7 +7636,7 @@ public:
         // target exactly
         for (auto it = target.cbegin(); it != target.cend(); ++it)
         {
-            diff_add(result, detail::concat<string_t>(path, '/', detail::escape(it.key())), it.value());
+            diff_add(result, diff_key_path(path, it.key()), it.value());
         }
         return false;
     }
@@ -7444,6 +7645,9 @@ public:
     @brief @ref diff, for values at nesting level @a depth, appending the
            operations to @a result
 
+    @a path is the path of the values; it is a buffer that is extended and
+    shortened again below, and is the same when the function returns.
+
     Diffing two arrays or objects calls this function again, once per nesting
     level, so values nested deeply enough used to exhaust the call stack and
     terminate the process. The descent is bounded here: once @ref
@@ -7451,7 +7655,7 @@ public:
     diff_iteratively diffs what is left without the call stack.
     */
     static void diff_recursively(basic_json& result, const basic_json& source, const basic_json& target,
-                                 const string_t& path, const std::size_t depth)
+                                 string_t& path, const std::size_t depth)
     {
         // if the values are the same, there is nothing to do
         if (source == target)
@@ -7463,9 +7667,9 @@ public:
     }
 
     /// @brief @ref diff_recursively for values known to be unequal; callers
-    ///        compare first so they build no path for equal ones
+    ///        compare first so they extend the path only for unequal ones
     static void diff_unequal(basic_json& result, const basic_json& source, const basic_json& target,
-                             const string_t& path, const std::size_t depth)
+                             string_t& path, const std::size_t depth)
     {
         if (JSON_HEDLEY_UNLIKELY(depth >= detail::recursion_depth_limit()))
         {
@@ -7484,6 +7688,8 @@ public:
         {
             case value_t::array:
             {
+                const std::size_t path_length = path.size();
+
                 // first pass: traverse common elements
                 std::size_t i = 0;
                 while (i < source.size() && i < target.size())
@@ -7491,7 +7697,9 @@ public:
                     // recursive call to compare array values at index i
                     if (!(source[i] == target[i]))
                     {
-                        diff_unequal(result, source[i], target[i], detail::concat<string_t>(path, '/', detail::to_string<string_t>(i)), depth + 1);
+                        diff_append_index(path, i);
+                        diff_unequal(result, source[i], target[i], path, depth + 1);
+                        path.resize(path_length);
                     }
                     ++i;
                 }
@@ -7513,6 +7721,7 @@ public:
                     // lookup is needed. Deleted keys are interleaved here
                     // too, in source's original order, to match the
                     // historical (pre-reordering-aware) output order.
+                    const std::size_t path_length = path.size();
                     std::size_t i = 0;
                     for (auto it = source.cbegin(); it != source.cend(); ++it, ++i)
                     {
@@ -7520,13 +7729,15 @@ public:
                         {
                             if (!(it.value() == *target_values[i]))
                             {
-                                diff_unequal(result, it.value(), *target_values[i], detail::concat<string_t>(path, '/', detail::escape(it.key())), depth + 1);
+                                diff_append_key(path, it.key());
+                                diff_unequal(result, it.value(), *target_values[i], path, depth + 1);
+                                path.resize(path_length);
                             }
                         }
                         else
                         {
                             // found a key that is not in target -> remove it
-                            diff_remove(result, detail::concat<string_t>(path, '/', detail::escape(it.key())));
+                            diff_remove(result, diff_key_path(path, it.key()));
                         }
                     }
 
@@ -7657,7 +7868,7 @@ public:
                 if (i < source_array.size() && i < target_array.size())
                 {
                     ++stack.back().index;
-                    detail::concat_into(current_path, '/', detail::to_string<string_t>(i));
+                    diff_append_index(current_path, i);
                     enter(source_array[i], target_array[i]);
                     if (stack.size() == depth)
                     {
@@ -7680,7 +7891,7 @@ public:
                     if (target_value_ptr != nullptr)
                     {
                         const basic_json& target_value = *target_value_ptr;
-                        detail::concat_into(current_path, '/', detail::escape(it.key()));
+                        diff_append_key(current_path, it.key());
                         enter(it.value(), target_value);
                         if (stack.size() == depth)
                         {
@@ -7690,7 +7901,7 @@ public:
                     else
                     {
                         // found a key that is not in target -> remove it
-                        diff_remove(result, detail::concat<string_t>(current_path, '/', detail::escape(it.key())));
+                        diff_remove(result, diff_key_path(current_path, it.key()));
                     }
                     continue;
                 }

tests/src/unit-custom-array-type.cpp

@@ -9,9 +9,7 @@
 #include "doctest_compatibility.h"
 
 #include <nlohmann/json.hpp>
-using nlohmann::json;
 
-#include <cstdint>
 #include <deque>
 #include <map>
 #include <memory>
@@ -166,47 +164,3 @@ TEST_CASE("array type without at()")
     CHECK(j.at(no_at_json::json_pointer("/1")) == 2);
     CHECK_THROWS_AS(j.at(no_at_json::json_pointer("/3")), no_at_json::out_of_range);
 }
-
-TEST_CASE("regression test #5476 - array type without reserve()")
-{
-    // the capacity reserved for definite-length arrays must not require the
-    // array type to have a reserve() member function
-    SECTION("std::deque")
-    {
-        const auto j = deque_json::parse(R"({"a":[1,[2,3]],"b":[]})");
-        CHECK(j.dump() == R"({"a":[1,[2,3]],"b":[]})");
-
-        // the binary formats pass a definite length to start_array()
-        CHECK(deque_json::from_cbor(deque_json::to_cbor(j)) == j);
-        CHECK(deque_json::from_msgpack(deque_json::to_msgpack(j)) == j);
-        CHECK(deque_json::from_bon8(deque_json::to_bon8(j)) == j);
-
-        // parse() instantiates the callback parser as well, which reserves too
-        const auto with_callback = deque_json::parse(R"([1,2,3])", [](int /*depth*/, deque_json::parse_event_t /*event*/, deque_json& /*parsed*/) noexcept
-        {
-            return true;
-        });
-        CHECK(with_callback == deque_json({1, 2, 3}));
-    }
-
-    SECTION("std::vector still reserves")
-    {
-        json array = json::array();
-        for (int i = 0; i < 100; ++i)
-        {
-            array.push_back(i);
-        }
-
-        const auto j = json::from_cbor(json::to_cbor(array));
-        CHECK(j == array);
-        CHECK(j.get_ref<const json::array_t&>().capacity() >= 100);
-    }
-
-    SECTION("the reservation stays capped")
-    {
-        // CBOR array announcing 2^32-1 elements, but truncated right after the
-        // header: the input must be rejected without reserving that capacity
-        const std::vector<std::uint8_t> truncated = {0x9A, 0xFF, 0xFF, 0xFF, 0xFF};
-        CHECK(json::from_cbor(truncated, true, false).is_discarded());
-    }
-}

tests/src/unit-custom-number-types.cpp

@@ -1,367 +0,0 @@
-//     __ _____ _____ _____
-//  __|  |   __|     |   | |  JSON for Modern C++ (supporting code)
-// |  |  |__   |  |  | | | |  version 3.12.0
-// |_____|_____|_____|_|___|  https://github.com/nlohmann/json
-//
-// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
-// SPDX-License-Identifier: MIT
-
-#include "doctest_compatibility.h"
-
-#include <nlohmann/json.hpp>
-
-#include <cstdint>
-#include <limits>
-#include <map>
-#include <string>
-#include <type_traits>
-#include <vector>
-
-#if JSON_HAS_THREE_WAY_COMPARISON
-    #include <compare>
-#endif
-
-namespace custom_number_types
-{
-
-// a signed integer of class type: std::is_signed is only true for arithmetic
-// types, so the library has to take the signedness from std::numeric_limits,
-// as it does for 128-bit and multiprecision integer classes such as
-// absl::int128 or boost::multiprecision::cpp_int
-class class_int
-{
-  public:
-    // trivial, like the 128-bit integer classes: the value is stored in a union
-    class_int() = default;
-
-    // implicit from built-in integers and explicit from floats, like absl::int128;
-    // non-template overloads (as absl::int128 has) because clang 3.4 and 3.5
-    // reject basic_json's defaulted data() for a number type with template
-    // constructors
-    class_int(int v) : value(v) {} // NOLINT(google-explicit-constructor,hicpp-explicit-conversions)
-    class_int(unsigned v) : value(v) {} // NOLINT(google-explicit-constructor,hicpp-explicit-conversions)
-    class_int(long v) : value(v) {} // NOLINT(google-explicit-constructor,hicpp-explicit-conversions)
-    class_int(unsigned long v) : value(static_cast<std::int64_t>(v)) {} // NOLINT(google-explicit-constructor,hicpp-explicit-conversions)
-    class_int(long long v) : value(v) {} // NOLINT(google-explicit-constructor,hicpp-explicit-conversions)
-    class_int(unsigned long long v) : value(static_cast<std::int64_t>(v)) {} // NOLINT(google-explicit-constructor,hicpp-explicit-conversions)
-    explicit class_int(double v) : value(static_cast<std::int64_t>(v)) {}
-
-    template<typename T, typename std::enable_if<std::is_arithmetic<T>::value, int>::type = 0>
-    explicit operator T() const
-    {
-        return static_cast<T>(value);
-    }
-
-    friend bool operator==(class_int lhs, class_int rhs)
-    {
-        return lhs.value == rhs.value;
-    }
-    friend bool operator!=(class_int lhs, class_int rhs)
-    {
-        return lhs.value != rhs.value;
-    }
-    friend bool operator<(class_int lhs, class_int rhs)
-    {
-        return lhs.value < rhs.value;
-    }
-#if JSON_HAS_THREE_WAY_COMPARISON
-    friend std::strong_ordering operator<=>(class_int lhs, class_int rhs) // *NOPAD*
-    {
-        return lhs.value <=> rhs.value; // *NOPAD*
-    }
-#endif
-
-  private:
-    std::int64_t value;
-};
-
-} // namespace custom_number_types
-
-using custom_number_types::class_int;
-
-namespace std
-{
-// only the members the library uses
-template<>
-class numeric_limits<class_int>
-{
-  public:
-    static constexpr bool is_signed = true;
-    static constexpr int digits = std::numeric_limits<std::int64_t>::digits;
-};
-} // namespace std
-
-namespace
-{
-
-using class_int_json = nlohmann::basic_json<std::map, std::vector, std::string, bool, class_int, std::uint64_t, double>;
-
-class_int_json make_class_int(std::int64_t v)
-{
-    class_int_json j(class_int_json::value_t::number_integer);
-    j.get_ref<class_int&>() = class_int(v);
-    return j;
-}
-
-} // namespace
-
-// the serializer cannot print an integer of class type, so doctest must not
-// try when an assertion fails
-namespace doctest
-{
-template<>
-struct StringMaker<class_int_json>
-{
-    static String convert(const class_int_json& j)
-    {
-        return j.type_name();
-    }
-};
-} // namespace doctest
-
-// __int128 as number type needs the std::numeric_limits (for the range checks)
-// and std::is_integral (for the serializer) specializations, which libc++
-// always provides and libstdc++ only outside strict ISO modes; the MSVC
-// standard library has none
-#if defined(__SIZEOF_INT128__) && (defined(_LIBCPP_VERSION) || defined(__GLIBCXX_TYPE_INT_N_0))
-    #define JSON_TEST_INT128_NUMBER_TYPES 1
-#else
-    #define JSON_TEST_INT128_NUMBER_TYPES 0
-#endif
-
-#if JSON_TEST_INT128_NUMBER_TYPES
-namespace
-{
-
-// __extension__ keeps -Wpedantic from flagging the non-standard type
-__extension__ typedef __int128 int128; // NOLINT(modernize-use-using)
-__extension__ typedef unsigned __int128 uint128; // NOLINT(modernize-use-using)
-
-static_assert(std::numeric_limits<int128>::digits == 127 && std::numeric_limits<uint128>::digits == 128 &&
-              std::is_integral<int128>::value && std::is_integral<uint128>::value, // NOLINT(misc-redundant-expression)
-              "__int128 is not fully supported by the standard library");
-
-using wide_json = nlohmann::basic_json<std::map, std::vector, std::string, bool, int128, uint128, double>;
-
-wide_json make_int(int128 v)
-{
-    wide_json j(wide_json::value_t::number_integer);
-    j.get_ref<int128&>() = v;
-    return j;
-}
-
-wide_json make_uint(uint128 v)
-{
-    wide_json j(wide_json::value_t::number_unsigned);
-    j.get_ref<uint128&>() = v;
-    return j;
-}
-
-wide_json wrap(const wide_json& value)
-{
-    wide_json o(wide_json::value_t::object);
-    o["v"] = value;
-    return o;
-}
-
-} // namespace
-
-TEST_CASE("custom number types: integers beyond 64 bits")
-{
-    using out_of_range = wide_json::out_of_range;
-
-    const int128 two_64 = static_cast<int128>(1) << 64;
-    const int128 two_100 = static_cast<int128>(1) << 100;
-    const int128 max_int64 = (std::numeric_limits<std::int64_t>::max)();
-    const int128 min_int64 = (std::numeric_limits<std::int64_t>::min)();
-    const uint128 max_uint64 = (std::numeric_limits<std::uint64_t>::max)();
-
-    // before the range checks, these values were silently truncated, e.g.,
-    // 2^100 was written as 0
-    const wide_json int_big = make_int(two_100);
-    const wide_json int_big_negative = make_int(-two_100);
-    const wide_json uint_big = make_uint(static_cast<uint128>(two_100));
-    std::vector<std::uint8_t> _;
-
-    SECTION("CBOR")
-    {
-        CHECK_THROWS_WITH_AS(_ = wide_json::to_cbor(int_big), "[json.exception.out_of_range.407] integer number 1267650600228229401496703205376 cannot be represented by CBOR as it does not fit [-2^64, 2^64-1]", out_of_range&);
-        CHECK_THROWS_WITH_AS(_ = wide_json::to_cbor(int_big_negative), "[json.exception.out_of_range.407] integer number -1267650600228229401496703205376 cannot be represented by CBOR as it does not fit [-2^64, 2^64-1]", out_of_range&);
-        CHECK_THROWS_WITH_AS(_ = wide_json::to_cbor(uint_big), "[json.exception.out_of_range.407] integer number 1267650600228229401496703205376 cannot be represented by CBOR as it does not fit uint64", out_of_range&);
-        // the extremes of the 128-bit types, whose digits are produced without
-        // negating the minimum as a whole
-        CHECK_THROWS_WITH_AS(_ = wide_json::to_cbor(make_int((std::numeric_limits<int128>::min)())), "[json.exception.out_of_range.407] integer number -170141183460469231731687303715884105728 cannot be represented by CBOR as it does not fit [-2^64, 2^64-1]", out_of_range&);
-        CHECK_THROWS_WITH_AS(_ = wide_json::to_cbor(make_int((std::numeric_limits<int128>::max)())), "[json.exception.out_of_range.407] integer number 170141183460469231731687303715884105727 cannot be represented by CBOR as it does not fit [-2^64, 2^64-1]", out_of_range&);
-        CHECK_THROWS_WITH_AS(_ = wide_json::to_cbor(make_uint((std::numeric_limits<uint128>::max)())), "[json.exception.out_of_range.407] integer number 340282366920938463463374607431768211455 cannot be represented by CBOR as it does not fit uint64", out_of_range&);
-        CHECK_THROWS_AS(_ = wide_json::to_cbor(make_int(two_64)), out_of_range&);
-        CHECK_THROWS_AS(_ = wide_json::to_cbor(make_int(-two_64 - 1)), out_of_range&);
-        CHECK_THROWS_AS(_ = wide_json::to_cbor(make_uint(static_cast<uint128>(max_uint64) + 1)), out_of_range&);
-
-        // CBOR's integers cover [-2^64, 2^64-1]
-        CHECK(wide_json::to_cbor(make_int(two_64 - 1)) == std::vector<std::uint8_t>({0x1B, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF}));
-        CHECK(wide_json::to_cbor(make_int(-two_64)) == std::vector<std::uint8_t>({0x3B, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF}));
-        CHECK(wide_json::to_cbor(make_uint(max_uint64)) == std::vector<std::uint8_t>({0x1B, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF}));
-    }
-
-    SECTION("MessagePack")
-    {
-        CHECK_THROWS_WITH_AS(_ = wide_json::to_msgpack(int_big), "[json.exception.out_of_range.407] integer number 1267650600228229401496703205376 cannot be represented by MessagePack as it does not fit [-2^63, 2^64-1]", out_of_range&);
-        CHECK_THROWS_WITH_AS(_ = wide_json::to_msgpack(int_big_negative), "[json.exception.out_of_range.407] integer number -1267650600228229401496703205376 cannot be represented by MessagePack as it does not fit [-2^63, 2^64-1]", out_of_range&);
-        CHECK_THROWS_WITH_AS(_ = wide_json::to_msgpack(uint_big), "[json.exception.out_of_range.407] integer number 1267650600228229401496703205376 cannot be represented by MessagePack as it does not fit uint64", out_of_range&);
-        CHECK_THROWS_AS(_ = wide_json::to_msgpack(make_int(two_64)), out_of_range&);
-        CHECK_THROWS_AS(_ = wide_json::to_msgpack(make_int(min_int64 - 1)), out_of_range&);
-
-        // MessagePack's integers cover [-2^63, 2^64-1]
-        CHECK(wide_json::to_msgpack(make_int(two_64 - 1)) == std::vector<std::uint8_t>({0xCF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF}));
-        CHECK(wide_json::to_msgpack(make_int(min_int64)) == std::vector<std::uint8_t>({0xD3, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00}));
-        CHECK(wide_json::to_msgpack(make_uint(max_uint64)) == std::vector<std::uint8_t>({0xCF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF}));
-    }
-
-    SECTION("BSON")
-    {
-        // wrap() stores the value at /v, which JSON_DIAGNOSTICS adds to the message
-#if JSON_DIAGNOSTICS
-        CHECK_THROWS_WITH_AS(_ = wide_json::to_bson(wrap(int_big)), "[json.exception.out_of_range.407] (/v) integer number 1267650600228229401496703205376 cannot be represented by BSON as it does not fit int64", out_of_range&);
-        CHECK_THROWS_WITH_AS(_ = wide_json::to_bson(wrap(int_big_negative)), "[json.exception.out_of_range.407] (/v) integer number -1267650600228229401496703205376 cannot be represented by BSON as it does not fit int64", out_of_range&);
-        CHECK_THROWS_WITH_AS(_ = wide_json::to_bson(wrap(uint_big)), "[json.exception.out_of_range.407] (/v) integer number 1267650600228229401496703205376 cannot be represented by BSON as it does not fit uint64", out_of_range&);
-#else
-        CHECK_THROWS_WITH_AS(_ = wide_json::to_bson(wrap(int_big)), "[json.exception.out_of_range.407] integer number 1267650600228229401496703205376 cannot be represented by BSON as it does not fit int64", out_of_range&);
-        CHECK_THROWS_WITH_AS(_ = wide_json::to_bson(wrap(int_big_negative)), "[json.exception.out_of_range.407] integer number -1267650600228229401496703205376 cannot be represented by BSON as it does not fit int64", out_of_range&);
-        CHECK_THROWS_WITH_AS(_ = wide_json::to_bson(wrap(uint_big)), "[json.exception.out_of_range.407] integer number 1267650600228229401496703205376 cannot be represented by BSON as it does not fit uint64", out_of_range&);
-#endif
-        CHECK_THROWS_AS(_ = wide_json::to_bson(wrap(make_int(max_int64 + 1))), out_of_range&);
-        CHECK_THROWS_AS(_ = wide_json::to_bson(wrap(make_int(min_int64 - 1))), out_of_range&);
-
-        // nested values are checked as well, before anything is written
-        wide_json nested(wide_json::value_t::object);
-        nested["a"] = wide_json::array({wrap(int_big)});
-        std::vector<std::uint8_t> out;
-        CHECK_THROWS_AS(wide_json::to_bson(nested, out), out_of_range&);
-        CHECK(out.empty());
-
-        CHECK(wide_json::from_bson(wide_json::to_bson(wrap(make_int(max_int64)))) == wrap(make_int(max_int64)));
-        CHECK(wide_json::from_bson(wide_json::to_bson(wrap(make_int(min_int64)))) == wrap(make_int(min_int64)));
-        CHECK_NOTHROW(wide_json::to_bson(wrap(make_uint(max_uint64))));
-    }
-
-    SECTION("BON8")
-    {
-        CHECK_THROWS_WITH_AS(_ = wide_json::to_bon8(int_big), "[json.exception.out_of_range.407] integer number 1267650600228229401496703205376 cannot be represented by BON8 as it does not fit int64", out_of_range&);
-        CHECK_THROWS_WITH_AS(_ = wide_json::to_bon8(int_big_negative), "[json.exception.out_of_range.407] integer number -1267650600228229401496703205376 cannot be represented by BON8 as it does not fit int64", out_of_range&);
-        CHECK_THROWS_WITH_AS(_ = wide_json::to_bon8(uint_big), "[json.exception.out_of_range.407] integer number 1267650600228229401496703205376 cannot be represented by BON8 as it does not fit int64", out_of_range&);
-        CHECK_THROWS_AS(_ = wide_json::to_bon8(make_int(max_int64 + 1)), out_of_range&);
-        CHECK_THROWS_AS(_ = wide_json::to_bon8(make_int(min_int64 - 1)), out_of_range&);
-
-        CHECK(wide_json::from_bon8(wide_json::to_bon8(make_int(max_int64))) == make_int(max_int64));
-        CHECK(wide_json::from_bon8(wide_json::to_bon8(make_int(min_int64))) == make_int(min_int64));
-    }
-
-    SECTION("UBJSON and BJData")
-    {
-        // integers beyond 64 bits are written exactly as high-precision numbers
-        const std::string digits = "1267650600228229401496703205376";
-        std::vector<std::uint8_t> expected = {'H', 'i', static_cast<std::uint8_t>(digits.size())};
-        expected.insert(expected.end(), digits.begin(), digits.end());
-        CHECK(wide_json::to_ubjson(int_big) == expected);
-        CHECK(wide_json::to_ubjson(uint_big) == expected);
-        CHECK(wide_json::to_bjdata(int_big) == expected);
-        // BJData's uint64 marker 'M' was used for any unsigned value beyond
-        // int64, which truncated the ones beyond 64 bits
-        CHECK(wide_json::to_bjdata(uint_big) == expected);
-
-        // the uint64 marker is still used where the value fits
-        CHECK(wide_json::to_bjdata(make_uint(max_uint64)) == std::vector<std::uint8_t>({'M', 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF}));
-
-        // the elements of an optimized container are written the same way;
-        // BJData does not allow 'H' as the type of an optimized container
-        std::vector<std::uint8_t> expected_ubjson_array = {'[', '$', 'H', '#', 'i', 2};
-        std::vector<std::uint8_t> expected_bjdata_array = {'[', '#', 'i', 2};
-        for (int i = 0; i < 2; ++i)
-        {
-            expected_ubjson_array.insert(expected_ubjson_array.end(), expected.begin() + 1, expected.end());
-            expected_bjdata_array.insert(expected_bjdata_array.end(), expected.begin(), expected.end());
-        }
-        CHECK(wide_json::to_ubjson(wide_json::array({uint_big, uint_big}), true, true) == expected_ubjson_array);
-        CHECK(wide_json::to_bjdata(wide_json::array({uint_big, uint_big}), true, true) == expected_bjdata_array);
-    }
-
-    SECTION("BJData ND-array")
-    {
-        // an ND-array element beyond 64 bits does not fit any dtype, so the
-        // annotated object is written as a plain object instead of truncating
-        // the element into range
-        wide_json element(wide_json::value_t::object);
-        element["_ArrayType_"] = "uint8";
-        element["_ArraySize_"] = wide_json::array({make_uint(2), make_uint(1)});
-        element["_ArrayData_"] = wide_json::array({make_uint(1), make_uint(static_cast<uint128>(two_64) + 1)});
-        const auto element_bytes = wide_json::to_bjdata(element, true, true);
-        REQUIRE(!element_bytes.empty());
-        CHECK(element_bytes[0] == '{');
-
-        wide_json signed_element = element;
-        signed_element["_ArrayType_"] = "int8";
-        signed_element["_ArrayData_"] = wide_json::array({make_int(1), make_int(-two_64 + 1)});
-        const auto signed_element_bytes = wide_json::to_bjdata(signed_element, true, true);
-        REQUIRE(!signed_element_bytes.empty());
-        CHECK(signed_element_bytes[0] == '{');
-
-        // the same for a dimension beyond 64 bits, which wrapped into a
-        // dimension matching the size of _ArrayData_ (here: 1)
-        wide_json dimension = element;
-        dimension["_ArraySize_"] = wide_json::array({make_uint(2), make_uint(static_cast<uint128>(two_64) + 1)});
-        dimension["_ArrayData_"] = wide_json::array({make_uint(1), make_uint(2)});
-        const auto dimension_bytes = wide_json::to_bjdata(dimension, true, true);
-        REQUIRE(!dimension_bytes.empty());
-        CHECK(dimension_bytes[0] == '{');
-
-        // in range, the ND-array is still written
-        wide_json fits = element;
-        fits["_ArrayData_"] = wide_json::array({make_uint(1), make_uint(2)});
-        const auto fits_bytes = wide_json::to_bjdata(fits, true, true);
-        REQUIRE(!fits_bytes.empty());
-        CHECK(fits_bytes[0] == '[');
-    }
-}
-
-#endif
-
-TEST_CASE("custom number types")
-{
-    SECTION("signed integer of class type compared with a float")
-    {
-        // std::is_signed is false for a class type, which made the comparison
-        // treat any float below zero as less than every integer
-        const class_int_json minus_five = make_class_int(-5);
-        const class_int_json minus_two = make_class_int(-2);
-        const class_int_json five = make_class_int(5);
-        const class_int_json minus_two_and_a_half = -2.5;
-        const class_int_json two_and_a_half = 2.5;
-        const class_int_json huge_negative = -1e30;
-        const class_int_json huge_positive = 1e30;
-
-        CHECK(minus_five < minus_two_and_a_half);
-        CHECK_FALSE(minus_two_and_a_half < minus_five);
-        CHECK(minus_two_and_a_half > minus_five);
-        CHECK(minus_five <= minus_two_and_a_half);
-        CHECK(minus_two_and_a_half >= minus_five);
-        CHECK(minus_five != minus_two_and_a_half);
-
-        CHECK(minus_two_and_a_half < minus_two);
-        CHECK_FALSE(minus_two < minus_two_and_a_half);
-
-        CHECK(two_and_a_half < five);
-        CHECK(minus_five < two_and_a_half);
-
-        // floats beyond the integer's range
-        CHECK(huge_negative < minus_five);
-        CHECK_FALSE(minus_five < huge_negative);
-        CHECK(five < huge_positive);
-        CHECK_FALSE(huge_positive < five);
-
-        // equality
-        const class_int_json minus_two_float = -2.0;
-        CHECK(minus_two == minus_two_float);
-        CHECK(minus_two_float == minus_two);
-    }
-
-}

tests/src/unit-msgpack-custom-types.cpp

@@ -1,296 +0,0 @@
-//     __ _____ _____ _____
-//  __|  |   __|     |   | |  JSON for Modern C++ (supporting code)
-// |  |  |__   |  |  | | | |  version 3.12.0
-// |_____|_____|_____|_|___|  https://github.com/nlohmann/json
-//
-// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
-// SPDX-License-Identifier: MIT
-
-// MessagePack tests with basic_json specializations other than nlohmann::json;
-// they are kept apart from unit-msgpack.cpp because every specialization adds
-// many sections to the object file, and MinGW cannot link objects with more
-// than 65535 sections (see the clang job in .github/workflows/windows.yml)
-
-#include "doctest_compatibility.h"
-
-#include <nlohmann/json.hpp>
-using nlohmann::json;
-
-#include <cstdint> // SIZE_MAX, UINT32_MAX
-#include <functional>
-#include <limits>
-#include <map>
-#include <memory>
-#include <string>
-#include <utility>
-#include <vector>
-
-// the fake sizes below do not fit into a 32-bit std::size_t
-// with clang and libstdc++ 10, the std::filesystem::path conversion that
-// C++17 builds consider for every string type is ambiguous for a class
-// derived from std::string, so the string case is not tested there
-#if !(defined(__clang__) && defined(_GLIBCXX_RELEASE) && _GLIBCXX_RELEASE < 11)
-    #define JSON_TEST_BEYOND_UINT32_STRING 1
-#endif
-
-#if SIZE_MAX > UINT32_MAX
-template<typename T, typename A = std::allocator<T>>
-struct huge_array : std::vector<T, A>
-{
-    using base = std::vector<T, A>;
-    using base::base;
-
-    bool fake_size = false;
-
-    std::size_t size() const noexcept
-    {
-        if (fake_size)
-        {
-            return (std::numeric_limits<std::uint32_t>::max)() + 1ULL;
-        }
-
-        return base::size();
-    }
-};
-
-using huge_array_json = nlohmann::json::with_array_t<huge_array>;
-
-TEST_CASE("MessagePack Size above uint32 for array")
-{
-    huge_array_json j = huge_array_json::array();
-
-    j.push_back(1);
-    j.push_back(2);
-    j.push_back(3);
-
-    auto& array = j.get_ref<huge_array_json::array_t&>();
-    array.fake_size = true;
-
-    // write into a caller-owned vector: to_msgpack(j) reserves space based on
-    // the (faked) element count, which fails with bad_alloc on Windows
-    std::vector<std::uint8_t> result;
-    CHECK_THROWS_WITH_AS(
-        huge_array_json::to_msgpack(j, result),
-        "[json.exception.out_of_range.412] MessagePack length 4294967296 exceeds maximum of 4294967295",
-        json::out_of_range&);
-
-    array.fake_size = false;
-}
-
-template<typename K, typename V,
-         typename C = std::less<K>,
-         typename A = std::allocator<std::pair<const K, V>>>
-                                     struct huge_map : std::map<K, V, C, A>
-{
-    using base = std::map<K, V, C, A>;
-    using base::base;
-
-    bool fake_size = false;
-
-    std::size_t size() const noexcept
-    {
-        if (fake_size)
-        {
-            return static_cast<std::size_t>(UINT32_MAX) + 1ULL;
-        }
-
-        return base::size();
-    }
-};
-
-using huge_object_json = nlohmann::json::with_object_t<huge_map>;
-
-TEST_CASE("MessagePack Size above uint32 for object")
-{
-
-    huge_object_json j = huge_object_json::object();
-
-    j["one"] = 1;
-    j["two"] = 2;
-
-    auto& object = j.get_ref<huge_object_json::object_t&>();
-    object.fake_size = true;
-
-    // write into a caller-owned vector: to_msgpack(j) reserves space based on
-    // the (faked) element count, which fails with bad_alloc on Windows
-    std::vector<std::uint8_t> result;
-    CHECK_THROWS_WITH_AS(
-        huge_object_json::to_msgpack(j, result),
-        "[json.exception.out_of_range.412] MessagePack length 4294967296 exceeds maximum of 4294967295",
-        json::out_of_range&);
-
-    object.fake_size = false;
-}
-
-#ifdef JSON_TEST_BEYOND_UINT32_STRING
-struct huge_string : std::string
-{
-    using std::string::string;
-
-    std::size_t size() const noexcept // NOLINT(readability-convert-member-functions-to-static)
-    {
-        return static_cast<std::size_t>(UINT32_MAX) + 1ULL;
-    }
-};
-
-using huge_string_json = nlohmann::json::with_string_t<huge_string>;
-
-TEST_CASE("MessagePack Size above uint32 for string")
-{
-    const huge_string_json j = "hello";
-
-    CHECK_THROWS_WITH_AS(
-        huge_string_json::to_msgpack(j),
-        "[json.exception.out_of_range.412] MessagePack length 4294967296 exceeds maximum of 4294967295",
-        json::out_of_range&);
-}
-#endif
-
-struct huge_binary : std::vector<std::uint8_t>
-{
-    using std::vector<std::uint8_t>::vector;
-
-    std::size_t size() const noexcept // NOLINT(readability-convert-member-functions-to-static)
-    {
-        return static_cast<std::size_t>(UINT32_MAX) + 1ULL;
-    }
-};
-
-using huge_binary_json = nlohmann::json::with_binary_t<huge_binary>;
-
-TEST_CASE("MessagePack Size above uint32 for binary")
-{
-
-    huge_binary_json j = huge_binary_json::binary(huge_binary{});
-
-    j.get_binary().push_back(0x01);
-    j.get_binary().push_back(0x02);
-
-    CHECK_THROWS_WITH_AS(
-        huge_binary_json::to_msgpack(j),
-        "[json.exception.out_of_range.412] MessagePack length 4294967296 exceeds maximum of 4294967295",
-        json::out_of_range&);
-}
-#endif
-
-namespace
-{
-// types that report a size beyond UINT32_MAX without allocating that much
-// memory, so the MessagePack length limit can be tested cheaply; see the
-// similar types in unit-bson.cpp
-std::size_t beyond_uint32_size()
-{
-    return static_cast<std::size_t>((std::numeric_limits<std::uint32_t>::max)()) + 1;
-}
-
-class beyond_uint32_binary_t : public std::vector<std::uint8_t>
-{
-  public:
-    using std::vector<std::uint8_t>::vector;
-
-    size_type size() const noexcept // NOLINT(readability-convert-member-functions-to-static)
-    {
-        return beyond_uint32_size();
-    }
-};
-
-#ifdef JSON_TEST_BEYOND_UINT32_STRING
-class beyond_uint32_string_t : public std::string
-{
-  public:
-    using std::string::string;
-
-    size_type size() const noexcept // NOLINT(readability-convert-member-functions-to-static)
-    {
-        return beyond_uint32_size();
-    }
-};
-
-using beyond_uint32_string_json = nlohmann::json::with_string_t<beyond_uint32_string_t>;
-#endif
-
-using beyond_uint32_binary_json = nlohmann::json::with_binary_t<beyond_uint32_binary_t>;
-} // namespace
-
-TEST_CASE("MessagePack lengths beyond UINT32_MAX cannot be serialized")
-{
-    // MessagePack stores the length of a string, binary value, array, or
-    // object in at most 32 bits; a larger one used to be written without any
-    // length at all
-#if SIZE_MAX > UINT32_MAX
-    {
-        const char* const expected = "[json.exception.out_of_range.412] MessagePack length 4294967296 exceeds maximum of 4294967295";
-
-        const beyond_uint32_binary_json binary = beyond_uint32_binary_json::binary(beyond_uint32_binary_t{});
-        CHECK_THROWS_WITH_AS(beyond_uint32_binary_json::to_msgpack(binary), expected, beyond_uint32_binary_json::out_of_range&);
-
-        const beyond_uint32_binary_json ext = beyond_uint32_binary_json::binary(beyond_uint32_binary_t{}, 42);
-        CHECK_THROWS_WITH_AS(beyond_uint32_binary_json::to_msgpack(ext), expected, beyond_uint32_binary_json::out_of_range&);
-
-#ifdef JSON_TEST_BEYOND_UINT32_STRING
-        // created from its type rather than from a beyond_uint32_string_t:
-        // that would consider the std::filesystem::path conversion, which
-        // libstdc++ 10 cannot decide for a class derived from std::string
-        const beyond_uint32_string_json string(beyond_uint32_string_json::value_t::string);
-        CHECK_THROWS_WITH_AS(beyond_uint32_string_json::to_msgpack(string), expected, beyond_uint32_string_json::out_of_range&);
-#endif
-    }
-#endif
-}
-
-TEST_CASE("MessagePack numbers use the active union member (see #5644)")
-{
-    // when number_integer_t is narrower than number_unsigned_t, to_msgpack()
-    // used to read the union member that was not the active one, writing
-    // wrong bytes for some values; std::int64_t/std::uint64_t (the default
-    // types, where both members have the same width) were not affected
-    using int32_json = nlohmann::json::with_integers_t<std::int32_t, std::uint64_t>;
-    using int16_json = nlohmann::json::with_integers_t<std::int16_t, std::uint64_t>;
-
-    SECTION("number_integer_t = std::int32_t")
-    {
-        SECTION("6442450944 (uint 64; the low 32 bits used to be sign-extended)")
-        {
-            const int32_json j = 6442450944ULL;
-            CHECK(j.is_number_unsigned());
-
-            std::vector<uint8_t> const expected{0xcf, 0x00, 0x00, 0x00, 0x01, 0x80, 0x00, 0x00, 0x00};
-            const auto result = int32_json::to_msgpack(j);
-            CHECK(result == expected);
-            CHECK(int32_json::from_msgpack(result) == j);
-        }
-
-        SECTION("4294967496 (uint 64; the low 32 bits used to be the whole value)")
-        {
-            const int32_json j = 4294967496ULL;
-            CHECK(j.is_number_unsigned());
-
-            std::vector<uint8_t> const expected{0xcf, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0xc8};
-            const auto result = int32_json::to_msgpack(j);
-            CHECK(result == expected);
-            CHECK(int32_json::from_msgpack(result) == j);
-        }
-    }
-
-    SECTION("number_integer_t = std::int16_t, 98304 (uint 32)")
-    {
-        const int16_json j = 98304ULL;
-        CHECK(j.is_number_unsigned());
-
-        std::vector<uint8_t> const expected{0xce, 0x00, 0x01, 0x80, 0x00};
-        const auto result = int16_json::to_msgpack(j);
-        CHECK(result == expected);
-        CHECK(int16_json::from_msgpack(result) == j);
-    }
-
-    SECTION("default types (std::int64_t/std::uint64_t) are unaffected")
-    {
-        const json j = 4294967496ULL;
-        CHECK(j.is_number_unsigned());
-
-        std::vector<uint8_t> const expected{0xcf, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0xc8};
-        const auto result = json::to_msgpack(j);
-        CHECK(result == expected);
-        CHECK(json::from_msgpack(result) == j);
-    }
-}

tests/src/unit-msgpack.cpp

@@ -2173,6 +2173,276 @@ TEST_CASE("MessagePack roundtrips" * doctest::skip())
     }
 }
 
+// the fake sizes below do not fit into a 32-bit std::size_t
+// with clang and libstdc++ 10, the std::filesystem::path conversion that
+// C++17 builds consider for every string type is ambiguous for a class
+// derived from std::string, so the string case is not tested there
+#if !(defined(__clang__) && defined(_GLIBCXX_RELEASE) && _GLIBCXX_RELEASE < 11)
+    #define JSON_TEST_BEYOND_UINT32_STRING 1
+#endif
+
+#if SIZE_MAX > UINT32_MAX
+template<typename T, typename A = std::allocator<T>>
+struct huge_array : std::vector<T, A>
+{
+    using base = std::vector<T, A>;
+    using base::base;
+
+    bool fake_size = false;
+
+    std::size_t size() const noexcept
+    {
+        if (fake_size)
+        {
+            return (std::numeric_limits<std::uint32_t>::max)() + 1ULL;
+        }
+
+        return base::size();
+    }
+};
+
+using huge_array_json = nlohmann::json::with_array_t<huge_array>;
+
+TEST_CASE("MessagePack Size above uint32 for array")
+{
+    huge_array_json j = huge_array_json::array();
+
+    j.push_back(1);
+    j.push_back(2);
+    j.push_back(3);
+
+    auto& array = j.get_ref<huge_array_json::array_t&>();
+    array.fake_size = true;
+
+    // write into a caller-owned vector: to_msgpack(j) reserves space based on
+    // the (faked) element count, which fails with bad_alloc on Windows
+    std::vector<std::uint8_t> result;
+    CHECK_THROWS_WITH_AS(
+        huge_array_json::to_msgpack(j, result),
+        "[json.exception.out_of_range.412] MessagePack length 4294967296 exceeds maximum of 4294967295",
+        json::out_of_range&);
+
+    array.fake_size = false;
+}
+
+template<typename K, typename V,
+         typename C = std::less<K>,
+         typename A = std::allocator<std::pair<const K, V>>>
+                                     struct huge_map : std::map<K, V, C, A>
+{
+    using base = std::map<K, V, C, A>;
+    using base::base;
+
+    bool fake_size = false;
+
+    std::size_t size() const noexcept
+    {
+        if (fake_size)
+        {
+            return static_cast<std::size_t>(UINT32_MAX) + 1ULL;
+        }
+
+        return base::size();
+    }
+};
+
+using huge_object_json = nlohmann::json::with_object_t<huge_map>;
+
+TEST_CASE("MessagePack Size above uint32 for object")
+{
+
+    huge_object_json j = huge_object_json::object();
+
+    j["one"] = 1;
+    j["two"] = 2;
+
+    auto& object = j.get_ref<huge_object_json::object_t&>();
+    object.fake_size = true;
+
+    // write into a caller-owned vector: to_msgpack(j) reserves space based on
+    // the (faked) element count, which fails with bad_alloc on Windows
+    std::vector<std::uint8_t> result;
+    CHECK_THROWS_WITH_AS(
+        huge_object_json::to_msgpack(j, result),
+        "[json.exception.out_of_range.412] MessagePack length 4294967296 exceeds maximum of 4294967295",
+        json::out_of_range&);
+
+    object.fake_size = false;
+}
+
+#ifdef JSON_TEST_BEYOND_UINT32_STRING
+struct huge_string : std::string
+{
+    using std::string::string;
+
+    std::size_t size() const noexcept // NOLINT(readability-convert-member-functions-to-static)
+    {
+        return static_cast<std::size_t>(UINT32_MAX) + 1ULL;
+    }
+};
+
+using huge_string_json = nlohmann::json::with_string_t<huge_string>;
+
+TEST_CASE("MessagePack Size above uint32 for string")
+{
+    const huge_string_json j = "hello";
+
+    CHECK_THROWS_WITH_AS(
+        huge_string_json::to_msgpack(j),
+        "[json.exception.out_of_range.412] MessagePack length 4294967296 exceeds maximum of 4294967295",
+        json::out_of_range&);
+}
+#endif
+
+struct huge_binary : std::vector<std::uint8_t>
+{
+    using std::vector<std::uint8_t>::vector;
+
+    std::size_t size() const noexcept // NOLINT(readability-convert-member-functions-to-static)
+    {
+        return static_cast<std::size_t>(UINT32_MAX) + 1ULL;
+    }
+};
+
+using huge_binary_json = nlohmann::json::with_binary_t<huge_binary>;
+
+TEST_CASE("MessagePack Size above uint32 for binary")
+{
+
+    huge_binary_json j = huge_binary_json::binary(huge_binary{});
+
+    j.get_binary().push_back(0x01);
+    j.get_binary().push_back(0x02);
+
+    CHECK_THROWS_WITH_AS(
+        huge_binary_json::to_msgpack(j),
+        "[json.exception.out_of_range.412] MessagePack length 4294967296 exceeds maximum of 4294967295",
+        json::out_of_range&);
+}
+#endif
+
+namespace
+{
+// types that report a size beyond UINT32_MAX without allocating that much
+// memory, so the MessagePack length limit can be tested cheaply; see the
+// similar types in unit-bson.cpp
+std::size_t beyond_uint32_size()
+{
+    return static_cast<std::size_t>((std::numeric_limits<std::uint32_t>::max)()) + 1;
+}
+
+class beyond_uint32_binary_t : public std::vector<std::uint8_t>
+{
+  public:
+    using std::vector<std::uint8_t>::vector;
+
+    size_type size() const noexcept // NOLINT(readability-convert-member-functions-to-static)
+    {
+        return beyond_uint32_size();
+    }
+};
+
+#ifdef JSON_TEST_BEYOND_UINT32_STRING
+class beyond_uint32_string_t : public std::string
+{
+  public:
+    using std::string::string;
+
+    size_type size() const noexcept // NOLINT(readability-convert-member-functions-to-static)
+    {
+        return beyond_uint32_size();
+    }
+};
+
+using beyond_uint32_string_json = nlohmann::json::with_string_t<beyond_uint32_string_t>;
+#endif
+
+using beyond_uint32_binary_json = nlohmann::json::with_binary_t<beyond_uint32_binary_t>;
+} // namespace
+
+TEST_CASE("MessagePack lengths beyond UINT32_MAX cannot be serialized")
+{
+    // MessagePack stores the length of a string, binary value, array, or
+    // object in at most 32 bits; a larger one used to be written without any
+    // length at all
+#if SIZE_MAX > UINT32_MAX
+    {
+        const char* const expected = "[json.exception.out_of_range.412] MessagePack length 4294967296 exceeds maximum of 4294967295";
+
+        const beyond_uint32_binary_json binary = beyond_uint32_binary_json::binary(beyond_uint32_binary_t{});
+        CHECK_THROWS_WITH_AS(beyond_uint32_binary_json::to_msgpack(binary), expected, beyond_uint32_binary_json::out_of_range&);
+
+        const beyond_uint32_binary_json ext = beyond_uint32_binary_json::binary(beyond_uint32_binary_t{}, 42);
+        CHECK_THROWS_WITH_AS(beyond_uint32_binary_json::to_msgpack(ext), expected, beyond_uint32_binary_json::out_of_range&);
+
+#ifdef JSON_TEST_BEYOND_UINT32_STRING
+        // created from its type rather than from a beyond_uint32_string_t:
+        // that would consider the std::filesystem::path conversion, which
+        // libstdc++ 10 cannot decide for a class derived from std::string
+        const beyond_uint32_string_json string(beyond_uint32_string_json::value_t::string);
+        CHECK_THROWS_WITH_AS(beyond_uint32_string_json::to_msgpack(string), expected, beyond_uint32_string_json::out_of_range&);
+#endif
+    }
+#endif
+}
+
+TEST_CASE("MessagePack numbers use the active union member (see #5644)")
+{
+    // when number_integer_t is narrower than number_unsigned_t, to_msgpack()
+    // used to read the union member that was not the active one, writing
+    // wrong bytes for some values; std::int64_t/std::uint64_t (the default
+    // types, where both members have the same width) were not affected
+    using int32_json = nlohmann::json::with_integers_t<std::int32_t, std::uint64_t>;
+    using int16_json = nlohmann::json::with_integers_t<std::int16_t, std::uint64_t>;
+
+    SECTION("number_integer_t = std::int32_t")
+    {
+        SECTION("6442450944 (uint 64; the low 32 bits used to be sign-extended)")
+        {
+            const int32_json j = 6442450944ULL;
+            CHECK(j.is_number_unsigned());
+
+            std::vector<uint8_t> const expected{0xcf, 0x00, 0x00, 0x00, 0x01, 0x80, 0x00, 0x00, 0x00};
+            const auto result = int32_json::to_msgpack(j);
+            CHECK(result == expected);
+            CHECK(int32_json::from_msgpack(result) == j);
+        }
+
+        SECTION("4294967496 (uint 64; the low 32 bits used to be the whole value)")
+        {
+            const int32_json j = 4294967496ULL;
+            CHECK(j.is_number_unsigned());
+
+            std::vector<uint8_t> const expected{0xcf, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0xc8};
+            const auto result = int32_json::to_msgpack(j);
+            CHECK(result == expected);
+            CHECK(int32_json::from_msgpack(result) == j);
+        }
+    }
+
+    SECTION("number_integer_t = std::int16_t, 98304 (uint 32)")
+    {
+        const int16_json j = 98304ULL;
+        CHECK(j.is_number_unsigned());
+
+        std::vector<uint8_t> const expected{0xce, 0x00, 0x01, 0x80, 0x00};
+        const auto result = int16_json::to_msgpack(j);
+        CHECK(result == expected);
+        CHECK(int16_json::from_msgpack(result) == j);
+    }
+
+    SECTION("default types (std::int64_t/std::uint64_t) are unaffected")
+    {
+        const json j = 4294967496ULL;
+        CHECK(j.is_number_unsigned());
+
+        std::vector<uint8_t> const expected{0xcf, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0xc8};
+        const auto result = json::to_msgpack(j);
+        CHECK(result == expected);
+        CHECK(json::from_msgpack(result) == j);
+    }
+}
+
 TEST_CASE("MessagePack large strings and binaries (chunked reader)")
 {
     // get_msgpack_string()/get_msgpack_binary() both read through get_binary(),

tests/src/unit-regression3.cpp

@@ -43,6 +43,7 @@ using ordered_json = nlohmann::ordered_json;
 #endif
 
 #include <cstdio>
+#include <deque>
 #include <list>
 #include <type_traits>
 #include <utility>
@@ -866,6 +867,52 @@ TEST_CASE("issue #5402 - update(merge_objects=true) overwrites a primitive with
 }
 
 
+TEST_CASE("regression test #5476 - array type without reserve()")
+{
+    // the capacity reserved for definite-length arrays must not require the
+    // array type to have a reserve() member function
+    using deque_json = nlohmann::json::with_array_t<std::deque>;
+
+    SECTION("std::deque")
+    {
+        const auto j = deque_json::parse(R"({"a":[1,[2,3]],"b":[]})");
+        CHECK(j.dump() == R"({"a":[1,[2,3]],"b":[]})");
+
+        // the binary formats pass a definite length to start_array()
+        CHECK(deque_json::from_cbor(deque_json::to_cbor(j)) == j);
+        CHECK(deque_json::from_msgpack(deque_json::to_msgpack(j)) == j);
+        CHECK(deque_json::from_bon8(deque_json::to_bon8(j)) == j);
+
+        // parse() instantiates the callback parser as well, which reserves too
+        const auto with_callback = deque_json::parse(R"([1,2,3])", [](int /*depth*/, deque_json::parse_event_t /*event*/, deque_json& /*parsed*/) noexcept
+        {
+            return true;
+        });
+        CHECK(with_callback == deque_json({1, 2, 3}));
+    }
+
+    SECTION("std::vector still reserves")
+    {
+        json array = json::array();
+        for (int i = 0; i < 100; ++i)
+        {
+            array.push_back(i);
+        }
+
+        const auto j = json::from_cbor(json::to_cbor(array));
+        CHECK(j == array);
+        CHECK(j.get_ref<const json::array_t&>().capacity() >= 100);
+    }
+
+    SECTION("the reservation stays capped")
+    {
+        // CBOR array announcing 2^32-1 elements, but truncated right after the
+        // header: the input must be rejected without reserving that capacity
+        const std::vector<std::uint8_t> truncated = {0x9A, 0xFF, 0xFF, 0xFF, 0xFF};
+        CHECK(json::from_cbor(truncated, true, false).is_discarded());
+    }
+}
+
 TEST_CASE("issue #5317 - nested indefinite-length CBOR string chunks are rejected")
 {
     json _;