Merge branch 'develop' into json-view/5869-7-check-utf8

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/at.md

@@ -90,8 +90,7 @@ Strong exception safety: if an exception occurs, the original value stays intact
 1. Constant.
 2. Logarithmic in the size of the container.
 3. Logarithmic in the size of the container.
-4. Linear in the number of reference tokens of `ptr`; for each of them, logarithmic in the size of the object, or
+4. Logarithmic in the size of the container.
-   constant for an array.
 
 ## Notes
 

docs/mkdocs/docs/api/basic_json/operator[].md

@@ -92,20 +92,12 @@ Strong exception safety: if an exception occurs, the original value stays intact
     For the **const** version, an object key or array index in `ptr` that does not exist is not reported by an
     exception, but is undefined behavior (see the notes below). Use [`at`](at.md) for checked access.
 
-    If the **non-const** version throws, the value is left unchanged: values it created for earlier reference tokens
-    of `ptr` are removed again. This also holds for exceptions such as `#!cpp std::bad_alloc` or
-    `#!cpp std::length_error` thrown while padding an array up to a very large index. With custom
-    [template parameters](../../features/types/template_parameters.md), this requires that `erase()` of the object and
-    array types and the move operations of the custom base class do not throw.
-
 ## Complexity
 
 1. Constant if `idx` is in the range of the array. Otherwise, linear in `idx - size()`.
 2. Logarithmic in the size of the container.
 3. Logarithmic in the size of the container.
-4. Linear in the number of reference tokens of `ptr`; for each of them, logarithmic in the size of the object, or
+4. Logarithmic in the size of the container.
-   constant for an array. When the non-const version pads an array with `#!json null` values, it is additionally
-   linear in `idx - size()` for that array.
 
 ## Notes
 
@@ -153,14 +145,6 @@ Strong exception safety: if an exception occurs, the original value stays intact
         specified by the JSON Pointer RFC; it is this library's own, intentional disambiguation rule. See also
         [JSON Pointer](../../features/json_pointer.md).
 
-    !!! warning "Large array indices"
-
-        As the array is filled up with `#!json null` values to the given index, the memory used grows with the index
-        and not with the size of the pointer: on a 64-bit platform, the index `10000000` allocates about 160 MB.
-        The library does not limit the index, so check the array indices of a JSON pointer from untrusted input
-        before using it with the non-const version. See
-        [Parsing untrusted input](../../features/parsing/untrusted_input.md#json-pointers-and-flattened-keys).
-
 !!! warning "Deprecation"
 
     Overload (4) also accepts a [`json_pointer`](../json_pointer/index.md) whose template argument is a `basic_json`
@@ -310,6 +294,4 @@ Strong exception safety: if an exception occurs, the original value stays intact
 4. Added in version 2.0.0. A missing array index in the const version is guarded by a runtime assertion since
    version 3.13.0. Throws [`parse_error.109`](../../home/exceptions.md#jsonexceptionparse_error109) instead of
 [`out_of_range.404`](../../home/exceptions.md#jsonexceptionout_of_range404) for a one-character array index that is not
-a digit (e.g., `/x`) in version 3.13.0, as it already did for longer ones. Fixed in version 3.13.0 to leave the value
+a digit (e.g., `/x`) in version 3.13.0, as it already did for longer ones.
-unchanged if the non-const version throws (strong exception safety, as documented); before, values created for earlier
-reference tokens of `ptr` were kept.

docs/mkdocs/docs/features/parsing/untrusted_input.md

@@ -130,17 +130,6 @@ parameters of [`parse()`](../../api/basic_json/parse.md) or [`accept()`](../../a
 enable either for input whose conformance you cannot otherwise control, since interoperability with strictly
 conforming JSON consumers is exactly what the default rejects.
 
-## JSON Pointers and flattened keys
-
-A [JSON Pointer](../json_pointer.md) can come from untrusted input as well, for instance as a path in a request or as a
-key of a flattened object. The non-const [`operator[]`](../../api/basic_json/operator%5B%5D.md) with a JSON Pointer and
-[`unflatten`](../../api/basic_json/unflatten.md) create missing values, and when an array index lies beyond the end of
-an array, they fill the array with `#!json null` values up to that index. The memory used therefore grows with the
-index, not with the size of the pointer: on a 64-bit platform, the index `10000000` allocates about 160 MB, the index
-`100000000` about 1.6 GB. The library does not limit the index. For pointers from untrusted input, check the array
-indices (or the pointer as a whole) before using them, or use [`at`](../../api/basic_json/at.md) or
-[`contains`](../../api/basic_json/contains.md), which never add values.
-
 ## Checklist
 
 - Wrap parsing in a `#!cpp try`/`#!cpp catch` block, or use `allow_exceptions=false`/`accept()` if your environment
@@ -156,8 +145,6 @@ indices (or the pointer as a whole) before using them, or use [`at`](../../api/b
   `JSON_STRICT_NUL_HANDLING`.
 - Decide whether duplicate object keys should be an error for your application, and add a callback if so.
 - Do not enable `ignore_comments` or `ignore_trailing_commas` for input that must be strictly conforming JSON.
-- Check the array indices of untrusted JSON Pointers and flattened keys before passing them to the non-const
-  `operator[]` or `unflatten()`, which fill arrays with `#!json null` values up to the index.
 
 For the broader design rationale and how it is tested (fuzzing, sanitizers, static analysis), see the
 [assurance case](../../community/assurance_case.md) and [quality assurance](../../community/quality_assurance.md). To

include/nlohmann/detail/json_pointer.hpp

@@ -8,7 +8,7 @@
 
 #pragma once
 
-#include <algorithm> // copy
+#include <algorithm> // all_of, copy
 #include <array> // array
 #include <cctype> // isdigit
 #include <cerrno> // errno, ERANGE
@@ -16,11 +16,9 @@
 #include <cstdint> // uint64_t
 #include <cstdlib> // strtoull
 #include <cstring> // memchr, memcmp, memcpy
-#include <functional> // less
 #ifndef JSON_NO_IO
     #include <iosfwd> // ostream
 #endif  // JSON_NO_IO
-#include <iterator> // next
 #include <limits> // max
 #include <string> // string
 #include <utility> // move
@@ -30,7 +28,6 @@
 #include <nlohmann/detail/macro_scope.hpp>
 #include <nlohmann/detail/meta/cpp_future.hpp>
 #include <nlohmann/detail/meta/type_traits.hpp>
-#include <nlohmann/detail/meta/void_t.hpp>
 #include <nlohmann/detail/string_concat.hpp>
 #include <nlohmann/detail/string_escape.hpp>
 #include <nlohmann/detail/string_utils.hpp>
@@ -137,26 +134,6 @@ JSON_HEDLEY_ALWAYS_INLINE array_index_status parse_array_index(const StringType&
     return array_index_status::ok;
 }
 
-/// whether two keys of type @a StringType are equivalent under @a Compare
-/// exactly if they are equal: std::less<StringType> and std::less<>
-template<typename Compare, typename StringType>
-struct is_less_comparator : std::false_type {};
-
-template<typename StringType>
-struct is_less_comparator<std::less<StringType>, StringType> : std::true_type {};
-
-template<typename StringType>
-struct is_less_comparator<std::less<void>, StringType> : std::true_type {};
-
-/// whether the objects of type @a ObjectType find a key of type @a StringType
-/// exactly if it equals a stored key
-template<typename ObjectType, typename StringType, typename = void>
-struct finds_equal_keys : std::false_type {};
-
-template<typename ObjectType, typename StringType>
-struct finds_equal_keys<ObjectType, StringType, void_t<typename ObjectType::key_compare, typename ObjectType::key_type>>
-    : conjunction<is_less_comparator<typename ObjectType::key_compare, StringType>, std::is_same<typename ObjectType::key_type, StringType>> {};
-
 }  // namespace detail
 
 /// @brief JSON Pointer defines a string syntax for identifying a specific value within a JSON document
@@ -612,228 +589,6 @@ class json_pointer
         return object_member_impl(object, key, detail::priority_tag<1> {});
     }
 
-    /// @brief append a null value to @a array, like operator[] with the size of @a array
-    template<typename BasicJsonType>
-    static BasicJsonType& array_append(BasicJsonType& array)
-    {
-        auto& elements = *array.m_data.m_value.array;
-        const auto old_capacity = array.container_capacity();
-        elements.emplace_back();
-        return array.set_parent_after_insert(elements.back(), old_capacity);
-    }
-
-    /// @brief add @a key, which is missing, to @a object
-    template<typename BasicJsonType>
-    static auto object_add_impl(BasicJsonType& object, const string_t& key, detail::priority_tag<1> /*unused*/)
-    -> decltype(object.m_data.m_value.object->emplace(key, nullptr), std::declval<BasicJsonType&>())
-    {
-        const auto old_capacity = object.container_capacity();
-        const auto it = object.m_data.m_value.object->emplace(key, nullptr).first;
-        return object.set_parent_after_insert(it->second, old_capacity);
-    }
-
-    template<typename BasicJsonType>
-    static BasicJsonType& object_add_impl(BasicJsonType& object, const string_t& key, detail::priority_tag<0> /*unused*/)
-    {
-        return object.operator[](key);
-    }
-
-    template<typename BasicJsonType>
-    static BasicJsonType& object_add(BasicJsonType& object, const string_t& key)
-    {
-        return object_add_impl(object, key, detail::priority_tag<1> {});
-    }
-
-    /// @brief search @a key in @a members: the result is where the key is or,
-    /// for objects that can insert at a position (std::map), where it belongs
-    template<typename ObjectType>
-    static auto object_search_impl(ObjectType& members, const string_t& key, detail::priority_tag<1> /*unused*/)
-    -> decltype(members.key_comp()(key, members.lower_bound(key)->first), members.lower_bound(key))
-    {
-        return members.lower_bound(key);
-    }
-
-    template<typename ObjectType>
-    static auto object_search_impl(ObjectType& members, const string_t& key, detail::priority_tag<0> /*unused*/)
-    -> decltype(members.find(key))
-    {
-        return members.find(key);
-    }
-
-    template<typename ObjectType>
-    static auto object_search(ObjectType& members, const string_t& key)
-    -> decltype(object_search_impl(members, key, detail::priority_tag<1> {}))
-    {
-        return object_search_impl(members, key, detail::priority_tag<1> {});
-    }
-
-    /// @brief whether @a it, the result of @ref object_search for @a key, points to @a key
-    ///
-    /// With std::less, the key at the lower bound is equivalent to @a key
-    /// exactly if it is equal to it, and comparing for equality is cheaper.
-    template<typename ObjectType, typename Iterator,
-             detail::enable_if_t<detail::finds_equal_keys<ObjectType, string_t>::value, int> = 0>
-    static auto object_search_found_impl(ObjectType& members, const Iterator& it, const string_t& key, detail::priority_tag<2> /*unused*/)
-    -> decltype(members.lower_bound(key), it->first == key)
-    {
-        return it != members.end() && it->first == key;
-    }
-
-    template<typename ObjectType, typename Iterator>
-    static auto object_search_found_impl(ObjectType& members, const Iterator& it, const string_t& key, detail::priority_tag<1> /*unused*/)
-    -> decltype(members.key_comp()(key, it->first), members.lower_bound(key), bool())
-    {
-        return it != members.end() && !members.key_comp()(key, it->first);
-    }
-
-    template<typename ObjectType, typename Iterator>
-    static bool object_search_found_impl(ObjectType& members, const Iterator& it, const string_t& /*key*/, detail::priority_tag<0> /*unused*/)
-    {
-        return it != members.end();
-    }
-
-    template<typename ObjectType, typename Iterator>
-    static bool object_search_found(ObjectType& members, const Iterator& it, const string_t& key)
-    {
-        return object_search_found_impl(members, it, key, detail::priority_tag<2> {});
-    }
-
-    /// @brief insert @a key, which is missing, into @a object at @a it, the
-    /// result of @ref object_search for @a key
-    template<typename BasicJsonType, typename Iterator>
-    static auto object_insert_impl(BasicJsonType& object, const Iterator& it, const string_t& key, detail::priority_tag<1> /*unused*/)
-    -> decltype(object.m_data.m_value.object->emplace_hint(it, key, nullptr), std::declval<BasicJsonType&>())
-    {
-        const auto old_capacity = object.container_capacity();
-        const auto pos = object.m_data.m_value.object->emplace_hint(it, key, nullptr);
-        return object.set_parent_after_insert(pos->second, old_capacity);
-    }
-
-    template<typename BasicJsonType, typename Iterator>
-    static BasicJsonType& object_insert_impl(BasicJsonType& object, const Iterator& /*it*/, const string_t& key, detail::priority_tag<0> /*unused*/)
-    {
-        return object_add(object, key);
-    }
-
-    template<typename BasicJsonType, typename Iterator>
-    static BasicJsonType& object_insert(BasicJsonType& object, const Iterator& it, const string_t& key)
-    {
-        return object_insert_impl(object, it, key, detail::priority_tag<1> {});
-    }
-
-    /*!
-    @brief undoes the change @ref get_unchecked_create made to an existing value
-
-    @ref get_unchecked walks existing values without changing them. At the
-    first reference token that does not select an existing value, @ref
-    get_unchecked_create makes exactly one change to an existing value: it
-    converts a null value, inserts a key into an object, or grows an array.
-    Everything it creates afterwards lies below that change. If a later step
-    throws, the destructor of this guard reverts that one change, which
-    removes everything created after it, so the value is left as it was
-    (strong exception safety). The guard needs no try/catch and therefore also
-    compiles with JSON_NOEXCEPTION.
-
-    Reverting the change must not throw, as it happens in a destructor. It
-    erases the inserted member by its iterator (no key is converted or
-    compared, nothing is allocated), erases the added elements at the end of
-    the array, or swaps the null value back. This holds for std::map,
-    std::vector, ordered_map, and every object_t, array_t, and
-    json_base_class_t whose erase() and move operations do not throw.
-    Between recording and reverting, nothing is inserted into or erased from
-    the object or array, so the iterator stays valid.
-    */
-    template<typename BasicJsonType>
-    class get_unchecked_rollback
-    {
-      public:
-        using object_iterator = typename BasicJsonType::object_t::iterator;
-
-        /// record that @a member was inserted into the object @a value
-        get_unchecked_rollback(BasicJsonType* value, const object_iterator& member)
-            : m_value(value)
-            , m_member(member)
-            , m_change(change::key_inserted)
-        {}
-
-        /// record the change that the next reference token makes to the null value or array @a value
-        explicit get_unchecked_rollback(BasicJsonType* value)
-            : m_value(value)
-        {
-            if (value->is_null())
-            {
-                // keep the null value (with its base class and positions) and
-                // continue with a fresh null value in its place, which is what
-                // assigning an array or object to it would have left there
-                m_saved_null.swap(*value);
-                m_change = change::null_converted;
-            }
-            else
-            {
-                JSON_ASSERT(value->is_array());
-                m_old_size = value->m_data.m_value.array->size();
-                m_change = change::array_grown;
-            }
-        }
-
-        get_unchecked_rollback(const get_unchecked_rollback&) = delete;
-        get_unchecked_rollback(get_unchecked_rollback&&) = delete;
-        get_unchecked_rollback& operator=(const get_unchecked_rollback&) = delete;
-        get_unchecked_rollback& operator=(get_unchecked_rollback&&) = delete;
-
-        ~get_unchecked_rollback()
-        {
-            if (JSON_HEDLEY_UNLIKELY(m_change != change::none))
-            {
-                undo();
-            }
-        }
-
-        /// keep the change
-        void commit() noexcept
-        {
-            m_change = change::none;
-        }
-
-      private:
-        JSON_HEDLEY_NEVER_INLINE void undo() noexcept
-        {
-            switch (m_change)
-            {
-                case change::null_converted:
-                    // the values created since go away with m_saved_null
-                    m_value->swap(m_saved_null);
-                    break;
-                case change::key_inserted:
-                    m_value->m_data.m_value.object->erase(m_member);
-                    break;
-                case change::array_grown:
-                {
-                    auto& elements = *m_value->m_data.m_value.array;
-                    elements.resize(m_old_size);
-                    break;
-                }
-                case change::none:
-                default:
-                    break;
-            }
-        }
-
-        enum class change
-        {
-            none,
-            null_converted,
-            key_inserted,
-            array_grown
-        };
-
-        BasicJsonType* m_value;
-        object_iterator m_member {};
-        typename BasicJsonType::size_type m_old_size = 0;
-        change m_change = change::none;
-        BasicJsonType m_saved_null {};
-    };
-
   JSON_PRIVATE_UNLESS_TESTED:
     json_pointer top() const
     {
@@ -1217,245 +972,47 @@ class json_pointer
 
     Complexity: Linear in the length of the JSON pointer.
 
-    Exception safety: Strong guarantee. This function walks existing values
+    @throw parse_error.106   if an array index begins with '0'
-    until the first reference token that does not select one. If that is the
+    @throw parse_error.109   if an array index was not a number
-    last reference token, inserting the key into the object or growing the
+    @throw out_of_range.404  if the JSON pointer can not be resolved
-    array is strongly exception safe by itself; any other change is made by
-    @ref get_unchecked_create, which undoes it if a later step throws.
-
-    @throw parse_error.106   if an array index of an existing array begins with '0'
-    @throw parse_error.109   if an array index of an existing array was not a number
-    @throw out_of_range.404  if the JSON pointer can not be resolved, or if an
-                             array index is too large for strtoull
-    @throw out_of_range.410  if an array index exceeds size_type
-    @throw std::length_error if padding an array exceeds its maximal size
-    @throw std::bad_alloc    if an allocation fails
     */
     template<typename BasicJsonType>
     BasicJsonType& get_unchecked(BasicJsonType* ptr) const
     {
-        const auto* token = reference_tokens.data();
+        for (const auto& reference_token : reference_tokens)
-        const auto* const end = token + reference_tokens.size();
-        if (token == end)
         {
-            return *ptr;
-        }
-        if (ptr->is_null())
-        {
-            // the null value is converted to an array or an object
-            return get_unchecked_create(ptr, token, end);
-        }
-
-        // walk existing values; the walk itself inserts nothing, which keeps
-        // the loop fast (insertions are done below or in get_unchecked_create)
-        typename BasicJsonType::object_t::iterator it;
-        do
-        {
-            if (ptr->is_object())
-            {
-                auto& members = *ptr->m_data.m_value.object;
-                it = object_search(members, *token);
-                if (!object_search_found(members, it, *token))
-                {
-                    // the key is inserted below or in get_unchecked_create
-                    break;
-                }
-                ptr = &it->second;
-            }
-            else if (ptr->is_array())
-            {
-                if (is_dash(*token))
-                {
-                    if (token + 1 == end)
-                    {
-                        // explicitly treat "-" as index beyond the end
-                        return array_append(*ptr);
-                    }
-                    // an element will be appended in get_unchecked_create
-                    break;
-                }
-                if (token + 1 == end)
-                {
-                    // convert array index to number; unchecked access
-                    return ptr->operator[](array_index<BasicJsonType>(*token));
-                }
-                const auto idx = array_index<BasicJsonType>(*token);
-                if (idx >= ptr->m_data.m_value.array->size())
-                {
-                    // the array will be padded in get_unchecked_create
-                    break;
-                }
-                ptr = &(*ptr->m_data.m_value.array)[idx];
-            }
-            else if (ptr->is_null())
-            {
-                // the null value will be converted in get_unchecked_create
-                break;
-            }
-            else
-            {
-                // primitive value with a reference token left
-                throw_unresolved(*token, ptr);
-            }
-        }
-        while (++token != end);
-
-        if (token == end)
-        {
-            return *ptr;
-        }
-        if (token + 1 == end && ptr->is_object())
-        {
-            // insert the missing key where the walk found it belongs
-            return object_insert(*ptr, it, *token);
-        }
-        return get_unchecked_create(ptr, token, end);
-    }
-
-    /*!
-    @brief create the values for the reference tokens from @a first on
-
-    Continues @ref get_unchecked at the first reference token that does not
-    select an existing value. That token changes @a ptr, an existing value (see
-    @ref get_unchecked_rollback); all later tokens only change values created
-    here.
-
-    If the emplace() of the object type finds a member that the lookup in @ref
-    get_unchecked did not find (the two disagree, e.g. for fifo_map), nothing
-    is inserted, and the walk continues with that member as in @ref
-    get_unchecked, which is what basic_json::operator[] does.
-
-    @param[in] ptr    the value the reference token @a first is applied to
-    @param[in] first  the first reference token to resolve (an element of
-                      @ref reference_tokens)
-    @param[in] end    the end of @ref reference_tokens
-
-    @throw parse_error.106   if an array index of an existing array begins with
-                             '0' (only after emplace() found a member, see above)
-    @throw parse_error.109   if an array index of an existing array was not a
-                             number (only after emplace() found a member)
-    @throw out_of_range.404  if an array index is too large for strtoull, or (only
-                             after emplace() found a member) if the JSON pointer
-                             can not be resolved
-    @throw out_of_range.410  if an array index exceeds size_type
-    @throw std::length_error if padding an array exceeds its maximal size
-    @throw std::bad_alloc    if an allocation fails
-    */
-    template<typename BasicJsonType>
-    JSON_HEDLEY_NEVER_INLINE static BasicJsonType& get_unchecked_create(BasicJsonType* ptr, const string_t* first, const string_t* end)
-    {
-        const auto* token = first;
-        while (ptr->is_object())
-        {
-            const auto old_capacity = ptr->container_capacity();
-            const auto inserted = ptr->m_data.m_value.object->emplace(*token, nullptr);
-            if (JSON_HEDLEY_LIKELY(inserted.second))
-            {
-                get_unchecked_rollback<BasicJsonType> rollback(ptr, inserted.first);
-                BasicJsonType& result = create_values(&ptr->set_parent_after_insert(inserted.first->second, old_capacity), token + 1, end);
-                rollback.commit();
-                return result;
-            }
-
-            // nothing was inserted: walk existing values from the member
-            // emplace() found on, like get_unchecked
-            ptr = &inserted.first->second;
-            while (++token != end)
-            {
-                if (ptr->is_object())
-                {
-                    auto& members = *ptr->m_data.m_value.object;
-                    const auto it = object_search(members, *token);
-                    if (!object_search_found(members, it, *token))
-                    {
-                        break;
-                    }
-                    ptr = &it->second;
-                }
-                else if (ptr->is_array())
-                {
-                    if (is_dash(*token))
-                    {
-                        break;
-                    }
-                    const auto idx = array_index<BasicJsonType>(*token);
-                    if (idx >= ptr->m_data.m_value.array->size())
-                    {
-                        break;
-                    }
-                    ptr = &(*ptr->m_data.m_value.array)[idx];
-                }
-                else if (ptr->is_null())
-                {
-                    break;
-                }
-                else
-                {
-                    // primitive value with a reference token left
-                    throw_unresolved(*token, ptr);
-                }
-            }
-            if (token == end)
-            {
-                return *ptr;
-            }
-        }
-
-        get_unchecked_rollback<BasicJsonType> rollback(ptr);
-        BasicJsonType& result = create_values(ptr, token, end);
-        rollback.commit();
-        return result;
-    }
-
-    /*!
-    @brief the loop of @ref get_unchecked_create
-
-    @a ptr is a null value or an array. Every value created here is a fresh
-    one, so every object a key is added to is empty.
-
-    @throw out_of_range.404  if an array index is too large for strtoull
-    @throw out_of_range.410  if an array index exceeds size_type
-    @throw std::length_error if padding an array exceeds its maximal size
-    @throw std::bad_alloc    if an allocation fails
-    */
-    template<typename BasicJsonType>
-    static BasicJsonType& create_values(BasicJsonType* ptr, const string_t* first, const string_t* end)
-    {
-        for (const auto* token = first; token != end; ++token)
-        {
-            const auto& reference_token = *token;
-
             // convert null values to arrays or objects before continuing
             if (ptr->is_null())
             {
-                // change value to an array for array indices or "-" or to an
+                // check if the reference token is a valid array index, that is
-                // object otherwise; every null value here is a fresh one (see
+                // a nonempty sequence of digits without a leading '0'
-                // get_unchecked_rollback), so converting it in place leaves
+                // (cf. RFC 6901, Sect. 4); tokens that could never be a valid
-                // the same value as assigning an array or object to it
+                // array index (such as "01" or "") are treated as object keys
-                if (is_array_index_token(reference_token) || is_dash(reference_token))
+                const bool nums = !reference_token.empty()
+                                  && (reference_token.size() == 1 || reference_token[0] != '0')
+                                  && std::all_of(reference_token.begin(), reference_token.end(),
+                                                 [](const unsigned char x)
                 {
-                    ptr->template convert_null_to<typename BasicJsonType::array_t>();
+                    return std::isdigit(x);
-                }
+                });
-                else
+
-                {
+                // change value to an array for array indices or "-" or to object otherwise
-                    ptr->template convert_null_to<typename BasicJsonType::object_t>();
+                *ptr = (nums || reference_token == "-")
-                }
+                       ? detail::value_t::array
+                       : detail::value_t::object;
             }
 
             if (ptr->is_object())
             {
-                // the object is new and empty
+                // use unchecked object access
-                ptr = &object_add(*ptr, reference_token);
+                ptr = &object_member(*ptr, reference_token);
             }
-            else
+            else if (ptr->is_array())
             {
-                // only the first value is not created here, and it is a null
+                if (reference_token == "-")
-                // value or an array (see get_unchecked_create)
-                JSON_ASSERT(ptr->is_array());
-                if (is_dash(reference_token))
                 {
                     // explicitly treat "-" as index beyond the end
-                    ptr = &array_append(*ptr);
+                    ptr = &ptr->operator[](ptr->m_data.m_value.array->size());
                 }
                 else
                 {
@@ -1463,31 +1020,16 @@ class json_pointer
                     ptr = &ptr->operator[](array_index<BasicJsonType>(reference_token));
                 }
             }
+            else
+            {
+                // primitive value with a reference token left
+                throw_unresolved(reference_token, ptr);
+            }
         }
 
         return *ptr;
     }
 
-    /// whether @a s is a valid array index, that is a nonempty sequence of
-    /// digits without a leading '0' (cf. RFC 6901, Sect. 4); tokens that could
-    /// never be a valid array index (such as "01" or "") are object keys
-    static bool is_array_index_token(const string_t& s)
-    {
-        const std::size_t size = s.size();
-        if (size == 0 || (size > 1 && s[0] == '0'))
-        {
-            return false;
-        }
-        for (std::size_t i = 0; i < size; ++i)
-        {
-            if (std::isdigit(static_cast<unsigned char>(s[i])) == 0)
-            {
-                return false;
-            }
-        }
-        return true;
-    }
-
     /*!
     @throw parse_error.106   if an array index begins with '0'
     @throw parse_error.109   if an array index was not a number

include/nlohmann/detail/view/edit.hpp

@@ -26,6 +26,7 @@
 #include <nlohmann/detail/view/lookup.hpp>
 #include <nlohmann/detail/view/macro_scope.hpp>
 #include <nlohmann/detail/view/node.hpp>
+#include <nlohmann/detail/view/scan.hpp>
 #include <nlohmann/detail/view/view_mode.hpp>
 
 NLOHMANN_JSON_NAMESPACE_BEGIN
@@ -46,10 +47,9 @@ struct first_true<false, Conditions...> : std::integral_constant < int, 1 + firs
 /// (type_error.316 with the same message), so that an editable document
 /// only holds valid UTF-8: the error is at the first byte that no
 /// well-formed sequence can continue with (Unicode, Table 3-7).
-inline void check_utf8(const char* s, std::size_t n)
+NLOHMANN_VIEW_ALWAYS_INLINE void check_utf8_bytewise(const unsigned char* p, std::size_t n, std::size_t i)
 {
-    const auto* const p = reinterpret_cast<const unsigned char*>(s); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
+    for (; i < n;)
-    for (std::size_t i = 0; i < n;)
     {
         const unsigned char c = p[i];
         if (c < 0x80)
@@ -79,6 +79,43 @@ inline void check_utf8(const char* s, std::size_t n)
     }
 }
 
+/// The long-string path of check_utf8(): skips well-formed runs with the
+/// parser's string scan (which stops at a quote, a backslash, or a control
+/// character, all legal here, and at ill-formed UTF-8); the byte-wise check
+/// runs only from where the scan found ill-formed UTF-8, to report it.
+NLOHMANN_VIEW_NOINLINE inline void check_utf8_scan(const unsigned char* p, std::size_t n)
+{
+    const unsigned char* const e = p + n;
+    const unsigned char* q = p;
+    for (;;)
+    {
+        q = scan_string_run<true>(q, e);
+        if (q == e)
+        {
+            return;
+        }
+        if (*q >= 0x80)
+        {
+            check_utf8_bytewise(p, n, static_cast<std::size_t>(q - p));
+            return; // not reached: the scan stopped at ill-formed UTF-8
+        }
+        ++q; // a quote, backslash, or control character
+    }
+}
+
+inline void check_utf8(const char* s, std::size_t n)
+{
+    const auto* const p = reinterpret_cast<const unsigned char*>(s); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
+    if (NLOHMANN_VIEW_UNLIKELY(n >= 32))
+    {
+        check_utf8_scan(p, n);
+        return;
+    }
+    // short strings: the vector scan costs more than it saves; (the loop is
+    // inlined with a constant start, so this path is the plain byte-wise check)
+    check_utf8_bytewise(p, n, 0);
+}
+
 /*!
 @brief the edits of an editable basic_json_document
 

single_include/nlohmann/json.hpp

@@ -24064,7 +24064,7 @@ NLOHMANN_JSON_NAMESPACE_END
 
 
 
-#include <algorithm> // copy
+#include <algorithm> // all_of, copy
 #include <array> // array
 #include <cctype> // isdigit
 #include <cerrno> // errno, ERANGE
@@ -24072,11 +24072,9 @@ NLOHMANN_JSON_NAMESPACE_END
 #include <cstdint> // uint64_t
 #include <cstdlib> // strtoull
 #include <cstring> // memchr, memcmp, memcpy
-#include <functional> // less
 #ifndef JSON_NO_IO
     #include <iosfwd> // ostream
 #endif  // JSON_NO_IO
-#include <iterator> // next
 #include <limits> // max
 #include <string> // string
 #include <utility> // move
@@ -24090,8 +24088,6 @@ NLOHMANN_JSON_NAMESPACE_END
 
 // #include <nlohmann/detail/meta/type_traits.hpp>
 
-// #include <nlohmann/detail/meta/void_t.hpp>
-
 // #include <nlohmann/detail/string_concat.hpp>
 
 // #include <nlohmann/detail/string_escape.hpp>
@@ -24202,26 +24198,6 @@ JSON_HEDLEY_ALWAYS_INLINE array_index_status parse_array_index(const StringType&
     return array_index_status::ok;
 }
 
-/// whether two keys of type @a StringType are equivalent under @a Compare
-/// exactly if they are equal: std::less<StringType> and std::less<>
-template<typename Compare, typename StringType>
-struct is_less_comparator : std::false_type {};
-
-template<typename StringType>
-struct is_less_comparator<std::less<StringType>, StringType> : std::true_type {};
-
-template<typename StringType>
-struct is_less_comparator<std::less<void>, StringType> : std::true_type {};
-
-/// whether the objects of type @a ObjectType find a key of type @a StringType
-/// exactly if it equals a stored key
-template<typename ObjectType, typename StringType, typename = void>
-struct finds_equal_keys : std::false_type {};
-
-template<typename ObjectType, typename StringType>
-struct finds_equal_keys<ObjectType, StringType, void_t<typename ObjectType::key_compare, typename ObjectType::key_type>>
-    : conjunction<is_less_comparator<typename ObjectType::key_compare, StringType>, std::is_same<typename ObjectType::key_type, StringType>> {};
-
 }  // namespace detail
 
 /// @brief JSON Pointer defines a string syntax for identifying a specific value within a JSON document
@@ -24677,228 +24653,6 @@ class json_pointer
         return object_member_impl(object, key, detail::priority_tag<1> {});
     }
 
-    /// @brief append a null value to @a array, like operator[] with the size of @a array
-    template<typename BasicJsonType>
-    static BasicJsonType& array_append(BasicJsonType& array)
-    {
-        auto& elements = *array.m_data.m_value.array;
-        const auto old_capacity = array.container_capacity();
-        elements.emplace_back();
-        return array.set_parent_after_insert(elements.back(), old_capacity);
-    }
-
-    /// @brief add @a key, which is missing, to @a object
-    template<typename BasicJsonType>
-    static auto object_add_impl(BasicJsonType& object, const string_t& key, detail::priority_tag<1> /*unused*/)
-    -> decltype(object.m_data.m_value.object->emplace(key, nullptr), std::declval<BasicJsonType&>())
-    {
-        const auto old_capacity = object.container_capacity();
-        const auto it = object.m_data.m_value.object->emplace(key, nullptr).first;
-        return object.set_parent_after_insert(it->second, old_capacity);
-    }
-
-    template<typename BasicJsonType>
-    static BasicJsonType& object_add_impl(BasicJsonType& object, const string_t& key, detail::priority_tag<0> /*unused*/)
-    {
-        return object.operator[](key);
-    }
-
-    template<typename BasicJsonType>
-    static BasicJsonType& object_add(BasicJsonType& object, const string_t& key)
-    {
-        return object_add_impl(object, key, detail::priority_tag<1> {});
-    }
-
-    /// @brief search @a key in @a members: the result is where the key is or,
-    /// for objects that can insert at a position (std::map), where it belongs
-    template<typename ObjectType>
-    static auto object_search_impl(ObjectType& members, const string_t& key, detail::priority_tag<1> /*unused*/)
-    -> decltype(members.key_comp()(key, members.lower_bound(key)->first), members.lower_bound(key))
-    {
-        return members.lower_bound(key);
-    }
-
-    template<typename ObjectType>
-    static auto object_search_impl(ObjectType& members, const string_t& key, detail::priority_tag<0> /*unused*/)
-    -> decltype(members.find(key))
-    {
-        return members.find(key);
-    }
-
-    template<typename ObjectType>
-    static auto object_search(ObjectType& members, const string_t& key)
-    -> decltype(object_search_impl(members, key, detail::priority_tag<1> {}))
-    {
-        return object_search_impl(members, key, detail::priority_tag<1> {});
-    }
-
-    /// @brief whether @a it, the result of @ref object_search for @a key, points to @a key
-    ///
-    /// With std::less, the key at the lower bound is equivalent to @a key
-    /// exactly if it is equal to it, and comparing for equality is cheaper.
-    template<typename ObjectType, typename Iterator,
-             detail::enable_if_t<detail::finds_equal_keys<ObjectType, string_t>::value, int> = 0>
-    static auto object_search_found_impl(ObjectType& members, const Iterator& it, const string_t& key, detail::priority_tag<2> /*unused*/)
-    -> decltype(members.lower_bound(key), it->first == key)
-    {
-        return it != members.end() && it->first == key;
-    }
-
-    template<typename ObjectType, typename Iterator>
-    static auto object_search_found_impl(ObjectType& members, const Iterator& it, const string_t& key, detail::priority_tag<1> /*unused*/)
-    -> decltype(members.key_comp()(key, it->first), members.lower_bound(key), bool())
-    {
-        return it != members.end() && !members.key_comp()(key, it->first);
-    }
-
-    template<typename ObjectType, typename Iterator>
-    static bool object_search_found_impl(ObjectType& members, const Iterator& it, const string_t& /*key*/, detail::priority_tag<0> /*unused*/)
-    {
-        return it != members.end();
-    }
-
-    template<typename ObjectType, typename Iterator>
-    static bool object_search_found(ObjectType& members, const Iterator& it, const string_t& key)
-    {
-        return object_search_found_impl(members, it, key, detail::priority_tag<2> {});
-    }
-
-    /// @brief insert @a key, which is missing, into @a object at @a it, the
-    /// result of @ref object_search for @a key
-    template<typename BasicJsonType, typename Iterator>
-    static auto object_insert_impl(BasicJsonType& object, const Iterator& it, const string_t& key, detail::priority_tag<1> /*unused*/)
-    -> decltype(object.m_data.m_value.object->emplace_hint(it, key, nullptr), std::declval<BasicJsonType&>())
-    {
-        const auto old_capacity = object.container_capacity();
-        const auto pos = object.m_data.m_value.object->emplace_hint(it, key, nullptr);
-        return object.set_parent_after_insert(pos->second, old_capacity);
-    }
-
-    template<typename BasicJsonType, typename Iterator>
-    static BasicJsonType& object_insert_impl(BasicJsonType& object, const Iterator& /*it*/, const string_t& key, detail::priority_tag<0> /*unused*/)
-    {
-        return object_add(object, key);
-    }
-
-    template<typename BasicJsonType, typename Iterator>
-    static BasicJsonType& object_insert(BasicJsonType& object, const Iterator& it, const string_t& key)
-    {
-        return object_insert_impl(object, it, key, detail::priority_tag<1> {});
-    }
-
-    /*!
-    @brief undoes the change @ref get_unchecked_create made to an existing value
-
-    @ref get_unchecked walks existing values without changing them. At the
-    first reference token that does not select an existing value, @ref
-    get_unchecked_create makes exactly one change to an existing value: it
-    converts a null value, inserts a key into an object, or grows an array.
-    Everything it creates afterwards lies below that change. If a later step
-    throws, the destructor of this guard reverts that one change, which
-    removes everything created after it, so the value is left as it was
-    (strong exception safety). The guard needs no try/catch and therefore also
-    compiles with JSON_NOEXCEPTION.
-
-    Reverting the change must not throw, as it happens in a destructor. It
-    erases the inserted member by its iterator (no key is converted or
-    compared, nothing is allocated), erases the added elements at the end of
-    the array, or swaps the null value back. This holds for std::map,
-    std::vector, ordered_map, and every object_t, array_t, and
-    json_base_class_t whose erase() and move operations do not throw.
-    Between recording and reverting, nothing is inserted into or erased from
-    the object or array, so the iterator stays valid.
-    */
-    template<typename BasicJsonType>
-    class get_unchecked_rollback
-    {
-      public:
-        using object_iterator = typename BasicJsonType::object_t::iterator;
-
-        /// record that @a member was inserted into the object @a value
-        get_unchecked_rollback(BasicJsonType* value, const object_iterator& member)
-            : m_value(value)
-            , m_member(member)
-            , m_change(change::key_inserted)
-        {}
-
-        /// record the change that the next reference token makes to the null value or array @a value
-        explicit get_unchecked_rollback(BasicJsonType* value)
-            : m_value(value)
-        {
-            if (value->is_null())
-            {
-                // keep the null value (with its base class and positions) and
-                // continue with a fresh null value in its place, which is what
-                // assigning an array or object to it would have left there
-                m_saved_null.swap(*value);
-                m_change = change::null_converted;
-            }
-            else
-            {
-                JSON_ASSERT(value->is_array());
-                m_old_size = value->m_data.m_value.array->size();
-                m_change = change::array_grown;
-            }
-        }
-
-        get_unchecked_rollback(const get_unchecked_rollback&) = delete;
-        get_unchecked_rollback(get_unchecked_rollback&&) = delete;
-        get_unchecked_rollback& operator=(const get_unchecked_rollback&) = delete;
-        get_unchecked_rollback& operator=(get_unchecked_rollback&&) = delete;
-
-        ~get_unchecked_rollback()
-        {
-            if (JSON_HEDLEY_UNLIKELY(m_change != change::none))
-            {
-                undo();
-            }
-        }
-
-        /// keep the change
-        void commit() noexcept
-        {
-            m_change = change::none;
-        }
-
-      private:
-        JSON_HEDLEY_NEVER_INLINE void undo() noexcept
-        {
-            switch (m_change)
-            {
-                case change::null_converted:
-                    // the values created since go away with m_saved_null
-                    m_value->swap(m_saved_null);
-                    break;
-                case change::key_inserted:
-                    m_value->m_data.m_value.object->erase(m_member);
-                    break;
-                case change::array_grown:
-                {
-                    auto& elements = *m_value->m_data.m_value.array;
-                    elements.resize(m_old_size);
-                    break;
-                }
-                case change::none:
-                default:
-                    break;
-            }
-        }
-
-        enum class change
-        {
-            none,
-            null_converted,
-            key_inserted,
-            array_grown
-        };
-
-        BasicJsonType* m_value;
-        object_iterator m_member {};
-        typename BasicJsonType::size_type m_old_size = 0;
-        change m_change = change::none;
-        BasicJsonType m_saved_null {};
-    };
-
   JSON_PRIVATE_UNLESS_TESTED:
     json_pointer top() const
     {
@@ -25282,245 +25036,47 @@ class json_pointer
 
     Complexity: Linear in the length of the JSON pointer.
 
-    Exception safety: Strong guarantee. This function walks existing values
+    @throw parse_error.106   if an array index begins with '0'
-    until the first reference token that does not select one. If that is the
+    @throw parse_error.109   if an array index was not a number
-    last reference token, inserting the key into the object or growing the
+    @throw out_of_range.404  if the JSON pointer can not be resolved
-    array is strongly exception safe by itself; any other change is made by
-    @ref get_unchecked_create, which undoes it if a later step throws.
-
-    @throw parse_error.106   if an array index of an existing array begins with '0'
-    @throw parse_error.109   if an array index of an existing array was not a number
-    @throw out_of_range.404  if the JSON pointer can not be resolved, or if an
-                             array index is too large for strtoull
-    @throw out_of_range.410  if an array index exceeds size_type
-    @throw std::length_error if padding an array exceeds its maximal size
-    @throw std::bad_alloc    if an allocation fails
     */
     template<typename BasicJsonType>
     BasicJsonType& get_unchecked(BasicJsonType* ptr) const
     {
-        const auto* token = reference_tokens.data();
+        for (const auto& reference_token : reference_tokens)
-        const auto* const end = token + reference_tokens.size();
-        if (token == end)
         {
-            return *ptr;
-        }
-        if (ptr->is_null())
-        {
-            // the null value is converted to an array or an object
-            return get_unchecked_create(ptr, token, end);
-        }
-
-        // walk existing values; the walk itself inserts nothing, which keeps
-        // the loop fast (insertions are done below or in get_unchecked_create)
-        typename BasicJsonType::object_t::iterator it;
-        do
-        {
-            if (ptr->is_object())
-            {
-                auto& members = *ptr->m_data.m_value.object;
-                it = object_search(members, *token);
-                if (!object_search_found(members, it, *token))
-                {
-                    // the key is inserted below or in get_unchecked_create
-                    break;
-                }
-                ptr = &it->second;
-            }
-            else if (ptr->is_array())
-            {
-                if (is_dash(*token))
-                {
-                    if (token + 1 == end)
-                    {
-                        // explicitly treat "-" as index beyond the end
-                        return array_append(*ptr);
-                    }
-                    // an element will be appended in get_unchecked_create
-                    break;
-                }
-                if (token + 1 == end)
-                {
-                    // convert array index to number; unchecked access
-                    return ptr->operator[](array_index<BasicJsonType>(*token));
-                }
-                const auto idx = array_index<BasicJsonType>(*token);
-                if (idx >= ptr->m_data.m_value.array->size())
-                {
-                    // the array will be padded in get_unchecked_create
-                    break;
-                }
-                ptr = &(*ptr->m_data.m_value.array)[idx];
-            }
-            else if (ptr->is_null())
-            {
-                // the null value will be converted in get_unchecked_create
-                break;
-            }
-            else
-            {
-                // primitive value with a reference token left
-                throw_unresolved(*token, ptr);
-            }
-        }
-        while (++token != end);
-
-        if (token == end)
-        {
-            return *ptr;
-        }
-        if (token + 1 == end && ptr->is_object())
-        {
-            // insert the missing key where the walk found it belongs
-            return object_insert(*ptr, it, *token);
-        }
-        return get_unchecked_create(ptr, token, end);
-    }
-
-    /*!
-    @brief create the values for the reference tokens from @a first on
-
-    Continues @ref get_unchecked at the first reference token that does not
-    select an existing value. That token changes @a ptr, an existing value (see
-    @ref get_unchecked_rollback); all later tokens only change values created
-    here.
-
-    If the emplace() of the object type finds a member that the lookup in @ref
-    get_unchecked did not find (the two disagree, e.g. for fifo_map), nothing
-    is inserted, and the walk continues with that member as in @ref
-    get_unchecked, which is what basic_json::operator[] does.
-
-    @param[in] ptr    the value the reference token @a first is applied to
-    @param[in] first  the first reference token to resolve (an element of
-                      @ref reference_tokens)
-    @param[in] end    the end of @ref reference_tokens
-
-    @throw parse_error.106   if an array index of an existing array begins with
-                             '0' (only after emplace() found a member, see above)
-    @throw parse_error.109   if an array index of an existing array was not a
-                             number (only after emplace() found a member)
-    @throw out_of_range.404  if an array index is too large for strtoull, or (only
-                             after emplace() found a member) if the JSON pointer
-                             can not be resolved
-    @throw out_of_range.410  if an array index exceeds size_type
-    @throw std::length_error if padding an array exceeds its maximal size
-    @throw std::bad_alloc    if an allocation fails
-    */
-    template<typename BasicJsonType>
-    JSON_HEDLEY_NEVER_INLINE static BasicJsonType& get_unchecked_create(BasicJsonType* ptr, const string_t* first, const string_t* end)
-    {
-        const auto* token = first;
-        while (ptr->is_object())
-        {
-            const auto old_capacity = ptr->container_capacity();
-            const auto inserted = ptr->m_data.m_value.object->emplace(*token, nullptr);
-            if (JSON_HEDLEY_LIKELY(inserted.second))
-            {
-                get_unchecked_rollback<BasicJsonType> rollback(ptr, inserted.first);
-                BasicJsonType& result = create_values(&ptr->set_parent_after_insert(inserted.first->second, old_capacity), token + 1, end);
-                rollback.commit();
-                return result;
-            }
-
-            // nothing was inserted: walk existing values from the member
-            // emplace() found on, like get_unchecked
-            ptr = &inserted.first->second;
-            while (++token != end)
-            {
-                if (ptr->is_object())
-                {
-                    auto& members = *ptr->m_data.m_value.object;
-                    const auto it = object_search(members, *token);
-                    if (!object_search_found(members, it, *token))
-                    {
-                        break;
-                    }
-                    ptr = &it->second;
-                }
-                else if (ptr->is_array())
-                {
-                    if (is_dash(*token))
-                    {
-                        break;
-                    }
-                    const auto idx = array_index<BasicJsonType>(*token);
-                    if (idx >= ptr->m_data.m_value.array->size())
-                    {
-                        break;
-                    }
-                    ptr = &(*ptr->m_data.m_value.array)[idx];
-                }
-                else if (ptr->is_null())
-                {
-                    break;
-                }
-                else
-                {
-                    // primitive value with a reference token left
-                    throw_unresolved(*token, ptr);
-                }
-            }
-            if (token == end)
-            {
-                return *ptr;
-            }
-        }
-
-        get_unchecked_rollback<BasicJsonType> rollback(ptr);
-        BasicJsonType& result = create_values(ptr, token, end);
-        rollback.commit();
-        return result;
-    }
-
-    /*!
-    @brief the loop of @ref get_unchecked_create
-
-    @a ptr is a null value or an array. Every value created here is a fresh
-    one, so every object a key is added to is empty.
-
-    @throw out_of_range.404  if an array index is too large for strtoull
-    @throw out_of_range.410  if an array index exceeds size_type
-    @throw std::length_error if padding an array exceeds its maximal size
-    @throw std::bad_alloc    if an allocation fails
-    */
-    template<typename BasicJsonType>
-    static BasicJsonType& create_values(BasicJsonType* ptr, const string_t* first, const string_t* end)
-    {
-        for (const auto* token = first; token != end; ++token)
-        {
-            const auto& reference_token = *token;
-
             // convert null values to arrays or objects before continuing
             if (ptr->is_null())
             {
-                // change value to an array for array indices or "-" or to an
+                // check if the reference token is a valid array index, that is
-                // object otherwise; every null value here is a fresh one (see
+                // a nonempty sequence of digits without a leading '0'
-                // get_unchecked_rollback), so converting it in place leaves
+                // (cf. RFC 6901, Sect. 4); tokens that could never be a valid
-                // the same value as assigning an array or object to it
+                // array index (such as "01" or "") are treated as object keys
-                if (is_array_index_token(reference_token) || is_dash(reference_token))
+                const bool nums = !reference_token.empty()
+                                  && (reference_token.size() == 1 || reference_token[0] != '0')
+                                  && std::all_of(reference_token.begin(), reference_token.end(),
+                                                 [](const unsigned char x)
                 {
-                    ptr->template convert_null_to<typename BasicJsonType::array_t>();
+                    return std::isdigit(x);
-                }
+                });
-                else
+
-                {
+                // change value to an array for array indices or "-" or to object otherwise
-                    ptr->template convert_null_to<typename BasicJsonType::object_t>();
+                *ptr = (nums || reference_token == "-")
-                }
+                       ? detail::value_t::array
+                       : detail::value_t::object;
             }
 
             if (ptr->is_object())
             {
-                // the object is new and empty
+                // use unchecked object access
-                ptr = &object_add(*ptr, reference_token);
+                ptr = &object_member(*ptr, reference_token);
             }
-            else
+            else if (ptr->is_array())
             {
-                // only the first value is not created here, and it is a null
+                if (reference_token == "-")
-                // value or an array (see get_unchecked_create)
-                JSON_ASSERT(ptr->is_array());
-                if (is_dash(reference_token))
                 {
                     // explicitly treat "-" as index beyond the end
-                    ptr = &array_append(*ptr);
+                    ptr = &ptr->operator[](ptr->m_data.m_value.array->size());
                 }
                 else
                 {
@@ -25528,31 +25084,16 @@ class json_pointer
                     ptr = &ptr->operator[](array_index<BasicJsonType>(reference_token));
                 }
             }
+            else
+            {
+                // primitive value with a reference token left
+                throw_unresolved(reference_token, ptr);
+            }
         }
 
         return *ptr;
     }
 
-    /// whether @a s is a valid array index, that is a nonempty sequence of
-    /// digits without a leading '0' (cf. RFC 6901, Sect. 4); tokens that could
-    /// never be a valid array index (such as "01" or "") are object keys
-    static bool is_array_index_token(const string_t& s)
-    {
-        const std::size_t size = s.size();
-        if (size == 0 || (size > 1 && s[0] == '0'))
-        {
-            return false;
-        }
-        for (std::size_t i = 0; i < size; ++i)
-        {
-            if (std::isdigit(static_cast<unsigned char>(s[i])) == 0)
-            {
-                return false;
-            }
-        }
-        return true;
-    }
-
     /*!
     @throw parse_error.106   if an array index begins with '0'
     @throw parse_error.109   if an array index was not a number

single_include/nlohmann/json_view.hpp

@@ -3740,6 +3740,8 @@ NLOHMANN_JSON_NAMESPACE_END
 
 // #include <nlohmann/detail/view/node.hpp>
 
+// #include <nlohmann/detail/view/scan.hpp>
+
 // #include <nlohmann/detail/view/view_mode.hpp>
 //     __ _____ _____ _____
 //  __|  |   __|     |   | |  JSON for Modern C++
@@ -3813,10 +3815,9 @@ struct first_true<false, Conditions...> : std::integral_constant < int, 1 + firs
 /// (type_error.316 with the same message), so that an editable document
 /// only holds valid UTF-8: the error is at the first byte that no
 /// well-formed sequence can continue with (Unicode, Table 3-7).
-inline void check_utf8(const char* s, std::size_t n)
+NLOHMANN_VIEW_ALWAYS_INLINE void check_utf8_bytewise(const unsigned char* p, std::size_t n, std::size_t i)
 {
-    const auto* const p = reinterpret_cast<const unsigned char*>(s); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
+    for (; i < n;)
-    for (std::size_t i = 0; i < n;)
     {
         const unsigned char c = p[i];
         if (c < 0x80)
@@ -3846,6 +3847,43 @@ inline void check_utf8(const char* s, std::size_t n)
     }
 }
 
+/// The long-string path of check_utf8(): skips well-formed runs with the
+/// parser's string scan (which stops at a quote, a backslash, or a control
+/// character, all legal here, and at ill-formed UTF-8); the byte-wise check
+/// runs only from where the scan found ill-formed UTF-8, to report it.
+NLOHMANN_VIEW_NOINLINE inline void check_utf8_scan(const unsigned char* p, std::size_t n)
+{
+    const unsigned char* const e = p + n;
+    const unsigned char* q = p;
+    for (;;)
+    {
+        q = scan_string_run<true>(q, e);
+        if (q == e)
+        {
+            return;
+        }
+        if (*q >= 0x80)
+        {
+            check_utf8_bytewise(p, n, static_cast<std::size_t>(q - p));
+            return; // not reached: the scan stopped at ill-formed UTF-8
+        }
+        ++q; // a quote, backslash, or control character
+    }
+}
+
+inline void check_utf8(const char* s, std::size_t n)
+{
+    const auto* const p = reinterpret_cast<const unsigned char*>(s); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
+    if (NLOHMANN_VIEW_UNLIKELY(n >= 32))
+    {
+        check_utf8_scan(p, n);
+        return;
+    }
+    // short strings: the vector scan costs more than it saves; (the loop is
+    // inlined with a constant start, so this path is the plain byte-wise check)
+    check_utf8_bytewise(p, n, 0);
+}
+
 /*!
 @brief the edits of an editable basic_json_document
 

tests/src/unit-alt-string.cpp

@@ -373,23 +373,6 @@ TEST_CASE("alternative string type")
         CHECK_FALSE(j.contains(alt_json::json_pointer("/foo/bar")));
     }
 
-    SECTION("non-const operator[](json_pointer)")
-    {
-        // creating values must compile without string_t::begin()
-        alt_json j = alt_json::parse(R"({"a":{"b":[1]},"c":null})");
-        const alt_json before = j;
-        j[alt_json::json_pointer("/a/b/3/x/-/y")] = 1; // pad, append, create
-        j[alt_json::json_pointer("/a/z")] = 2;         // insert a key
-        j[alt_json::json_pointer("/c/0/q")] = 3;       // convert null
-        j[alt_json::json_pointer("/a/b/-")] = 4;       // append
-        CHECK(j.dump() == R"({"a":{"b":[1,null,null,{"x":[{"y":1}]},4],"z":2},"c":[{"q":3}]})");
-
-        // a failing access leaves the value unchanged
-        j = before;
-        CHECK_THROWS_AS(j[alt_json::json_pointer("/a/n/99999999999999999999999")], alt_json::out_of_range&);
-        CHECK(j == before);
-    }
-
     SECTION("operator/(std::size_t)")
     {
         // json_pointer::operator/=(std::size_t) must compile without string_t

tests/src/unit-json_pointer.cpp

@@ -29,64 +29,9 @@ using nlohmann::json;
 #include <map>
 #include <random>
 #include <sstream>
-#include <stdexcept>
 #include <string>
 #include <vector>
 
-namespace
-{
-// the non-const operator[] must throw for pointer and leave the value parsed
-// from init unchanged (strong exception safety)
-template<typename Json>
-void check_strong_guarantee(const char* init, const std::string& pointer, const char* message)
-{
-    CAPTURE(init)
-    CAPTURE(pointer)
-    Json j = Json::parse(init);
-    const Json before = Json::parse(init);
-    const typename Json::json_pointer ptr(pointer);
-    CHECK_THROWS_WITH_AS(j[ptr], message, typename Json::exception&);
-    CHECK(j == before);
-}
-
-// as above, for an index that fits into size_type, but not into an array
-template<typename Json>
-void check_strong_guarantee_length_error(const char* init, const std::string& pointer)
-{
-    CAPTURE(init)
-    CAPTURE(pointer)
-    Json j = Json::parse(init);
-    const Json before = Json::parse(init);
-    const typename Json::json_pointer ptr(pointer);
-    CHECK_THROWS_AS(j[ptr], std::length_error&);
-    CHECK(j == before);
-}
-
-template<typename Json>
-void check_strong_guarantee_cases()
-{
-    const char* const unresolved = "[json.exception.out_of_range.404] unresolved reference token '99999999999999999999999'";
-
-    // the first change converts a null value
-    check_strong_guarantee<Json>("null", "/99999999999999999999999", unresolved);
-    check_strong_guarantee<Json>("null", "/a/99999999999999999999999", unresolved);
-    check_strong_guarantee<Json>(R"({"a":null})", "/a/b/-/99999999999999999999999", unresolved);
-    // the first change inserts a key into an object
-    check_strong_guarantee<Json>("{}", "/a/99999999999999999999999", unresolved);
-    check_strong_guarantee<Json>(R"({"a":{"b":1}})", "/a/c/d/99999999999999999999999", unresolved);
-    // the first change pads or appends to an array
-    check_strong_guarantee<Json>(R"({"a":[1]})", "/a/5/99999999999999999999999", unresolved);
-    check_strong_guarantee<Json>(R"({"a":[1]})", "/a/-/x/1/99999999999999999999999", unresolved);
-    // an index that does not fit into size_type
-    check_strong_guarantee<Json>("null", "/x/18446744073709551615",
-                                 "[json.exception.out_of_range.410] array index 18446744073709551615 exceeds size_type");
-    // an index that fits into size_type, but not into an array
-    const auto too_large = std::to_string((std::numeric_limits<typename Json::size_type>::max)() - 1);
-    check_strong_guarantee_length_error<Json>("null", "/" + too_large);
-    check_strong_guarantee_length_error<Json>(R"({"a":[[1]]})", "/a/0/3/" + too_large);
-}
-} // namespace
-
 TEST_CASE("JSON pointers")
 {
     SECTION("errors")
@@ -612,23 +557,6 @@ TEST_CASE("JSON pointers")
             CHECK_THROWS_WITH_AS(j.at("/01"_json_pointer),
                                  "[json.exception.parse_error.106] parse error: array index '01' must not begin with '0'", json::parse_error&);
         }
-
-        SECTION("a failing operator[] leaves the value unchanged")
-        {
-            check_strong_guarantee_cases<json>();
-            check_strong_guarantee_cases<nlohmann::ordered_json>();
-        }
-
-        SECTION("a successful operator[] still creates the same values")
-        {
-            json j = {{"a", {1}}};
-            j["/a/3/x/-/y"_json_pointer] = 1;
-            CHECK(j == json::parse(R"({"a":[1,null,null,{"x":[{"y":1}]}]})"));
-            j["/a/-"_json_pointer] = 2;
-            j["/a/0"_json_pointer] = 3;
-            j["/b"_json_pointer] = 4;
-            CHECK(j == json::parse(R"({"a":[3,null,null,{"x":[{"y":1}]},2],"b":4})"));
-        }
     }
 
     SECTION("flatten")

tests/src/unit-json_view_edit.cpp

@@ -941,3 +941,123 @@ TEST_CASE("json_view edits: duplicate keys")
         CHECK(d.root()["o"]["d"].get<int>() == 0);
     }
 }
+
+#if !defined(JSON_NOEXCEPTION)
+TEST_CASE("json_view edits: check_utf8 agrees with the byte-wise check")
+{
+    namespace dv = nlohmann::detail::view;
+
+    // "ok" or the exception message
+    const auto run = [](bool fast, const std::string & s) -> std::string
+    {
+        try
+        {
+            if (fast)
+            {
+                dv::check_utf8(s.data(), s.size());
+            }
+            else
+            {
+                dv::check_utf8_bytewise(reinterpret_cast<const unsigned char*>(s.data()), s.size(), 0);
+            }
+        }
+        catch (const json::type_error& e)
+        {
+            return {e.what()};
+        }
+        return {"ok"};
+    };
+    std::size_t errors = 0;
+    const auto same = [&](const std::string & s)
+    {
+        const std::string a = run(true, s);
+        const std::string b = run(false, s);
+        if (a != b)
+        {
+            INFO("fast: " << a << ", byte-wise: " << b << ", length " << s.size());
+            CHECK(a == b);
+        }
+        errors += b != "ok" ? 1 : 0;
+    };
+
+    const std::vector<std::string> pieces =
+    {
+        "a", "Z", " ", "\"", "\\", "\n", "\x01", "\x7f", "\xc3\xa4", "\xe4\xb8\xad", "\xf0\x9f\x98\x80",
+        "\x80", "\xbf", "\xc0\x80", "\xc1\xbf", "\xc2", "\xe0\x80\x80", "\xe0\x9f\xbf", "\xed\xa0\x80", "\xed\xbf\xbf",
+        "\xf0\x80\x80\x80", "\xf0\x8f\xbf\xbf", "\xf4\x90\x80\x80", "\xf4\x8f\xbf\xbf", "\xf5", "\xf8", "\xff", "\xfe",
+        "\xe4\xb8", "\xf0\x9f\x98", "\xf0\x9f", "\xf0"
+    };
+
+    // every piece behind prefixes of every length, and followed by more bytes
+    for (std::size_t prefix = 0; prefix <= 64; ++prefix)
+    {
+        for (const auto& piece : pieces)
+        {
+            same(std::string(prefix, 'x') + piece);
+            same(std::string(prefix, 'x') + piece + "yz");
+            same(std::string(prefix, '"') + piece);
+            same(std::string(prefix, '\xc3') + piece);
+            same(std::string(prefix / 2, '\xc3').append(prefix / 2, '\xa4') + piece);
+            same(std::string(prefix, 'x') + "\xe4\xb8\xad" + piece);
+        }
+    }
+    // an invalid byte at every position of valid strings of every length
+    for (std::size_t len = 0; len <= 64; ++len)
+    {
+        const std::string ascii(len, 'a');
+        std::string mixed;
+        while (mixed.size() < len)
+        {
+            if (mixed.size() % 7 == 0)
+            {
+                mixed += "\xc3\xa4";
+            }
+            else if (mixed.size() % 11 == 0)
+            {
+                mixed += "\xf0\x9f\x98\x80";
+            }
+            else
+            {
+                mixed += "a";
+            }
+        }
+        same(ascii);
+        same(mixed);
+        for (std::size_t pos = 0; pos < len; ++pos)
+        {
+            for (const char bad :
+                    {'\x80', '\xc0', '\xc2', '\xe0', '\xf4', '\xff', '"', '\\', '\x1f'
+                    })
+            {
+                std::string s = ascii;
+                s[pos] = bad;
+                same(s);
+                s = mixed;
+                s[pos] = bad;
+                same(s);
+            }
+        }
+    }
+    // random bytes, biased to the interesting ones
+    std::mt19937 gen(5869); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed): reproducible
+    static const std::array<unsigned char, 23> interesting = {{0x00, 0x1f, 0x22, 0x5c, 0x7f, 0x80, 0x8f, 0x90, 0x9f, 0xa0, 0xbf, 0xc0, 0xc2, 0xdf, 0xe0, 0xe1, 0xed, 0xef, 0xf0, 0xf1, 0xf4, 0xf5, 0xff}};
+    for (int i = 0; i < 100000; ++i)
+    {
+        std::string s(gen() % 80, 'a');
+        for (auto& c : s)
+        {
+            const unsigned r = gen() % 8;
+            if (r == 0)
+            {
+                c = static_cast<char>(gen() & 0xff);
+            }
+            else if (r == 1)
+            {
+                c = static_cast<char>(interesting[gen() % interesting.size()]);
+            }
+        }
+        same(s);
+    }
+    CHECK(errors > 1000);
+}
+#endif

tests/src/unit-regression1.cpp

@@ -1499,23 +1499,6 @@ TEST_CASE("regression tests 1")
         CHECK(order.dump() == R"({"a":2,"m":{"y":1,"b":2},"new_key":3})");
     }
 
-    SECTION("operator[](json_pointer) when emplace() finds a key the lookup does not")
-    {
-        // a copied fifo_map restarts its timestamps, so a new key "k" gets
-        // the timestamp of "x": the lookup does not find "k", but emplace()
-        // finds "x" and inserts nothing; the pointer then continues with the
-        // member of "x", as basic_json::operator[] does
-        const my_json orig = my_json::parse(R"({"x":1})");
-        const std::array<const char*, 3> pointers = {{"/k/0", "/k/a/b", "/k/-"}};
-        for (const char* pointer : pointers)
-        {
-            CAPTURE(pointer)
-            my_json j = orig;
-            CHECK_THROWS_AS(j[my_json::json_pointer(pointer)], my_json::out_of_range&);
-            CHECK(j.dump() == R"({"x":1})");
-        }
-    }
-
     SECTION("issue #977 - Assigning between different json types")
     {
         foo_json lj = ns::foo{3};