From 78a0687147acaca2c8484df38bfaae33ec91fc70 Mon Sep 17 00:00:00 2001 From: Niels Lohmann Date: Tue, 27 Oct 2026 17:15:55 +0100 Subject: [PATCH 1/3] Strong exception guarantee for operator[](json_pointer) (#5940) * Keep the value unchanged when operator[] with a JSON pointer throws The non-const operator[](json_pointer) is documented with strong exception safety, but get_unchecked created intermediate values while walking the pointer and kept them when a later reference token failed, e.g. json j; j["/a/99999999999999999999999"_json_pointer] threw 404 and left {"a":[]}; padding an existing array or appending with "-" behaved alike, and so did std::length_error/std::bad_alloc from a huge index. get_unchecked now only walks existing values up to the last reference token. The last token's key insertion or array growth is strongly exception safe by itself. Any other change goes through get_unchecked_create, which records the one change it makes to an existing value (null converted, key inserted, array grown) in a RAII guard and reverts it if a later step throws; everything else it creates lies below that change. The guard needs no try/catch, so JSON_NOEXCEPTION builds are unaffected. Successful calls create the same values as before. Docs: state the guarantee in the exceptions list and version history of operator[], fix the complexity of the pointer overloads of operator[] and at, and warn that padding arrays up to an untrusted index allocates memory proportional to the index (operator[].md, untrusted_input.md). Signed-off-by: Niels Lohmann * Make the strongly exception-safe pointer operator[] faster than before The previous commit made get_unchecked strongly exception safe but slower with GCC 16 when it creates values (up to +6 % instructions for appending with "-"). This restructures it so that no benchmark row is slower than before the strong guarantee, with GCC 16, Clang 22 and Apple Clang. - get_unchecked walks existing values in one loop that inserts nothing. The last reference token is resolved in that walk: "-" appends with emplace_back(), an index grows the array with operator[], and a missing key is inserted with emplace_hint() at the position the walk's lower_bound() found, so the key is searched once. - With std::less or std::less<> as comparator, the walk checks the key at the lower bound for equality instead of equivalence, which is the same for these comparators and cheaper. - get_unchecked_create (not inlined) handles all other changes. Its guard swaps a null value with a fresh null value instead of copying it and reverts the change in a non-inlined function, so a successful call pays one compare for it. As every null value converted afterwards is a fresh one, it is converted in place instead of assigning a new array or object. - Keys are added to new objects with emplace(), which also replaces the basic_json::operator[] call for ordered_json. Signed-off-by: Niels Lohmann * Fix rollback of pointer operator[] for unusual object and string types Follow-up to the strong exception guarantee of the non-const operator[](json_pointer): - get_unchecked_create assumed that emplace() always inserts the key that the walk did not find. For object types whose lookup and emplace() disagree (e.g. a copied fifo_map), nothing was inserted, the rollback erased the existing member, and the walk continued with a value of the wrong type (assertion, type_error.305, or a crash). It now checks the result of emplace(), arms the rollback only if a key was inserted, and otherwise continues the walk with the member found, as basic_json's operator[] does: such pointers throw out_of_range.404 and leave the value unchanged, as before the strong guarantee. - The rollback erased the inserted key by key, which converts string_t to object_t::key_type and may allocate inside a noexcept function (std::terminate on bad_alloc with custom key types). It now erases by the iterator emplace() returned, and shrinks a grown array with erase() at the end instead of resize(). The documentation states the remaining requirement for custom template parameters. - Creating values no longer calls string_t::begin(), so the non-const operator[](json_pointer) compiles with string types that lack it (as the alternative string type of the tests). - Correct the @throw lists of get_unchecked and get_unchecked_create. Signed-off-by: Niels Lohmann * Address review comments on operator[](json_pointer) rollback - undo() shrinks the array with resize() instead of erase() - comments say what "will be" done in get_unchecked_create - tests: avoid a clang-tidy performance-unnecessary-copy-initialization hit (unit-alt-string) and the word "access" in section names, which Flawfinder flags as race/access Signed-off-by: Niels Lohmann * Let the pointer rollback guard call undo() unconditionally The check in the destructor and JSON_HEDLEY_NEVER_INLINE on undo() made no measurable difference: instruction counts of j[ptr] are identical with GCC 13 -O3 and within 0.4 % with clang, timings are within noise. Signed-off-by: Niels Lohmann --------- Signed-off-by: Niels Lohmann --- docs/mkdocs/docs/api/basic_json/at.md | 3 +- docs/mkdocs/docs/api/basic_json/operator[].md | 22 +- .../docs/features/parsing/untrusted_input.md | 13 + include/nlohmann/detail/json_pointer.hpp | 515 ++++++++++++++++- single_include/nlohmann/json.hpp | 516 +++++++++++++++++- tests/src/unit-alt-string.cpp | 17 + tests/src/unit-json_pointer.cpp | 72 +++ tests/src/unit-regression1.cpp | 17 + 8 files changed, 1112 insertions(+), 63 deletions(-) diff --git a/docs/mkdocs/docs/api/basic_json/at.md b/docs/mkdocs/docs/api/basic_json/at.md index f707b9835..b698443b3 100644 --- a/docs/mkdocs/docs/api/basic_json/at.md +++ b/docs/mkdocs/docs/api/basic_json/at.md @@ -90,7 +90,8 @@ Strong exception safety: if an exception occurs, the original value stays intact 1. Constant. 2. Logarithmic in the size of the container. 3. Logarithmic in the size of the container. -4. Logarithmic in the size of the container. +4. Linear in the number of reference tokens of `ptr`; for each of them, logarithmic in the size of the object, or + constant for an array. ## Notes diff --git a/docs/mkdocs/docs/api/basic_json/operator[].md b/docs/mkdocs/docs/api/basic_json/operator[].md index 59e48d984..a2b2acf5a 100644 --- a/docs/mkdocs/docs/api/basic_json/operator[].md +++ b/docs/mkdocs/docs/api/basic_json/operator[].md @@ -92,12 +92,20 @@ Strong exception safety: if an exception occurs, the original value stays intact For the **const** version, an object key or array index in `ptr` that does not exist is not reported by an exception, but is undefined behavior (see the notes below). Use [`at`](at.md) for checked access. + If the **non-const** version throws, the value is left unchanged: values it created for earlier reference tokens + of `ptr` are removed again. This also holds for exceptions such as `#!cpp std::bad_alloc` or + `#!cpp std::length_error` thrown while padding an array up to a very large index. With custom + [template parameters](../../features/types/template_parameters.md), this requires that `erase()` of the object and + array types and the move operations of the custom base class do not throw. + ## Complexity 1. Constant if `idx` is in the range of the array. Otherwise, linear in `idx - size()`. 2. Logarithmic in the size of the container. 3. Logarithmic in the size of the container. -4. Logarithmic in the size of the container. +4. Linear in the number of reference tokens of `ptr`; for each of them, logarithmic in the size of the object, or + constant for an array. When the non-const version pads an array with `#!json null` values, it is additionally + linear in `idx - size()` for that array. ## Notes @@ -145,6 +153,14 @@ Strong exception safety: if an exception occurs, the original value stays intact specified by the JSON Pointer RFC; it is this library's own, intentional disambiguation rule. See also [JSON Pointer](../../features/json_pointer.md). + !!! warning "Large array indices" + + As the array is filled up with `#!json null` values to the given index, the memory used grows with the index + and not with the size of the pointer: on a 64-bit platform, the index `10000000` allocates about 160 MB. + The library does not limit the index, so check the array indices of a JSON pointer from untrusted input + before using it with the non-const version. See + [Parsing untrusted input](../../features/parsing/untrusted_input.md#json-pointers-and-flattened-keys). + !!! warning "Deprecation" Overload (4) also accepts a [`json_pointer`](../json_pointer/index.md) whose template argument is a `basic_json` @@ -296,4 +312,6 @@ Strong exception safety: if an exception occurs, the original value stays intact 4. Added in version 2.0.0. A missing array index in the const version is guarded by a runtime assertion since version 3.13.0. Throws [`parse_error.109`](../../home/exceptions.md#jsonexceptionparse_error109) instead of [`out_of_range.404`](../../home/exceptions.md#jsonexceptionout_of_range404) for a one-character array index that is not -a digit (e.g., `/x`) in version 3.13.0, as it already did for longer ones. +a digit (e.g., `/x`) in version 3.13.0, as it already did for longer ones. Fixed in version 3.13.0 to leave the value +unchanged if the non-const version throws (strong exception safety, as documented); before, values created for earlier +reference tokens of `ptr` were kept. diff --git a/docs/mkdocs/docs/features/parsing/untrusted_input.md b/docs/mkdocs/docs/features/parsing/untrusted_input.md index e89a7a7a3..1fe7b64aa 100644 --- a/docs/mkdocs/docs/features/parsing/untrusted_input.md +++ b/docs/mkdocs/docs/features/parsing/untrusted_input.md @@ -130,6 +130,17 @@ parameters of [`parse()`](../../api/basic_json/parse.md) or [`accept()`](../../a enable either for input whose conformance you cannot otherwise control, since interoperability with strictly conforming JSON consumers is exactly what the default rejects. +## JSON Pointers and flattened keys + +A [JSON Pointer](../json_pointer.md) can come from untrusted input as well, for instance as a path in a request or as a +key of a flattened object. The non-const [`operator[]`](../../api/basic_json/operator%5B%5D.md) with a JSON Pointer and +[`unflatten`](../../api/basic_json/unflatten.md) create missing values, and when an array index lies beyond the end of +an array, they fill the array with `#!json null` values up to that index. The memory used therefore grows with the +index, not with the size of the pointer: on a 64-bit platform, the index `10000000` allocates about 160 MB, the index +`100000000` about 1.6 GB. The library does not limit the index. For pointers from untrusted input, check the array +indices (or the pointer as a whole) before using them, or use [`at`](../../api/basic_json/at.md) or +[`contains`](../../api/basic_json/contains.md), which never add values. + ## Checklist - Wrap parsing in a `#!cpp try`/`#!cpp catch` block, or use `allow_exceptions=false`/`accept()` if your environment @@ -145,6 +156,8 @@ conforming JSON consumers is exactly what the default rejects. `JSON_STRICT_NUL_HANDLING`. - Decide whether duplicate object keys should be an error for your application, and add a callback if so. - Do not enable `ignore_comments` or `ignore_trailing_commas` for input that must be strictly conforming JSON. +- Check the array indices of untrusted JSON Pointers and flattened keys before passing them to the non-const + `operator[]` or `unflatten()`, which fill arrays with `#!json null` values up to the index. For the broader design rationale and how it is tested (fuzzing, sanitizers, static analysis), see the [assurance case](../../community/assurance_case.md) and [quality assurance](../../community/quality_assurance.md). To diff --git a/include/nlohmann/detail/json_pointer.hpp b/include/nlohmann/detail/json_pointer.hpp index 457985480..eef4ce0a4 100644 --- a/include/nlohmann/detail/json_pointer.hpp +++ b/include/nlohmann/detail/json_pointer.hpp @@ -8,7 +8,7 @@ #pragma once -#include // all_of, copy +#include // copy #include // array #include // isdigit #include // errno, ERANGE @@ -16,9 +16,11 @@ #include // uint64_t #include // strtoull #include // memchr, memcmp, memcpy +#include // less #ifndef JSON_NO_IO #include // ostream #endif // JSON_NO_IO +#include // next #include // max #include // string #include // move @@ -28,6 +30,7 @@ #include #include #include +#include #include #include #include @@ -134,6 +137,26 @@ JSON_HEDLEY_ALWAYS_INLINE array_index_status parse_array_index(const StringType& return array_index_status::ok; } +/// whether two keys of type @a StringType are equivalent under @a Compare +/// exactly if they are equal: std::less and std::less<> +template +struct is_less_comparator : std::false_type {}; + +template +struct is_less_comparator, StringType> : std::true_type {}; + +template +struct is_less_comparator, StringType> : std::true_type {}; + +/// whether the objects of type @a ObjectType find a key of type @a StringType +/// exactly if it equals a stored key +template +struct finds_equal_keys : std::false_type {}; + +template +struct finds_equal_keys> + : conjunction, std::is_same> {}; + } // namespace detail /// @brief JSON Pointer defines a string syntax for identifying a specific value within a JSON document @@ -589,6 +612,225 @@ class json_pointer return object_member_impl(object, key, detail::priority_tag<1> {}); } + /// @brief append a null value to @a array, like operator[] with the size of @a array + template + static BasicJsonType& array_append(BasicJsonType& array) + { + auto& elements = *array.m_data.m_value.array; + const auto old_capacity = array.container_capacity(); + elements.emplace_back(); + return array.set_parent_after_insert(elements.back(), old_capacity); + } + + /// @brief add @a key, which is missing, to @a object + template + static auto object_add_impl(BasicJsonType& object, const string_t& key, detail::priority_tag<1> /*unused*/) + -> decltype(object.m_data.m_value.object->emplace(key, nullptr), std::declval()) + { + const auto old_capacity = object.container_capacity(); + const auto it = object.m_data.m_value.object->emplace(key, nullptr).first; + return object.set_parent_after_insert(it->second, old_capacity); + } + + template + static BasicJsonType& object_add_impl(BasicJsonType& object, const string_t& key, detail::priority_tag<0> /*unused*/) + { + return object.operator[](key); + } + + template + static BasicJsonType& object_add(BasicJsonType& object, const string_t& key) + { + return object_add_impl(object, key, detail::priority_tag<1> {}); + } + + /// @brief search @a key in @a members: the result is where the key is or, + /// for objects that can insert at a position (std::map), where it belongs + template + static auto object_search_impl(ObjectType& members, const string_t& key, detail::priority_tag<1> /*unused*/) + -> decltype(members.key_comp()(key, members.lower_bound(key)->first), members.lower_bound(key)) + { + return members.lower_bound(key); + } + + template + static auto object_search_impl(ObjectType& members, const string_t& key, detail::priority_tag<0> /*unused*/) + -> decltype(members.find(key)) + { + return members.find(key); + } + + template + static auto object_search(ObjectType& members, const string_t& key) + -> decltype(object_search_impl(members, key, detail::priority_tag<1> {})) + { + return object_search_impl(members, key, detail::priority_tag<1> {}); + } + + /// @brief whether @a it, the result of @ref object_search for @a key, points to @a key + /// + /// With std::less, the key at the lower bound is equivalent to @a key + /// exactly if it is equal to it, and comparing for equality is cheaper. + template::value, int> = 0> + static auto object_search_found_impl(ObjectType& members, const Iterator& it, const string_t& key, detail::priority_tag<2> /*unused*/) + -> decltype(members.lower_bound(key), it->first == key) + { + return it != members.end() && it->first == key; + } + + template + static auto object_search_found_impl(ObjectType& members, const Iterator& it, const string_t& key, detail::priority_tag<1> /*unused*/) + -> decltype(members.key_comp()(key, it->first), members.lower_bound(key), bool()) + { + return it != members.end() && !members.key_comp()(key, it->first); + } + + template + static bool object_search_found_impl(ObjectType& members, const Iterator& it, const string_t& /*key*/, detail::priority_tag<0> /*unused*/) + { + return it != members.end(); + } + + template + static bool object_search_found(ObjectType& members, const Iterator& it, const string_t& key) + { + return object_search_found_impl(members, it, key, detail::priority_tag<2> {}); + } + + /// @brief insert @a key, which is missing, into @a object at @a it, the + /// result of @ref object_search for @a key + template + static auto object_insert_impl(BasicJsonType& object, const Iterator& it, const string_t& key, detail::priority_tag<1> /*unused*/) + -> decltype(object.m_data.m_value.object->emplace_hint(it, key, nullptr), std::declval()) + { + const auto old_capacity = object.container_capacity(); + const auto pos = object.m_data.m_value.object->emplace_hint(it, key, nullptr); + return object.set_parent_after_insert(pos->second, old_capacity); + } + + template + static BasicJsonType& object_insert_impl(BasicJsonType& object, const Iterator& /*it*/, const string_t& key, detail::priority_tag<0> /*unused*/) + { + return object_add(object, key); + } + + template + static BasicJsonType& object_insert(BasicJsonType& object, const Iterator& it, const string_t& key) + { + return object_insert_impl(object, it, key, detail::priority_tag<1> {}); + } + + /*! + @brief undoes the change @ref get_unchecked_create made to an existing value + + @ref get_unchecked walks existing values without changing them. At the + first reference token that does not select an existing value, @ref + get_unchecked_create makes exactly one change to an existing value: it + converts a null value, inserts a key into an object, or grows an array. + Everything it creates afterwards lies below that change. If a later step + throws, the destructor of this guard reverts that one change, which + removes everything created after it, so the value is left as it was + (strong exception safety). The guard needs no try/catch and therefore also + compiles with JSON_NOEXCEPTION. + + Reverting the change must not throw, as it happens in a destructor. It + erases the inserted member by its iterator (no key is converted or + compared, nothing is allocated), erases the added elements at the end of + the array, or swaps the null value back. This holds for std::map, + std::vector, ordered_map, and every object_t, array_t, and + json_base_class_t whose erase() and move operations do not throw. + Between recording and reverting, nothing is inserted into or erased from + the object or array, so the iterator stays valid. + */ + template + class get_unchecked_rollback + { + public: + using object_iterator = typename BasicJsonType::object_t::iterator; + + /// record that @a member was inserted into the object @a value + get_unchecked_rollback(BasicJsonType* value, const object_iterator& member) + : m_value(value) + , m_member(member) + , m_change(change::key_inserted) + {} + + /// record the change that the next reference token makes to the null value or array @a value + explicit get_unchecked_rollback(BasicJsonType* value) + : m_value(value) + { + if (value->is_null()) + { + // keep the null value (with its base class and positions) and + // continue with a fresh null value in its place, which is what + // assigning an array or object to it would have left there + m_saved_null.swap(*value); + m_change = change::null_converted; + } + else + { + JSON_ASSERT(value->is_array()); + m_old_size = value->m_data.m_value.array->size(); + m_change = change::array_grown; + } + } + + get_unchecked_rollback(const get_unchecked_rollback&) = delete; + get_unchecked_rollback(get_unchecked_rollback&&) = delete; + get_unchecked_rollback& operator=(const get_unchecked_rollback&) = delete; + get_unchecked_rollback& operator=(get_unchecked_rollback&&) = delete; + + ~get_unchecked_rollback() + { + undo(); + } + + /// keep the change + void commit() noexcept + { + m_change = change::none; + } + + private: + void undo() noexcept + { + switch (m_change) + { + case change::null_converted: + // the values created since go away with m_saved_null + m_value->swap(m_saved_null); + break; + case change::key_inserted: + m_value->m_data.m_value.object->erase(m_member); + break; + case change::array_grown: + { + auto& elements = *m_value->m_data.m_value.array; + elements.resize(m_old_size); + break; + } + case change::none: + default: + break; + } + } + + enum class change + { + none, + null_converted, + key_inserted, + array_grown + }; + + BasicJsonType* m_value; + object_iterator m_member {}; + typename BasicJsonType::size_type m_old_size = 0; + change m_change = change::none; + BasicJsonType m_saved_null {}; + }; + JSON_PRIVATE_UNLESS_TESTED: json_pointer top() const { @@ -972,47 +1214,245 @@ class json_pointer Complexity: Linear in the length of the JSON pointer. - @throw parse_error.106 if an array index begins with '0' - @throw parse_error.109 if an array index was not a number - @throw out_of_range.404 if the JSON pointer can not be resolved + Exception safety: Strong guarantee. This function walks existing values + until the first reference token that does not select one. If that is the + last reference token, inserting the key into the object or growing the + array is strongly exception safe by itself; any other change is made by + @ref get_unchecked_create, which undoes it if a later step throws. + + @throw parse_error.106 if an array index of an existing array begins with '0' + @throw parse_error.109 if an array index of an existing array was not a number + @throw out_of_range.404 if the JSON pointer can not be resolved, or if an + array index is too large for strtoull + @throw out_of_range.410 if an array index exceeds size_type + @throw std::length_error if padding an array exceeds its maximal size + @throw std::bad_alloc if an allocation fails */ template BasicJsonType& get_unchecked(BasicJsonType* ptr) const { - for (const auto& reference_token : reference_tokens) + const auto* token = reference_tokens.data(); + const auto* const end = token + reference_tokens.size(); + if (token == end) { + return *ptr; + } + if (ptr->is_null()) + { + // the null value is converted to an array or an object + return get_unchecked_create(ptr, token, end); + } + + // walk existing values; the walk itself inserts nothing, which keeps + // the loop fast (insertions are done below or in get_unchecked_create) + typename BasicJsonType::object_t::iterator it; + do + { + if (ptr->is_object()) + { + auto& members = *ptr->m_data.m_value.object; + it = object_search(members, *token); + if (!object_search_found(members, it, *token)) + { + // the key is inserted below or in get_unchecked_create + break; + } + ptr = &it->second; + } + else if (ptr->is_array()) + { + if (is_dash(*token)) + { + if (token + 1 == end) + { + // explicitly treat "-" as index beyond the end + return array_append(*ptr); + } + // an element will be appended in get_unchecked_create + break; + } + if (token + 1 == end) + { + // convert array index to number; unchecked access + return ptr->operator[](array_index(*token)); + } + const auto idx = array_index(*token); + if (idx >= ptr->m_data.m_value.array->size()) + { + // the array will be padded in get_unchecked_create + break; + } + ptr = &(*ptr->m_data.m_value.array)[idx]; + } + else if (ptr->is_null()) + { + // the null value will be converted in get_unchecked_create + break; + } + else + { + // primitive value with a reference token left + throw_unresolved(*token, ptr); + } + } + while (++token != end); + + if (token == end) + { + return *ptr; + } + if (token + 1 == end && ptr->is_object()) + { + // insert the missing key where the walk found it belongs + return object_insert(*ptr, it, *token); + } + return get_unchecked_create(ptr, token, end); + } + + /*! + @brief create the values for the reference tokens from @a first on + + Continues @ref get_unchecked at the first reference token that does not + select an existing value. That token changes @a ptr, an existing value (see + @ref get_unchecked_rollback); all later tokens only change values created + here. + + If the emplace() of the object type finds a member that the lookup in @ref + get_unchecked did not find (the two disagree, e.g. for fifo_map), nothing + is inserted, and the walk continues with that member as in @ref + get_unchecked, which is what basic_json::operator[] does. + + @param[in] ptr the value the reference token @a first is applied to + @param[in] first the first reference token to resolve (an element of + @ref reference_tokens) + @param[in] end the end of @ref reference_tokens + + @throw parse_error.106 if an array index of an existing array begins with + '0' (only after emplace() found a member, see above) + @throw parse_error.109 if an array index of an existing array was not a + number (only after emplace() found a member) + @throw out_of_range.404 if an array index is too large for strtoull, or (only + after emplace() found a member) if the JSON pointer + can not be resolved + @throw out_of_range.410 if an array index exceeds size_type + @throw std::length_error if padding an array exceeds its maximal size + @throw std::bad_alloc if an allocation fails + */ + template + JSON_HEDLEY_NEVER_INLINE static BasicJsonType& get_unchecked_create(BasicJsonType* ptr, const string_t* first, const string_t* end) + { + const auto* token = first; + while (ptr->is_object()) + { + const auto old_capacity = ptr->container_capacity(); + const auto inserted = ptr->m_data.m_value.object->emplace(*token, nullptr); + if (JSON_HEDLEY_LIKELY(inserted.second)) + { + get_unchecked_rollback rollback(ptr, inserted.first); + BasicJsonType& result = create_values(&ptr->set_parent_after_insert(inserted.first->second, old_capacity), token + 1, end); + rollback.commit(); + return result; + } + + // nothing was inserted: walk existing values from the member + // emplace() found on, like get_unchecked + ptr = &inserted.first->second; + while (++token != end) + { + if (ptr->is_object()) + { + auto& members = *ptr->m_data.m_value.object; + const auto it = object_search(members, *token); + if (!object_search_found(members, it, *token)) + { + break; + } + ptr = &it->second; + } + else if (ptr->is_array()) + { + if (is_dash(*token)) + { + break; + } + const auto idx = array_index(*token); + if (idx >= ptr->m_data.m_value.array->size()) + { + break; + } + ptr = &(*ptr->m_data.m_value.array)[idx]; + } + else if (ptr->is_null()) + { + break; + } + else + { + // primitive value with a reference token left + throw_unresolved(*token, ptr); + } + } + if (token == end) + { + return *ptr; + } + } + + get_unchecked_rollback rollback(ptr); + BasicJsonType& result = create_values(ptr, token, end); + rollback.commit(); + return result; + } + + /*! + @brief the loop of @ref get_unchecked_create + + @a ptr is a null value or an array. Every value created here is a fresh + one, so every object a key is added to is empty. + + @throw out_of_range.404 if an array index is too large for strtoull + @throw out_of_range.410 if an array index exceeds size_type + @throw std::length_error if padding an array exceeds its maximal size + @throw std::bad_alloc if an allocation fails + */ + template + static BasicJsonType& create_values(BasicJsonType* ptr, const string_t* first, const string_t* end) + { + for (const auto* token = first; token != end; ++token) + { + const auto& reference_token = *token; + // convert null values to arrays or objects before continuing if (ptr->is_null()) { - // check if the reference token is a valid array index, that is - // a nonempty sequence of digits without a leading '0' - // (cf. RFC 6901, Sect. 4); tokens that could never be a valid - // array index (such as "01" or "") are treated as object keys - const bool nums = !reference_token.empty() - && (reference_token.size() == 1 || reference_token[0] != '0') - && std::all_of(reference_token.begin(), reference_token.end(), - [](const unsigned char x) + // change value to an array for array indices or "-" or to an + // object otherwise; every null value here is a fresh one (see + // get_unchecked_rollback), so converting it in place leaves + // the same value as assigning an array or object to it + if (is_array_index_token(reference_token) || is_dash(reference_token)) { - return std::isdigit(x); - }); - - // change value to an array for array indices or "-" or to object otherwise - *ptr = (nums || reference_token == "-") - ? detail::value_t::array - : detail::value_t::object; + ptr->template convert_null_to(); + } + else + { + ptr->template convert_null_to(); + } } if (ptr->is_object()) { - // use unchecked object access - ptr = &object_member(*ptr, reference_token); + // the object is new and empty + ptr = &object_add(*ptr, reference_token); } - else if (ptr->is_array()) + else { - if (reference_token == "-") + // only the first value is not created here, and it is a null + // value or an array (see get_unchecked_create) + JSON_ASSERT(ptr->is_array()); + if (is_dash(reference_token)) { // explicitly treat "-" as index beyond the end - ptr = &ptr->operator[](ptr->m_data.m_value.array->size()); + ptr = &array_append(*ptr); } else { @@ -1020,16 +1460,31 @@ class json_pointer ptr = &ptr->operator[](array_index(reference_token)); } } - else - { - // primitive value with a reference token left - throw_unresolved(reference_token, ptr); - } } return *ptr; } + /// whether @a s is a valid array index, that is a nonempty sequence of + /// digits without a leading '0' (cf. RFC 6901, Sect. 4); tokens that could + /// never be a valid array index (such as "01" or "") are object keys + static bool is_array_index_token(const string_t& s) + { + const std::size_t size = s.size(); + if (size == 0 || (size > 1 && s[0] == '0')) + { + return false; + } + for (std::size_t i = 0; i < size; ++i) + { + if (std::isdigit(static_cast(s[i])) == 0) + { + return false; + } + } + return true; + } + /*! @throw parse_error.106 if an array index begins with '0' @throw parse_error.109 if an array index was not a number diff --git a/single_include/nlohmann/json.hpp b/single_include/nlohmann/json.hpp index 0c4753d41..085b55afb 100644 --- a/single_include/nlohmann/json.hpp +++ b/single_include/nlohmann/json.hpp @@ -24064,7 +24064,7 @@ NLOHMANN_JSON_NAMESPACE_END -#include // all_of, copy +#include // copy #include // array #include // isdigit #include // errno, ERANGE @@ -24072,9 +24072,11 @@ NLOHMANN_JSON_NAMESPACE_END #include // uint64_t #include // strtoull #include // memchr, memcmp, memcpy +#include // less #ifndef JSON_NO_IO #include // ostream #endif // JSON_NO_IO +#include // next #include // max #include // string #include // move @@ -24088,6 +24090,8 @@ NLOHMANN_JSON_NAMESPACE_END // #include +// #include + // #include // #include @@ -24198,6 +24202,26 @@ JSON_HEDLEY_ALWAYS_INLINE array_index_status parse_array_index(const StringType& return array_index_status::ok; } +/// whether two keys of type @a StringType are equivalent under @a Compare +/// exactly if they are equal: std::less and std::less<> +template +struct is_less_comparator : std::false_type {}; + +template +struct is_less_comparator, StringType> : std::true_type {}; + +template +struct is_less_comparator, StringType> : std::true_type {}; + +/// whether the objects of type @a ObjectType find a key of type @a StringType +/// exactly if it equals a stored key +template +struct finds_equal_keys : std::false_type {}; + +template +struct finds_equal_keys> + : conjunction, std::is_same> {}; + } // namespace detail /// @brief JSON Pointer defines a string syntax for identifying a specific value within a JSON document @@ -24653,6 +24677,225 @@ class json_pointer return object_member_impl(object, key, detail::priority_tag<1> {}); } + /// @brief append a null value to @a array, like operator[] with the size of @a array + template + static BasicJsonType& array_append(BasicJsonType& array) + { + auto& elements = *array.m_data.m_value.array; + const auto old_capacity = array.container_capacity(); + elements.emplace_back(); + return array.set_parent_after_insert(elements.back(), old_capacity); + } + + /// @brief add @a key, which is missing, to @a object + template + static auto object_add_impl(BasicJsonType& object, const string_t& key, detail::priority_tag<1> /*unused*/) + -> decltype(object.m_data.m_value.object->emplace(key, nullptr), std::declval()) + { + const auto old_capacity = object.container_capacity(); + const auto it = object.m_data.m_value.object->emplace(key, nullptr).first; + return object.set_parent_after_insert(it->second, old_capacity); + } + + template + static BasicJsonType& object_add_impl(BasicJsonType& object, const string_t& key, detail::priority_tag<0> /*unused*/) + { + return object.operator[](key); + } + + template + static BasicJsonType& object_add(BasicJsonType& object, const string_t& key) + { + return object_add_impl(object, key, detail::priority_tag<1> {}); + } + + /// @brief search @a key in @a members: the result is where the key is or, + /// for objects that can insert at a position (std::map), where it belongs + template + static auto object_search_impl(ObjectType& members, const string_t& key, detail::priority_tag<1> /*unused*/) + -> decltype(members.key_comp()(key, members.lower_bound(key)->first), members.lower_bound(key)) + { + return members.lower_bound(key); + } + + template + static auto object_search_impl(ObjectType& members, const string_t& key, detail::priority_tag<0> /*unused*/) + -> decltype(members.find(key)) + { + return members.find(key); + } + + template + static auto object_search(ObjectType& members, const string_t& key) + -> decltype(object_search_impl(members, key, detail::priority_tag<1> {})) + { + return object_search_impl(members, key, detail::priority_tag<1> {}); + } + + /// @brief whether @a it, the result of @ref object_search for @a key, points to @a key + /// + /// With std::less, the key at the lower bound is equivalent to @a key + /// exactly if it is equal to it, and comparing for equality is cheaper. + template::value, int> = 0> + static auto object_search_found_impl(ObjectType& members, const Iterator& it, const string_t& key, detail::priority_tag<2> /*unused*/) + -> decltype(members.lower_bound(key), it->first == key) + { + return it != members.end() && it->first == key; + } + + template + static auto object_search_found_impl(ObjectType& members, const Iterator& it, const string_t& key, detail::priority_tag<1> /*unused*/) + -> decltype(members.key_comp()(key, it->first), members.lower_bound(key), bool()) + { + return it != members.end() && !members.key_comp()(key, it->first); + } + + template + static bool object_search_found_impl(ObjectType& members, const Iterator& it, const string_t& /*key*/, detail::priority_tag<0> /*unused*/) + { + return it != members.end(); + } + + template + static bool object_search_found(ObjectType& members, const Iterator& it, const string_t& key) + { + return object_search_found_impl(members, it, key, detail::priority_tag<2> {}); + } + + /// @brief insert @a key, which is missing, into @a object at @a it, the + /// result of @ref object_search for @a key + template + static auto object_insert_impl(BasicJsonType& object, const Iterator& it, const string_t& key, detail::priority_tag<1> /*unused*/) + -> decltype(object.m_data.m_value.object->emplace_hint(it, key, nullptr), std::declval()) + { + const auto old_capacity = object.container_capacity(); + const auto pos = object.m_data.m_value.object->emplace_hint(it, key, nullptr); + return object.set_parent_after_insert(pos->second, old_capacity); + } + + template + static BasicJsonType& object_insert_impl(BasicJsonType& object, const Iterator& /*it*/, const string_t& key, detail::priority_tag<0> /*unused*/) + { + return object_add(object, key); + } + + template + static BasicJsonType& object_insert(BasicJsonType& object, const Iterator& it, const string_t& key) + { + return object_insert_impl(object, it, key, detail::priority_tag<1> {}); + } + + /*! + @brief undoes the change @ref get_unchecked_create made to an existing value + + @ref get_unchecked walks existing values without changing them. At the + first reference token that does not select an existing value, @ref + get_unchecked_create makes exactly one change to an existing value: it + converts a null value, inserts a key into an object, or grows an array. + Everything it creates afterwards lies below that change. If a later step + throws, the destructor of this guard reverts that one change, which + removes everything created after it, so the value is left as it was + (strong exception safety). The guard needs no try/catch and therefore also + compiles with JSON_NOEXCEPTION. + + Reverting the change must not throw, as it happens in a destructor. It + erases the inserted member by its iterator (no key is converted or + compared, nothing is allocated), erases the added elements at the end of + the array, or swaps the null value back. This holds for std::map, + std::vector, ordered_map, and every object_t, array_t, and + json_base_class_t whose erase() and move operations do not throw. + Between recording and reverting, nothing is inserted into or erased from + the object or array, so the iterator stays valid. + */ + template + class get_unchecked_rollback + { + public: + using object_iterator = typename BasicJsonType::object_t::iterator; + + /// record that @a member was inserted into the object @a value + get_unchecked_rollback(BasicJsonType* value, const object_iterator& member) + : m_value(value) + , m_member(member) + , m_change(change::key_inserted) + {} + + /// record the change that the next reference token makes to the null value or array @a value + explicit get_unchecked_rollback(BasicJsonType* value) + : m_value(value) + { + if (value->is_null()) + { + // keep the null value (with its base class and positions) and + // continue with a fresh null value in its place, which is what + // assigning an array or object to it would have left there + m_saved_null.swap(*value); + m_change = change::null_converted; + } + else + { + JSON_ASSERT(value->is_array()); + m_old_size = value->m_data.m_value.array->size(); + m_change = change::array_grown; + } + } + + get_unchecked_rollback(const get_unchecked_rollback&) = delete; + get_unchecked_rollback(get_unchecked_rollback&&) = delete; + get_unchecked_rollback& operator=(const get_unchecked_rollback&) = delete; + get_unchecked_rollback& operator=(get_unchecked_rollback&&) = delete; + + ~get_unchecked_rollback() + { + undo(); + } + + /// keep the change + void commit() noexcept + { + m_change = change::none; + } + + private: + void undo() noexcept + { + switch (m_change) + { + case change::null_converted: + // the values created since go away with m_saved_null + m_value->swap(m_saved_null); + break; + case change::key_inserted: + m_value->m_data.m_value.object->erase(m_member); + break; + case change::array_grown: + { + auto& elements = *m_value->m_data.m_value.array; + elements.resize(m_old_size); + break; + } + case change::none: + default: + break; + } + } + + enum class change + { + none, + null_converted, + key_inserted, + array_grown + }; + + BasicJsonType* m_value; + object_iterator m_member {}; + typename BasicJsonType::size_type m_old_size = 0; + change m_change = change::none; + BasicJsonType m_saved_null {}; + }; + JSON_PRIVATE_UNLESS_TESTED: json_pointer top() const { @@ -25036,47 +25279,245 @@ class json_pointer Complexity: Linear in the length of the JSON pointer. - @throw parse_error.106 if an array index begins with '0' - @throw parse_error.109 if an array index was not a number - @throw out_of_range.404 if the JSON pointer can not be resolved + Exception safety: Strong guarantee. This function walks existing values + until the first reference token that does not select one. If that is the + last reference token, inserting the key into the object or growing the + array is strongly exception safe by itself; any other change is made by + @ref get_unchecked_create, which undoes it if a later step throws. + + @throw parse_error.106 if an array index of an existing array begins with '0' + @throw parse_error.109 if an array index of an existing array was not a number + @throw out_of_range.404 if the JSON pointer can not be resolved, or if an + array index is too large for strtoull + @throw out_of_range.410 if an array index exceeds size_type + @throw std::length_error if padding an array exceeds its maximal size + @throw std::bad_alloc if an allocation fails */ template BasicJsonType& get_unchecked(BasicJsonType* ptr) const { - for (const auto& reference_token : reference_tokens) + const auto* token = reference_tokens.data(); + const auto* const end = token + reference_tokens.size(); + if (token == end) { + return *ptr; + } + if (ptr->is_null()) + { + // the null value is converted to an array or an object + return get_unchecked_create(ptr, token, end); + } + + // walk existing values; the walk itself inserts nothing, which keeps + // the loop fast (insertions are done below or in get_unchecked_create) + typename BasicJsonType::object_t::iterator it; + do + { + if (ptr->is_object()) + { + auto& members = *ptr->m_data.m_value.object; + it = object_search(members, *token); + if (!object_search_found(members, it, *token)) + { + // the key is inserted below or in get_unchecked_create + break; + } + ptr = &it->second; + } + else if (ptr->is_array()) + { + if (is_dash(*token)) + { + if (token + 1 == end) + { + // explicitly treat "-" as index beyond the end + return array_append(*ptr); + } + // an element will be appended in get_unchecked_create + break; + } + if (token + 1 == end) + { + // convert array index to number; unchecked access + return ptr->operator[](array_index(*token)); + } + const auto idx = array_index(*token); + if (idx >= ptr->m_data.m_value.array->size()) + { + // the array will be padded in get_unchecked_create + break; + } + ptr = &(*ptr->m_data.m_value.array)[idx]; + } + else if (ptr->is_null()) + { + // the null value will be converted in get_unchecked_create + break; + } + else + { + // primitive value with a reference token left + throw_unresolved(*token, ptr); + } + } + while (++token != end); + + if (token == end) + { + return *ptr; + } + if (token + 1 == end && ptr->is_object()) + { + // insert the missing key where the walk found it belongs + return object_insert(*ptr, it, *token); + } + return get_unchecked_create(ptr, token, end); + } + + /*! + @brief create the values for the reference tokens from @a first on + + Continues @ref get_unchecked at the first reference token that does not + select an existing value. That token changes @a ptr, an existing value (see + @ref get_unchecked_rollback); all later tokens only change values created + here. + + If the emplace() of the object type finds a member that the lookup in @ref + get_unchecked did not find (the two disagree, e.g. for fifo_map), nothing + is inserted, and the walk continues with that member as in @ref + get_unchecked, which is what basic_json::operator[] does. + + @param[in] ptr the value the reference token @a first is applied to + @param[in] first the first reference token to resolve (an element of + @ref reference_tokens) + @param[in] end the end of @ref reference_tokens + + @throw parse_error.106 if an array index of an existing array begins with + '0' (only after emplace() found a member, see above) + @throw parse_error.109 if an array index of an existing array was not a + number (only after emplace() found a member) + @throw out_of_range.404 if an array index is too large for strtoull, or (only + after emplace() found a member) if the JSON pointer + can not be resolved + @throw out_of_range.410 if an array index exceeds size_type + @throw std::length_error if padding an array exceeds its maximal size + @throw std::bad_alloc if an allocation fails + */ + template + JSON_HEDLEY_NEVER_INLINE static BasicJsonType& get_unchecked_create(BasicJsonType* ptr, const string_t* first, const string_t* end) + { + const auto* token = first; + while (ptr->is_object()) + { + const auto old_capacity = ptr->container_capacity(); + const auto inserted = ptr->m_data.m_value.object->emplace(*token, nullptr); + if (JSON_HEDLEY_LIKELY(inserted.second)) + { + get_unchecked_rollback rollback(ptr, inserted.first); + BasicJsonType& result = create_values(&ptr->set_parent_after_insert(inserted.first->second, old_capacity), token + 1, end); + rollback.commit(); + return result; + } + + // nothing was inserted: walk existing values from the member + // emplace() found on, like get_unchecked + ptr = &inserted.first->second; + while (++token != end) + { + if (ptr->is_object()) + { + auto& members = *ptr->m_data.m_value.object; + const auto it = object_search(members, *token); + if (!object_search_found(members, it, *token)) + { + break; + } + ptr = &it->second; + } + else if (ptr->is_array()) + { + if (is_dash(*token)) + { + break; + } + const auto idx = array_index(*token); + if (idx >= ptr->m_data.m_value.array->size()) + { + break; + } + ptr = &(*ptr->m_data.m_value.array)[idx]; + } + else if (ptr->is_null()) + { + break; + } + else + { + // primitive value with a reference token left + throw_unresolved(*token, ptr); + } + } + if (token == end) + { + return *ptr; + } + } + + get_unchecked_rollback rollback(ptr); + BasicJsonType& result = create_values(ptr, token, end); + rollback.commit(); + return result; + } + + /*! + @brief the loop of @ref get_unchecked_create + + @a ptr is a null value or an array. Every value created here is a fresh + one, so every object a key is added to is empty. + + @throw out_of_range.404 if an array index is too large for strtoull + @throw out_of_range.410 if an array index exceeds size_type + @throw std::length_error if padding an array exceeds its maximal size + @throw std::bad_alloc if an allocation fails + */ + template + static BasicJsonType& create_values(BasicJsonType* ptr, const string_t* first, const string_t* end) + { + for (const auto* token = first; token != end; ++token) + { + const auto& reference_token = *token; + // convert null values to arrays or objects before continuing if (ptr->is_null()) { - // check if the reference token is a valid array index, that is - // a nonempty sequence of digits without a leading '0' - // (cf. RFC 6901, Sect. 4); tokens that could never be a valid - // array index (such as "01" or "") are treated as object keys - const bool nums = !reference_token.empty() - && (reference_token.size() == 1 || reference_token[0] != '0') - && std::all_of(reference_token.begin(), reference_token.end(), - [](const unsigned char x) + // change value to an array for array indices or "-" or to an + // object otherwise; every null value here is a fresh one (see + // get_unchecked_rollback), so converting it in place leaves + // the same value as assigning an array or object to it + if (is_array_index_token(reference_token) || is_dash(reference_token)) { - return std::isdigit(x); - }); - - // change value to an array for array indices or "-" or to object otherwise - *ptr = (nums || reference_token == "-") - ? detail::value_t::array - : detail::value_t::object; + ptr->template convert_null_to(); + } + else + { + ptr->template convert_null_to(); + } } if (ptr->is_object()) { - // use unchecked object access - ptr = &object_member(*ptr, reference_token); + // the object is new and empty + ptr = &object_add(*ptr, reference_token); } - else if (ptr->is_array()) + else { - if (reference_token == "-") + // only the first value is not created here, and it is a null + // value or an array (see get_unchecked_create) + JSON_ASSERT(ptr->is_array()); + if (is_dash(reference_token)) { // explicitly treat "-" as index beyond the end - ptr = &ptr->operator[](ptr->m_data.m_value.array->size()); + ptr = &array_append(*ptr); } else { @@ -25084,16 +25525,31 @@ class json_pointer ptr = &ptr->operator[](array_index(reference_token)); } } - else - { - // primitive value with a reference token left - throw_unresolved(reference_token, ptr); - } } return *ptr; } + /// whether @a s is a valid array index, that is a nonempty sequence of + /// digits without a leading '0' (cf. RFC 6901, Sect. 4); tokens that could + /// never be a valid array index (such as "01" or "") are object keys + static bool is_array_index_token(const string_t& s) + { + const std::size_t size = s.size(); + if (size == 0 || (size > 1 && s[0] == '0')) + { + return false; + } + for (std::size_t i = 0; i < size; ++i) + { + if (std::isdigit(static_cast(s[i])) == 0) + { + return false; + } + } + return true; + } + /*! @throw parse_error.106 if an array index begins with '0' @throw parse_error.109 if an array index was not a number diff --git a/tests/src/unit-alt-string.cpp b/tests/src/unit-alt-string.cpp index 961cd742b..75feb74e8 100644 --- a/tests/src/unit-alt-string.cpp +++ b/tests/src/unit-alt-string.cpp @@ -373,6 +373,23 @@ TEST_CASE("alternative string type") CHECK_FALSE(j.contains(alt_json::json_pointer("/foo/bar"))); } + SECTION("non-const operator[](json_pointer)") + { + // creating values must compile without string_t::begin() + alt_json j = alt_json::parse(R"({"a":{"b":[1]},"c":null})"); + const alt_json before = j; + j[alt_json::json_pointer("/a/b/3/x/-/y")] = 1; // pad, append, create + j[alt_json::json_pointer("/a/z")] = 2; // insert a key + j[alt_json::json_pointer("/c/0/q")] = 3; // convert null + j[alt_json::json_pointer("/a/b/-")] = 4; // append + CHECK(j.dump() == R"({"a":{"b":[1,null,null,{"x":[{"y":1}]},4],"z":2},"c":[{"q":3}]})"); + + // a failing access leaves the value unchanged + j = before; + CHECK_THROWS_AS(j[alt_json::json_pointer("/a/n/99999999999999999999999")], alt_json::out_of_range&); + CHECK(j == before); + } + SECTION("operator/(std::size_t)") { // json_pointer::operator/=(std::size_t) must compile without string_t diff --git a/tests/src/unit-json_pointer.cpp b/tests/src/unit-json_pointer.cpp index b28ac4023..81c588544 100644 --- a/tests/src/unit-json_pointer.cpp +++ b/tests/src/unit-json_pointer.cpp @@ -29,9 +29,64 @@ using nlohmann::json; #include #include #include +#include #include #include +namespace +{ +// the non-const operator[] must throw for pointer and leave the value parsed +// from init unchanged (strong exception safety) +template +void check_strong_guarantee(const char* init, const std::string& pointer, const char* message) +{ + CAPTURE(init) + CAPTURE(pointer) + Json j = Json::parse(init); + const Json before = Json::parse(init); + const typename Json::json_pointer ptr(pointer); + CHECK_THROWS_WITH_AS(j[ptr], message, typename Json::exception&); + CHECK(j == before); +} + +// as above, for an index that fits into size_type, but not into an array +template +void check_strong_guarantee_length_error(const char* init, const std::string& pointer) +{ + CAPTURE(init) + CAPTURE(pointer) + Json j = Json::parse(init); + const Json before = Json::parse(init); + const typename Json::json_pointer ptr(pointer); + CHECK_THROWS_AS(j[ptr], std::length_error&); + CHECK(j == before); +} + +template +void check_strong_guarantee_cases() +{ + const char* const unresolved = "[json.exception.out_of_range.404] unresolved reference token '99999999999999999999999'"; + + // the first change converts a null value + check_strong_guarantee("null", "/99999999999999999999999", unresolved); + check_strong_guarantee("null", "/a/99999999999999999999999", unresolved); + check_strong_guarantee(R"({"a":null})", "/a/b/-/99999999999999999999999", unresolved); + // the first change inserts a key into an object + check_strong_guarantee("{}", "/a/99999999999999999999999", unresolved); + check_strong_guarantee(R"({"a":{"b":1}})", "/a/c/d/99999999999999999999999", unresolved); + // the first change pads or appends to an array + check_strong_guarantee(R"({"a":[1]})", "/a/5/99999999999999999999999", unresolved); + check_strong_guarantee(R"({"a":[1]})", "/a/-/x/1/99999999999999999999999", unresolved); + // an index that does not fit into size_type + check_strong_guarantee("null", "/x/18446744073709551615", + "[json.exception.out_of_range.410] array index 18446744073709551615 exceeds size_type"); + // an index that fits into size_type, but not into an array + const auto too_large = std::to_string((std::numeric_limits::max)() - 1); + check_strong_guarantee_length_error("null", "/" + too_large); + check_strong_guarantee_length_error(R"({"a":[[1]]})", "/a/0/3/" + too_large); +} +} // namespace + TEST_CASE("JSON pointers") { SECTION("errors") @@ -557,6 +612,23 @@ TEST_CASE("JSON pointers") CHECK_THROWS_WITH_AS(j.at("/01"_json_pointer), "[json.exception.parse_error.106] parse error: array index '01' must not begin with '0'", json::parse_error&); } + + SECTION("a failing operator[] leaves the value unchanged") + { + check_strong_guarantee_cases(); + check_strong_guarantee_cases(); + } + + SECTION("a successful operator[] still creates the same values") + { + json j = {{"a", {1}}}; + j["/a/3/x/-/y"_json_pointer] = 1; + CHECK(j == json::parse(R"({"a":[1,null,null,{"x":[{"y":1}]}]})")); + j["/a/-"_json_pointer] = 2; + j["/a/0"_json_pointer] = 3; + j["/b"_json_pointer] = 4; + CHECK(j == json::parse(R"({"a":[3,null,null,{"x":[{"y":1}]},2],"b":4})")); + } } SECTION("flatten") diff --git a/tests/src/unit-regression1.cpp b/tests/src/unit-regression1.cpp index 084eb79fd..fae261ebf 100644 --- a/tests/src/unit-regression1.cpp +++ b/tests/src/unit-regression1.cpp @@ -1499,6 +1499,23 @@ TEST_CASE("regression tests 1") CHECK(order.dump() == R"({"a":2,"m":{"y":1,"b":2},"new_key":3})"); } + SECTION("operator[](json_pointer) when emplace() finds a key the lookup does not") + { + // a copied fifo_map restarts its timestamps, so a new key "k" gets + // the timestamp of "x": the lookup does not find "k", but emplace() + // finds "x" and inserts nothing; the pointer then continues with the + // member of "x", as basic_json::operator[] does + const my_json orig = my_json::parse(R"({"x":1})"); + const std::array pointers = {{"/k/0", "/k/a/b", "/k/-"}}; + for (const char* pointer : pointers) + { + CAPTURE(pointer) + my_json j = orig; + CHECK_THROWS_AS(j[my_json::json_pointer(pointer)], my_json::out_of_range&); + CHECK(j.dump() == R"({"x":1})"); + } + } + SECTION("issue #977 - Assigning between different json types") { foo_json lj = ns::foo{3}; From eb8f675bc9f28a3c9edc2708a4e5dfabb379edbc Mon Sep 17 00:00:00 2001 From: Niels Lohmann Date: Tue, 27 Oct 2026 17:16:08 +0100 Subject: [PATCH 2/3] docs: explain signed zero in the diff() round-trip exceptions (#5944) Spell out that numbers comparing equal produce no operation even if one is zero and the other negative zero, in diff.md and features/json_patch.md. Signed-off-by: Niels Lohmann --- docs/mkdocs/docs/api/basic_json/diff.md | 7 ++++--- docs/mkdocs/docs/features/json_patch.md | 5 +++-- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/mkdocs/docs/api/basic_json/diff.md b/docs/mkdocs/docs/api/basic_json/diff.md index 655261a3d..162882cdb 100644 --- a/docs/mkdocs/docs/api/basic_json/diff.md +++ b/docs/mkdocs/docs/api/basic_json/diff.md @@ -19,9 +19,10 @@ The equality holds up to [`operator==`](operator_eq.md), so it has the following - A value that is not equal to itself cannot round-trip. This applies to `NaN` (also nested in an array or object) and to [discarded](is_discarded.md) values: `diff` emits a `replace` operation for them. A `NaN` in the patch is serialized as `#!json null`, so a patch that was dumped and parsed again would set `#!json null`. -- Numbers that compare equal produce no operation, even if their types or signs differ: `diff(1, 1.0)`, - `diff(1u, 1)` (unsigned and signed integer) and `diff(0.0, -0.0)` all return `#!json []`. The patched value is then - the `source` and keeps its number type. +- Numbers that compare equal produce no operation, even if their types differ or if one is zero and the other is + negative zero: `diff(1, 1.0)` (integer and floating-point), `diff(1u, 1)` (unsigned and signed integer) and + `diff(0.0, -0.0)` (positive and negative zero) all return `#!json []`. The patched value is then the `source` and + keeps its number type and sign. ## Parameters diff --git a/docs/mkdocs/docs/features/json_patch.md b/docs/mkdocs/docs/features/json_patch.md index 3329c6cba..ea22b4e15 100644 --- a/docs/mkdocs/docs/features/json_patch.md +++ b/docs/mkdocs/docs/features/json_patch.md @@ -78,8 +78,9 @@ flowchart LR - `NaN` and [discarded](../api/basic_json/is_discarded.md) values are not equal to themselves, so the invariant cannot hold for them. `diff` emits a `replace` operation for such a value, and a `NaN` in a patch is serialized as `#!json null`. - - Numbers that compare equal produce no operation even if their types differ: `diff(1, 1.0)`, `diff(1u, 1)`, and - `diff(0.0, -0.0)` return `#!json []`, so the patched value keeps the number type of *source*. + - Numbers that compare equal produce no operation even if their types differ or if one is zero and the other is + negative zero: `diff(1, 1.0)`, `diff(1u, 1)`, and `diff(0.0, -0.0)` return `#!json []`, so the patched value + keeps the number type and sign of *source*. - A [binary value](binary_values.md) in a patch is serialized as an object (`#!json {"bytes":[...],"subtype":null}`), so a patch that was dumped and parsed again no longer holds a binary value. From 8617d9c94cc9f72ded869ac9c2e5cfda540a69da Mon Sep 17 00:00:00 2001 From: Niels Lohmann Date: Tue, 27 Oct 2026 17:25:18 +0100 Subject: [PATCH 3/3] Fix test lints that are red on develop (effc++, clang-tidy) (#5946) - unit-bson: initialize the SAX recorder's member (-Weffc++) - unit-json_patch: build the reference document independently instead of copying it (performance-unnecessary-copy-initialization) - unit-json_view_builder: use node_kind::link instead of an out-of-range enum cast (clang-analyzer-optin.core.EnumCastOutOfRange) Signed-off-by: Niels Lohmann --- tests/src/unit-bson.cpp | 2 +- tests/src/unit-json_patch.cpp | 11 ++++++++--- tests/src/unit-json_view_builder.cpp | 4 ++-- 3 files changed, 11 insertions(+), 6 deletions(-) diff --git a/tests/src/unit-bson.cpp b/tests/src/unit-bson.cpp index b4756c9b5..09321e3d5 100644 --- a/tests/src/unit-bson.cpp +++ b/tests/src/unit-bson.cpp @@ -1311,7 +1311,7 @@ TEST_CASE("BSON declared document size bounds the content") // records the events that a SAX parser receives struct recorder : nlohmann::json_sax { - std::string events; + std::string events {}; bool null() override { diff --git a/tests/src/unit-json_patch.cpp b/tests/src/unit-json_patch.cpp index b6c78532d..cc1b26a40 100644 --- a/tests/src/unit-json_patch.cpp +++ b/tests/src/unit-json_patch.cpp @@ -2178,8 +2178,13 @@ TEST_CASE("JSON patch - behavior documented in the API reference") SECTION("patch() has the strong guarantee") { - const json doc = R"({"a": [1, 2], "o": {"k": "v"}})"_json; - const json before = doc; + // independent values, so that the comparison below checks the untouched document + const auto make_doc = [] + { + return R"({"a": [1, 2], "o": {"k": "v"}})"_json; + }; + const json doc = make_doc(); + const json before = make_doc(); const std::vector failing = { @@ -2202,7 +2207,7 @@ TEST_CASE("JSON patch - behavior documented in the API reference") CHECK_THROWS_AS(doc.patch(patch), json::exception&); CHECK(doc == before); - json copy = doc; + json copy = make_doc(); CHECK_THROWS_AS(copy.patch(patch), json::exception&); CHECK(copy == before); } diff --git a/tests/src/unit-json_view_builder.cpp b/tests/src/unit-json_view_builder.cpp index 1074f7018..4023c7848 100644 --- a/tests/src/unit-json_view_builder.cpp +++ b/tests/src/unit-json_view_builder.cpp @@ -592,7 +592,7 @@ TEST_CASE("json_view node integer bits") }) { CAPTURE(v) - n.kind = static_cast(0x5A); + n.kind = nlohmann::detail::view::node_kind::link; n.flags = 0xA5; n.extra = 0x1234; n.off = 0x89ABCDEFu; @@ -601,7 +601,7 @@ TEST_CASE("json_view node integer bits") CHECK(n.len == static_cast(v)); CHECK(n.next == static_cast(v >> 32)); // the other fields are untouched - CHECK(static_cast(n.kind) == 0x5A); + CHECK(n.kind == nlohmann::detail::view::node_kind::link); CHECK(n.flags == 0xA5); CHECK(n.extra == 0x1234); CHECK(n.off == 0x89ABCDEFu);