Fix self-aliasing in json_pointer::operator/= and patch_inplace

json_pointer::operator/=(const json_pointer&) passed iterators into its
own token vector to insert() when a pointer was appended to itself
(p /= p), which is undefined behavior; libstdc++ produced "/a/b//" for
"/a/b". The pointer now appends a copy in that case.

basic_json::patch_inplace iterated the patch while changing *this, so
j.patch_inplace(j) read freed memory. It now applies a copy of the
patch when the patch is the value itself. A patch stored inside the
value remains undefined behavior and is documented as such; copying the
patch unconditionally would double the time of a large patch.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/contains.md

@@ -61,18 +61,11 @@ Strong exception safety: if an exception occurs, the original value stays intact
 
 ## Complexity
 
-1. Logarithmic in the size of the JSON object.
-2. Logarithmic in the size of the JSON object.
-3. At most one lookup per reference token of `ptr`, each logarithmic in the size of the container at that level:
-   O(*n* log *m*) for *n* reference tokens and containers of size at most *m*.
-4. Deleted; a call with an integral argument does not compile.
+Logarithmic in the size of the JSON object.
 
 ## Notes
 
-- Overloads (1) and (2) always return `#!cpp false` when executed on a JSON type that is not an object. Overload (3)
-  also works on arrays and on primitive values: the empty JSON pointer resolves in every JSON value (so
-  `#!cpp json(1).contains(""_json_pointer)` returns `#!cpp true`), and `#!cpp json({1, 2}).contains("/1"_json_pointer)`
-  returns `#!cpp true`.
+- This method always returns `#!cpp false` when executed on a JSON type that is not an object.
 - This method can be executed on any JSON value type.
 - Calling this function with an integer argument (for example, `#!cpp contains(0)`) does not compile: such an argument
   would otherwise implicitly convert to a null `#!cpp const char*` and, from there, cause undefined behavior when

docs/mkdocs/docs/api/basic_json/patch_inplace.md

@@ -62,6 +62,16 @@ Unlike [`patch`](patch.md), `patch_inplace` applies the operation "in place" and
 That makes it faster for large documents by avoiding the copy. However, the JSON value might be corrupted if the
 function throws an exception.
 
+The JSON patch may be the JSON value itself (`#!cpp j.patch_inplace(j)`): the operations are then read from a copy of
+the patch made before the first operation is applied, so the result is the same as that of
+`#!cpp j = j.patch(j)`, and the exception safety is the same as above.
+
+!!! danger "Undefined behavior"
+
+    The JSON patch must not be a value stored inside the JSON value (e.g., `#!cpp j.patch_inplace(j["patch"])`), as the
+    operations may change or destroy it while it is read. Pass a copy (`#!cpp j.patch_inplace(json(j["patch"]))`) or use
+    [`patch`](patch.md) instead.
+
 ## Examples
 
 ??? example "Example: apply a JSON patch in place"
@@ -113,3 +123,4 @@ function throws an exception.
 - Throws [`parse_error.109`](../../home/exceptions.md#jsonexceptionparse_error109) instead of
   [`out_of_range.404`](../../home/exceptions.md#jsonexceptionout_of_range404) for a one-character array index that is
   not a digit (e.g., `/x`) in version 3.13.0, as it already did for longer ones.
+- Applying a JSON value as patch to itself no longer results in undefined behavior in version 3.13.0.

docs/mkdocs/docs/api/basic_json/unflatten.md

@@ -36,9 +36,9 @@ The function can throw the following exceptions:
 - Throws [`parse_error.109`](../../home/exceptions.md#jsonexceptionparse_error109) if an array index in a key is not a
   number; example: `"array index 'one' is not a number"`
 - Throws [`out_of_range.404`](../../home/exceptions.md#jsonexceptionout_of_range404) if a level becomes an array
-  (because one of its keys is `0`) and another key at that level is empty (as in `/a/`), is `-`, or begins with a digit
-  but is not a valid array index (such as `1a`); examples: `"unresolved reference token '-'"`,
-  `"unresolved reference token ''"`
+  (because one of its keys is `0`) and another key at that level begins with a digit but is not a valid array index
+  (such as `1a`), or is `-`; example:
+  `"unresolved reference token '-'"`
 
 ## Complexity
 
@@ -46,7 +46,7 @@ Linear in the size of the JSON value.
 
 ## Notes
 
-Empty objects and arrays are flattened by [`flatten()`](flatten.md) to `#!json null` values and cannot be unflattened to
+Empty objects and arrays are flattened by [`flatten()`](flatten.md) to `#!json null` values and cannot unflattened to
 their original type.
 
 A flattened array and a flattened object whose keys are array indices are indistinguishable, because both are
@@ -54,12 +54,7 @@ described by the same JSON pointers. A value is therefore restored as an array i
 reference token `0`, and as an object otherwise: `#!json {"2": 1}` is restored unchanged, whereas `#!json {"0": 1}` is
 restored as `#!json [1]`. This decision does not depend on the order in which the flattened object is iterated.
 
-A third case is an object that has the key `0` next to a key that is not a valid array index: `#!json {"0": 1, "b": 2}`
-is flattened to `#!json {"/0": 1, "/b": 2}`, and unflatten treats the level as an array and throws
-[`parse_error.109`](../../home/exceptions.md#jsonexceptionparse_error109) because `b` is not a number. (For the keys
-`-`, `1a`, or the empty key instead of `b`, it throws [`out_of_range.404`](../../home/exceptions.md#jsonexceptionout_of_range404).)
-
-Apart from these three cases, for a JSON value `j`, the following is always true:
+Apart from these two cases, for a JSON value `j`, the following is always true:
 `#!cpp j == j.flatten().unflatten()`.
 
 ## Examples

docs/mkdocs/docs/api/basic_json/value.md

@@ -45,9 +45,6 @@ This is equivalent to Python's `dict.get(key, default)`.
        return default_value;
     }
     ```
-    with one exception: if the JSON value is neither an array nor an object, the function throws
-    [`type_error.306`](../../home/exceptions.md#jsonexceptiontype_error306) before looking at `ptr`, even for the empty
-    pointer, whereas `at` returns the value itself for the empty pointer.
 
 !!! note "Differences to `at` and `operator[]`"
 

docs/mkdocs/docs/api/json_pointer/operator_slasheq.md

@@ -18,7 +18,8 @@ json_pointer& operator/=(std::size_t array_idx)
 ## Parameters
 
 `ptr` (in)
-:    JSON pointer to append
+:    JSON pointer to append; may be the JSON pointer itself (`#!cpp p /= p`), which then appends a copy of its
+     reference tokens
 
 `token` (in)
 :    reference token to append
@@ -68,6 +69,7 @@ json_pointer& operator/=(std::size_t array_idx)
 
 ## Version history
 
-1. Added in version 3.6.0.
+1. Added in version 3.6.0. Appending a JSON pointer to itself no longer results in undefined behavior in version
+   3.13.0.
 2. Added in version 3.6.0. Changed type of `token` to `string_t` in version 3.11.0.
 3. Added in version 3.6.0.

docs/mkdocs/docs/api/json_pointer/parent_pointer.md

@@ -12,8 +12,7 @@ Parent of this JSON pointer; in case this JSON pointer is the root, the root its
 
 ## Exception safety
 
-Strong exception safety: if an exception occurs, the original value stays intact. The function is not `#!cpp noexcept`
-and allocates a new JSON pointer, so it can throw `#!cpp std::bad_alloc`; it throws no library exceptions.
+No-throw guarantee: this function never throws exceptions.
 
 ## Complexity
 

docs/mkdocs/docs/api/operator_ltlt.md

@@ -37,8 +37,7 @@ the stream `o`
 1. Throws [`type_error.316`](../home/exceptions.md#jsonexceptiontype_error316) if a string stored inside the JSON
    value is not UTF-8 encoded. Note that unlike the [`dump`](basic_json/dump.md) member functions, no `error_handler`
    can be set.
-2. Throws no library exceptions. As the string representation is built with [`to_string`](json_pointer/to_string.md),
-   `#!cpp std::bad_alloc` is possible.
+2. None.
 
 ## Complexity
 

docs/mkdocs/docs/features/json_pointer.md

@@ -37,10 +37,6 @@ Then every value inside the JSON document can be identified as follows:
 Note `/` does not identify the root (i.e., the whole document), but an object entry with empty key `""`. See
 [RFC 6901](https://tools.ietf.org/html/rfc6901) for more information.
 
-[RFC 6901, section 4](https://tools.ietf.org/html/rfc6901#section-4) states that evaluation fails if an object has
-non-unique member names. A `basic_json` object holds one value per key (see [`object_t`](../api/basic_json/object_t.md)
-for how duplicate keys are handled while parsing), so evaluating a JSON Pointer never encounters duplicates.
-
 ## JSON Pointer creation
 
 JSON Pointers can be created from a string:
@@ -58,13 +54,6 @@ auto p = "/nested/one"_json_pointer;
 The escaping rules of [RFC 6901](https://tools.ietf.org/html/rfc6901) are implemented. See the
 [constructor documentation](../api/json_pointer/json_pointer.md) for more information.
 
-!!! note "URI fragment representation"
-
-    [RFC 6901, section 6](https://tools.ietf.org/html/rfc6901#section-6) also defines a URI fragment identifier
-    representation (`#/a%20b`). It is not supported: `#!cpp json::json_pointer("#/a")` throws
-    [`parse_error.107`](../home/exceptions.md#jsonexceptionparse_error107). Strip the leading `#` and
-    percent-decode the remainder before creating the JSON pointer.
-
 ## Value access
 
 JSON Pointers can be used in the [`at`](../api/basic_json/at.md), [`operator[]`](../api/basic_json/operator%5B%5D.md),

docs/mkdocs/docs/features/types/template_parameters.md

@@ -390,7 +390,6 @@ using array_t = ArrayType<basic_json, AllocatorType<basic_json>>;
   described in RFC 6901. Neither `find(const StringType&, size_type)`, nor `substr(pos, count)`, nor
   `replace(pos, count, const StringType&)` is required.
 - `empty()`.
-- `resize(n)` (one argument) -- used by `to_string` to size the result before the reference tokens are copied into it.
 - `begin()` and `end()` -- used by
   [`operator[](const json_pointer&)`](../../api/basic_json/operator%5B%5D.md) to decide whether a reference token
   denotes an array index.

docs/mkdocs/docs/home/exceptions.md

@@ -1083,14 +1083,9 @@ a JSON pointer exceeds the range of `size_type` (e.g., on 32-bit platforms).
 
     ```
     enum value out of range
-    array index 18446744073709551615 exceeds size_type
+    array index 18446744073709551616 exceeds size_type
     ```
 
-    The second message is thrown, for instance, by `#!cpp json::parse("[1, 2]").at("/18446744073709551615"_json_pointer)`
-    on a 64-bit platform, where `size_type` is `#!cpp std::size_t` and this index is `SIZE_MAX`. An index that does not
-    fit into 64 bits (such as `18446744073709551616`) is not reported as 410 but as
-    [`out_of_range.404`](#jsonexceptionout_of_range404).
-
 ### json.exception.out_of_range.411
 
 A JSON Patch `add` operation cannot be applied because the target location's parent is neither an object nor an array. Per [RFC 6902](https://datatracker.ietf.org/doc/html/rfc6902), an `add` target must reference a member of an existing object or an element of an existing array; a primitive value (string, number, boolean, etc.) cannot receive a new member or element.

include/nlohmann/detail/json_pointer.hpp

@@ -243,6 +243,13 @@ class json_pointer
     /// @sa https://json.nlohmann.me/api/json_pointer/operator_slasheq/
     json_pointer& operator/=(const json_pointer& ptr)
     {
+        // insert() must not be given iterators into the vector it inserts
+        // into: append a copy when a pointer is appended to itself
+        if (JSON_HEDLEY_UNLIKELY(&ptr == this))
+        {
+            return *this /= json_pointer(ptr);
+        }
+
         reference_tokens.insert(reference_tokens.end(),
                                 ptr.reference_tokens.begin(),
                                 ptr.reference_tokens.end());

include/nlohmann/json.hpp

@@ -6950,6 +6950,15 @@ public:
     /// @sa https://json.nlohmann.me/api/basic_json/patch/
     void patch_inplace(const basic_json& json_patch)
     {
+        // the operations below change *this while iterating json_patch: apply
+        // a copy when a value is applied as patch to itself
+        if (JSON_HEDLEY_UNLIKELY(&json_patch == this))
+        {
+            const basic_json json_patch_copy = json_patch;
+            patch_inplace(json_patch_copy);
+            return;
+        }
+
         basic_json& result = *this;
         // the valid JSON Patch operations
         enum class patch_operations {add, remove, replace, move, copy, test, invalid};

single_include/nlohmann/json.hpp

@@ -23828,6 +23828,13 @@ class json_pointer
     /// @sa https://json.nlohmann.me/api/json_pointer/operator_slasheq/
     json_pointer& operator/=(const json_pointer& ptr)
     {
+        // insert() must not be given iterators into the vector it inserts
+        // into: append a copy when a pointer is appended to itself
+        if (JSON_HEDLEY_UNLIKELY(&ptr == this))
+        {
+            return *this /= json_pointer(ptr);
+        }
+
         reference_tokens.insert(reference_tokens.end(),
                                 ptr.reference_tokens.begin(),
                                 ptr.reference_tokens.end());
@@ -42740,6 +42747,15 @@ public:
     /// @sa https://json.nlohmann.me/api/basic_json/patch/
     void patch_inplace(const basic_json& json_patch)
     {
+        // the operations below change *this while iterating json_patch: apply
+        // a copy when a value is applied as patch to itself
+        if (JSON_HEDLEY_UNLIKELY(&json_patch == this))
+        {
+            const basic_json json_patch_copy = json_patch;
+            patch_inplace(json_patch_copy);
+            return;
+        }
+
         basic_json& result = *this;
         // the valid JSON Patch operations
         enum class patch_operations {add, remove, replace, move, copy, test, invalid};

tests/src/unit-json_patch.cpp

@@ -823,6 +823,109 @@ TEST_CASE("JSON patch")
             CHECK(doc.at("foo") == "bar");
         }
 #endif // !defined(JSON_NOEXCEPTION)
+
+        SECTION("applying a value as patch to itself")
+        {
+            // a JSON patch is an array, so an array of operations can be
+            // applied to itself; the operations are read from the patch as
+            // it was before the first operation, as patch() does it
+            const json copy_op = {{"op", "copy"}, {"from", "/0"}, {"path", "/-"}};
+            json copy_ops = json::array();
+            for (int i = 0; i < 5; ++i)
+            {
+                copy_ops.push_back(copy_op);
+            }
+
+            json expected_copies = copy_ops;
+            for (int i = 0; i < 5; ++i)
+            {
+                expected_copies.push_back(copy_op);
+            }
+
+            SECTION("copy operations without spare capacity")
+            {
+                json doc = copy_ops;
+                doc.get_ref<json::array_t&>().shrink_to_fit();
+                CHECK(doc.patch(doc) == expected_copies);
+
+                doc.patch_inplace(doc);
+                CHECK(doc == expected_copies);
+            }
+
+            SECTION("copy operations with spare capacity")
+            {
+                json doc = copy_ops;
+                doc.get_ref<json::array_t&>().reserve(100);
+                CHECK(doc.patch(doc) == expected_copies);
+
+                doc.patch_inplace(doc);
+                CHECK(doc == expected_copies);
+            }
+
+            SECTION("remove operations")
+            {
+                json doc = R"([
+                        {"op": "remove", "path": "/0"},
+                        {"op": "remove", "path": "/0"},
+                        {"op": "remove", "path": "/0"}
+                    ])"_json;
+                CHECK(doc.patch(doc) == json::array());
+
+                doc.patch_inplace(doc);
+                CHECK(doc == json::array());
+            }
+
+            SECTION("operations that read a changed element")
+            {
+                // the second operation is still a "test" operation, although
+                // the first one replaced it in the document
+                json doc = R"([
+                        {"op": "replace", "path": "/1", "value": 1},
+                        {"op": "test", "path": "/1", "value": 1},
+                        {"op": "add", "path": "/-", "value": 2}
+                    ])"_json;
+                const json expected = R"([
+                        {"op": "replace", "path": "/1", "value": 1},
+                        1,
+                        {"op": "add", "path": "/-", "value": 2},
+                        2
+                    ])"_json;
+                CHECK(doc.patch(doc) == expected);
+
+                doc.patch_inplace(doc);
+                CHECK(doc == expected);
+            }
+
+            SECTION("replacing the root")
+            {
+                json doc = R"([
+                        {"op": "add", "path": "", "value": [1, 2]},
+                        {"op": "add", "path": "/-", "value": 3},
+                        {"op": "replace", "path": "", "value": {"a": 4}}
+                    ])"_json;
+                const json expected = {{"a", 4}};
+                CHECK(doc.patch(doc) == expected);
+
+                doc.patch_inplace(doc);
+                CHECK(doc == expected);
+            }
+
+#if !defined(JSON_NOEXCEPTION)
+            SECTION("failing operation")
+            {
+                // the operations before the failing one remain applied
+                json doc = R"([
+                        {"op": "add", "path": "/-", "value": 1},
+                        {"op": "test", "path": "/0/op", "value": "remove"}
+                    ])"_json;
+                CHECK_THROWS_AS(doc.patch(doc), json::other_error&);
+
+                CHECK_THROWS_AS(doc.patch_inplace(doc), json::other_error&);
+                CHECK(doc.size() == 3);
+                CHECK(doc.back() == 1);
+            }
+#endif // !defined(JSON_NOEXCEPTION)
+        }
     }
 
     SECTION("errors")

tests/src/unit-json_pointer.cpp

@@ -779,6 +779,44 @@ TEST_CASE("JSON pointers")
         CHECK(ptr.to_string() == "/object/~1");
     }
 
+    SECTION("appending a pointer to itself")
+    {
+        // operator/= must not pass iterators into its own token vector to
+        // insert(); each pointer is appended to itself and must yield the
+        // pointer whose string representation is the original one twice
+        const std::vector<std::string> pointers =
+        {
+            "",
+            "/",
+            "/a",
+            "/a/b",
+            "/foo/0/-",
+            "/a~1b/~0c/~01",
+            "/1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17"
+        };
+
+        for (const auto& s : pointers)
+        {
+            CAPTURE(s)
+            json::json_pointer ptr(s);
+            const json::json_pointer expected(s + s);
+
+            json::json_pointer& ret = (ptr /= ptr);
+            CHECK(&ret == &ptr);
+            CHECK(ptr == expected);
+            CHECK(ptr.to_string() == s + s);
+
+            // appending to itself again doubles the pointer once more
+            ptr /= ptr;
+            CHECK(ptr.to_string() == s + s + s + s);
+
+            // operator/ appends to a copy and leaves the operand unchanged
+            const json::json_pointer original(s);
+            CHECK((original / original) == expected);
+            CHECK(original.to_string() == s);
+        }
+    }
+
     SECTION("equality comparison")
     {
         std::string ptr_string{"/foo/bar"};
@@ -1044,56 +1082,6 @@ TEST_CASE("flatten of structured values")
         CHECK(flat_object.unflatten() == nlohmann::ordered_json::parse(R"({"b": {"1": 2}})"));
     }
 
-#if !defined(JSON_NOEXCEPTION)
-    SECTION("an object with the key \"0\" next to a non-index key does not round-trip")
-    {
-        // the third documented case in which j != j.flatten().unflatten()
-        const json value = {{"a", {{"0", 1}, {"b", 2}}}};
-        const json flat = value.flatten();
-        CHECK(flat == json({{"/a/0", 1}, {"/a/b", 2}}));
-        CHECK_THROWS_WITH_AS(flat.unflatten(),
-                             "[json.exception.parse_error.109] parse error: array index 'b' is not a number", json::parse_error&);
-
-        // the same for other keys that are not valid array indices
-        CHECK_THROWS_WITH_AS(json({{"/a/0", 1}, {"/a/1a", 2}}).unflatten(),
-                             "[json.exception.out_of_range.404] unresolved reference token '1a'", json::out_of_range&);
-        CHECK_THROWS_WITH_AS(json({{"/a/0", 1}, {"/a/-", 2}}).unflatten(),
-                             "[json.exception.out_of_range.404] unresolved reference token '-'", json::out_of_range&);
-    }
-
-    SECTION("unflatten with the empty token among array indices")
-    {
-        CHECK_THROWS_WITH_AS(json({{"/a/0", 1}, {"/a/", 2}}).unflatten(),
-                             "[json.exception.out_of_range.404] unresolved reference token ''", json::out_of_range&);
-        CHECK_THROWS_WITH_AS(json({{"/a/0", 1}, {"/a/1", 2}, {"/a/", 3}}).unflatten(),
-                             "[json.exception.out_of_range.404] unresolved reference token ''", json::out_of_range&);
-
-        // without an index "0", the empty token is an ordinary object key
-        CHECK(json({{"/a/", 2}}).unflatten() == json({{"a", {{"", 2}}}}));
-    }
-
-    SECTION("value() with the empty pointer on a primitive root")
-    {
-        // unlike at(), value() rejects primitive values before it looks at the pointer
-        const json number = 1;
-        CHECK(number.at(""_json_pointer) == 1);
-        CHECK_THROWS_WITH_AS(number.value(""_json_pointer, 7),
-                             "[json.exception.type_error.306] cannot use value() with number", json::type_error&);
-
-        const json string = "s";
-        CHECK_THROWS_WITH_AS(string.value(""_json_pointer, 7),
-                             "[json.exception.type_error.306] cannot use value() with string", json::type_error&);
-
-        // contains() accepts the empty pointer on every value
-        CHECK(number.contains(""_json_pointer));
-        CHECK_FALSE(number.contains("/a"_json_pointer));
-
-        // arrays and objects return the default for a missing location
-        CHECK(json::array().value(""_json_pointer, json(7)) == json::array());
-        CHECK(json::array().value("/0"_json_pointer, 7) == 7);
-    }
-#endif
-
     SECTION("objects and arrays interleaved")
     {
         const json value =