Add root-pointer and array-append-token edge case tests for the move prefix check

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/patch.md

@@ -34,6 +34,10 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
   ("add", "remove", "move")
 - Throws [`out_of_range.411`](../../home/exceptions.md#jsonexceptionout_of_range411) if an "add" operation's target
   location has a parent that is neither an object nor an array.
+- Throws [`out_of_range.413`](../../home/exceptions.md#jsonexceptionout_of_range413) if a "remove" operation's target
+  location has a parent that is neither an object nor an array.
+- Throws [`out_of_range.414`](../../home/exceptions.md#jsonexceptionout_of_range414) if a "move" operation's "from"
+  location is a proper prefix of its "path" location.
 - Throws [`other_error.501`](../../home/exceptions.md#jsonexceptionother_error501) if "test" operation was
   unsuccessful.
 
@@ -75,3 +79,7 @@ is thrown. In any case, the original value is not changed: the patch is applied
 - Added in version 2.0.0.
 - Added [`out_of_range.411`](../../home/exceptions.md#jsonexceptionout_of_range411) and stopped relying on an internal assertion when an "add" operation's
   target location has a non-object/non-array parent in version 3.13.0.
+- Added [`out_of_range.413`](../../home/exceptions.md#jsonexceptionout_of_range413) and stopped silently ignoring a "remove" operation whose target
+  location has a non-object/non-array parent in version 3.13.0.
+- Added [`out_of_range.414`](../../home/exceptions.md#jsonexceptionout_of_range414) and rejected a "move" operation whose "from" location is a proper
+  prefix of its "path" location instead of silently producing a corrupted result in version 3.13.0.

docs/mkdocs/docs/api/basic_json/patch_inplace.md

@@ -30,6 +30,10 @@ No guarantees, value may be corrupted by an unsuccessful patch operation.
   ("add", "remove", "move")
 - Throws [`out_of_range.411`](../../home/exceptions.md#jsonexceptionout_of_range411) if an "add" operation's target
   location has a parent that is neither an object nor an array.
+- Throws [`out_of_range.413`](../../home/exceptions.md#jsonexceptionout_of_range413) if a "remove" operation's target
+  location has a parent that is neither an object nor an array.
+- Throws [`out_of_range.414`](../../home/exceptions.md#jsonexceptionout_of_range414) if a "move" operation's "from"
+  location is a proper prefix of its "path" location.
 - Throws [`other_error.501`](../../home/exceptions.md#jsonexceptionother_error501) if "test" operation was
   unsuccessful.
 
@@ -72,3 +76,7 @@ function throws an exception.
 - Added in version 3.11.0.
 - Added [`out_of_range.411`](../../home/exceptions.md#jsonexceptionout_of_range411) and stopped relying on an internal assertion when an "add" operation's
   target location has a non-object/non-array parent in version 3.13.0.
+- Added [`out_of_range.413`](../../home/exceptions.md#jsonexceptionout_of_range413) and stopped silently ignoring a "remove" operation whose target
+  location has a non-object/non-array parent in version 3.13.0.
+- Added [`out_of_range.414`](../../home/exceptions.md#jsonexceptionout_of_range414) and rejected a "move" operation whose "from" location is a proper
+  prefix of its "path" location instead of silently producing a corrupted result in version 3.13.0.

docs/mkdocs/docs/home/exceptions.md

@@ -933,6 +933,34 @@ BSON stores the length of documents, arrays, strings, and binary values in a sig
     [`to_bson`](../api/basic_json/to_bson.md) produced documents with negative length prefixes that
     [`from_bson`](../api/basic_json/from_bson.md) rejected.
 
+### json.exception.out_of_range.413
+
+A JSON Patch `remove` operation cannot be applied because the target location's parent is neither an object nor an array. Per [RFC 6902](https://datatracker.ietf.org/doc/html/rfc6902), a `remove` target must reference a member of an existing object or an element of an existing array; a primitive value (string, number, boolean, etc.) or `null` has no members or elements to remove.
+
+!!! failure "Example message"
+
+    ```
+    cannot remove value: the JSON Patch 'remove' target's parent is of type number, but must be an object or array
+    ```
+
+!!! note
+
+    This exception was added in version 3.13.0. Before that, this situation was silently ignored (the `remove` operation had no effect).
+
+### json.exception.out_of_range.414
+
+A JSON Patch `move` operation's `"from"` location is a proper prefix of its `"path"` location. Per [RFC 6902](https://datatracker.ietf.org/doc/html/rfc6902) (section 4.4), a location cannot be moved into one of its own children.
+
+!!! failure "Example message"
+
+    ```
+    cannot move value: 'from' path '/0' is a proper prefix of 'path' '/0/0'
+    ```
+
+!!! note
+
+    This exception was added in version 3.13.0. Before that, this situation could succeed with a corrupted result: for an array target, removing the "from" element before the "add" step shifted subsequent indices, so "path" silently re-resolved to a different element than intended.
+
 ## Further exceptions
 
 This exception is thrown in case of errors that cannot be classified with the

include/nlohmann/json.hpp

@@ -4939,6 +4939,12 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
                 // note erase performs range check
                 parent.erase(json_pointer::template array_index<basic_json_t>(last_path));
             }
+            else
+            {
+                // the parent of a "remove" target must be an object or array
+                // (see #5396)
+                JSON_THROW(out_of_range::create(413, detail::concat("cannot remove value: the JSON Patch 'remove' target's parent is of type ", parent.type_name(), ", but must be an object or array"), &parent));
+            }
         };
 
         // type check: top level value must be an array
@@ -5016,6 +5022,19 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
                     const auto from_path = get_value("move", "from", true).template get<string_t>();
                     json_pointer from_ptr(from_path);
 
+                    // RFC 6902 (section 4.4) forbids "from" from being a
+                    // proper prefix of "path": a location cannot be moved
+                    // into one of its own children. Compare the pointers'
+                    // reference tokens (already unescaped by json_pointer's
+                    // parser) rather than the raw pointer strings, since a
+                    // token may itself contain an escaped '/' or '~' that
+                    // would defeat a naive string-prefix comparison.
+                    if (JSON_HEDLEY_UNLIKELY(from_ptr.reference_tokens.size() < ptr.reference_tokens.size()
+                                             && std::equal(from_ptr.reference_tokens.begin(), from_ptr.reference_tokens.end(), ptr.reference_tokens.begin())))
+                    {
+                        JSON_THROW(out_of_range::create(414, detail::concat("cannot move value: 'from' path '", from_path, "' is a proper prefix of 'path' '", path, "'"), &result));
+                    }
+
                     // the "from" location must exist - use at()
                     basic_json const v = result.at(from_ptr);
 

single_include/nlohmann/json.hpp

@@ -26367,6 +26367,12 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
                 // note erase performs range check
                 parent.erase(json_pointer::template array_index<basic_json_t>(last_path));
             }
+            else
+            {
+                // the parent of a "remove" target must be an object or array
+                // (see #5396)
+                JSON_THROW(out_of_range::create(413, detail::concat("cannot remove value: the JSON Patch 'remove' target's parent is of type ", parent.type_name(), ", but must be an object or array"), &parent));
+            }
         };
 
         // type check: top level value must be an array
@@ -26444,6 +26450,19 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
                     const auto from_path = get_value("move", "from", true).template get<string_t>();
                     json_pointer from_ptr(from_path);
 
+                    // RFC 6902 (section 4.4) forbids "from" from being a
+                    // proper prefix of "path": a location cannot be moved
+                    // into one of its own children. Compare the pointers'
+                    // reference tokens (already unescaped by json_pointer's
+                    // parser) rather than the raw pointer strings, since a
+                    // token may itself contain an escaped '/' or '~' that
+                    // would defeat a naive string-prefix comparison.
+                    if (JSON_HEDLEY_UNLIKELY(from_ptr.reference_tokens.size() < ptr.reference_tokens.size()
+                                             && std::equal(from_ptr.reference_tokens.begin(), from_ptr.reference_tokens.end(), ptr.reference_tokens.begin())))
+                    {
+                        JSON_THROW(out_of_range::create(414, detail::concat("cannot move value: 'from' path '", from_path, "' is a proper prefix of 'path' '", path, "'"), &result));
+                    }
+
                     // the "from" location must exist - use at()
                     basic_json const v = result.at(from_ptr);
 

tests/src/unit-json_patch.cpp

@@ -1389,6 +1389,172 @@ TEST_CASE("JSON patch - add to a primitive parent (regression #4292)")
     }
 }
 
+TEST_CASE("JSON patch - remove with primitive or null parent (regression #5396)")
+{
+    // Regression test for https://github.com/nlohmann/json/issues/5396
+    //
+    // RFC 6902 (§4.2) requires the target location of a "remove" operation
+    // to exist. When the target's parent resolves to a primitive value or
+    // null, the operation must fail. Previously operation_remove silently
+    // did nothing in this case (neither the "is_object" nor the "is_array"
+    // branch matched, and there was no final "else"), so the patch appeared
+    // to succeed without changing the document. It now throws
+    // out_of_range.413.
+
+    SECTION("parent is a primitive (number)")
+    {
+        json const doc = {{"a", 1}};
+        json const patch = {{{"op", "remove"}, {"path", "/a/b"}}};
+#if JSON_DIAGNOSTICS
+        CHECK_THROWS_WITH_AS(doc.patch(patch), "[json.exception.out_of_range.413] (/a) cannot remove value: the JSON Patch 'remove' target's parent is of type number, but must be an object or array", json::out_of_range&);
+#else
+        CHECK_THROWS_WITH_AS(doc.patch(patch), "[json.exception.out_of_range.413] cannot remove value: the JSON Patch 'remove' target's parent is of type number, but must be an object or array", json::out_of_range&);
+#endif
+    }
+
+    SECTION("parent is a primitive (string)")
+    {
+        json const doc = {{"foo", {{"bar", "a string"}}}};
+        json const patch = {{{"op", "remove"}, {"path", "/foo/bar/baz"}}};
+#if JSON_DIAGNOSTICS
+        CHECK_THROWS_WITH_AS(doc.patch(patch), "[json.exception.out_of_range.413] (/foo/bar) cannot remove value: the JSON Patch 'remove' target's parent is of type string, but must be an object or array", json::out_of_range&);
+#else
+        CHECK_THROWS_WITH_AS(doc.patch(patch), "[json.exception.out_of_range.413] cannot remove value: the JSON Patch 'remove' target's parent is of type string, but must be an object or array", json::out_of_range&);
+#endif
+    }
+
+    SECTION("top-level document is null")
+    {
+        json const doc = nullptr;
+        json const patch = {{{"op", "remove"}, {"path", "/a"}}};
+        CHECK_THROWS_WITH_AS(doc.patch(patch), "[json.exception.out_of_range.413] cannot remove value: the JSON Patch 'remove' target's parent is of type null, but must be an object or array", json::out_of_range&);
+    }
+
+    SECTION("legitimate removes still work")
+    {
+        // object member
+        json const doc1 = {{"a", 1}, {"b", 2}};
+        json const patch1 = {{{"op", "remove"}, {"path", "/a"}}};
+        CHECK(doc1.patch(patch1) == json({{"b", 2}}));
+
+        // array element
+        json const doc2 = R"([1, 2, 3])"_json;
+        json const patch2 = {{{"op", "remove"}, {"path", "/1"}}};
+        CHECK(doc2.patch(patch2) == R"([1, 3])"_json);
+    }
+}
+
+TEST_CASE("JSON patch - move where 'from' is a proper prefix of 'path' (regression #5397)")
+{
+    // Regression test for https://github.com/nlohmann/json/issues/5397
+    //
+    // RFC 6902 (§4.4) forbids "from" from being a proper prefix of "path"
+    // for a "move" operation: "a location cannot be moved into one of its
+    // children." "move" is implemented as remove-then-add; for an object
+    // target this happened to throw anyway as a side effect of the "add"
+    // step re-resolving through the now-removed parent, but for an array
+    // target the removal shifted subsequent indices, so "path" silently
+    // re-resolved to a different element and the operation "succeeded"
+    // with a corrupted result. It now throws out_of_range.414 for both
+    // object and array targets.
+
+    SECTION("array target (from the issue)")
+    {
+        json const doc = R"([[1,2],[3]])"_json;
+        json const patch = {{{"op", "move"}, {"from", "/0"}, {"path", "/0/0"}}};
+        CHECK_THROWS_WITH_AS(doc.patch(patch), "[json.exception.out_of_range.414] cannot move value: 'from' path '/0' is a proper prefix of 'path' '/0/0'", json::out_of_range&);
+    }
+
+    SECTION("object target")
+    {
+        json const doc = R"({"a": {"b": 1}})"_json;
+        json const patch = {{{"op", "move"}, {"from", "/a"}, {"path", "/a/b"}}};
+        CHECK_THROWS_WITH_AS(doc.patch(patch), "[json.exception.out_of_range.414] cannot move value: 'from' path '/a' is a proper prefix of 'path' '/a/b'", json::out_of_range&);
+    }
+
+    SECTION("from == path is not a proper prefix and must not be rejected")
+    {
+        // "from" equal to "path" is a no-op move; it is not a *proper*
+        // prefix relationship, so this new check must not reject it.
+        json const doc = R"({"a": 1, "b": 2})"_json;
+        json const patch = {{{"op", "move"}, {"from", "/a"}, {"path", "/a"}}};
+        CHECK(doc.patch(patch) == doc);
+    }
+
+    SECTION("raw string prefix that is not a pointer-token prefix must be allowed")
+    {
+        // "/ab" is a string-prefix of "/abc/x" as raw text, but "ab" and
+        // "abc" are different reference tokens, so this is NOT a
+        // pointer-token prefix relationship and the move must succeed.
+        // This is the key case proving the check compares tokens, not
+        // raw pointer text (a naive std::string prefix/rfind check on
+        // the undecoded pointer would wrongly reject this).
+        json const doc = R"({"ab": 1, "abc": {"x": 2}})"_json;
+        json const patch = {{{"op", "move"}, {"from", "/ab"}, {"path", "/abc/x"}}};
+        json const result = R"({"abc": {"x": 1}})"_json;
+        CHECK(doc.patch(patch) == result);
+    }
+
+    SECTION("escaped reference tokens are compared unescaped")
+    {
+        // "from" is the single token "a/b" (escaped as "a~1b"); "path"
+        // addresses member "x" of that same value, so "from" is a
+        // proper (token-level) prefix of "path" and must be rejected.
+        json const doc = R"({"a/b": {"x": 1}})"_json;
+        json const patch = {{{"op", "move"}, {"from", "/a~1b"}, {"path", "/a~1b/x"}}};
+        CHECK_THROWS_WITH_AS(doc.patch(patch), "[json.exception.out_of_range.414] cannot move value: 'from' path '/a~1b' is a proper prefix of 'path' '/a~1b/x'", json::out_of_range&);
+    }
+
+    SECTION("ordinary valid moves still work")
+    {
+        // unrelated top-level members
+        json const doc1 = R"({"a": 1, "b": 2})"_json;
+        json const patch1 = {{{"op", "move"}, {"from", "/a"}, {"path", "/c"}}};
+        CHECK(doc1.patch(patch1) == R"({"b": 2, "c": 1})"_json);
+
+        // sibling paths that share a textual prefix but are unrelated
+        json const doc2 = R"({"a": {"x": 1}, "b": {"y": 2}})"_json;
+        json const patch2 = {{{"op", "move"}, {"from", "/a/x"}, {"path", "/b/z"}}};
+        CHECK(doc2.patch(patch2) == R"({"a": {}, "b": {"y": 2, "z": 1}})"_json);
+
+        // "path" is a proper prefix of "from" (the reverse relationship,
+        // which RFC 6902 does not forbid)
+        json const doc3 = R"({"a": {"b": 1}})"_json;
+        json const patch3 = {{{"op", "move"}, {"from", "/a/b"}, {"path", "/a"}}};
+        CHECK(doc3.patch(patch3) == R"({"a": 1})"_json);
+    }
+
+    SECTION("root 'from' is a proper prefix of every non-root 'path'")
+    {
+        // the whole document is a proper prefix of any location inside it
+        json const doc = R"({"a": 1})"_json;
+        json const patch = {{{"op", "move"}, {"from", ""}, {"path", "/a"}}};
+        CHECK_THROWS_WITH_AS(doc.patch(patch), "[json.exception.out_of_range.414] cannot move value: 'from' path '' is a proper prefix of 'path' '/a'", json::out_of_range&);
+    }
+
+    SECTION("root 'path' is never a proper prefix violation for a non-root 'from'")
+    {
+        // the reverse of the above: moving a non-root location to the root
+        // is the "path is a prefix of from" relationship, which RFC 6902
+        // permits (already covered generally above; this pins the root
+        // case specifically, since root is the one path with no reference
+        // tokens at all)
+        json const doc = R"({"a": {"b": 1}})"_json;
+        json const patch = {{{"op", "move"}, {"from", "/a"}, {"path", ""}}};
+        CHECK(doc.patch(patch) == R"({"b": 1})"_json);
+    }
+
+    SECTION("the array-append token '-' is an ordinary child token")
+    {
+        // "-" (append-to-array) addresses a location *inside* the array,
+        // so "from" pointing at the array is still a proper prefix of
+        // "path" ending in "-" and must be rejected like any other child.
+        json const doc = R"({"a": [1, 2]})"_json;
+        json const patch = {{{"op", "move"}, {"from", "/a"}, {"path", "/a/-"}}};
+        CHECK_THROWS_WITH_AS(doc.patch(patch), "[json.exception.out_of_range.414] cannot move value: 'from' path '/a' is a proper prefix of 'path' '/a/-'", json::out_of_range&);
+    }
+}
+
 TEST_CASE("JSON patch - diff emits array removals in descending index order")
 {
     SECTION("array shrunk to empty")

tests/src/unit-unicode3.cpp

@@ -304,8 +304,8 @@ TEST_CASE("Unicode (3/5)" * doctest::skip())
                         {
                             for (int byte4 = 0x00; byte4 <= 0xFF; ++byte4)
                             {
-                                // skip correct fourth byte
-                                if (0x80 <= byte4 && byte4 <= 0xBF)
+                                // skip fourth second byte
+                                if (0x80 <= byte3 && byte3 <= 0xBF)
                                 {
                                     continue;
                                 }

tests/src/unit-unicode4.cpp

@@ -305,7 +305,7 @@ TEST_CASE("Unicode (4/5)" * doctest::skip())
                             for (int byte4 = 0x00; byte4 <= 0xFF; ++byte4)
                             {
                                 // skip correct fourth byte
-                                if (0x80 <= byte4 && byte4 <= 0xBF)
+                                if (0x80 <= byte3 && byte3 <= 0xBF)
                                 {
                                     continue;
                                 }

tests/src/unit-unicode5.cpp

@@ -305,7 +305,7 @@ TEST_CASE("Unicode (5/5)" * doctest::skip())
                             for (int byte4 = 0x00; byte4 <= 0xFF; ++byte4)
                             {
                                 // skip correct fourth byte
-                                if (0x80 <= byte4 && byte4 <= 0xBF)
+                                if (0x80 <= byte3 && byte3 <= 0xBF)
                                 {
                                     continue;
                                 }