support JSON Merge Patch (RFC 7396) diff creation (#4965)

* support JSON Merge Patch (RFC 7396) diff creation

The JSON merge patch document format describes the set of modifications
to a resource's content, that more closely mimics the syntax of the
resource being modified.

However, and in contrast to JSON Patch (RFC 6902), a JSON Merge Patch
cannot express certain modifications, e.g., changing an array element
at a specific index, or setting a specific object value to null.
The null value in a JSON Merge Patch is used to remove the key from the
object.

The diff algorithm is not part of the RFC 7396, but it was tested
against all examples provided, plus additional cases on how null values
are handled.

Signed-off-by: Luís Murta <luis@murta.dev>

* improve algorithm efficiency

Signed-off-by: Luís Murta <luis@murta.dev>

* fix inconsistent nulls and empty arrays

Also add CHECKs for the patch, documentation and run `make amalgamate`.

Signed-off-by: Luís Murta <luis@murta.dev>

* documentation cleanups

Signed-off-by: Luís Murta <luis@murta.dev>

* add an iterative fallback to merge_diff

Bound the recursion depth like diff() and merge_patch(), and test values
nested past the bound.

Signed-off-by: Luís Murta <luis@murta.dev>

* address the remaining review comments

Signed-off-by: Luís Murta <luis@murta.dev>

* move multi-line raw strings out of CHECK macros

GCC 4.8/5, ICPC and NVHPC mis-tokenize multi-line raw string literals
inside macro arguments, breaking the build or the parsed JSON.

Signed-off-by: Luís Murta <luis@murta.dev>

---------

Signed-off-by: Luís Murta <luis@murta.dev>

docs/mkdocs/docs/api/basic_json/diff.md

@@ -59,6 +59,7 @@ Currently, only `remove`, `add`, and `replace` operations are generated.
 - [patch](patch.md) applies a JSON Patch
 - [patch_inplace](patch_inplace.md) applies a JSON Patch in place
 - [merge_patch](merge_patch.md) applies a JSON Merge Patch
+- [merge_diff](merge_diff.md) creates a diff as a JSON Merge Patch
 
 ## Version history
 

docs/mkdocs/docs/api/basic_json/index.md

@@ -285,6 +285,7 @@ Access to the JSON value
 ### JSON Merge Patch functions
 
 - [**merge_patch**](merge_patch.md) - applies a JSON Merge Patch
+- [**merge_diff**](merge_diff.md) (_static_) - creates a diff as a JSON Merge Patch
 
 ## Static functions
 

docs/mkdocs/docs/api/basic_json/merge_diff.md

@@ -0,0 +1,79 @@
+# <small>nlohmann::basic_json::</small>merge_diff
+
+```cpp
+static basic_json merge_diff(const basic_json& source,
+                             const basic_json& target);
+```
+
+Creates a [JSON Merge Patch](https://tools.ietf.org/html/rfc7396) so that value `source` can be changed into the value
+`target` by calling the [`merge_patch`](merge_patch.md) function. The patch contains only what differs: changed and
+added members with their new values, and removed members with the value `#!json null`. Nested objects are compared
+member by member; any other value that differs, including an array, is replaced as a whole.
+
+For two JSON values `source` and `target`, where `target` contains no object member whose value is `#!json null`, the
+following code always yields `#!cpp true`:
+```cpp
+basic_json patched = source;
+patched.merge_patch(merge_diff(source, target));
+patched == target;
+```
+
+## Parameters
+
+`source` (in)
+:   JSON value to compare from
+
+`target` (in)
+:   JSON value to compare against
+
+## Return value
+
+a JSON Merge Patch to convert the `source` to `target`
+
+## Exception safety
+
+Strong guarantee: `source` and `target` are never modified.
+
+## Complexity
+
+Linear in the total number of values (including nested ones) in `source` and `target`, times the cost of a key-based
+object operation (looking up a key, or adding one to the patch): logarithmic for [`json`](../json.md), and linear for
+[`ordered_json`](../ordered_json.md) (see [`ordered_map` complexity](../ordered_map.md#complexity)). For
+`ordered_json`, diffing two objects with `n` and `m` members therefore takes O((n+m)²).
+
+## Notes
+
+- A merge patch uses `#!json null` to remove a member, so it cannot set a member to `#!json null`. As in RFC 7396, a
+  `#!json null` member of `target` is treated as absent: the patch removes the member from `source` instead of setting
+  it to `#!json null`. A member that is `#!json null` in both `source` and `target` is unchanged and left out of the
+  patch, so `source` keeps it.
+- A merge patch cannot reorder object members, and [`merge_patch`](merge_patch.md) appends added members at the end.
+  For [`ordered_json`](../ordered_json.md), the patched value therefore has the same members as `target`, but not
+  necessarily in the same order, and may compare unequal to it. The patch lists changed and removed members in the
+  order of `source`, followed by added members in the order of `target`.
+
+## Examples
+
+??? example
+
+    The following code shows how a JSON Merge Patch is created as a diff for two JSON values.
+
+    ```cpp
+    --8<-- "examples/merge_diff.cpp"
+    ```
+
+    Output:
+
+    ```json
+    --8<-- "examples/merge_diff.output"
+    ```
+
+## See also
+
+- [RFC 7396 (JSON Merge Patch)](https://tools.ietf.org/html/rfc7396)
+- [merge_patch](merge_patch.md) applies a JSON Merge Patch
+- [diff](diff.md) creates a diff as a JSON Patch
+
+## Version history
+
+- Added in version 3.13.0.

docs/mkdocs/docs/api/basic_json/merge_patch.md

@@ -67,6 +67,7 @@ returned by `#!cpp (*this)[key]`); it is read as it was when `merge_patch()` was
 
 - [RFC 7396 (JSON Merge Patch)](https://tools.ietf.org/html/rfc7396)
 - [patch](patch.md) apply a JSON patch
+- [merge_diff](merge_diff.md) creates a diff as a JSON Merge Patch
 
 ## Version history
 

docs/mkdocs/docs/examples/merge_diff.cpp

@@ -0,0 +1,51 @@
+#include <iostream>
+#include <iomanip>
+#include <nlohmann/json.hpp>
+
+using json = nlohmann::json;
+using namespace nlohmann::literals;
+
+int main()
+{
+    // the source document
+    json source = R"(
+        {
+            "title": "Goodbye!",
+            "author": {
+                "givenName": "John",
+                "familyName": "Doe"
+            },
+            "tags": [
+                "example",
+                "sample"
+            ],
+            "content": "This will be unchanged"
+        }
+    )"_json;
+
+    // the target document
+    json target = R"(
+        {
+            "title": "Hello!",
+            "author": {
+                "givenName": "John"
+            },
+            "tags": [
+                "example"
+            ],
+            "content": "This will be unchanged",
+            "phoneNumber": "+01-123-456-7890"
+        }
+    )"_json;
+
+    // create the patch
+    json patch = json::merge_diff(source, target);
+
+    // roundtrip
+    json patched_source = source;
+    patched_source.merge_patch(patch);
+
+    // output patch and roundtrip result
+    std::cout << std::setw(4) << patch << "\n\n"
+              << std::setw(4) << patched_source << std::endl;
+}

docs/mkdocs/docs/examples/merge_diff.output

@@ -0,0 +1,22 @@
+{
+    "author": {
+        "familyName": null
+    },
+    "phoneNumber": "+01-123-456-7890",
+    "tags": [
+        "example"
+    ],
+    "title": "Hello!"
+}
+
+{
+    "author": {
+        "givenName": "John"
+    },
+    "content": "This will be unchanged",
+    "phoneNumber": "+01-123-456-7890",
+    "tags": [
+        "example"
+    ],
+    "title": "Hello!"
+}

docs/mkdocs/docs/features/merge_patch.md

@@ -9,6 +9,9 @@ syntax that closely mimics the document being modified. Unlike [JSON Patch](json
 express every kind of change (e.g., it cannot reorder array elements or remove a specific array element), but it is
 easier to read and write for object-shaped documents.
 
+The function [`merge_diff`](../api/basic_json/merge_diff.md) creates a merge patch that changes one JSON value into
+another.
+
 !!! tip "Not a general deep merge"
 
     A merge patch is not a general deep merge: a `#!json null` value in the patch deletes the key from the target.
@@ -35,4 +38,5 @@ easier to read and write for object-shaped documents.
 - [JSON Patch and Diff](json_patch.md) - a more expressive alternative that describes a sequence of operations
 - [JSON Pointer](json_pointer.md) - the addressing scheme used by JSON Patch
 - Function [`merge_patch`](../api/basic_json/merge_patch.md)
+- Function [`merge_diff`](../api/basic_json/merge_diff.md)
 - Function [`update`](../api/basic_json/update.md) - merge objects, optionally recursively

docs/mkdocs/mkdocs.yml

@@ -182,6 +182,7 @@ nav:
           - 'json_serializer': api/basic_json/json_serializer.md
           - 'max_size': api/basic_json/max_size.md
           - 'meta': api/basic_json/meta.md
+          - 'merge_diff': api/basic_json/merge_diff.md
           - 'merge_patch': api/basic_json/merge_patch.md
           - 'number_float_t': api/basic_json/number_float_t.md
           - 'number_integer_t': api/basic_json/number_integer_t.md

include/nlohmann/json.hpp

@@ -7449,6 +7449,176 @@ public:
     }
 
   public:
+    /// @brief creates a diff as a JSON Merge Patch
+    /// @sa https://json.nlohmann.me/api/basic_json/merge_diff/
+    JSON_HEDLEY_WARN_UNUSED_RESULT
+    static basic_json merge_diff(const basic_json& source, const basic_json& target)
+    {
+        // unless both are objects, the patch is the target itself
+        if (!source.is_object() || !target.is_object())
+        {
+            return target;
+        }
+
+        return merge_diff_recursively(source, target, 0);
+    }
+
+  private:
+    /// @brief two objects @ref merge_diff_iteratively is diffing
+    struct merge_diff_frame
+    {
+        merge_diff_frame(const basic_json* source_, const basic_json* target_)
+            : source(source_), target(target_), member(source_->cbegin()), patch(value_t::object)
+        {}
+
+        // declared for GCC's -Weffc++, as in diff_frame
+        merge_diff_frame(const merge_diff_frame&) = default;
+        merge_diff_frame(merge_diff_frame&&) = default;
+        merge_diff_frame& operator=(const merge_diff_frame&) = default;
+        merge_diff_frame& operator=(merge_diff_frame&&) = default;
+        ~merge_diff_frame() = default;
+
+        /// the objects being diffed
+        const basic_json* source;
+        const basic_json* target;
+        /// the member of source being diffed; advanced once it is done
+        const_iterator member;
+        /// the patch for the members of source diffed so far
+        basic_json patch;
+    };
+
+    /// @brief add the members of object @a target that @a source does not
+    ///        have to the merge patch @a result
+    static void merge_diff_added_members(basic_json& result, const basic_json& source, const basic_json& target)
+    {
+        for (auto it = target.begin(); it != target.end(); ++it)
+        {
+            auto itf = source.find(it.key());
+            if (itf == source.end())
+            {
+                result[it.key()] = it.value();
+            }
+        }
+    }
+
+    /*!
+    @brief @ref merge_diff of two objects at nesting level @a depth
+
+    Diffing two nested objects calls this function again, once per nesting
+    level, so values nested deeply enough would exhaust the call stack and
+    terminate the process. The descent is bounded here: once @ref
+    detail::recursion_depth_limit levels have been entered, @ref
+    merge_diff_iteratively diffs what is left without the call stack.
+    */
+    static basic_json merge_diff_recursively(const basic_json& source, const basic_json& target, const std::size_t depth)
+    {
+        JSON_ASSERT(source.is_object() && target.is_object());
+
+        if (JSON_HEDLEY_UNLIKELY(depth >= detail::recursion_depth_limit()))
+        {
+            return merge_diff_iteratively(source, target);
+        }
+
+        basic_json result(value_t::object);
+
+        for (auto it = source.begin(); it != source.end(); ++it)
+        {
+            auto itf = target.find(it.key());
+            if (itf != target.end())
+            {
+                if (it.value().is_object() && itf.value().is_object())
+                {
+                    auto diff = merge_diff_recursively(it.value(), itf.value(), depth + 1);
+                    if (!diff.empty())
+                    {
+                        result[it.key()] = std::move(diff);
+                    }
+                }
+                else if (it.value() != itf.value())
+                {
+                    result[it.key()] = itf.value();
+                }
+            }
+            else
+            {
+                result[it.key()] = value_t::null;
+            }
+        }
+
+        merge_diff_added_members(result, source, target);
+        return result;
+    }
+
+    /*!
+    @brief @ref merge_diff of two objects without the call stack
+
+    Produces the same patch as @ref merge_diff_recursively, keeping the objects
+    being diffed on an explicit stack instead of descending into them. Two
+    nested objects are still diffed completely before the next member, so the
+    patch has its members in the same order as in the recursive version. Only
+    reached for objects nested more deeply than @ref
+    detail::recursion_depth_limit.
+    */
+    static basic_json merge_diff_iteratively(const basic_json& source, const basic_json& target)
+    {
+        // the patch of source and target, set once their frame is popped
+        basic_json result;
+        std::vector<merge_diff_frame> stack;
+        stack.emplace_back(&source, &target);
+        while (!stack.empty())
+        {
+            // the frame's members are copied out and it is changed through
+            // stack.back(): pushing a frame for nested objects or popping
+            // this one would invalidate a reference to it
+            const basic_json* const s = stack.back().source;
+            const basic_json* const t = stack.back().target;
+            const const_iterator it = stack.back().member;
+
+            if (it != s->cend())
+            {
+                const const_iterator itf = t->find(it.key());
+                if (itf != t->cend())
+                {
+                    if (it.value().is_object() && itf.value().is_object())
+                    {
+                        // member is advanced once their patch is under its key
+                        stack.emplace_back(&it.value(), &itf.value());
+                        continue;
+                    }
+                    if (it.value() != itf.value())
+                    {
+                        stack.back().patch[it.key()] = itf.value();
+                    }
+                }
+                else
+                {
+                    stack.back().patch[it.key()] = value_t::null;
+                }
+                ++stack.back().member;
+                continue;
+            }
+
+            merge_diff_added_members(stack.back().patch, *s, *t);
+
+            // these objects are done: continue with the ones they are in
+            basic_json patch = std::move(stack.back().patch);
+            stack.pop_back();
+            if (stack.empty())
+            {
+                result = std::move(patch);
+            }
+            else
+            {
+                if (!patch.empty())
+                {
+                    stack.back().patch[stack.back().member.key()] = std::move(patch);
+                }
+                ++stack.back().member;
+            }
+        }
+        return result;
+    }
+
     /// @}
 };
 

single_include/nlohmann/json.hpp

@@ -35570,6 +35570,176 @@ public:
     }
 
   public:
+    /// @brief creates a diff as a JSON Merge Patch
+    /// @sa https://json.nlohmann.me/api/basic_json/merge_diff/
+    JSON_HEDLEY_WARN_UNUSED_RESULT
+    static basic_json merge_diff(const basic_json& source, const basic_json& target)
+    {
+        // unless both are objects, the patch is the target itself
+        if (!source.is_object() || !target.is_object())
+        {
+            return target;
+        }
+
+        return merge_diff_recursively(source, target, 0);
+    }
+
+  private:
+    /// @brief two objects @ref merge_diff_iteratively is diffing
+    struct merge_diff_frame
+    {
+        merge_diff_frame(const basic_json* source_, const basic_json* target_)
+            : source(source_), target(target_), member(source_->cbegin()), patch(value_t::object)
+        {}
+
+        // declared for GCC's -Weffc++, as in diff_frame
+        merge_diff_frame(const merge_diff_frame&) = default;
+        merge_diff_frame(merge_diff_frame&&) = default;
+        merge_diff_frame& operator=(const merge_diff_frame&) = default;
+        merge_diff_frame& operator=(merge_diff_frame&&) = default;
+        ~merge_diff_frame() = default;
+
+        /// the objects being diffed
+        const basic_json* source;
+        const basic_json* target;
+        /// the member of source being diffed; advanced once it is done
+        const_iterator member;
+        /// the patch for the members of source diffed so far
+        basic_json patch;
+    };
+
+    /// @brief add the members of object @a target that @a source does not
+    ///        have to the merge patch @a result
+    static void merge_diff_added_members(basic_json& result, const basic_json& source, const basic_json& target)
+    {
+        for (auto it = target.begin(); it != target.end(); ++it)
+        {
+            auto itf = source.find(it.key());
+            if (itf == source.end())
+            {
+                result[it.key()] = it.value();
+            }
+        }
+    }
+
+    /*!
+    @brief @ref merge_diff of two objects at nesting level @a depth
+
+    Diffing two nested objects calls this function again, once per nesting
+    level, so values nested deeply enough would exhaust the call stack and
+    terminate the process. The descent is bounded here: once @ref
+    detail::recursion_depth_limit levels have been entered, @ref
+    merge_diff_iteratively diffs what is left without the call stack.
+    */
+    static basic_json merge_diff_recursively(const basic_json& source, const basic_json& target, const std::size_t depth)
+    {
+        JSON_ASSERT(source.is_object() && target.is_object());
+
+        if (JSON_HEDLEY_UNLIKELY(depth >= detail::recursion_depth_limit()))
+        {
+            return merge_diff_iteratively(source, target);
+        }
+
+        basic_json result(value_t::object);
+
+        for (auto it = source.begin(); it != source.end(); ++it)
+        {
+            auto itf = target.find(it.key());
+            if (itf != target.end())
+            {
+                if (it.value().is_object() && itf.value().is_object())
+                {
+                    auto diff = merge_diff_recursively(it.value(), itf.value(), depth + 1);
+                    if (!diff.empty())
+                    {
+                        result[it.key()] = std::move(diff);
+                    }
+                }
+                else if (it.value() != itf.value())
+                {
+                    result[it.key()] = itf.value();
+                }
+            }
+            else
+            {
+                result[it.key()] = value_t::null;
+            }
+        }
+
+        merge_diff_added_members(result, source, target);
+        return result;
+    }
+
+    /*!
+    @brief @ref merge_diff of two objects without the call stack
+
+    Produces the same patch as @ref merge_diff_recursively, keeping the objects
+    being diffed on an explicit stack instead of descending into them. Two
+    nested objects are still diffed completely before the next member, so the
+    patch has its members in the same order as in the recursive version. Only
+    reached for objects nested more deeply than @ref
+    detail::recursion_depth_limit.
+    */
+    static basic_json merge_diff_iteratively(const basic_json& source, const basic_json& target)
+    {
+        // the patch of source and target, set once their frame is popped
+        basic_json result;
+        std::vector<merge_diff_frame> stack;
+        stack.emplace_back(&source, &target);
+        while (!stack.empty())
+        {
+            // the frame's members are copied out and it is changed through
+            // stack.back(): pushing a frame for nested objects or popping
+            // this one would invalidate a reference to it
+            const basic_json* const s = stack.back().source;
+            const basic_json* const t = stack.back().target;
+            const const_iterator it = stack.back().member;
+
+            if (it != s->cend())
+            {
+                const const_iterator itf = t->find(it.key());
+                if (itf != t->cend())
+                {
+                    if (it.value().is_object() && itf.value().is_object())
+                    {
+                        // member is advanced once their patch is under its key
+                        stack.emplace_back(&it.value(), &itf.value());
+                        continue;
+                    }
+                    if (it.value() != itf.value())
+                    {
+                        stack.back().patch[it.key()] = itf.value();
+                    }
+                }
+                else
+                {
+                    stack.back().patch[it.key()] = value_t::null;
+                }
+                ++stack.back().member;
+                continue;
+            }
+
+            merge_diff_added_members(stack.back().patch, *s, *t);
+
+            // these objects are done: continue with the ones they are in
+            basic_json patch = std::move(stack.back().patch);
+            stack.pop_back();
+            if (stack.empty())
+            {
+                result = std::move(patch);
+            }
+            else
+            {
+                if (!patch.empty())
+                {
+                    stack.back().patch[stack.back().member.key()] = std::move(patch);
+                }
+                ++stack.back().member;
+            }
+        }
+        return result;
+    }
+
     /// @}
 };
 

tests/src/unit-diagnostics.cpp

@@ -567,9 +567,9 @@ TEST_CASE("Regression tests for extended diagnostics")
     }
 }
 
-TEST_CASE("Better diagnostics past the descent bound of update() and merge_patch()")
+TEST_CASE("Better diagnostics past the descent bound of update(), merge_patch() and merge_diff()")
 {
-    // Both merge objects nested more than detail::recursion_depth_limit()
+    // All three handle objects nested more than detail::recursion_depth_limit()
     // (128) levels deep without recursing; the values they add or replace
     // there must still know their parents.
     // The values are built rather than parsed, so that the expected messages
@@ -615,4 +615,19 @@ TEST_CASE("Better diagnostics past the descent bound of update() and merge_patch
         CHECK_THROWS_WITH_AS(p->at("x").at(0), expected_x.c_str(), json::type_error);
         CHECK_THROWS_WITH_AS(p->at("y").at(0), expected_y.c_str(), json::type_error);
     }
+
+    SECTION("merge_diff()")
+    {
+        // target and patch differ only in their innermost members, so the
+        // diff of the two is nested as deeply: {"a": ... {"x": null, "y": 2}}
+        const json diff = json::merge_diff(target, patch); // NOLINT(readability-suspicious-call-argument)
+
+        const json* p = &diff;
+        for (std::size_t i = 0; i < depth; ++i)
+        {
+            p = &p->at("a");
+        }
+        CHECK(p->at("x").is_null());
+        CHECK_THROWS_WITH_AS(p->at("y").at(0), expected_y.c_str(), json::type_error);
+    }
 }