Point users to update(…, true), JSON Pointer defaults, and built-in std::optional

A code search of client code found many hand-written versions of
functionality the library already has: recursive object merges
(update(j, true) since 3.10.5), dotted-path getters (value() and
contains() with a JSON Pointer), get_or helpers (value() throws for a
member that is present but null), and adl_serializer specializations
for std::optional (supported since 3.12.0).

- modifying_values.md: describe both modes of update(), add a
  defaults + user settings recipe, and stop recommending Merge Patch
  for recursive merges.
- merge_patch.md: note that null deletes keys, so a merge patch is not
  a general deep merge; link update().
- default_value.md: add "Nested values" (value/contains with a JSON
  Pointer, building a pointer from a dotted path with operator/=) and
  a warning that null and mistyped members are not missing.
- value.md: note that a null member is converted, not replaced by the
  default.
- json_pointer.md: link value/contains/at with JSON Pointers.
- arbitrary_types.md: note that std::optional needs no serializer.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/docset/generate_docset.py

@@ -337,14 +337,17 @@ def is_remote(url) -> bool:
 def download(url, docs) -> str:
     """Download url into assets/external and return the path relative to docs."""
     u = urllib.parse.urlparse(url if not url.startswith('//') else 'https:' + url)
+    if u.scheme.lower() not in ('http', 'https'):
+        raise ValueError(f'not an http(s) URL: {url}')
     req = urllib.request.Request(u.geturl(), headers={'User-Agent': USER_AGENT})
-    with urllib.request.urlopen(req, timeout=20) as r:
+    # (the scheme is checked above)
+    with urllib.request.urlopen(req, timeout=20) as r:  # nosec B310
         data = r.read()
         ctype = r.headers.get_content_type()
     path = urllib.parse.unquote(u.path).lstrip('/')
     ext = os.path.splitext(path)[1]
     if u.query or not ext or path.endswith('/'):
-        digest = hashlib.sha1(url.encode()).hexdigest()[:12]
+        digest = hashlib.sha1(url.encode(), usedforsecurity=False).hexdigest()[:12]
         path = os.path.join(os.path.dirname(path), digest + CONTENT_TYPE_EXT.get(ctype, ext or '.bin'))
     rel = os.path.normpath(os.path.join('assets', 'external', u.hostname, path))
     out = os.path.join(docs, rel)
@@ -385,7 +388,8 @@ def localize_images(docs) -> None:
 def load_mkdocs_yml() -> dict:
     """Load mkdocs.yml, ignoring tags like !ENV and !!python/name."""
     with open(MKDOCS_YML, encoding='utf-8') as f:
-        return yaml.load(f, Loader=Loader)
+        # (Loader is a yaml.SafeLoader)
+        return yaml.load(f, Loader=Loader)  # nosec B506
 
 
 def localize_site_urls(docs, site_url) -> None:

docs/mkdocs/docs/api/basic_json/value.md

@@ -119,6 +119,14 @@ changes to any JSON value.
 
 ## Notes
 
+!!! warning "`null` members are not missing"
+
+    The default value is used only if the key (or JSON Pointer) does not exist. A member that exists but is
+    `#!json null` is converted like any other value, so `#!cpp j.value("k", 0)` throws a
+    [`type_error.302`](../../home/exceptions.md#jsonexceptiontype_error302) if `"k"` is `#!json null`. See
+    [Access with default value](../../features/element_access/default_value.md)
+    for alternatives.
+
 !!! warning "Return type"
 
     The value function is a template, and the return type of the function is determined by the type of the provided

docs/mkdocs/docs/features/arbitrary_types.md

@@ -347,6 +347,13 @@ struct adl_serializer<boost::optional<T>> {
 NLOHMANN_JSON_NAMESPACE_END
 ```
 
+!!! tip "`std::optional` needs no serializer"
+
+    Since version 3.12.0, `std::optional<T>` is supported out of the box when compiling with C++17
+    (`std::nullopt` is converted to and from `null`). Do not write an `adl_serializer` for it; this pattern is only
+    needed for types such as `boost::optional` or for custom semantics. See [Conversions](conversions.md) and
+    [Omitting a field when serializing `std::optional`](conversions.md#omitting-a-field-when-serializing-stdoptional).
+
 !!! note "ABI compatibility"
 
     Use [`NLOHMANN_JSON_NAMESPACE_BEGIN`](../api/macros/nlohmann_json_namespace_begin.md) and `NLOHMANN_JSON_NAMESPACE_END`

docs/mkdocs/docs/features/element_access/default_value.md

@@ -30,6 +30,47 @@ you want to access and a default value in case there is no value stored with tha
     | `#!cpp j.value("append", false)`            | `#!json true`                                        |
     | `#!cpp j.value("logLevel", "verbose")`      | `#!json "verbose"`                                   |
 
+## Nested values
+
+To read a value deep inside a document, pass a [JSON Pointer](../json_pointer.md) instead of a key. The default value is
+returned if the value at the pointer does not exist, including the case that an intermediate key is missing. There is
+no need to check each level with [`contains`](../../api/basic_json/contains.md) first.
+
+```cpp
+json j = {{"server", {{"port", 8080}}}, {"list", {10, 20}}};
+
+int port = j.value("/server/port"_json_pointer, 80);        // 8080
+int timeout = j.value("/server/limits/timeout"_json_pointer, 30); // 30 (missing intermediate key)
+int second = j.value("/list/1"_json_pointer, 0);            // 20 (numeric tokens index arrays)
+int third = j.value("/list/5"_json_pointer, 0);             // 0 (index out of range)
+
+bool has_port = j.contains("/server/port"_json_pointer);    // true
+bool has_host = j.contains("/server/host/name"_json_pointer); // false
+```
+
+If the path is only available as a dotted string such as `#!cpp "server.port"`, do not build the pointer by
+concatenating `#!cpp "/"` and the parts: keys containing `/` or `~` would be misinterpreted. Append each part as a
+reference token with [`operator/=`](../../api/json_pointer/operator_slasheq.md) instead. It escapes the token for you.
+
+```cpp
+json::json_pointer to_pointer(const std::string& dotted)
+{
+    json::json_pointer ptr;
+    std::istringstream in(dotted);
+    for (std::string token; std::getline(in, token, '.');)
+    {
+        ptr /= token;
+    }
+    return ptr;
+}
+
+int port = j.value(to_pointer("server.port"), 80);          // 8080
+int first = j.value(to_pointer("list.0"), 0);               // 10
+```
+
+The key `#!cpp "a/b"` yields the pointer `#!cpp "/a~1b"`; the dot-splitting itself is up to the caller, so keys
+containing `.` need a different separator.
+
 ## Notes
 
 !!! failure "Exceptions"
@@ -37,6 +78,31 @@ you want to access and a default value in case there is no value stored with tha
     - With string keys, `value` can only be used with objects. For other types, a [`basic_json::type_error`](../../home/exceptions.md#jsonexceptiontype_error306) is thrown.
     - With JSON Pointers, `value` can be used with both objects and arrays. For other types (null, boolean, number, string), a [`basic_json::type_error`](../../home/exceptions.md#jsonexceptiontype_error306) is thrown.
 
+!!! warning "`null` and mistyped members are not missing"
+
+    `value` returns the default value only if the key is **absent**. If the member exists, it is converted to the type
+    of the default value, even if it is `#!json null`. For `#!json {"k": null}`, the call `#!cpp j.value("k", 0)` throws
+    a [`basic_json::type_error`](../../home/exceptions.md#jsonexceptiontype_error302), and so does a member of another
+    type such as a string where a number is expected. The same holds for JSON Pointers.
+
+    To treat `#!json null` like a missing value, check for it explicitly:
+
+    ```cpp
+    int n = (j.contains("k") && !j["k"].is_null()) ? j["k"].get<int>() : 0;
+    ```
+
+    With C++17, [`get<std::optional<T>>()`](../../api/basic_json/get.md) maps `#!json null` to an empty optional. As
+    [`at`](../../api/basic_json/at.md) throws [`out_of_range`](../../home/exceptions.md#jsonexceptionout_of_range403)
+    for an absent key, use [`find`](../../api/basic_json/find.md) to cover both cases:
+
+    ```cpp
+    std::optional<int> n;                        // empty if "k" is absent or null
+    if (const auto it = j.find("k"); it != j.end())
+    {
+        n = it->get<std::optional<int>>();       // still throws for a non-number such as "text"
+    }
+    ```
+
 !!! warning "Return type"
 
     The value function is a template, and the return type of the function is determined by the type of the provided
@@ -63,3 +129,6 @@ you want to access and a default value in case there is no value stored with tha
 
 - [`value`](../../api/basic_json/value.md) for access with default value
 - documentation on [checked access](checked_access.md)
+- documentation on [JSON Pointer](../json_pointer.md)
+- [`contains`](../../api/basic_json/contains.md) to check whether a key or JSON Pointer exists
+- [`json_pointer::operator/=`](../../api/json_pointer/operator_slasheq.md) to build a pointer token by token

docs/mkdocs/docs/features/json_pointer.md

@@ -77,6 +77,10 @@ auto val2 = j.at(json::json_pointer("/nested/three/1"));    // false
 auto val3 = j.value(json::json_pointer("/nested/four"), 0); // 0
 ```
 
+To read a value with a fallback, use [`value`](../api/basic_json/value.md) with a JSON Pointer; to test for existence,
+use [`contains`](../api/basic_json/contains.md). Neither needs intermediate checks, see
+[Nested values](element_access/default_value.md#nested-values).
+
 !!! note "Creating intermediate levels that don't exist"
 
     See the [`operator[]` notes](../api/basic_json/operator%5B%5D.md#return-value) for how array vs. object is
@@ -126,6 +130,8 @@ auto j_original = j_flat.unflatten();
 ## See also
 
 - Class [`json_pointer`](../api/json_pointer/index.md)
+- Functions [`value`](../api/basic_json/value.md), [`contains`](../api/basic_json/contains.md), and
+  [`at`](../api/basic_json/at.md) accept JSON Pointers; see [Nested values](element_access/default_value.md#nested-values)
 - Function [`flatten`](../api/basic_json/flatten.md)
 - Function [`unflatten`](../api/basic_json/unflatten.md)
 - [JSON Patch](json_patch.md) - paths inside a patch are JSON Pointers

docs/mkdocs/docs/features/merge_patch.md

@@ -9,6 +9,13 @@ syntax that closely mimics the document being modified. Unlike [JSON Patch](json
 express every kind of change (e.g., it cannot reorder array elements or remove a specific array element), but it is
 easier to read and write for object-shaped documents.
 
+!!! tip "Not a general deep merge"
+
+    A merge patch is not a general deep merge: a `#!json null` value in the patch deletes the key from the target.
+    To merge two objects recursively (e.g., defaults and user settings), use
+    [`update`](../api/basic_json/update.md) with `merge_objects` set to `#!cpp true`; see
+    [Merging objects](modifying_values.md#merging-objects).
+
 ??? example
 
     The following code shows how a JSON Merge Patch is applied to a JSON document.
@@ -28,3 +35,4 @@ easier to read and write for object-shaped documents.
 - [JSON Patch and Diff](json_patch.md) - a more expressive alternative that describes a sequence of operations
 - [JSON Pointer](json_pointer.md) - the addressing scheme used by JSON Patch
 - Function [`merge_patch`](../api/basic_json/merge_patch.md)
+- Function [`update`](../api/basic_json/update.md) - merge objects, optionally recursively

docs/mkdocs/docs/features/modifying_values.md

@@ -35,8 +35,12 @@ the insertion happened — useful for "add if absent" semantics.
 
 ## Merging objects
 
-To merge one object into another, [`update`](../api/basic_json/update.md) copies all members from another object,
-overwriting existing keys (similar to Python's `dict.update`). This is the idiomatic way to combine two objects.
+To merge one object into another, [`update`](../api/basic_json/update.md) copies all members from another object
+(similar to Python's `dict.update`). This is the idiomatic way to combine two objects. It has two modes:
+
+- By default, the merge is shallow: existing keys are overwritten, even if both values are objects.
+- With `merge_objects = #!cpp true`, keys whose values are objects in both JSON values are merged recursively.
+  Everything else is overwritten. In particular, arrays are replaced, not concatenated.
 
 ??? example
 
@@ -50,9 +54,22 @@ overwriting existing keys (similar to Python's `dict.update`). This is the idiom
     --8<-- "examples/update.output"
     ```
 
-For a recursive merge that follows [RFC 7386](https://tools.ietf.org/html/rfc7386), see
-[JSON Merge Patch](merge_patch.md). To apply a sequence of well-defined edit operations, see
-[JSON Patch](json_patch.md).
+A common use of the recursive mode is combining defaults with user settings. Nested defaults that the user did not set
+are kept:
+
+```cpp
+json defaults = {{"log", {{"level", "info"}, {"file", "app.log"}}}, {"retries", 3}};
+json user_settings = {{"log", {{"level", "debug"}}}};
+
+json config = defaults;
+config.update(user_settings, true);
+// {"log":{"file":"app.log","level":"debug"},"retries":3}
+```
+
+[JSON Merge Patch](merge_patch.md) ([RFC 7386](https://tools.ietf.org/html/rfc7386)) also merges objects recursively,
+but it is a different tool: a `#!json null` in the patch means "remove this key". It is meant for applying merge patch
+documents (e.g., received via HTTP PATCH). To merge configuration-like objects, use `#!cpp update(..., true)`. To apply
+a sequence of well-defined edit operations, see [JSON Patch](json_patch.md).
 
 ## Removing elements
 
@@ -72,6 +89,7 @@ a.erase(1);             // [1,3,4]  (erase by index)
 
 - [`push_back`](../api/basic_json/push_back.md) / [`emplace_back`](../api/basic_json/emplace_back.md) - append to an array
 - [`emplace`](../api/basic_json/emplace.md) - insert into an object if the key is absent
-- [`update`](../api/basic_json/update.md) - merge objects
+- [`update`](../api/basic_json/update.md) - merge objects (shallow, or recursive with `merge_objects`)
+- [`merge_patch`](../api/basic_json/merge_patch.md) - apply an RFC 7386 merge patch
 - [`erase`](../api/basic_json/erase.md) / [`clear`](../api/basic_json/clear.md) - remove elements
 - [JSON Patch and Diff](json_patch.md) and [JSON Merge Patch](merge_patch.md) - structured modifications