Merge branch 'develop' into claude/issue-5340-restore-unget

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/parser_callback_t.md

@@ -29,7 +29,14 @@ Discarding a value (i.e., returning `#!cpp false`) has different effects dependi
 called:
 
 - Discarded values in structured types are skipped. That is, the parser will behave as if the discarded value was never
-  read.
+  read. This holds for every value type and for both kinds of parent: a discarded element is removed from the
+  surrounding array, and a discarded member is removed from the surrounding object together with its key.
+- Arrays and objects can be discarded either at their `parse_event_t::array_start`/`parse_event_t::object_start` event
+  or at their `parse_event_t::array_end`/`parse_event_t::object_end` event, and both remove the whole value. Discarding
+  it at the start event also means the callback is called neither for the content of the value nor for its matching end
+  event.
+- Discarding a `parse_event_t::key` event discards the whole object member. The callback is still called for the
+  associated value, but its return value has no further effect.
 - In case a value outside a structured type is skipped, it is replaced with `null`. This case happens if the top-level
   element is skipped.
 
@@ -49,7 +56,7 @@ called:
 ## Return value
 
 Whether the JSON value which called the function during parsing should be kept (`#!cpp true`) or not (`#!cpp false`). In
-the latter case, it is either skipped completely or replaced by an empty discarded object.
+the latter case, it is skipped completely, or replaced by `null` if it is the top-level value.
 
 ## Examples
 
@@ -68,6 +75,21 @@ the latter case, it is either skipped completely or replaced by an empty discard
     --8<-- "examples/parse__string__parser_callback_t.output"
     ```
 
+??? example
+
+    The example below shows where discarded values are removed. The array and the number are discarded in different
+    ways, but in each case the parse result contains neither the value nor its key.
+
+    ```cpp
+    --8<-- "examples/parser_callback_t.cpp"
+    ```
+
+    Output:
+
+    ```json
+    --8<-- "examples/parser_callback_t.output"
+    ```
+
 ## See also
 
 - [parse](parse.md) deserialize from a compatible input
@@ -76,3 +98,5 @@ the latter case, it is either skipped completely or replaced by an empty discard
 ## Version history
 
 - Added in version 1.0.0.
+- Fixed in version 3.13.0 to also remove discarded values from a parent object; before, discarding an array or a value
+  stored under an object key left a discarded member behind, which made the parse result serialize to invalid JSON.

docs/mkdocs/docs/api/ordered_json.md

@@ -13,6 +13,12 @@ Therefore, adding object elements can yield a reallocation in which case all ite
 [`end()`](basic_json/end.md) iterator) and all references to the elements are invalidated. Also, any iterator or
 reference after the insertion point will point to the same index, which is now a different value.
 
+## Complexity
+
+[`ordered_map`](ordered_map.md) has no lookup index: every key-based object operation is a linear scan, so building or
+parsing an object of `n` keys costs O(n²) rather than O(n log n). See
+[`ordered_map` complexity](ordered_map.md#complexity) for the per-operation table and for measured numbers.
+
 ## Examples
 
 ??? example

docs/mkdocs/docs/api/ordered_map.md

@@ -56,6 +56,48 @@ std::equal_to<>     // since C++14
 - **find**
 - **insert**
 
+## Complexity
+
+Because the elements are stored in a `std::vector` in insertion order, there is no index to look a key up by. Every
+key-based operation performs a **linear scan** over the stored elements. With `n` denoting the number of elements in the
+container:
+
+| Operation                              | Complexity     | Note                                                     |
+|----------------------------------------|----------------|----------------------------------------------------------|
+| **emplace**                            | O(n)           | scans for an existing key, then appends (amortized O(1)) |
+| **operator\[\]**                       | O(n)           | delegates to **emplace** (non-const) or **at** (const)   |
+| **at**                                 | O(n)           | throws `#!cpp std::out_of_range` if the key is not found |
+| **find**                               | O(n)           |                                                          |
+| **count**                              | O(n)           | the result is always 0 or 1                              |
+| **erase(key)**                         | O(n)           | scan, then move the remaining elements one position down |
+| **erase(pos)**, **erase(first, last)** | O(n)           | moves all elements after the erased range                |
+| **insert(value)**                      | O(n)           | equivalent to **emplace**                                |
+| **insert(first, last)**                | O((n + m) * m) | for `m` inserted elements                                |
+
+This differs from `#!cpp std::map`, where the same operations are O(log n).
+
+!!! warning "Quadratic cost of building large objects"
+
+    Because every insertion scans all elements inserted so far, building an object of `n` distinct keys costs
+    **O(n²)** in total. This applies to filling an [`ordered_json`](ordered_json.md) object key by key as well as to
+    parsing one, since the parser inserts each key as it is read.
+
+    The cost is negligible for the object sizes typically found in configuration files or API payloads, but it grows
+    steeply for machine-generated objects with many thousands of keys. Measured with `-O2 -DNDEBUG` for parsing a flat
+    object of `n` keys, relative to `#!cpp nlohmann::json` (which uses `#!cpp std::map`):
+
+    | `n`    | `json` | `ordered_json` | factor |
+    |--------|--------|----------------|--------|
+    | 2000   | 0.7 ms | 3.6 ms         | 5×     |
+    | 4000   | 0.8 ms | 14.0 ms        | 19×    |
+    | 8000   | 1.6 ms | 67.8 ms        | 43×    |
+    | 16 000 | 3.3 ms | 181.6 ms       | 54×    |
+
+    If key order matters for objects of that size, consider a container with a lookup index, such as
+    [`tsl::ordered_map`](https://github.com/Tessil/ordered-map)
+    ([integration](https://github.com/nlohmann/json/issues/546#issuecomment-304447518)), as the object type -- see
+    [object order](../features/object_order.md).
+
 ## Examples
 
 ??? example

docs/mkdocs/docs/examples/parser_callback_t.cpp

@@ -0,0 +1,47 @@
+#include <iostream>
+#include <nlohmann/json.hpp>
+
+using json = nlohmann::json;
+
+int main()
+{
+    // a JSON text with an array and a number inside an object
+    auto text = R"({"IDs": [116, 943], "Width": 800})";
+
+    // discard the array when the parser reads its opening bracket
+    json j_array_start = json::parse(text, [](int /*depth*/, json::parse_event_t event, json & /*parsed*/)
+    {
+        return event != json::parse_event_t::array_start;
+    });
+
+    // discard the same array when the parser reads its closing bracket
+    json j_array_end = json::parse(text, [](int /*depth*/, json::parse_event_t event, json & /*parsed*/)
+    {
+        return event != json::parse_event_t::array_end;
+    });
+
+    // discard the number, but keep its key
+    json j_value = json::parse(text, [](int /*depth*/, json::parse_event_t event, json & parsed)
+    {
+        return !(event == json::parse_event_t::value && parsed == json(800));
+    });
+
+    // discard the key of the number
+    json j_key = json::parse(text, [](int /*depth*/, json::parse_event_t event, json & parsed)
+    {
+        return !(event == json::parse_event_t::key && parsed == json("Width"));
+    });
+
+    // discard the top-level object
+    json j_root = json::parse(text, [](int /*depth*/, json::parse_event_t event, json & /*parsed*/)
+    {
+        return event != json::parse_event_t::object_end;
+    });
+
+    // in every case, the discarded value is removed together with its key
+    std::cout << j_array_start << '\n'
+              << j_array_end << '\n'
+              << j_value << '\n'
+              << j_key << '\n'
+              << j_root << '\n';
+}

docs/mkdocs/docs/examples/parser_callback_t.output

@@ -0,0 +1,5 @@
+{"Width":800}
+{"Width":800}
+{"IDs":[116,943]}
+{"IDs":[116,943]}
+null

docs/mkdocs/docs/features/conversions.md

@@ -66,8 +66,14 @@ auto t = j.get<std::tuple<double, std::string, int>>();  // {1.0, "hello", 42}
     std::get<1>(refs) = "world";  // modifies j[1] in place
     ```
 
-    A referenced type must be one the library actually stores (or an arithmetic type it can convert to/from);
-    otherwise this is a compile error.
+    A referenced element must name the type the library actually *stores* — one of [`boolean_t`](../api/basic_json/boolean_t.md),
+    [`number_integer_t`](../api/basic_json/number_integer_t.md), [`number_unsigned_t`](../api/basic_json/number_unsigned_t.md),
+    [`number_float_t`](../api/basic_json/number_float_t.md), [`string_t`](../api/basic_json/string_t.md),
+    [`binary_t`](../api/basic_json/binary_t.md), [`array_t`](../api/basic_json/array_t.md), or
+    [`object_t`](../api/basic_json/object_t.md). There is nothing else to refer to, so a reference to any other type is a
+    compile error even when a conversion would exist: `#!cpp std::tuple<int&>` is rejected, because the library stores a
+    `#!cpp number_integer_t` (`#!cpp std::int64_t` by default) and not an `#!cpp int`. This restriction applies only to
+    reference elements — a plain `#!cpp std::tuple<int>` converts by value as usual.
 
 ## Implicit conversions
 
@@ -116,17 +122,34 @@ which forces the explicit `get` form and can catch unintended conversions at com
     with a custom `adl_serializer<std::optional<T>>` specialization. Prefer `get<std::optional<T>>()`/`get_to()`
     over `static_cast` for optional types.
 
-!!! warning "Converting to a fixed-size `std::array` does not check length"
+!!! warning "Converting to a fixed-size destination does not check the array size"
 
-    Converting a JSON array to `#!cpp std::array<T, N>` does not check that the JSON array's size matches `N`:
-    if the JSON array is longer, the extra elements are silently dropped; if it is shorter, the remaining
-    `std::array` elements are left default-constructed. No exception is thrown in either case.
+    Some destination types have a size that is fixed by their C++ type rather than by the JSON value:
+    `#!cpp std::pair<A, B>`, `#!cpp std::tuple<Ts...>`, `#!cpp std::array<T, N>`, C arrays `#!cpp T[N]`, and
+    `#!cpp std::map`/`#!cpp std::unordered_map` with a non-string key type (which is read from an array of
+    two-element arrays). All of them read exactly as many elements as they need via
+    [`at`](../api/basic_json/at.md) and **never compare the JSON array's size to that number**. The two
+    mismatch directions therefore behave differently:
+
+    - The JSON array has **too many** elements: the surplus is **silently discarded**, and no exception is
+      thrown.
+    - The JSON array has **too few** elements: `at` throws
+      [`out_of_range.401`](../home/exceptions.md#jsonexceptionout_of_range401) for the first missing index --
+      an out-of-range error, not a [`type_error`](../home/exceptions.md#type-errors), even though the cause
+      is a shape mismatch.
 
     ```cpp
     json j = {1, 2, 3, 4, 5};
-    auto a = j.get<std::array<int, 3>>();  // {1, 2, 3} -- elements 4 and 5 silently dropped
+
+    auto a = j.get<std::array<int, 3>>();       // {1, 2, 3} -- elements 4 and 5 silently dropped
+    auto p = j.get<std::pair<int, int>>();      // (1, 2)    -- elements 3, 4, and 5 silently dropped
+
+    json k = {1};
+    auto q = k.get<std::pair<int, int>>();      // ❌ throws out_of_range.401
     ```
 
+    If a size mismatch is an error in your application, check the size yourself before converting.
+
 ## Omitting a field when serializing `std::optional`
 
 By default, `to_json` for `std::optional<T>` writes either the value or `#!json null` -- there is no built-in way

docs/mkdocs/docs/features/object_order.md

@@ -53,6 +53,12 @@ If you do want to preserve the **insertion order**, you can use the type [`nlohm
 
 Alternatively, you can use a more sophisticated ordered map like [`tsl::ordered_map`](https://github.com/Tessil/ordered-map) ([integration](https://github.com/nlohmann/json/issues/546#issuecomment-304447518)) or [`nlohmann::fifo_map`](https://github.com/nlohmann/fifo_map) ([integration](https://github.com/nlohmann/json/issues/485#issuecomment-333652309)).
 
+The [`ordered_map`](../api/ordered_map.md) behind `nlohmann::ordered_json` is deliberately minimal and has no lookup
+index, so every key access is a linear scan and building an object of `n` keys costs O(n²). This is unnoticeable at
+typical object sizes but becomes significant for objects with many thousands of keys; see
+[`ordered_map` complexity](../api/ordered_map.md#complexity). The alternatives above keep a lookup index and do not
+have this cost.
+
 ### Notes on parsing
 
 Note that you also need to call the right [`parse`](../api/basic_json/parse.md) function when reading from a file.

docs/mkdocs/docs/features/parsing/index.md

@@ -28,6 +28,22 @@ Inputs consisting of multiple values separated by newlines are handled by the [J
 By default, the library rejects comments and trailing commas. Both can be enabled with parameters of the `parse`
 function — see [comments](../comments.md) and [trailing commas](../trailing_commas.md).
 
+## Strictness and trailing data
+
+[`parse`](../../api/basic_json/parse.md) reads a single JSON value and requires the whole input to be consumed: any
+non-whitespace data after the value is reported as a parse error. Use it when you want to guarantee that an input is
+exactly one complete JSON document.
+
+[`operator>>`](../../api/operator_gtgt.md) follows relaxed `#!cpp std::istream` semantics instead: it parses one JSON
+value and leaves the stream positioned right after it, without requiring the rest of the stream to be consumed. This is
+what makes it possible to read several concatenated values from the same stream, but it also means that "a valid
+document followed by trailing bytes" is accepted rather than rejected. If you are validating conformance, or need to
+reject any input that is not exactly one JSON document, prefer `parse`.
+
+When using `operator>>` to read several concatenated values this way, a value that is a number must be followed by
+whitespace, because `operator>>` consumes the character that terminates a number — see the
+[`operator>>` notes](../../api/operator_gtgt.md#notes) for details and examples.
+
 ## SAX vs. DOM parsing
 
 The library offers two parsing models:

docs/mkdocs/docs/features/parsing/json_lines.md

@@ -49,4 +49,5 @@ JSON Lines input with more than one value is treated as invalid JSON by the [`pa
     with a JSON Lines input does not work, because the parser will try to parse one value after the last one.
 
     This is different from parsing a stream of *concatenated* (non-newline-delimited) JSON values, for which
-    `operator>>` does work -- see its [notes](../../api/operator_gtgt.md#notes) for details.
+    `operator>>` does work, provided that a value that is a number is followed by whitespace -- see its
+    [notes](../../api/operator_gtgt.md#notes) for details.

docs/mkdocs/docs/home/exceptions.md

@@ -291,9 +291,10 @@ A JSON Pointer array index must be a number.
 
 ### json.exception.parse_error.110
 
-When parsing CBOR or MessagePack, the byte vector ends before the complete value has been read.
+When parsing a [binary format](../features/binary_formats/index.md), the byte vector ends before the complete value has
+been read.
 
-!!! failure "Example message"
+!!! failure "Example messages"
 
     ```
     [json.exception.parse_error.110] parse error at byte 5: syntax error while parsing CBOR string: unexpected end of input
@@ -301,6 +302,9 @@ When parsing CBOR or MessagePack, the byte vector ends before the complete value
     ```
     [json.exception.parse_error.110] parse error at byte 2: syntax error while parsing UBJSON value: expected end of input; last byte: 0x5A
     ```
+    ```
+    [json.exception.parse_error.110] parse error at byte 8: syntax error while parsing BSON number: unexpected end of input
+    ```
 
 ### json.exception.parse_error.112
 
@@ -329,10 +333,14 @@ An unexpected byte was read in a [binary format](../features/binary_formats/inde
     ```
     [json.exception.parse_error.112] parse error at byte 9: syntax error while parsing CBOR value: negative integer overflow
     ```
+    ```
+    [json.exception.parse_error.112] parse error at byte 5: syntax error while parsing BSON document: document size 6 does not match the number of bytes read (5)
+    ```
 
 ### json.exception.parse_error.113
 
-While parsing a map key, a value that is not a string has been read.
+A string could not be read from a [binary format](../features/binary_formats/index.md): either a value that is not a
+string was read where one was required (for instance as a map key), or the string's length specification is invalid.
 
 !!! failure "Example messages"
 
@@ -345,6 +353,9 @@ While parsing a map key, a value that is not a string has been read.
     ```
     [json.exception.parse_error.113] parse error at byte 2: syntax error while parsing UBJSON char: byte after 'C' must be in range 0x00..0x7F; last byte: 0x82
     ```
+    ```
+    [json.exception.parse_error.113] parse error at byte 3: syntax error while parsing BJData string: string length must not be negative
+    ```
 
 ### json.exception.parse_error.114
 
@@ -853,13 +864,21 @@ and this exception no longer occurs.
 
 ### json.exception.out_of_range.408
 
-The size (following `#`) of an UBJSON array or object exceeds the maximal capacity.
+The size of an array or object in a [binary format](../features/binary_formats/index.md) exceeds the maximal capacity:
+the size following `#` for [UBJSON](../features/binary_formats/ubjson.md)/[BJData](../features/binary_formats/bjdata.md),
+or the encoded length for [CBOR](../features/binary_formats/cbor.md).
 
-!!! failure "Example message"
+!!! failure "Example messages"
 
     ```
     excessive array size: 8658170730974374167
     ```
+    ```
+    [json.exception.out_of_range.408] syntax error while parsing CBOR size: excessive array size
+    ```
+    ```
+    [json.exception.out_of_range.408] syntax error while parsing CBOR size: excessive map size
+    ```
 
 ### json.exception.out_of_range.409