docs: drop discarded-value remarks

Remove the mention that a discarded value cannot be serialized to CBOR
(type_error.321) from the exception-safety and mapping sections. The
integer table and all added prose already follow the table-format and
Oxford-comma rules.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/cbor_tag_handler_t.md

@@ -15,11 +15,13 @@ error
 :   report a parse error in case of a tag (the `from_cbor` overloads throw a `parse_error` exception by default)
 
 ignore
-:   ignore tags
+:   ignore tags (also in front of a text string map key)
 
 store
 :   store tagged byte strings (for bytes 0xd8..0xdb) as binary values with the tag as subtype; other tagged values are
-    read as if the tag were ignored. If several tags precede a byte string, only the innermost one is stored.
+    read as if the tag were ignored. If several tags precede a byte string, only the tag directly in front of it is
+    stored, and only when it has the form 0xd8..0xdb (a tag 0..23 written as 0xc0..0xd7 is skipped). Tags in front of a
+    text string map key are skipped.
 
 ## Examples
 

docs/mkdocs/docs/api/basic_json/from_cbor.md

@@ -32,8 +32,10 @@ The exact mapping and its limitations are described on a [dedicated page](../../
     
     - an `std::istream` object
     - a `FILE` pointer
-    - a C-style array of characters
+    - a C-style array of characters (its terminating NUL is read as trailing data: `parse_error.110` unless `strict` is
-    - a pointer to a null-terminated string of single byte characters
+      `false`)
+    - a pointer to a null-terminated string of single byte characters (read up to the first 0x00 byte, which is a valid
+      CBOR byte)
     - a container `obj` for which `begin(obj)` and `end(obj)` produce a valid pair of iterators
       (as found via ADL or member functions, with semantics compatible to `std::begin` and `std::end`)
 
@@ -89,8 +91,14 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
 - Throws [parse_error.112](../../home/exceptions.md#jsonexceptionparse_error112) if unsupported features from CBOR were
   used in the given input or if the input is not valid CBOR
 - Throws [parse_error.113](../../home/exceptions.md#jsonexceptionparse_error113) if a map key is not a string (keys of
-  other types are not supported, as JSON object keys are always strings), or if a string value or object key is not
+  other types are not supported, as JSON object keys are always strings), if the head of a string or byte string is
-  valid UTF-8 and `error_handler` is `strict`
+  malformed (including a chunk of the wrong type or an indefinite-length string nested inside an indefinite-length
+  string), or if a string value or object key (or one chunk of an indefinite-length string) is not valid UTF-8 and
+  `error_handler` is `strict`
+- Throws [out_of_range.408](../../home/exceptions.md#jsonexceptionout_of_range408) if an array or map announces
+  a length that `std::size_t` cannot hold, or that equals `std::size_t(-1)` (for example `9B FF FF FF FF FF FF FF FF`)
+- Throws [out_of_range.406](../../home/exceptions.md#jsonexceptionout_of_range406) if a floating-point number does not
+  fit into `number_float_t`
 
 ## Complexity
 

docs/mkdocs/docs/api/basic_json/to_cbor.md

@@ -42,7 +42,8 @@ The exact mapping and its limitations are described on a [dedicated page](../../
 
 ## Exception safety
 
-Strong guarantee: if an exception is thrown, there are no changes in the JSON value.
+Strong guarantee: if an exception is thrown, there are no changes in the JSON value. Bytes that were already written to
+the output adapter are not taken back.
 
 ## Exceptions
 

docs/mkdocs/docs/features/binary_formats/cbor.md

@@ -11,7 +11,7 @@ extremely small code sizes, fairly small message size, and extensibility without
 
 ## Byte layout
 
-CBOR encodes every data item as a *head* followed by optional content. The first byte of the head holds the *major type* in its upper three bits (unsigned integer, negative integer, byte string, text string, array, map, tag, or simple/float) and an *additional information* value in its lower five bits. Additional information below 24 is the argument itself (for example, a small integer or a short length); the values 24 to 27 announce that the argument follows in 1, 2, 4, or 8 big-endian bytes. Strings, arrays, and maps are prefixed with their length (or count of elements/pairs), and content follows directly, so a reader always knows how many items to expect without a terminator.
+CBOR encodes every data item as a *head* followed by optional content. The first byte of the head holds the *major type* in its upper three bits (unsigned integer, negative integer, byte string, text string, array, map, tag, or simple/float) and an *additional information* value in its lower five bits. Additional information below 24 is the argument itself (for example, a small integer or a short length); the values 24 to 27 announce that the argument follows in 1, 2, 4, or 8 big-endian bytes. Strings, arrays, and maps are prefixed with their length (or count of elements/pairs), and content follows directly. A reader of definite-length items knows how many items to expect without a terminator; indefinite-length items (additional information 31) are instead closed by the "break" byte 0xFF, which `from_cbor` reads but `to_cbor` never writes.
 
 Consider the following JSON value (shown formatted for readability):
 
@@ -52,16 +52,16 @@ The library uses the following mapping from JSON values types to CBOR types acco
 | null            | `null`                                     | Null                              | 0xF6       |
 | boolean         | `true`                                     | True                              | 0xF5       |
 | boolean         | `false`                                    | False                             | 0xF4       |
-| number_integer  | -9223372036854775808..-2147483649          | Negative integer (8 bytes follow) | 0x3B       |
+| number_integer  | -9223372036854775808..-4294967297          | Negative integer (8 bytes follow) | 0x3B       |
-| number_integer  | -2147483648..-32769                        | Negative integer (4 bytes follow) | 0x3A       |
+| number_integer  | -4294967296..-65537                        | Negative integer (4 bytes follow) | 0x3A       |
-| number_integer  | -32768..-129                               | Negative integer (2 bytes follow) | 0x39       |
+| number_integer  | -65536..-257                               | Negative integer (2 bytes follow) | 0x39       |
-| number_integer  | -128..-25                                  | Negative integer (1 byte follow)  | 0x38       |
+| number_integer  | -256..-25                                  | Negative integer (1 byte follow)  | 0x38       |
 | number_integer  | -24..-1                                    | Negative integer                  | 0x20..0x37 |
 | number_integer  | 0..23                                      | Integer                           | 0x00..0x17 |
 | number_integer  | 24..255                                    | Unsigned integer (1 byte follow)  | 0x18       |
 | number_integer  | 256..65535                                 | Unsigned integer (2 bytes follow) | 0x19       |
 | number_integer  | 65536..4294967295                          | Unsigned integer (4 bytes follow) | 0x1A       |
-| number_integer  | 4294967296..18446744073709551615           | Unsigned integer (8 bytes follow) | 0x1B       |
+| number_integer  | 4294967296..9223372036854775807            | Unsigned integer (8 bytes follow) | 0x1B       |
 | number_unsigned | 0..23                                      | Integer                           | 0x00..0x17 |
 | number_unsigned | 24..255                                    | Unsigned integer (1 byte follow)  | 0x18       |
 | number_unsigned | 256..65535                                 | Unsigned integer (2 bytes follow) | 0x19       |
@@ -95,7 +95,24 @@ see "binary" cells in the table above.
 
 !!! success "Complete mapping"
 
-    The mapping is **complete** in the sense that any JSON value type can be converted to a CBOR value.
+    The mapping is **complete** in the sense that any JSON value type can be converted to a CBOR value, with one
+    exception: an integer outside the range of CBOR integers (only possible with a number type wider than 64 bits)
+    throws [`out_of_range.407`](../../home/exceptions.md#jsonexceptionout_of_range407).
+
+!!! info "Not a deterministic encoding"
+
+    `to_cbor` does not produce the deterministic encoding of
+    [RFC 8949, Section 4.2.1](https://www.rfc-editor.org/rfc/rfc8949.html#section-4.2.1), and not always the preferred
+    serialization of Section 4.1:
+
+    - floating-point numbers are written as single precision (0xFA) when that is exact and as double precision (0xFB)
+      otherwise; half precision (0xF9) is used only for `NaN` and `Infinity`/`-Infinity` (`1.0` is `fa 3f 80 00 00`);
+    - a binary subtype 0..23 is written as `d8 xx` instead of the one-byte tag 0xC0..0xD7 (`d8 01 41 01`);
+    - object members are written in the order of `object_t` (for the default `std::map`: sorted by the key string, not
+      by the encoded key bytes).
+
+    Integers and lengths use the shortest form, and indefinite-length items are never written. Decoders must accept all
+    of this; the RFC does not require the preferred serialization.
 
 !!! info "NaN/infinity handling"
 
@@ -105,9 +122,8 @@ see "binary" cells in the table above.
 
 !!! note
 
-    Prior to version 3.13.0, NaN and Infinity were instead serialized as a CBOR double-precision float (type 0xFB,
+    Prior to version 3.8.0, NaN and Infinity were instead serialized as a CBOR double-precision float (type 0xFB,
-    9 bytes total), because the check used to select a smaller encoding compared magnitudes with NaN, which is
+    9 bytes total).
-    always `false` and caused the intended half-precision path to be skipped.
 
 !!! info "Unused CBOR types"
 
@@ -124,7 +140,6 @@ see "binary" cells in the table above.
       - expected conversions (0xD5..0xD7)
       - simple values (0xE0..0xF3, 0xF8)
       - undefined (0xF7)
-      - half-precision floats (0xF9)
       - break (0xFF)
 
 !!! info "Tagged items"
@@ -226,10 +241,20 @@ string of the same major type, as required by [RFC 8949, Section 3.2.3](https://
     ([RFC 8392](https://www.rfc-editor.org/rfc/rfc8392.html)), cannot be read with this library and need a
     general-purpose CBOR library instead.
 
+    A tag in front of a text string key is handled like a tag in front of a value: it is skipped with
+    `cbor_tag_handler_t::ignore` or `cbor_tag_handler_t::store` and rejected with the default `error`. A tag does not
+    turn a key of another type into a text string.
+
+    A map with duplicate keys is well-formed, but invalid under
+    [RFC 8949, Section 5.3.1](https://www.rfc-editor.org/rfc/rfc8949.html#section-5.3.1). `from_cbor` accepts it and the
+    last value wins; the [SAX interface](../parsing/sax_interface.md) reports every occurrence.
+
 !!! warning "Ill-formed UTF-8 in text strings"
 
     [RFC 8949, Section 3.1](https://www.rfc-editor.org/rfc/rfc8949.html#section-3.1) requires CBOR text strings (major
-    type 3) to be valid UTF-8, but leaves it up to the decoder whether to enforce this, so checking is opt-in: with the
+    type 3) to be valid UTF-8 (a string with ill-formed UTF-8 is "well-formed but invalid"), and
+    [Section 5.3.1](https://www.rfc-editor.org/rfc/rfc8949.html#section-5.3.1) leaves it up to the decoder whether to
+    verify this, so checking is opt-in: with the
     [`error_handler`](../../api/basic_json/from_cbor.md) parameter left at `keep` (the default), `from_cbor()` accepts a
     text string (object keys included) whose bytes are not valid UTF-8 and hands them back unchanged. Passing
     `error_handler_t::strict` makes `from_cbor()` check and throw
@@ -242,9 +267,14 @@ string of the same major type, as required by [RFC 8949, Section 3.2.3](https://
     [`JSON_STRICT_BINARY_UTF8`](../../api/macros/json_strict_binary_utf8.md) is enabled), it throws the same exception
     instead. Byte strings (major type 2) are unaffected, since they are not required to hold text.
 
+    The chunks of an indefinite-length text string are checked one by one, as required by
+    [RFC 8949, Section 3.2.3](https://www.rfc-editor.org/rfc/rfc8949.html#section-3.2.3): a code point cannot be spread
+    over two chunks. With `error_handler_t::strict` such a string throws `parse_error.113`; `replace` and `ignore`
+    sanitize each chunk on its own.
+
 !!! warning "Tagged items"
 
-    Tagged items (0xC0..0xDB) will throw a parse error by default. They can be ignored by passing `cbor_tag_handler_t::ignore` to function `from_cbor`, in which case the tag is skipped and the enclosed data item is parsed on its own. Passing `cbor_tag_handler_t::store` to function `from_cbor` stores tagged byte strings (for bytes 0xd8..0xdb) as binary values with the tag as subtype; other tagged values are read as if the tag were ignored. If several tags precede a byte string, only the innermost one is stored. Note that no tag is ever interpreted: for instance, a text string tagged with tag 0 (date/time) stays a string.
+    Tagged items (0xC0..0xDB) will throw a parse error by default. They can be ignored by passing `cbor_tag_handler_t::ignore` to function `from_cbor`, in which case the tag is skipped and the enclosed data item is parsed on its own. Passing `cbor_tag_handler_t::store` to function `from_cbor` stores tagged byte strings (for bytes 0xd8..0xdb) as binary values with the tag as subtype; other tagged values are read as if the tag were ignored. If several tags precede a byte string, only the tag directly in front of it is stored, and only when it has the form 0xD8..0xDB (a tag 0..23 written as 0xC0..0xD7 is skipped); for example, `D8 05 C1 41 AA` yields subtype `null`, whereas `C1 D8 05 41 AA` yields 5. Tags in front of a text string map key are skipped as well. Note that no tag is ever interpreted: for instance, a text string tagged with tag 0 (date/time) stays a string.
 
 ??? example "Example: deserialize a JSON value from CBOR"
 

docs/mkdocs/docs/features/binary_values.md

@@ -301,6 +301,8 @@ representation using the length of the byte array.
     [`json::from_cbor`](../api/basic_json/from_cbor.md) (see
     [`cbor_tag_handler_t`](../api/basic_json/cbor_tag_handler_t.md)).
 
+    With `ignore`, the subtype is dropped (`"subtype": null`); with `store`, the tag is kept (`"subtype": 42`):
+
     ```json
     {
       "binary": {
@@ -310,6 +312,15 @@ representation using the length of the byte array.
     }
     ```
 
+    ```json
+    {
+      "binary": {
+        "bytes": [202, 254, 186, 190],
+        "subtype": 42
+      }
+    }
+    ```
+
 ### MessagePack
 
 [MessagePack](binary_formats/messagepack.md) supports binary values and subtypes. If a subtype is given, the ext family

docs/mkdocs/docs/features/types/template_parameters.md

@@ -368,8 +368,7 @@ using array_t = ArrayType<basic_json, AllocatorType<basic_json>>;
   `#!cpp std::from_chars` is not available or declines the token, and `std::strtold` reads up to the null character. A type whose `data()`
   is not null-terminated does not fail to compile; with such a `NumberFloatType` it can silently misparse
   floating-point numbers.
-- `append(const char*, size_type)`, used by [`dump`](../../api/basic_json/dump.md), and `append(const StringType&)`,
+- `append(const char*, size_type)`, used by [`dump`](../../api/basic_json/dump.md) and by the binary readers. The library's internal string concatenation additionally has
-  used by the CBOR reader for indefinite-length strings. The library's internal string concatenation additionally has
   to append a `#!cpp char` and a `#!cpp const char*`; for each it selects between `append(arg)`, `#!cpp operator+=`,
   `append(first, last)`, and `append(data, size)`.
 - The comparison operator `==` against another `StringType`, and `<` for use as a key of the chosen
@@ -678,8 +677,7 @@ using binary_t = nlohmann::byte_container_with_subtype<BinaryType>;
   [`StringType`](#stringtype), whose non-`#!cpp const` `data()` is C++17 and later only.
 - Default-constructible, copy-constructible, and move-constructible.
 - Member functions `size()`, `empty()`, `data()`, `resize()`, `operator[]`, `back()`, `begin()`, `end()`, `cbegin()`,
-  and `cend()` with random-access iterators, and `insert(pos, first, last)`, which the CBOR reader uses to join the
+  and `cend()` with random-access iterators. `push_back()` is **not** required.
-  chunks of an indefinite-length byte string. `push_back()` is **not** required.
 - Comparison operators: `==` is used by
   [`byte_container_with_subtype`](../../api/byte_container_with_subtype/index.md), the relational operators by
   [`basic_json`'s comparison operators](../../api/basic_json/operator_le.md).

include/nlohmann/detail/input/binary_reader.hpp

@@ -1218,6 +1218,10 @@ class binary_reader
         // level as required by RFC 8949, Section 3.2.3
         bool indefinite = false;
 
+        // the chunk currently being validated; only used for the chunks of an
+        // indefinite-length string when the UTF-8 check can do something
+        string_t chunk;
+
         while (true)
         {
             if (JSON_HEDLEY_UNLIKELY(!unexpect_eof("string")))
@@ -1237,26 +1241,82 @@ class binary_reader
             }
 
             // a break marker closes the indefinite-length string; outside
-            // of one it falls through to the error below
+            // of one it falls through to the error below. Every chunk has
+            // been checked on its own, so there is nothing left to check.
             if (indefinite && current == 0xFF)
             {
-                return check_string_utf8(result, context);
+                return true;
-            }
-
-            if (JSON_HEDLEY_UNLIKELY(!get_cbor_string_chunk(result, indefinite)))
-            {
-                return false;
             }
 
             if (!indefinite)
             {
-                return check_string_utf8(result, context);
+                return get_cbor_string_chunk(result, false) && check_string_utf8(result, context);
+            }
+
+            if (error_handler == error_handler_t::keep)
+            {
+                // nothing to check: read the chunk directly into the result
+                if (JSON_HEDLEY_UNLIKELY(!get_cbor_string_chunk(result, true)))
+                {
+                    return false;
+                }
+            }
+            else
+            {
+                // RFC 8949, Section 3.2.3: each chunk is a text string of its
+                // own and must be valid UTF-8 by itself, i.e., a code point
+                // cannot be spread over two chunks
+                chunk.clear();
+                if (JSON_HEDLEY_UNLIKELY(!get_cbor_string_chunk(chunk, true) || !check_string_utf8(chunk, context)))
+                {
+                    return false;
+                }
+                result.append(chunk.data(), chunk.size());
             }
 
             get();
         }
     }
 
+    /*!
+    @brief skips the tags in front of a CBOR object key
+
+    A tag may wrap any data item, so it may also wrap a key. Like in front of a
+    value, the tags are skipped for the handlers ignore and store; with the
+    handler error they are left to @ref get_cbor_object_key, which rejects the
+    key. The tag loop is iterative, so the number of tags is not limited by the
+    recursion depth.
+
+    @param[in] tag_handler  how CBOR tags should be treated
+
+    @return whether the tags could be skipped (false on EOF in a tag argument)
+
+    @pre @a current is the first byte of the key
+    */
+    bool skip_cbor_key_tags(const cbor_tag_handler_t tag_handler)
+    {
+        if (tag_handler == cbor_tag_handler_t::error)
+        {
+            return true;
+        }
+
+        // 0xC0..0xDB: tag with the tag number in the head (0xC0..0xD7) or in
+        // the 1, 2, 4, or 8 bytes that follow (0xD8..0xDB)
+        while (current >= 0xC0 && current <= 0xDB)
+        {
+            if (current >= 0xD8)
+            {
+                std::uint64_t tag_to_ignore{};
+                if (JSON_HEDLEY_UNLIKELY(!get_cbor_argument(tag_to_ignore)))
+                {
+                    return false;
+                }
+            }
+            get();
+        }
+        return true;
+    }
+
     /*!
     @brief reads a CBOR object key
 
@@ -1623,6 +1683,10 @@ class binary_reader
 
                 if (top.is_object)
                 {
+                    if (JSON_HEDLEY_UNLIKELY(!skip_cbor_key_tags(tag_handler)))
+                    {
+                        return false;
+                    }
                     if (JSON_HEDLEY_UNLIKELY(!read_object_key<&binary_reader::get_cbor_object_key>(key)))
                     {
                         return false;
@@ -4130,9 +4194,9 @@ class binary_reader
         // sequence), UBJSON, BJData, or BSON requires a decoder to reject
         // ill-formed UTF-8. Checking (and, with @ref error_handler_t::strict,
         // rejecting, or with `replace`/`ignore`, sanitizing) is opt-in via
-        // @ref error_handler, applied once the whole string (all chunks of
+        // @ref error_handler, applied by @ref check_string_utf8 at the call
-        // an indefinite-length CBOR string included) has been assembled, by
+        // site to the whole string or, for an indefinite-length CBOR string,
-        // @ref check_string_utf8 at the call site.
+        // to each chunk on its own (RFC 8949, Section 3.2.3).
         return get_bytes(len, "string", result);
     }
 

single_include/nlohmann/json.hpp

@@ -18685,6 +18685,10 @@ class binary_reader
         // level as required by RFC 8949, Section 3.2.3
         bool indefinite = false;
 
+        // the chunk currently being validated; only used for the chunks of an
+        // indefinite-length string when the UTF-8 check can do something
+        string_t chunk;
+
         while (true)
         {
             if (JSON_HEDLEY_UNLIKELY(!unexpect_eof("string")))
@@ -18704,26 +18708,82 @@ class binary_reader
             }
 
             // a break marker closes the indefinite-length string; outside
-            // of one it falls through to the error below
+            // of one it falls through to the error below. Every chunk has
+            // been checked on its own, so there is nothing left to check.
             if (indefinite && current == 0xFF)
             {
-                return check_string_utf8(result, context);
+                return true;
-            }
-
-            if (JSON_HEDLEY_UNLIKELY(!get_cbor_string_chunk(result, indefinite)))
-            {
-                return false;
             }
 
             if (!indefinite)
             {
-                return check_string_utf8(result, context);
+                return get_cbor_string_chunk(result, false) && check_string_utf8(result, context);
+            }
+
+            if (error_handler == error_handler_t::keep)
+            {
+                // nothing to check: read the chunk directly into the result
+                if (JSON_HEDLEY_UNLIKELY(!get_cbor_string_chunk(result, true)))
+                {
+                    return false;
+                }
+            }
+            else
+            {
+                // RFC 8949, Section 3.2.3: each chunk is a text string of its
+                // own and must be valid UTF-8 by itself, i.e., a code point
+                // cannot be spread over two chunks
+                chunk.clear();
+                if (JSON_HEDLEY_UNLIKELY(!get_cbor_string_chunk(chunk, true) || !check_string_utf8(chunk, context)))
+                {
+                    return false;
+                }
+                result.append(chunk.data(), chunk.size());
             }
 
             get();
         }
     }
 
+    /*!
+    @brief skips the tags in front of a CBOR object key
+
+    A tag may wrap any data item, so it may also wrap a key. Like in front of a
+    value, the tags are skipped for the handlers ignore and store; with the
+    handler error they are left to @ref get_cbor_object_key, which rejects the
+    key. The tag loop is iterative, so the number of tags is not limited by the
+    recursion depth.
+
+    @param[in] tag_handler  how CBOR tags should be treated
+
+    @return whether the tags could be skipped (false on EOF in a tag argument)
+
+    @pre @a current is the first byte of the key
+    */
+    bool skip_cbor_key_tags(const cbor_tag_handler_t tag_handler)
+    {
+        if (tag_handler == cbor_tag_handler_t::error)
+        {
+            return true;
+        }
+
+        // 0xC0..0xDB: tag with the tag number in the head (0xC0..0xD7) or in
+        // the 1, 2, 4, or 8 bytes that follow (0xD8..0xDB)
+        while (current >= 0xC0 && current <= 0xDB)
+        {
+            if (current >= 0xD8)
+            {
+                std::uint64_t tag_to_ignore{};
+                if (JSON_HEDLEY_UNLIKELY(!get_cbor_argument(tag_to_ignore)))
+                {
+                    return false;
+                }
+            }
+            get();
+        }
+        return true;
+    }
+
     /*!
     @brief reads a CBOR object key
 
@@ -19090,6 +19150,10 @@ class binary_reader
 
                 if (top.is_object)
                 {
+                    if (JSON_HEDLEY_UNLIKELY(!skip_cbor_key_tags(tag_handler)))
+                    {
+                        return false;
+                    }
                     if (JSON_HEDLEY_UNLIKELY(!read_object_key<&binary_reader::get_cbor_object_key>(key)))
                     {
                         return false;
@@ -21597,9 +21661,9 @@ class binary_reader
         // sequence), UBJSON, BJData, or BSON requires a decoder to reject
         // ill-formed UTF-8. Checking (and, with @ref error_handler_t::strict,
         // rejecting, or with `replace`/`ignore`, sanitizing) is opt-in via
-        // @ref error_handler, applied once the whole string (all chunks of
+        // @ref error_handler, applied by @ref check_string_utf8 at the call
-        // an indefinite-length CBOR string included) has been assembled, by
+        // site to the whole string or, for an indefinite-length CBOR string,
-        // @ref check_string_utf8 at the call site.
+        // to each chunk on its own (RFC 8949, Section 3.2.3).
         return get_bytes(len, "string", result);
     }
 

tests/benchmarks/json_view/README.md

@@ -21,10 +21,6 @@ python3 tests/benchmarks/json_view/compare.py --data <json_test_data directory>
 - The other libraries come from the system: pkg-config, or Homebrew (`brew install yyjson simdjson boost`). With
   `--download`, pinned releases are downloaded instead and checked against their SHA-256. Without Boost headers (or
   with `--no-boost`), the Boost.JSON columns are skipped, and the results say so.
-- `--aligned` additionally builds and runs everything with all functions aligned to 64 bytes (`-mllvm
-  -align-all-functions=6` for Clang, `-falign-functions=64 -falign-loops=64` for GCC), and reports both result
-  tables as "default build" and "aligned build". Code layout alone can move a result by a few percent; a difference
-  that appears in only one of the two tables is a layout artifact. Off by default, which doubles the run time.
 - `--corpus file...` adds files to the corpus benchmark, e.g. those of
   [simdjson-data](https://github.com/simdjson/simdjson-data) or the
   [yyjson benchmark](https://github.com/ibireme/yyjson_benchmark).
@@ -43,7 +39,7 @@ where `json_view` stands on another architecture; they are not meant for publica
 
 ## What is measured
 
-`bench_view.cpp` runs these workloads on twitter, citm_catalog, canada, jeopardy, a single tweet (`status`), and a
+`bench_view.cpp` runs four workloads on twitter, citm_catalog, canada, jeopardy, a single tweet (`status`), and a
 JSON-RPC request (`rpc`):
 
 | workload | what it does |
@@ -52,7 +48,6 @@ JSON-RPC request (`rpc`):
 | traverse | parse, then visit every value, convert every number, touch every string and key |
 | select | parse, then read a few fields per record (e.g. id, user name, and retweet count of each tweet) |
 | dump | serialize a parsed document (compact) |
-| parse+dump | build a fresh document, then serialize it (compact), in one timed call |
 
 `bench_corpus.cpp` runs parse, traverse, and dump on any list of files, so that no library is tuned to a handful of
 documents. Its dump also writes the numbers as they are in the input: `json_view` with `number_format::source`, and
@@ -71,32 +66,11 @@ call follows an untimed call of the same engine: otherwise the engine after `jso
 cleaning up the tens of thousands of nodes `json::parse` just freed (with glibc, this made `json_view` look 1.7 times
 slower on citm_catalog traverse).
 
-`json_view` is an index into the source text, not a tree of owning nodes. Its like-for-like peers are yyjson and
-simdjson DOM (immutable, indexed, no per-node allocation). Boost.JSON and `json::parse` build mutable, owning DOMs and
-are reference points for what that costs, not equals; simdjson On-Demand is lazy and forward-only, so it is the other
-way round: it has no random access to pay for. All engines validate UTF-8 and syntax by default.
-
-Number conversion differs: `json_view` converts integers while parsing but converts floating-point numbers lazily,
-when they are read or dumped; yyjson (default flags) and simdjson DOM convert every number while parsing. The cost of
-float conversion therefore appears in the traverse and dump rows for `json_view`, and in the parse row for the others.
-For float-heavy documents (canada), the fair reading is a sum of parse and use: the `traverse` workload already
-includes the parse, and `parse+dump` measures a fresh document and its dump in one timed call; the `dump` row alone
-leaves the conversion in `json_view` and out of the others. Eager conversion at parse time was prototyped as an opt-in
-and measured: canada traverse (with its parse) was 4-7 % faster, but parse+dump 11-30 % slower and parse 36-90 % slower. It was
-dropped, so lazy conversion is the only mode.
-
-`json_view` can also be edited: `json_editable_document` has the same read API, and the `(editable)` rows in
-`bench_view` measure it on the same workloads. They are the fair counterpart of Boost.JSON and `json::parse`, which are
-mutable too; the plain `json_view` rows are the counterpart of yyjson and simdjson DOM, which are not. `bench_edit` adds
-the read-modify-write workloads. Read the tables with that in mind: an immutable-only library is not compared with an
-editable `json_view` unless it says so.
-
 The engines do not all offer the same features, which the numbers should be read with:
 
 | engine | document | random access | editable | notes |
 |---|---|---|---|---|
 | `json_view` | immutable index into the text | yes | no | a fresh document per parse; "reused" parses into the same document |
-| `json_view` (editable) | index into the text plus edits | yes | yes, in place | `json_editable_document`; keeps member order and number spellings |
 | yyjson | immutable (`yyjson_read`) | yes | via a mutable copy | |
 | simdjson DOM | immutable, parser reused | yes | no | "fresh" uses a new parser per parse |
 | simdjson On-Demand | none: forward-only, lazy | no | no | only traverse and select |

tests/benchmarks/json_view/bench_view.cpp

@@ -8,8 +8,7 @@
 
 // Same-feature-set benchmark: read-only JSON documents with random access.
 //
-//   json_view      nlohmann/json_view.hpp (fresh document per parse / reused;
+//   json_view      nlohmann/json_view.hpp (fresh document per parse / reused)
-//                  "(editable)": json_editable_document, same read API)
 //   yyjson         yyjson_read(): immutable document, random access
 //   simdjson DOM   dom::parser (reused, as recommended; "fresh": a new parser
 //                  per parse): immutable, random access
@@ -20,11 +19,7 @@
 //
 // Workloads: parse (build + free), traverse (visit everything, convert every
 // number, touch every string and key), select (a few fields per document),
-// dump (compact serialization of the parsed document), and parse+dump (a fresh
+// dump (compact serialization of the parsed document).
-// document, then its dump, in one timed call). json_view converts floating-point
-// numbers lazily (when read or dumped), yyjson and simdjson DOM while parsing:
-// for float-heavy documents, traverse (which includes the parse) and parse+dump
-// are the fair reading of the parse and dump rows.
 // All engines run interleaved in every round, each timed call after an untimed
 // one of the same engine; the best round is reported.
 #include <nlohmann/json_view.hpp>
@@ -48,8 +43,6 @@
 
 using nlohmann::json;
 using nlohmann::json_document;
-using nlohmann::json_editable_document;
-using nlohmann::json_editable_view;
 using nlohmann::json_view;
 
 static volatile double g_sink;
@@ -62,8 +55,7 @@ struct stats
 
 // ---------------- traversal ----------------
 
-template<class V>
+static void walk(json_view v, stats& st)
-static void walk_view(V v, stats& st)
 {
     ++st.nodes;
     switch (v.type())
@@ -72,42 +64,32 @@ static void walk_view(V v, stats& st)
             for (auto it = v.begin(); it != v.end(); ++it)
             {
                 st.str += it.key().size();
-                walk_view<V>(*it, st);
+                walk(*it, st);
             }
             break;
         case json::value_t::array:
-            for (const V e : v)
+            for (const json_view e : v)
             {
-                walk_view<V>(e, st);
+                walk(e, st);
             }
             break;
         case json::value_t::string:
             st.str += v.get_string().size();
             break;
         case json::value_t::number_integer:
-            st.num += static_cast<double>(v.template get<std::int64_t>());
+            st.num += static_cast<double>(v.get<std::int64_t>());
             break;
         case json::value_t::number_unsigned:
-            st.num += static_cast<double>(v.template get<std::uint64_t>());
+            st.num += static_cast<double>(v.get<std::uint64_t>());
             break;
         case json::value_t::number_float:
-            st.num += v.template get<double>();
+            st.num += v.get<double>();
             break;
         default:
             break;
     }
 }
 
-static void walk(json_view v, stats& st)
-{
-    walk_view<json_view>(v, st);
-}
-
-static void walk(json_editable_view v, stats& st)
-{
-    walk_view<json_editable_view>(v, st);
-}
-
 static void walk(const json& j, stats& st)
 {
     ++st.nodes;
@@ -318,21 +300,20 @@ static void walk(const boost::json::value& v, stats& st)
 // status (one tweet): id, user.screen_name, retweet_count
 // rpc: method, params.minuend, id
 
-template<class V>
+static double pick(const std::string& name, json_view r)
-static double pick_view(const std::string& name, V r)
 {
     double acc = 0;
     if (name == "twitter" || name == "status")
     {
-        auto one = [&](V s)
+        auto one = [&](json_view s)
         {
-            acc += static_cast<double>(s["id"].template get<std::uint64_t>());
+            acc += static_cast<double>(s["id"].get<std::uint64_t>());
             acc += static_cast<double>(s["user"]["screen_name"].get_string().size());
-            acc += static_cast<double>(s["retweet_count"].template get<std::int64_t>());
+            acc += static_cast<double>(s["retweet_count"].get<std::int64_t>());
         };
         if (name == "twitter")
         {
-            for (const V s : r["statuses"])
+            for (const json_view s : r["statuses"])
             {
                 one(s);
             }
@@ -344,20 +325,20 @@ static double pick_view(const std::string& name, V r)
     }
     else if (name == "citm_catalog")
     {
-        for (const V p : r["performances"])
+        for (const json_view p : r["performances"])
         {
-            acc += static_cast<double>(p["id"].template get<std::uint64_t>() + p["eventId"].template get<std::uint64_t>() + p["seatCategories"].size());
+            acc += static_cast<double>(p["id"].get<std::uint64_t>() + p["eventId"].get<std::uint64_t>() + p["seatCategories"].size());
         }
         acc += static_cast<double>(r["events"].size());
     }
     else if (name == "canada")
     {
-        const V g = r["features"][0]["geometry"];
+        const json_view g = r["features"][0]["geometry"];
         acc += static_cast<double>(r["type"].get_string().size() + g["type"].get_string().size() + g["coordinates"].size());
     }
     else if (name == "jeopardy")
     {
-        for (const V q : r)
+        for (const json_view q : r)
         {
             acc += q["round"].get_string() == "Final Jeopardy!" ? 1 : 0;
             acc += static_cast<double>(q["category"].get_string().size());
@@ -366,21 +347,11 @@ static double pick_view(const std::string& name, V r)
     else if (name == "rpc")
     {
         acc += static_cast<double>(r["method"].get_string().size());
-        acc += static_cast<double>(r["params"]["minuend"].template get<std::int64_t>() + r["id"].template get<std::int64_t>());
+        acc += static_cast<double>(r["params"]["minuend"].get<std::int64_t>() + r["id"].get<std::int64_t>());
     }
     return acc;
 }
 
-static double pick(const std::string& name, json_view r)
-{
-    return pick_view<json_view>(name, r);
-}
-
-static double pick(const std::string& name, json_editable_view r)
-{
-    return pick_view<json_editable_view>(name, r);
-}
-
 static double pick(const std::string& name, const json& r)
 {
     double acc = 0;
@@ -663,35 +634,15 @@ int main(int argc, char** argv)
         walk(json::parse(dc.text), a);
         auto d = json_document::parse(dc.text);
         walk(d.root(), b);
-        auto ed = json_editable_document::parse(dc.text);
-        stats e;
-        walk(ed.root(), e);
         yyjson_doc* y = yyjson_read(dc.text.data(), dc.text.size(), 0);
         walk(yyjson_doc_get_root(y), c);
         simdjson::dom::parser p;
         walk(p.parse(dc.text).value(), dd);
-        const bool ok = a.nodes == b.nodes && a.nodes == e.nodes && a.str == e.str && std::fabs(a.num - e.num) <= 1e-9 * std::fabs(a.num) && a.nodes == c.nodes && a.nodes == dd.nodes && a.str == b.str && a.str == c.str && a.str == dd.str
+        const bool ok = a.nodes == b.nodes && a.nodes == c.nodes && a.nodes == dd.nodes && a.str == b.str && a.str == c.str && a.str == dd.str
                         && std::fabs(a.num - b.num) <= 1e-9 * std::fabs(a.num) && std::fabs(a.num - c.num) <= 1e-9 * std::fabs(a.num);
-        const double pa = pick(dc.name, json::parse(dc.text)), pb = pick(dc.name, d.root()), pe = pick(dc.name, ed.root()), pc = pick(dc.name, yyjson_doc_get_root(y)), pd = pick(dc.name, p.parse(dc.text).value());
+        const double pa = pick(dc.name, json::parse(dc.text)), pb = pick(dc.name, d.root()), pc = pick(dc.name, yyjson_doc_get_root(y)), pd = pick(dc.name, p.parse(dc.text).value());
-        std::printf("check %-13s traverse %s select %s\n", dc.name.c_str(), ok ? "OK" : "MISMATCH", (pa == pb && pa == pe && pa == pc && pa == pd) ? "OK" : "MISMATCH");
+        std::printf("check %-13s traverse %s select %s\n", dc.name.c_str(), ok ? "OK" : "MISMATCH", (pa == pb && pa == pc && pa == pd) ? "OK" : "MISMATCH");
         yyjson_doc_free(y);
-
-        // the dumps of the engines may format floats differently, so only: every dump is non-empty, and the
-        // dump of json_view (also editable) and of json::parse parse back to the same number of nodes
-        const std::string dj = json::parse(dc.text).dump();
-        const std::string dv = d.root().dump();
-        const std::string de = ed.root().dump();
-        std::size_t yn = 0;
-        yyjson_doc* y2 = yyjson_read(dc.text.data(), dc.text.size(), 0);
-        char* yo = yyjson_write(y2, 0, &yn);
-        const bool dumps_ok = !dj.empty() && !dv.empty() && !de.empty() && yo != nullptr && yn > 0 && !simdjson::to_string(p.parse(dc.text).value()).empty();
-        std::free(yo);
-        yyjson_doc_free(y2);
-        stats sj_nodes, sv_nodes, se_nodes;
-        walk(json::parse(dj), sj_nodes);
-        walk(json::parse(dv), sv_nodes);
-        walk(json::parse(de), se_nodes);
-        std::printf("check %-13s dump %s\n", dc.name.c_str(), (dumps_ok && sj_nodes.nodes == a.nodes && sv_nodes.nodes == a.nodes && se_nodes.nodes == a.nodes) ? "OK" : "MISMATCH");
     }
 
     std::FILE* csv = std::fopen("bench_view.csv", "w");
@@ -718,7 +669,6 @@ int main(int argc, char** argv)
         workloads.push_back({"parse", {
                 {"json_view", [&] { auto d = json_document::parse(s); g_sink = static_cast<double>(d.node_count()); }},
                 {"json_view (reused)", [&] { reused.read(s); g_sink = static_cast<double>(reused.node_count()); }},
-                {"json_view (editable)", [&] { auto d = json_editable_document::parse(s); g_sink = static_cast<double>(d.root().size()); }},
                 {"yyjson", [&] { yyjson_doc* d = yyjson_read(s.data(), s.size(), 0); g_sink = static_cast<double>(yyjson_doc_get_val_count(d)); yyjson_doc_free(d); }},
                 {"simdjson DOM", [&] { auto e = sj.parse(ps).value_unsafe(); g_sink = e.is_object(); }},
                 {"simdjson DOM (fresh)", [&] { simdjson::dom::parser p; auto e = p.parse(ps).value_unsafe(); g_sink = e.is_object(); }},
@@ -730,7 +680,6 @@ int main(int argc, char** argv)
         workloads.push_back({"traverse", {
                 {"json_view", [&] { auto d = json_document::parse(s); stats st; walk(d.root(), st); g_sink = st.num; }},
                 {"json_view (reused)", [&] { reused.read(s); stats st; walk(reused.root(), st); g_sink = st.num; }},
-                {"json_view (editable)", [&] { auto d = json_editable_document::parse(s); stats st; walk(d.root(), st); g_sink = st.num; }},
                 {"yyjson", [&] { yyjson_doc* d = yyjson_read(s.data(), s.size(), 0); stats st; walk(yyjson_doc_get_root(d), st); g_sink = st.num; yyjson_doc_free(d); }},
                 {"simdjson DOM", [&] { stats st; walk(sj.parse(ps).value_unsafe(), st); g_sink = st.num; }},
                 {"simdjson OD", [&] { auto d = od.iterate(ps).value_unsafe(); stats st; walk_od(d.get_value().value_unsafe(), st); g_sink = st.num; }},
@@ -742,31 +691,15 @@ int main(int argc, char** argv)
         workloads.push_back({"select", {
                 {"json_view", [&] { auto d = json_document::parse(s); g_sink = pick(name, d.root()); }},
                 {"json_view (reused)", [&] { reused.read(s); g_sink = pick(name, reused.root()); }},
-                {"json_view (editable)", [&] { auto d = json_editable_document::parse(s); g_sink = pick(name, d.root()); }},
                 {"yyjson", [&] { yyjson_doc* d = yyjson_read(s.data(), s.size(), 0); g_sink = pick(name, yyjson_doc_get_root(d)); yyjson_doc_free(d); }},
                 {"simdjson DOM", [&] { g_sink = pick(name, sj.parse(ps).value_unsafe()); }},
                 {"simdjson OD", [&] { auto d = od.iterate(ps).value_unsafe(); g_sink = pick_od(name, d); }},
                 {"json::parse", [&] { json j = json::parse(s); g_sink = pick(name, j); }},
             }});
-        // pipeline: a fresh document, then a dump of it, in one timed call
-        // (simdjson DOM reuses its parser, as in the parse row); the traverse
-        // workload above is already parse+traverse
-        workloads.push_back({"parse+dump", {
-                {"json_view", [&] { auto d = json_document::parse(s); std::string o = d.root().dump(); g_sink = static_cast<double>(o.size()); }},
-                {"json_view (editable)", [&] { auto d = json_editable_document::parse(s); std::string o = d.root().dump(); g_sink = static_cast<double>(o.size()); }},
-                {"yyjson", [&] { yyjson_doc* d = yyjson_read(s.data(), s.size(), 0); std::size_t n = 0; char* o = yyjson_write(d, 0, &n); g_sink = static_cast<double>(n); std::free(o); yyjson_doc_free(d); }},
-                {"simdjson DOM", [&] { std::string o = simdjson::to_string(sj.parse(ps).value_unsafe()); g_sink = static_cast<double>(o.size()); }},
-#if JSON_VIEW_BENCH_BOOST
-                {"Boost.JSON", [&] { boost::json::monotonic_resource mr; auto v = boost::json::parse(s, &mr); std::string o = boost::json::serialize(v); g_sink = static_cast<double>(o.size()); }},
-#endif
-                {"json::parse", [&] { json j = json::parse(s); std::string o = j.dump(); g_sink = static_cast<double>(o.size()); }},
-            }});
         {
             // serialization of an already parsed document
             static json_document vd;
             vd.read(s);
-            static json_editable_document ed = json_editable_document::parse(s);
-            ed = json_editable_document::parse(s);
             static yyjson_doc* yd = nullptr;
             if (yd)
             {
@@ -780,7 +713,6 @@ int main(int argc, char** argv)
             jd = json::parse(s);
             workloads.push_back({"dump", {
                     {"json_view", [&] { std::string o = vd.root().dump(); g_sink = static_cast<double>(o.size()); }},
-                    {"json_view (editable)", [&] { std::string o = ed.root().dump(); g_sink = static_cast<double>(o.size()); }},
                     {"yyjson", [&] { std::size_t n = 0; char* o = yyjson_write(yd, 0, &n); g_sink = static_cast<double>(n); std::free(o); }},
                     {"simdjson DOM", [&] { std::string o = simdjson::to_string(se); g_sink = static_cast<double>(o.size()); }},
                     {"json::parse", [&] { std::string o = jd.dump(); g_sink = static_cast<double>(o.size()); }},
@@ -808,7 +740,7 @@ int main(int argc, char** argv)
                 }
             }
             const double ref = best[0];
-            std::printf("%-13s %-14s", dc.name.c_str(), wl.first.c_str());
+            std::printf("%-13s %-9s", dc.name.c_str(), wl.first.c_str());
             for (std::size_t k = 0; k < wl.second.size(); ++k)
             {
                 const double us = best[k] / 1e3;

tests/benchmarks/json_view/compare.py

@@ -239,8 +239,6 @@ def main():
     ap.add_argument('--download', action='store_true', help='use pinned downloads instead of system libraries')
     ap.add_argument('--no-boost', action='store_true', help='skip Boost.JSON')
     ap.add_argument('--native', action='store_true', help='compile for this CPU (-march=native / -mcpu=native)')
-    ap.add_argument('--aligned', action='store_true',
-                    help='also build and run with all functions (and loops) aligned, to make code layout artifacts visible')
     ap.add_argument('--rounds', type=int, default=30, help='rounds of bench_view (default: 30)')
     ap.add_argument('--corpus', nargs='*', default=[], help='more files for bench_corpus')
     ap.add_argument('--build-dir', default=os.path.join(HERE, 'build'), help='where to build (default: build/ next to this script)')
@@ -284,44 +282,30 @@ def main():
                 + ['-I' + d for d in lib.include])
             objects.append(obj)
 
-    cxx_version = output(cxx + ['--version'])
-    is_clang = 'clang' in cxx_version.lower()
-    # alignment flags: functions and (gcc) loops on 64-byte boundaries
-    align_flags = ['-mllvm', '-align-all-functions=6'] if is_clang else ['-falign-functions=64', '-falign-loops=64']
-
-    # the builds: the default one and, with --aligned, one with aligned code
-    variants = [('default build', args.build_dir, [])]
-    if args.aligned:
-        variants.append(('aligned build', os.path.join(args.build_dir, 'aligned'), align_flags))
-
-    corpus = [os.path.join(args.data, f) for f in DEFAULT_CORPUS] + args.corpus
-    # label -> {benchmark: output}
-    all_outputs = {}
-    for label, build_dir, extra in variants:
-        os.makedirs(build_dir, exist_ok=True)
     binaries = {}
     for bench in ['bench_view', 'bench_corpus', 'bench_edit']:
-            exe = os.path.join(build_dir, bench)
+        exe = os.path.join(args.build_dir, bench)
-            run(cxx + flags + extra + include + [os.path.join(HERE, bench + '.cpp')] + objects + link + ['-o', exe])
+        run(cxx + flags + include + [os.path.join(HERE, bench + '.cpp')] + objects + link + ['-o', exe])
         binaries[bench] = exe
 
     # run: bench_view on its documents, bench_corpus on those and the given files
+    corpus = [os.path.join(args.data, f) for f in DEFAULT_CORPUS] + args.corpus
     outputs = {}
-        outputs['bench_view'] = run([binaries['bench_view'], args.data, str(args.rounds)], cwd=build_dir,
+    outputs['bench_view'] = run([binaries['bench_view'], args.data, str(args.rounds)], cwd=args.build_dir,
                                 capture_output=True, text=True).stdout
-        outputs['bench_corpus'] = run([binaries['bench_corpus']] + corpus, cwd=build_dir,
+    outputs['bench_corpus'] = run([binaries['bench_corpus']] + corpus, cwd=args.build_dir,
                                   capture_output=True, text=True).stdout
-        outputs['bench_edit'] = run([binaries['bench_edit'], args.data, str(max(1, args.rounds // 2))], cwd=build_dir,
+    outputs['bench_edit'] = run([binaries['bench_edit'], args.data, str(max(1, args.rounds // 2))], cwd=args.build_dir,
                                 capture_output=True, text=True).stdout
     for name, text in outputs.items():
         print(text)
-        all_outputs[label] = outputs
 
     # results with their metadata
     now = datetime.datetime.now()
     host = re.sub(r'[^A-Za-z0-9-]+', '-', platform.node().split('.')[0]) or 'host'
     stem = os.path.join(HERE, 'results', f'{now:%Y-%m-%d}-{host}')
     os.makedirs(os.path.dirname(stem), exist_ok=True)
+    cxx_version = output(cxx + ['--version'])
     compiler_version = cxx_version.splitlines()[0] if cxx_version else ' '.join(cxx)
     meta = [
         ('date', f'{now:%Y-%m-%d %H:%M}'),
@@ -330,7 +314,6 @@ def main():
         ('OS', f'{platform.system()} {platform.release()} ({platform.machine()})'),
         ('compiler', compiler_version),
         ('flags', ' '.join(flags)),
-        ('aligned results', ('included (flags: ' + ' '.join(align_flags) + ')') if args.aligned else 'not included'),
         ('yyjson', libs['yyjson'].version),
         ('simdjson', libs['simdjson'].version),
         ('Boost.JSON', libs['boost'].version if with_boost else 'skipped (not found)'),
@@ -343,20 +326,15 @@ def main():
         f.write('| | |\n|---|---|\n')
         for key, value in meta:
             f.write(f'| {key} | {value} |\n')
-        for label, outputs in all_outputs.items():
-            # (without --aligned there is only the default build, and no extra heading level)
-            prefix = f'{label}: ' if args.aligned else ''
         for name, text in outputs.items():
-                f.write(f'\n## {prefix}{name}\n\n```\n{text.rstrip()}\n```\n')
+            f.write(f'\n## {name}\n\n```\n{text.rstrip()}\n```\n')
     with open(stem + '.csv', 'w', encoding='utf-8') as out:
         out.write(''.join(f'# {key}: {value}\n' for key, value in meta))
-        for label, build_dir, _ in variants:
         for name in ['bench_view', 'bench_corpus', 'bench_edit']:
-                path = os.path.join(build_dir, name + '.csv')
+            path = os.path.join(args.build_dir, name + '.csv')
             if os.path.isfile(path):
                 with open(path, encoding='utf-8') as f:
-                        # (the default build keeps its plain section names)
+                    out.write(f'# {name}\n' + f.read())
-                        out.write(f'# {name}' + (f' ({label})' if args.aligned else '') + '\n' + f.read())
     print(f'results: {stem}.md, {stem}.csv')
 
 

tests/src/unit-cbor.cpp

@@ -1807,6 +1807,93 @@ TEST_CASE("CBOR")
             CHECK(json::from_cbor(std::vector<uint8_t>({0xa1, 0xff, 0x01}), true, false).is_discarded());
         }
 
+        SECTION("tag in front of a key")
+        {
+            const auto ignore = json::cbor_tag_handler_t::ignore;
+            const auto store = json::cbor_tag_handler_t::store;
+            const json expected = {{"a", 10}};
+
+            // the tags are skipped in front of a text string key, like in
+            // front of a value
+            const std::vector<std::vector<std::uint8_t>> tagged =
+            {
+                {0xA1, 0xC0, 0x61, 0x61, 0x0A},                         // tag in the head
+                {0xA1, 0xD8, 0x05, 0x61, 0x61, 0x0A},                   // 1-byte tag number
+                {0xA1, 0xD9, 0x01, 0x2C, 0x61, 0x61, 0x0A},             // 2-byte tag number
+                {0xA1, 0xDA, 0x00, 0x01, 0x00, 0x00, 0x61, 0x61, 0x0A}, // 4-byte tag number
+                {0xA1, 0xDB, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x61, 0x61, 0x0A}, // 8-byte tag number
+                {0xA1, 0xC0, 0xD8, 0x05, 0xC1, 0x61, 0x61, 0x0A},       // chained tags
+                {0xBF, 0xC0, 0x61, 0x61, 0x0A, 0xFF},                   // indefinite-length map
+                {0xBF, 0x61, 0x62, 0x01, 0xC0, 0x61, 0x61, 0x0A, 0xFF}, // second key tagged
+                {0xA1, 0xC0, 0x7F, 0x61, 0x61, 0xFF, 0x0A},             // indefinite-length text key
+                {0xA1, 0xC0, 0x61, 0x61, 0xC0, 0x0A},                   // tagged key and value
+            };
+
+            for (const auto& input : tagged)
+            {
+                CAPTURE(input)
+                json const j_ignore = json::from_cbor(input, true, true, ignore);
+                json const j_store = json::from_cbor(input, true, true, store);
+                CHECK(j_ignore.at("a") == 10);
+                CHECK(j_store.at("a") == 10);
+                CHECK(j_ignore == j_store);
+            }
+            CHECK(json::from_cbor(tagged[0], true, true, ignore) == expected);
+            CHECK(json::from_cbor(tagged[1], true, true, store) == expected);
+
+            // nested maps
+            CHECK(json::from_cbor(std::vector<std::uint8_t>({0xA1, 0xC0, 0x61, 0x61, 0xA1, 0xD8, 0x05, 0x61, 0x62, 0x01}), true, true, ignore) == json{{"a", {{"b", 1}}}});
+
+            // the tag handler error still rejects every tag
+            for (const auto& input : tagged)
+            {
+                CAPTURE(input)
+                json _;
+                CHECK_THROWS_AS(_ = json::from_cbor(input), json::parse_error&);
+            }
+            {
+                json _;
+                CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<std::uint8_t>({0xA1, 0xD8, 0x05, 0x61, 0x61, 0x0A})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR object key: only string keys are supported, but found a tag; last byte: 0xD8", json::parse_error&);
+            }
+
+            // a tag does not turn a key of another type into a text string
+            {
+                json _;
+                CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<std::uint8_t>({0xA1, 0xC0, 0x01, 0x01}), true, true, ignore), "[json.exception.parse_error.113] parse error at byte 3: syntax error while parsing CBOR object key: only string keys are supported, but found an unsigned integer; last byte: 0x01", json::parse_error&);
+                CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<std::uint8_t>({0xA1, 0xD8, 0x05, 0x41, 0x61, 0x01}), true, true, store), "[json.exception.parse_error.113] parse error at byte 4: syntax error while parsing CBOR object key: only string keys are supported, but found a byte string; last byte: 0x41", json::parse_error&);
+                CHECK(json::from_cbor(std::vector<std::uint8_t>({0xA1, 0xC0, 0x01, 0x01}), true, false, ignore).is_discarded());
+            }
+
+            // end of input after the tag, or inside the tag number
+            {
+                json _;
+                CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<std::uint8_t>({0xA1, 0xC0}), true, true, ignore), "[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing CBOR string: unexpected end of input", json::parse_error&);
+                CHECK_THROWS_AS(_ = json::from_cbor(std::vector<std::uint8_t>({0xA1, 0xD8}), true, true, ignore), json::parse_error&);
+                CHECK_THROWS_AS(_ = json::from_cbor(std::vector<std::uint8_t>({0xA1, 0xDB, 0x00, 0x00}), true, true, store), json::parse_error&);
+                CHECK_THROWS_AS(_ = json::from_cbor(std::vector<std::uint8_t>({0xBF, 0xC0}), true, true, store), json::parse_error&);
+            }
+
+            // SAX: the key event is reported without the tag (start_object,
+            // key, number_unsigned, end_object are four events)
+            {
+                const std::vector<std::uint8_t> input = {0xA1, 0xC0, 0x61, 0x61, 0x0A};
+                SaxCountdown four(4);
+                CHECK(json::sax_parse(input, &four, json::input_format_t::cbor, true, false, false, ignore));
+                SaxCountdown three(3);
+                CHECK(!json::sax_parse(input, &three, json::input_format_t::cbor, true, false, false, ignore));
+                SaxCountdown error_handler(4);
+                CHECK(!json::sax_parse(input, &error_handler, json::input_format_t::cbor));
+            }
+
+            // a long chain of tags needs no recursion
+            {
+                std::vector<std::uint8_t> input = {0xA1};
+                input.insert(input.end(), 100000, 0xC0);
+                input.insert(input.end(), {0x61, 0x61, 0x0A});
+                CHECK(json::from_cbor(input, true, true, ignore) == expected);
+            }
+        }
+
         SECTION("non-string key (see #2766 and #3381)")
         {
             // only text strings map to JSON object keys; any other key is
@@ -1915,9 +2002,10 @@ TEST_CASE("CBOR")
         {
             json _;
 
-            // the chunks are concatenated as is, without checking that each
+            // with the default error handler keep, the chunks are concatenated
-            // chunk is valid UTF-8 on its own (RFC 8949, Section 3.2.3), so
+            // as is, without checking that each chunk is valid UTF-8 on its
-            // a code point split across two chunks yields a valid string
+            // own (RFC 8949, Section 3.2.3), so a code point split across two
+            // chunks yields a valid string
             CHECK_NOTHROW(_ = json::from_cbor(std::vector<uint8_t>({0x7f, 0x61, 0xc3, 0x61, 0xa9, 0xff})));
             CHECK(_ == "\xc3\xa9");
             CHECK(_.dump() == "\"\xc3\xa9\"");
@@ -1937,6 +2025,63 @@ TEST_CASE("CBOR")
             CHECK(json::from_cbor(std::vector<uint8_t>({0x7f, 0x62, 0xc3, 0xa9, 0x62, 0xc3, 0xb6, 0xff})) == "\xc3\xa9\xc3\xb6");
         }
 
+        SECTION("ill-formed UTF-8 in a chunk of an indefinite-length string (RFC 8949, Section 3.2.3)")
+        {
+            const auto strict = json::error_handler_t::strict;
+            const auto replace = json::error_handler_t::replace;
+            const auto ignore = json::error_handler_t::ignore;
+            const auto keep = json::error_handler_t::keep;
+            const auto tags = json::cbor_tag_handler_t::error;
+
+            // a code point split over two chunks
+            const std::vector<std::uint8_t> split = {0x7f, 0x61, 0xc3, 0x61, 0xa9, 0xff};
+            // a 4-byte code point split over two chunks
+            const std::vector<std::uint8_t> split_emoji = {0x7f, 0x62, 0xf0, 0x9f, 0x62, 0x98, 0x80, 0xff};
+            // the same in an object key
+            const std::vector<std::uint8_t> split_key = {0xa1, 0x7f, 0x61, 0xc3, 0x61, 0xa9, 0xff, 0x01};
+            // a valid chunk followed by one with a truncated code point
+            const std::vector<std::uint8_t> truncated = {0x7f, 0x62, 0x41, 0xc3, 0x61, 0xa9, 0xff};
+
+            // strict: the string is invalid
+            for (const auto& input :
+                    {
+                        split, split_emoji, split_key, truncated
+                    })
+            {
+                CAPTURE(input)
+                json _;
+                CHECK_THROWS_AS(_ = json::from_cbor(input, true, true, tags, strict), json::parse_error&);
+                CHECK(json::from_cbor(input, true, false, tags, strict).is_discarded());
+            }
+            {
+                json _;
+                CHECK_THROWS_WITH_AS(_ = json::from_cbor(split, true, true, tags, strict), "[json.exception.parse_error.113] parse error at byte 3: syntax error while parsing CBOR string: invalid string: ill-formed UTF-8 byte", json::parse_error&);
+                CHECK_THROWS_WITH_AS(_ = json::from_cbor(split_key, true, true, tags, strict), "[json.exception.parse_error.113] parse error at byte 4: syntax error while parsing CBOR key: invalid string: ill-formed UTF-8 byte", json::parse_error&);
+            }
+
+            // replace and ignore: every chunk is sanitized on its own
+            CHECK(json::from_cbor(split, true, true, tags, replace) == "\xEF\xBF\xBD\xEF\xBF\xBD");
+            CHECK(json::from_cbor(split, true, true, tags, ignore) == "");
+            CHECK(json::from_cbor(truncated, true, true, tags, replace) == "A\xEF\xBF\xBD\xEF\xBF\xBD");
+            CHECK(json::from_cbor(truncated, true, true, tags, ignore) == "A");
+            CHECK(json::from_cbor(split_key, true, true, tags, replace) == json{{"\xEF\xBF\xBD\xEF\xBF\xBD", 1}});
+
+            // keep: unchanged, the chunks are concatenated as is
+            CHECK(json::from_cbor(split, true, true, tags, keep) == "\xc3\xa9");
+            CHECK(json::from_cbor(split_emoji, true, true, tags, keep) == "\xf0\x9f\x98\x80");
+
+            // conforming input is unaffected by the error handler
+            const std::vector<std::uint8_t> valid = {0x7f, 0x62, 0xc3, 0xa9, 0x64, 0xf0, 0x9f, 0x98, 0x80, 0x60, 0xff};
+            for (const auto handler :
+                    {
+                        strict, replace, ignore, keep
+                    })
+            {
+                CHECK(json::from_cbor(valid, true, true, tags, handler) == "\xc3\xa9\xf0\x9f\x98\x80");
+                CHECK(json::from_cbor(std::vector<std::uint8_t>({0x7f, 0xff}), true, true, tags, handler) == "");
+            }
+        }
+
         SECTION("many chunks in indefinite-length string")
         {
             constexpr std::size_t chunks = 100000;