Fix BSON binary subtype 0x02 and check the declared document size earlier (#5930)

* Write and read the inner length of BSON binary subtype 0x02

The BSON specification defines the payload of the "binary (old)" subtype
0x02 as an int32 followed by that many bytes. to_bson wrote the plain
payload instead, so decoders such as pymongo and libbson-based readers
rejected the bytes, and from_bson returned the inner length as part of the
payload (documented as lenient behavior in gh-5333).

The writer now emits the outer length (payload + 4), the subtype, the inner
int32 holding the payload size, and the payload. The reader requires an
outer length of at least 4 and an inner length equal to the outer length
minus 4 (the cases the bson-corpus lists as decode errors), reports parse
error 112 otherwise, and returns only the payload. Both directions change
together so that the round trip keeps working.

The single-reservation fast path for scalar entries declines subtype 0x02
because it has no room for the inner length.

This is a deliberate exception to the opt-in rule for behavior changes:
the previous output was invalid BSON.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Check the declared BSON document size before reading its content

The BSON reader compared a document's declared size with the number of bytes
read only when it reached the terminator. The declared size never bounded
the reads, so a SAX handler saw the keys and values of a document whose size
was too small before parse error 112 was reported.

The container frame now stores the position where the document ends
according to its declared size. open_bson_document rejects a size below 5
(size prefix plus terminator) and an embedded document that does not end
before the terminator of its enclosing document; the element loop rejects an
element whose type byte leaves no room for the name terminator and the
document terminator. The terminator check stays and still catches a declared
size that is too large. No input that was accepted before is rejected now;
documents with a wrong size are reported earlier, with fewer SAX events, a
different byte position and, in three cases, a different message. The error
id (parse_error.112) is unchanged and strict=false still does not skip the
check.

The tests that asserted the old messages and positions are updated, with the
outer sizes of two inputs corrected so that they still exercise the embedded
document. New tests cover the SAX events, a size below 5, and an embedded
document that fits exactly.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Document BSON subtype 0x02, document size checks, and lenient reads

Describe the inner length of binary subtype 0x02 that to_bson now writes and
from_bson checks and removes. Add the document size checks (parse_error.112,
not affected by strict=false) to bson.md and from_bson.md, and extend the
list of lenient reads with the UUID and MD5 payload lengths (not checked),
duplicate keys (last value wins), embedded NUL bytes in strings, and the
ignored trailing bytes with strict=false. State that the members of an object
are written in iteration order.

Correct the statement that element names are never validated on read: they
are checked and sanitized like string values by the error handler. Add the
BSON message to the examples of out_of_range.415 and type_error.321, the
subtype limit of 255 to the binary values page, CBOR to the formats that use
byte_container_with_subtype, and fix a typo in the incomplete mapping box.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Regenerate single_include and apply astyle

Result of make amalgamate (including make pretty) for the preceding changes.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Address review of the BSON documentation

Describe exactly when a too small document size is detected and note that subtype 2 data written by earlier versions is not readable.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: use the Oxford comma, drop discarded-value remarks

Add the serial comma in a list added by this PR and remove the BSON
example of type_error.321 (with the type_error.317 remark) so the docs do
not mention discarded values for BSON; restore the original singular
heading of that section. No tables were changed by this PR.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix -Weffc++ in the BSON recorder test and a clang-tidy use-auto hit

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix clang-tidy readability-redundant-member-init in BSON test

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/from_bson.md

@@ -80,7 +80,8 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
 - Throws [`parse_error.110`](../../home/exceptions.md#jsonexceptionparse_error110) if the given input ends prematurely or
   the end of the input was not reached when `strict` was set to true
 - Throws [`parse_error.112`](../../home/exceptions.md#jsonexceptionparse_error112) if a parse error occurs (e.g., an
-  invalid string or byte array length)
+  invalid string or byte array length, an invalid inner length of a binary value with subtype `0x02`, or a document size
+  that does not match the content; `strict` does not affect the document size checks)
 - Throws [`parse_error.114`](../../home/exceptions.md#jsonexceptionparse_error114) if an unsupported BSON record type is
   encountered
 - Throws [`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113) if a string value or object key is

docs/mkdocs/docs/api/basic_json/to_bson.md

@@ -13,7 +13,7 @@ static void to_bson(const basic_json& j, detail::output_adapter<char> o,
 ```
 
 BSON (Binary JSON) is a binary format in which zero or more ordered key/value pairs are stored as a single entity (a
-so-called document).
+so-called document). The members of an object are written in the order in which iteration visits them.
 
 1. Returns a byte vector containing the BSON serialization.
 2. Writes the BSON serialization to an output adapter.

docs/mkdocs/docs/api/byte_container_with_subtype/index.md

@@ -6,7 +6,7 @@ class byte_container_with_subtype : public BinaryType;
 ```
 
 This type extends the template parameter `BinaryType` provided to [`basic_json`](../basic_json/index.md) with a subtype
-used by BSON and MessagePack. This type exists so that the user does not have to specify a type themselves with a
+used by BSON, CBOR, and MessagePack. This type exists so that the user does not have to specify a type themselves with a
 specific naming scheme in order to override the binary type.
 
 ## Template parameters

docs/mkdocs/docs/features/binary_formats/bson.md

@@ -66,7 +66,12 @@ The library uses the following mapping from JSON values types to BSON types:
 !!! warning "Incomplete mapping"
 
     The mapping is **incomplete**, since only JSON-objects (and things contained therein) can be serialized to BSON.
-    Also, keys may not contain U+0000, since they are serialized a zero-terminated c-strings.
+    Also, keys may not contain U+0000, since they are serialized as zero-terminated C strings.
+
+!!! info "Order of object members"
+
+    The members of an object are written in the order in which iteration visits them: sorted by key for
+    [`json`](../../api/json.md), in insertion order for [`ordered_json`](../../api/ordered_json.md).
 
 !!! warning "BSON type 0x11 interoperability"
 
@@ -81,6 +86,19 @@ The library uses the following mapping from JSON values types to BSON types:
     As a result, serializing and deserializing a JSON object containing such a value produces a different JSON object,
     even though the binary data is unchanged.
 
+!!! info "Binary subtype 0x02 (old binary)"
+
+    The BSON specification defines the payload of the "binary (old)" subtype `0x02` as an int32 followed by that many
+    bytes, so the value is preceded by two lengths: the length of the binary element and, as the first four bytes of
+    the payload, the number of bytes that follow. A binary value with subtype `2` is therefore written with the outer
+    length 4 bytes larger than its size and the size as inner length: `binary({1, 2, 3}, 2)` becomes
+    `07 00 00 00 02 03 00 00 00 01 02 03`. When reading, the inner length is checked and removed, so the value
+    contains only the bytes that follow it. `from_bson()` throws
+    [`parse_error.112`](../../home/exceptions.md#jsonexceptionparse_error112) if the outer length is less than 4 or the
+    inner length is not the outer length minus 4. Versions before 3.13.0 did not write the inner length and returned
+    it as part of the payload; BSON data written by those versions with subtype 2 is not readable any more and must be
+    written again.
+
 ??? example "Example: serialize a JSON value to BSON"
 
     ```cpp
@@ -135,12 +153,29 @@ The library maps BSON record types to JSON value types as follows:
     The BSON reader is lenient in a few areas where the BSON specification is more restrictive:
 
     - array element keys are not checked against the required decimal sequence (`0`, `1`, `2`, ...),
-    - any non-zero byte is accepted as `true` for the boolean type, and
-    - the payload for binary subtype `0x02` is returned as-is, including its inner length prefix.
+    - any non-zero byte is accepted as `true` for the boolean type,
+    - the payloads of the binary subtypes `0x03` (UUID, old), `0x04` (UUID), and `0x05` (MD5) are not checked to have
+      16 bytes,
+    - duplicate keys within a document are accepted; the last value wins (`{"a": 1, "a": 2}` is read as `{"a": 2}`),
+      while a SAX handler receives every occurrence,
+    - a `string` value may contain embedded `0x00` bytes, and
+    - with `strict` set to `false`, bytes after the end of the document are ignored.
 
     If BSON input must be validated for strict specification compliance, validate it separately before passing it to
     `from_bson()`.
 
+!!! note "Document size"
+
+    Every document starts with its size. `from_bson()` rejects a document with
+    [`parse_error.112`](../../home/exceptions.md#jsonexceptionparse_error112) if the size is less than 5 (the size prefix and the terminator), if an embedded document does not end before the
+    end of the document that contains it, if the elements do not fit into the declared size, or if the size does not
+    match the number of bytes read when the terminating `0x00` byte is reached. A size that is too small is detected
+    before an element is read whose type byte and name no longer fit; an element that starts inside the declared size
+    but extends past it is read, and reported to a SAX handler, before the mismatch is found at the terminator. A size
+    that is too large is likewise found only at the terminator. Passing `strict` as `false` skips only the check for bytes after
+    the end of the document ([`parse_error.110`](../../home/exceptions.md#jsonexceptionparse_error110)), not the
+    document size checks.
+
 !!! warning "Ill-formed UTF-8 in string values"
 
     The BSON specification requires `string` values (type `0x02`) to be valid UTF-8, but this is not required of a
@@ -154,8 +189,9 @@ The library maps BSON record types to JSON value types as follows:
     ill-formed bytes. `to_bson()`'s own `error_handler` parameter defaults to `keep`, so such a string value or element
     (key) name is written unchanged; with `strict` (the default if
     [`JSON_STRICT_BINARY_UTF8`](../../api/macros/json_strict_binary_utf8.md) is enabled), it throws the same exception
-    instead. Element (key) names are never validated on read, since they are read byte-by-byte as a C string. `binary`
-    values (type `0x05`) are unaffected, since they are not required to hold text.
+    instead. Element (key) names are treated like string values: with `keep` they are read unchanged, with `strict`,
+    `replace`, or `ignore` they are checked and handled in the same way. `binary` values (type `0x05`) are unaffected,
+    since they are not required to hold text.
 
 ??? example "Example: deserialize a JSON value from BSON"
 

docs/mkdocs/docs/features/binary_values.md

@@ -227,7 +227,10 @@ array of integers.
 ### BSON
 
 [BSON](binary_formats/bson.md) supports binary values and subtypes. If a subtype is given, it is used and added as an
-unsigned 8-bit integer. If no subtype is given, the generic binary subtype 0x00 is used.
+unsigned 8-bit integer; a subtype above 255 throws
+[`out_of_range.415`](../home/exceptions.md#jsonexceptionout_of_range415). If no subtype is given, the generic binary
+subtype 0x00 is used. The payload of the "old binary" subtype 0x02 is preceded by a second length, which the library
+writes and removes again when reading; it is not part of the payload.
 
 ??? example "Example: serialize a binary value to BSON"
 

docs/mkdocs/docs/home/exceptions.md

@@ -1162,12 +1162,16 @@ A JSON Patch `move` operation's `"from"` location is a proper prefix of its `"pa
 MessagePack's ext type and BSON's binary subtype are each stored in a single byte. This exception is thrown when serializing a
 [`byte_container_with_subtype`](../api/byte_container_with_subtype/index.md) whose subtype exceeds 255, the largest value the library can store in that byte.
 
-!!! failure "Example message"
+!!! failure "Example messages"
 
     ```
     [json.exception.out_of_range.415] subtype 70000 is too large for the MessagePack ext type (max 255)
     ```
 
+    ```
+    [json.exception.out_of_range.415] subtype 70000 is too large for the BSON binary subtype (max 255)
+    ```
+
 !!! note
 
     This exception was added in version 3.13.0. Before that, subtypes above 255 were silently truncated modulo 256 instead of raising an error.