From a8323598597d6019239eb9687150450d4274c98f Mon Sep 17 00:00:00 2001 From: Niels Lohmann Date: Sun, 25 Oct 2026 17:29:56 +0100 Subject: [PATCH] docs: fix BON8 documentation gaps found in the format review Verified each claim with small C++ probes before writing it. - index.md: BON8 serialization is also incomplete for invalid UTF-8. - bon8.md: document duplicate keys (last wins), the error ids for malformed input incl. 0xF5..0xF7 + 0x80..0xBF, exact string termination wording, wide integers and discarded values in the unsupported list, and title two untitled admonitions. - exceptions.md: to_bon8 in type_error.316, BON8 in parse_error.112 (with example) and .113, wide integers in out_of_range.407. - binary_values.md: bytes 0..39 take one byte in BON8. - from_bon8.md: note on strict UTF-8 and duplicate keys. Signed-off-by: Niels Lohmann --- docs/mkdocs/docs/api/basic_json/from_bon8.md | 5 ++++ .../docs/features/binary_formats/bon8.md | 25 +++++++++++++------ .../docs/features/binary_formats/index.md | 16 ++++++------ docs/mkdocs/docs/features/binary_values.md | 2 +- docs/mkdocs/docs/home/exceptions.md | 16 +++++++++--- 5 files changed, 45 insertions(+), 19 deletions(-) diff --git a/docs/mkdocs/docs/api/basic_json/from_bon8.md b/docs/mkdocs/docs/api/basic_json/from_bon8.md index e566b8688..cd0bac558 100644 --- a/docs/mkdocs/docs/api/basic_json/from_bon8.md +++ b/docs/mkdocs/docs/api/basic_json/from_bon8.md @@ -74,6 +74,11 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va - Throws [parse_error.112](../../home/exceptions.md#jsonexceptionparse_error112) if a parse error occurs, for instance an invalid byte, a string that is not valid UTF-8, or an object key that is not a string +The UTF-8 validity of strings is always checked, independent of +[`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md), because the end of a BON8 string is determined from +its encoding. Duplicate object keys are accepted; the last value wins (see +[BON8](../../features/binary_formats/bon8.md)). + ## Complexity Linear in the size of the input. diff --git a/docs/mkdocs/docs/features/binary_formats/bon8.md b/docs/mkdocs/docs/features/binary_formats/bon8.md index b972dc035..a703e5f0f 100644 --- a/docs/mkdocs/docs/features/binary_formats/bon8.md +++ b/docs/mkdocs/docs/features/binary_formats/bon8.md @@ -1,7 +1,7 @@ # BON8 -BON8 (Binary Object Notation 8) is a compact binary serialization format for JSON values. It uses the byte values that -cannot begin a UTF-8 character as type markers, so strings are stored as plain UTF-8 without a length prefix: a string +BON8 (Binary Object Notation 8) is a compact binary serialization format for JSON values. It uses byte values that +cannot occur in valid UTF-8 as type markers, so strings are stored as plain UTF-8 without a length prefix: a string ends at the first byte that cannot continue it. Integers from -10 to 39, `true`, `false`, `null`, and the floating-point values -1.0, 0.0, and 1.0 take a single byte, and arrays and objects with up to four elements need no terminator. @@ -12,7 +12,7 @@ values -1.0, 0.0, and 1.0 take a single byte, and arrays and objects with up to ## Byte layout -BON8 exploits the fact that some byte values can never start a UTF-8 character. These bytes serve as *type markers*, so strings can be stored as plain UTF-8 without a length prefix: a string ends at the first byte that cannot continue it (often the marker of the next value). Small integers, `true`, `false`, `null`, and a few common floats are a single byte. Arrays and objects are introduced by a marker; those with up to four elements encode their size in the marker and need no terminator, while larger ones are closed by the end marker `0xFE`. +BON8 exploits the fact that some byte values can never occur in valid UTF-8. These bytes serve as *type markers*, so strings can be stored as plain UTF-8 without a length prefix: a string ends at the first byte that is not part of a valid UTF-8 character: a marker byte, or a lead byte 0xC2..0xF7 followed by a byte outside 0x80..0xBF (often the marker of the next value). Small integers, `true`, `false`, `null`, and a few common floats are a single byte. Arrays and objects are introduced by a marker; those with up to four elements encode their size in the marker and need no terminator, while larger ones are closed by the end marker `0xFE`. Consider the following JSON value (shown formatted for readability): @@ -33,7 +33,7 @@ Consider the following JSON value (shown formatted for readability):
- + @@ -100,10 +100,12 @@ serialized as 0x81 0x65 0xFF, but `[1,2,3,4,"e"]` as 0x85 0x91 0x92 0x93 0x94 0x The following values can **not** be converted to a BON8 value: - - unsigned integers above 9223372036854775807, because BON8 has no unsigned 64-bit integer type + - integers outside the int64 range (unsigned integers above 9223372036854775807 and, for number types wider than + 64 bits, any larger or smaller integer), because BON8 integers are limited to int64 ([out_of_range.407](../../home/exceptions.md#jsonexceptionout_of_range407)) - strings that are not valid UTF-8, because the end of a string is determined from its encoding ([type_error.316](../../home/exceptions.md#jsonexceptiontype_error316)) + - discarded values ([type_error.321](../../home/exceptions.md#jsonexceptiontype_error321)) !!! info "NaN/infinity handling" @@ -168,7 +170,13 @@ The library maps BON8 types to JSON value types as follows: Non-negative integers are read as number_unsigned, negative integers as number_integer. -!!! info +A byte 0xF5..0xF7 followed by 0x80..0xBF is neither a string nor an integer and is rejected with +[parse_error.112](../../home/exceptions.md#jsonexceptionparse_error112) (invalid UTF-8 byte). The same error is +reported for any other invalid byte (such as a stray 0xFE), for ill-formed UTF-8, and for an object key that is not a +string. Input that ends prematurely, trailing data after the value, and a missing final 0xFF after a string at the end +of the message are reported as [parse_error.110](../../home/exceptions.md#jsonexceptionparse_error110). + +!!! info "Non-canonical input" Values that do not use the canonical representation, such as integers with a longer encoding than necessary, arrays and objects with up to four elements that are terminated by 0xFE, unsorted object keys, or a 0xFF after a @@ -176,7 +184,10 @@ Non-negative integers are read as number_unsigned, negative integers as number_i Strings must be valid UTF-8, and a string at the very end of a message must be terminated by 0xFF. -!!! info + Duplicate object keys are accepted; the last value wins (`ordered_json` keeps the position of the first occurrence). + The specification does not say how to treat them; the HikoGUI reference decoder keeps the first value. + +!!! info "Round trips" Any BON8 output created by `to_bon8` can be successfully parsed by `from_bon8`. diff --git a/docs/mkdocs/docs/features/binary_formats/index.md b/docs/mkdocs/docs/features/binary_formats/index.md index 6e1ada110..15ff4b90b 100644 --- a/docs/mkdocs/docs/features/binary_formats/index.md +++ b/docs/mkdocs/docs/features/binary_formats/index.md @@ -16,14 +16,14 @@ to efficiently encode JSON values to byte vectors and to decode such vectors. ### Completeness -| Format | Serialization | Deserialization | -|-------------|-----------------------------------------------|-----------------------------------------------------------------------| -| BJData | complete | incomplete: no column-major ND-arrays, no Draft 4 SoA/extension types | -| BON8 | incomplete: no unsigned integers above int64 | complete | -| BSON | incomplete: top-level value must be an object | incomplete, but all JSON types are supported | -| CBOR | complete | incomplete, but all JSON types are supported | -| MessagePack | complete | incomplete, but all JSON types are supported | -| UBJSON | complete | complete | +| Format | Serialization | Deserialization | +|-------------|------------------------------------------------------------------------|-----------------------------------------------------------------------| +| BJData | complete | incomplete: no column-major ND-arrays, no Draft 4 SoA/extension types | +| BON8 | incomplete: no unsigned integers above int64, no invalid UTF-8 strings | complete | +| BSON | incomplete: top-level value must be an object | incomplete, but all JSON types are supported | +| CBOR | complete | incomplete, but all JSON types are supported | +| MessagePack | complete | incomplete, but all JSON types are supported | +| UBJSON | complete | complete | ### Binary values diff --git a/docs/mkdocs/docs/features/binary_values.md b/docs/mkdocs/docs/features/binary_values.md index f0e10247a..5be9e5ba7 100644 --- a/docs/mkdocs/docs/features/binary_values.md +++ b/docs/mkdocs/docs/features/binary_values.md @@ -216,7 +216,7 @@ array of integers. 0x87 // object with 1 member 0x62 0x69 0x6E 0x61 0x72 0x79 // "binary" 0x84 // array with 4 elements - 0xC3 0x22 0xC3 0x56 0xC3 0x12 0xC3 0x16 // content (each byte as a 2-byte integer) + 0xC3 0x22 0xC3 0x56 0xC3 0x12 0xC3 0x16 // content (each byte, here all above 39, as a 2-byte integer) ``` Note that the subtype is lost, and deserializing `v` would yield the following value: diff --git a/docs/mkdocs/docs/home/exceptions.md b/docs/mkdocs/docs/home/exceptions.md index 6af904138..7fdac8008 100644 --- a/docs/mkdocs/docs/home/exceptions.md +++ b/docs/mkdocs/docs/home/exceptions.md @@ -346,7 +346,7 @@ been read, or (with `strict = true`) further bytes follow the value. ### json.exception.parse_error.112 -An unexpected byte was read in a [binary format](../features/binary_formats/index.md) or length information is invalid ([BSON](../features/binary_formats/bson.md)). +An unexpected byte was read in a [binary format](../features/binary_formats/index.md) or length information is invalid ([BSON](../features/binary_formats/bson.md)). In [BON8](../features/binary_formats/bon8.md), this exception is also thrown for a key that is not a string and for ill-formed UTF-8. !!! failure "Example messages" @@ -371,15 +371,21 @@ An unexpected byte was read in a [binary format](../features/binary_formats/inde ``` [json.exception.parse_error.112] parse error at byte 5: syntax error while parsing BSON document: document size 6 does not match the number of bytes read (5) ``` + ``` + [json.exception.parse_error.112] parse error at byte 2: syntax error while parsing BON8 string: invalid UTF-8 byte: 0x80 + ``` ### json.exception.parse_error.113 A string could not be read from a [binary format](../features/binary_formats/index.md): either a value that is not a string was read where one was required (for instance as a map key), the string's length specification is invalid, or the string's bytes are not valid UTF-8 and the `error_handler` parameter of the corresponding `from_*` function is -set to `strict`. By default (`error_handler_t::keep`), the bytes of a string are not checked for valid UTF-8 on read; +set to `strict`. By default (`error_handler_t::keep`), the bytes of a string are not checked for valid UTF-8 on read +for the formats whose `from_*` function has an `error_handler` parameter; see the ill-formed UTF-8 notes on the individual [binary format](../features/binary_formats/index.md) pages for how such a string is handled depending on `error_handler`. +[`from_bon8()`](../api/basic_json/from_bon8.md) never throws this exception: BON8 non-string keys and ill-formed UTF-8 +are reported as [parse_error.112](#jsonexceptionparse_error112). CBOR and MessagePack allow map keys of any type, but JSON object keys are always strings. Maps with keys of any other type (for instance integers or `null`) are therefore not supported; see the notes on @@ -815,6 +821,9 @@ The binary writers [`to_cbor()`](../api/basic_json/to_cbor.md), [`to_ubjson()`]( as well for a string value or object key that is not valid UTF-8 if their `error_handler` is `strict` (the default if [`JSON_STRICT_BINARY_UTF8`](../api/macros/json_strict_binary_utf8.md) is enabled). So does [`to_msgpack()`](../api/basic_json/to_msgpack.md) if `error_handler_t::strict` is passed. +[`to_bon8()`](../api/basic_json/to_bon8.md) always throws this exception for such a string or key, regardless of +[`JSON_STRICT_BINARY_UTF8`](../api/macros/json_strict_binary_utf8.md), because the end of a BON8 string is determined +from its UTF-8 encoding. !!! failure "Example message" @@ -1009,7 +1018,8 @@ double-precision number when `number_float_t` is `#!cpp float`. An integer number cannot be represented by the binary format it is serialized to: - [BON8](../features/binary_formats/bon8.md) only stores integers that fit into int64, so - [`to_bon8`](../api/basic_json/to_bon8.md) throws for unsigned integers greater than 9223372036854775807. + [`to_bon8`](../api/basic_json/to_bon8.md) throws for unsigned integers greater than 9223372036854775807 and, with a + number type wider than 64 bits, for any integer outside the int64 range. - [CBOR](../features/binary_formats/cbor.md), [MessagePack](../features/binary_formats/messagepack.md), and [BSON](../features/binary_formats/bson.md) store integers in at most 64 bits. With the default number types, every integer fits, but a [`number_integer_t`](../api/basic_json/number_integer_t.md) or
JSONBytesMeaning
{8Aobject with 4 members
"id": 300,69 64 C4 04key: UTF-8 bytes; the string ends at the next byte, which cannot continue a UTF-8 character; value: 2-byte positive integer (lead byte, then a byte below 0x80)
"id": 300,69 64 C4 04key: UTF-8 bytes; the string ends at 0xC4, because the byte after it (0x04) is not a UTF-8 continuation byte, so 0xC4 starts an integer; value: 2-byte positive integer (lead byte, then a byte below 0x80)
"name": "json",6E 61 6D 65 FF 6A 73 6F 6E FFkey: UTF-8 bytes; 0xFF terminates the string as another string follows; value: UTF-8 bytes; terminated as a string key follows
"pi": 3.14,70 69 8F 40 09 1E B8 51 EB 85 1Fkey: UTF-8 bytes; value: binary64 (IEEE 754, big-endian); its marker also ends "pi"
"tags": [74 61 67 73 82key: UTF-8 bytes; value: array with 2 elements; its marker also ends "tags"