Add BON8 support (#2998)

* Add BON8 support Add to_bon8/from_bon8 and input_format_t::bon8 for BON8, a binary format that uses the byte values that cannot begin a UTF-8 character as type markers, so strings need no length prefix. It is the most compact of the supported binary formats on the benchmark files. The reader is non-recursive like the other binary readers. A string ends at the first byte that cannot continue it, so the reader hands the one or two bytes it reads past a string back to the value that follows. The writer produces the canonical representation of the specification, except for NFC normalization; its output is identical to that of the reference implementation (HikoGUI) on all files of the test data. The round-trip tests need the .bon8 files of json_test_data 3.2.0. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review comments - Reuse detail::validate_one_utf8 to check strings in to_bon8; the error now names the first byte of the invalid sequence. - Document that to_bon8 leaves bytes in the output adapter on an exception, and that string_open is only an output of write_bon8_marker. - Explain why the pushback buffer of the BON8 reader cannot overflow. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Select the BON8 float prefix by type get_bon8_float_prefix only depends on the type of its argument, so make the type a template parameter instead of passing an unused value. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Rename a test variable that Flawfinder mistakes for read() Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the BON8 CI failures - compare the float in write_bon8_float with number_float_t constants, so GCC does not warn about a float-to-double conversion - mark check_bon8_utf8's context as used when exceptions are disabled - choose the compact float prefix in a helper rather than with nested conditional operators (clang-tidy) - use auto for the cast in the BON8 integer reader (clang-tidy) - write the int32 minimum test values as long long literals (MSVC C4146) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Amalgamate Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Read BON8 strings in bulk from contiguous input - copy the valid UTF-8 of a string in one step when the input is contiguous (twitter.json is read in 1.68 instead of 2.52 ms, jeopardy.json in 196 instead of 297 ms, close to CBOR and MessagePack) - share the new valid_utf8_prefix() with the writer's UTF-8 check, which now skips ASCII 8 bytes at a time - let the fuzzer check that contiguous and stream input give the same value or error, and test both paths in the unit tests - clarify that a second 0xFF after a string is an empty string Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Link the BON8 functions from the other binary format pages Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Name the bulk scan flag after the input, not BON8 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Read BSON keys in bulk from contiguous input BSON keys (and array indices) are C-style strings, which were read byte by byte. For contiguous input they are now read up to their \x00-byte in one step, using the same bulk_scan flag as BON8 strings: twitter.json is read in 1.46 instead of 2.01 ms, citm_catalog.json in 2.93 instead of 3.33 ms, jeopardy.json in 182 instead of 207 ms. canada.json, whose keys are almost all one-digit array indices, takes 2 % longer. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the BON8 CI failures of the bulk-read tests - skip the contiguous-versus-stream tests of BON8 strings and BSON keys when exceptions are disabled: they catch the parse errors of invalid input, and without exceptions the library aborts instead - use static_cast for the int64 test value (google-readability-casting) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Move the explicit basic_json instantiation into its own test file Linking test-regression3_cpp20 with clang and MinGW failed with "relocation truncated to fit: IMAGE_REL_AMD64_REL32 against `.rdata'", as test-regression2 did before #5511. The explicit instantiation of basic_json<> for #4825 compiles every member function, including the BON8 reader and writer, into that object, and it was already close to the limit (2,226,104 bytes on develop, 2,234,960 with BON8; clang -O1, C++20). Give the instantiation a file of its own: unit-regression3 is now 1,594,736 bytes and unit-explicit_instantiation 1,095,064. The new file mentions JSON_HAS_CPP_17 and JSON_HAS_CPP_20 so it keeps being built for the C++17 standard the regression was about. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Convert the bytes of the BON8 test strings explicitly The str() helper constructed a std::string from a byte range, which converts each unsigned char implicitly; -fsanitize=integer reports that for bytes of 0x80 and above (ci_test_clang_sanitizer). Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-03 05:00:30 +00:00 · 2026-09-27 16:56:21 +02:00
parent f682cd2ef1
commit 1e101ecac1
58 changed files with 3950 additions and 53 deletions
@@ -104,6 +104,7 @@ Linear in the size of the input.
 - [from_msgpack](from_msgpack.md) create a JSON value from an input in MessagePack format
 - [from_bson](from_bson.md) create a JSON value from an input in BSON format
 - [from_ubjson](from_ubjson.md) create a JSON value from an input in UBJSON format
+- [from_bon8](from_bon8.md) create a JSON value from an input in BON8 format

 ## Version history

@@ -0,0 +1,108 @@
+# <small>nlohmann::basic_json::</small>from_bon8
+
+```cpp
+// (1)
+template<typename InputType>
+static basic_json from_bon8(InputType&& i,
+                            const bool strict = true,
+                            const bool allow_exceptions = true);
+// (2)
+template<typename IteratorType, typename SentinelType = IteratorType>
+static basic_json from_bon8(IteratorType first, SentinelType last,
+                            const bool strict = true,
+                            const bool allow_exceptions = true);
+```
+
+Deserializes a given input to a JSON value using the BON8 (Binary Object Notation 8) serialization format.
+
+1. Reads from a compatible input.
+2. Reads from an iterator range, or an iterator and a sentinel of a different type (C++20 ranges support).
+
+The exact mapping and its limitations are described on a [dedicated page](../../features/binary_formats/bon8.md).
+
+## Template parameters
+
+`InputType`
+:   A compatible input, for instance:
+
+    - an `std::istream` object
+    - a `FILE` pointer
+    - a C-style array of characters
+    - a pointer to a null-terminated string of single byte characters
+    - a container `obj` for which `begin(obj)` and `end(obj)` produce a valid pair of iterators
+      (as found via ADL or member functions, with semantics compatible to `std::begin` and `std::end`)
+
+`IteratorType`
+:   a compatible iterator type
+
+`SentinelType`
+:   defaults to `IteratorType`; may be a different type comparable to `IteratorType` via `operator!=`, for instance.
+
+    - a custom sentinel type for C++20 ranges
+    - `std::default_sentinel_t`, when `IteratorType` is `std::counted_iterator`
+
+## Parameters
+
+`i` (in)
+:   an input in BON8 format convertible to an input adapter
+
+`first` (in)
+:   iterator to the start of the input
+
+`last` (in)
+:   iterator to the end of the input, or a sentinel value that compares equal to the end iterator with `operator!=`
+
+`strict` (in)
+:   whether to expect the input to be consumed until EOF (`#!cpp true` by default)
+
+`allow_exceptions` (in)
+:   whether to throw exceptions in case of a parse error (optional, `#!cpp true` by default)
+
+## Return value
+
+deserialized JSON value; in case of a parse error and `allow_exceptions` set to `#!cpp false`, the return value will be
+`value_t::discarded`. The latter can be checked with [`is_discarded`](is_discarded.md).
+
+## Exception safety
+
+Strong guarantee: if an exception is thrown, there are no changes in the JSON value.
+
+## Exceptions
+
+- Throws [parse_error.110](../../home/exceptions.md#jsonexceptionparse_error110) if the given input ends prematurely or
+  the end of the file was not reached when `strict` was set to true
+- Throws [parse_error.112](../../home/exceptions.md#jsonexceptionparse_error112) if a parse error occurs, for instance
+  an invalid byte, a string that is not valid UTF-8, or an object key that is not a string
+
+## Complexity
+
+Linear in the size of the input.
+
+## Examples
+
+??? example
+
+    The example shows the deserialization of a byte vector in BON8 format to a JSON value.
+     
+    ```cpp
+    --8<-- "examples/from_bon8.cpp"
+    ```
+    
+    Output:
+    
+    ```json
+    --8<-- "examples/from_bon8.output"
+    ```
+
+## See also
+
+- [to_bon8](to_bon8.md) create a BON8 serialization of a JSON value
+- [from_cbor](from_cbor.md) create a JSON value from an input in CBOR format
+- [from_msgpack](from_msgpack.md) create a JSON value from an input in MessagePack format
+- [from_bson](from_bson.md) create a JSON value from an input in BSON format
+- [from_ubjson](from_ubjson.md) create a JSON value from an input in UBJSON format
+- [from_bjdata](from_bjdata.md) create a JSON value from an input in BJData format
+
+## Version history
+
+- Added in version 3.13.0.
@@ -104,6 +104,7 @@ Linear in the size of the input.
 - [from_msgpack](from_msgpack.md) for the related MessagePack format
 - [from_ubjson](from_ubjson.md) for the related UBJSON format
 - [from_bjdata](from_bjdata.md) for the related BJData format
+- [from_bon8](from_bon8.md) for the related BON8 format

 ## Version history

@@ -110,6 +110,7 @@ Linear in the size of the input.
 - [from_bson](from_bson.md) create a JSON value from an input in BSON format
 - [from_ubjson](from_ubjson.md) create a JSON value from an input in UBJSON format
 - [from_bjdata](from_bjdata.md) create a JSON value from an input in BJData format
+- [from_bon8](from_bon8.md) create a JSON value from an input in BON8 format

 ## Version history

@@ -103,6 +103,7 @@ Linear in the size of the input.
 - [from_bson](from_bson.md) create a JSON value from an input in BSON format
 - [from_ubjson](from_ubjson.md) create a JSON value from an input in UBJSON format
 - [from_bjdata](from_bjdata.md) create a JSON value from an input in BJData format
+- [from_bon8](from_bon8.md) create a JSON value from an input in BON8 format

 ## Version history

@@ -104,6 +104,7 @@ Linear in the size of the input.
 - [from_msgpack](from_msgpack.md) create a JSON value from an input in MessagePack format
 - [from_bson](from_bson.md) create a JSON value from an input in BSON format
 - [from_bjdata](from_bjdata.md) create a JSON value from an input in BJData format
+- [from_bon8](from_bon8.md) create a JSON value from an input in BON8 format

 ## Version history

@@ -290,11 +290,13 @@ Access to the JSON value
 ### Binary formats

 - [**from_bjdata**](from_bjdata.md) (_static_) - create a JSON value from an input in BJData format
+- [**from_bon8**](from_bon8.md) (_static_) - create a JSON value from an input in BON8 format
 - [**from_bson**](from_bson.md) (_static_) - create a JSON value from an input in BSON format
 - [**from_cbor**](from_cbor.md) (_static_) - create a JSON value from an input in CBOR format
 - [**from_msgpack**](from_msgpack.md) (_static_) - create a JSON value from an input in MessagePack format
 - [**from_ubjson**](from_ubjson.md) (_static_) - create a JSON value from an input in UBJSON format
 - [**to_bjdata**](to_bjdata.md) (_static_) - create a BJData serialization of a given JSON value
+- [**to_bon8**](to_bon8.md) (_static_) - create a BON8 serialization of a given JSON value
 - [**to_bson**](to_bson.md) (_static_) - create a BSON serialization of a given JSON value
 - [**to_cbor**](to_cbor.md) (_static_) - create a CBOR serialization of a given JSON value
 - [**to_msgpack**](to_msgpack.md) (_static_) - create a MessagePack serialization of a given JSON value
@@ -7,7 +7,8 @@ enum class input_format_t {
    msgpack,
    ubjson,
    bson,
-    bjdata
+    bjdata,
+    bon8
 };
 ```

@@ -31,6 +32,9 @@ bson
 bjdata
 :   BJData (Binary JData)

+bon8
+:   BON8 (Binary Object Notation 8)
+
 ## Examples

 ??? example
@@ -5,7 +5,7 @@ class parse_error : public exception;
 ```

 The library throws this exception when a parse error occurs. Parse errors can occur during the deserialization of
-JSON text, BSON, CBOR, MessagePack, UBJSON, as well as when using JSON Patch.
+JSON text, BJData, BON8, BSON, CBOR, MessagePack, UBJSON, as well as when using JSON Patch.

 Member `byte` holds the byte index of the last read character in the input file (see note below).

@@ -65,7 +65,8 @@ The SAX event lister must follow the interface of [`json_sax`](../json_sax/index
 :   SAX event listener (must not be null)

 `format` (in)
-:    the format to parse (JSON, CBOR, MessagePack, or UBJSON) (optional, `input_format_t::json` by default), see
+:    the format to parse (JSON, BJData, BON8, BSON, CBOR, MessagePack, or UBJSON) (optional, `input_format_t::json` by
+     default), see
     [`input_format_t`](input_format_t.md) for more information

 `strict` (in)
@@ -84,6 +84,7 @@ Linear in the size of the JSON value `j`.
 - [to_msgpack](to_msgpack.md) create a MessagePack serialization of a JSON value
 - [to_bson](to_bson.md) create a BSON serialization of a JSON value
 - [to_ubjson](to_ubjson.md) create a UBJSON serialization of a JSON value
+- [to_bon8](to_bon8.md) create a BON8 serialization of a JSON value

 ## Version history

@@ -0,0 +1,76 @@
+# <small>nlohmann::basic_json::</small>to_bon8
+
+```cpp
+// (1)
+static std::vector<std::uint8_t> to_bon8(const basic_json& j);
+
+// (2)
+static void to_bon8(const basic_json& j, detail::output_adapter<std::uint8_t> o);
+static void to_bon8(const basic_json& j, detail::output_adapter<char> o);
+```
+
+Serializes a given JSON value `j` to a byte vector using the BON8 (Binary Object Notation 8) serialization format. BON8
+is a compact binary serialization format that stores strings as UTF-8 without a length prefix.
+
+1. Returns a byte vector containing the BON8 serialization.
+2. Writes the BON8 serialization to an output adapter.
+
+The exact mapping and its limitations are described on a [dedicated page](../../features/binary_formats/bon8.md).
+
+## Parameters
+
+`j` (in)
+:   JSON value to serialize
+
+`o` (in)
+:   output adapter to write serialization to
+
+## Return value
+
+1. BON8 serialization as a byte vector
+2. (none)
+
+## Exception safety
+
+Strong guarantee: if an exception is thrown, there are no changes in the JSON value `j`, which is never modified.
+With (2), the bytes written before the exception remain in the output adapter.
+
+## Exceptions
+
+- Throws [out_of_range.407](../../home/exceptions.md#jsonexceptionout_of_range407) if `j` contains an unsigned integer
+  above 9223372036854775807, which BON8 cannot represent
+- Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if `j` contains a string that is not
+  valid UTF-8
+
+## Complexity
+
+Linear in the size of the JSON value `j`.
+
+## Examples
+
+??? example
+
+    The example shows the serialization of a JSON value to a byte vector in BON8 format.
+     
+    ```cpp
+    --8<-- "examples/to_bon8.cpp"
+    ```
+    
+    Output:
+    
+    ```json
+    --8<-- "examples/to_bon8.output"
+    ```
+
+## See also
+
+- [from_bon8](from_bon8.md) create a JSON value from an input in BON8 format
+- [to_cbor](to_cbor.md) create a CBOR serialization of a JSON value
+- [to_msgpack](to_msgpack.md) create a MessagePack serialization of a JSON value
+- [to_bson](to_bson.md) create a BSON serialization of a JSON value
+- [to_ubjson](to_ubjson.md) create a UBJSON serialization of a JSON value
+- [to_bjdata](to_bjdata.md) create a BJData serialization of a JSON value
+
+## Version history
+
+- Added in version 3.13.0.
@@ -72,6 +72,7 @@ pass before anything is written.
 - [to_msgpack](to_msgpack.md) create a MessagePack serialization of a JSON value
 - [to_ubjson](to_ubjson.md) create a UBJSON serialization of a JSON value
 - [to_bjdata](to_bjdata.md) create a BJData serialization of a JSON value
+- [to_bon8](to_bon8.md) create a BON8 serialization of a JSON value

 ## Version history

@@ -62,6 +62,7 @@ Linear in the size of the JSON value `j`.
 - [to_bson](to_bson.md) create a BSON serialization of a JSON value
 - [to_ubjson](to_ubjson.md) create a UBJSON serialization of a JSON value
 - [to_bjdata](to_bjdata.md) create a BJData serialization of a JSON value
+- [to_bon8](to_bon8.md) create a BON8 serialization of a JSON value

 ## Version history

@@ -70,6 +70,7 @@ Linear in the size of the JSON value `j`.
 - [to_bson](to_bson.md) create a BSON serialization of a JSON value
 - [to_ubjson](to_ubjson.md) create a UBJSON serialization of a JSON value
 - [to_bjdata](to_bjdata.md) create a BJData serialization of a JSON value
+- [to_bon8](to_bon8.md) create a BON8 serialization of a JSON value

 ## Version history

@@ -77,6 +77,7 @@ Linear in the size of the JSON value `j`.
 - [to_msgpack](to_msgpack.md) create a MessagePack serialization of a JSON value
 - [to_bson](to_bson.md) create a BSON serialization of a JSON value
 - [to_bjdata](to_bjdata.md) create a BJData serialization of a JSON value
+- [to_bon8](to_bon8.md) create a BON8 serialization of a JSON value

 ## Version history

@@ -11,9 +11,9 @@ The macro only affects the JSON text parser ([`parse`](../basic_json/parse.md),
 [`sax_parse`](../basic_json/sax_parse.md), and [`operator>>`](../operator_gtgt.md)). There are three cases where a NUL
 byte is still not rejected:

- The binary formats ([`from_bjdata`](../basic_json/from_bjdata.md), [`from_bson`](../basic_json/from_bson.md),
-  [`from_cbor`](../basic_json/from_cbor.md), [`from_msgpack`](../basic_json/from_msgpack.md),
-  [`from_ubjson`](../basic_json/from_ubjson.md)) are never affected: there, `0x00` is ordinary data.
+- The binary formats ([`from_bjdata`](../basic_json/from_bjdata.md), [`from_bon8`](../basic_json/from_bon8.md),
+  [`from_bson`](../basic_json/from_bson.md), [`from_cbor`](../basic_json/from_cbor.md),
+  [`from_msgpack`](../basic_json/from_msgpack.md), [`from_ubjson`](../basic_json/from_ubjson.md)) are never affected: there, `0x00` is ordinary data.
 - A bare `const char*` pointer has no length of its own, so its length is still determined with `strlen()`. The first
  NUL byte therefore still marks the end of the input, and nothing after it is read.
 - One trailing `'\0'` at the end of a `char` array (e.g., a string literal) is trimmed; see the warning below.