Recover from parse errors when parse_error() returns true (#3989)

The return value of json_sax::parse_error() was documented both as "must
return false" and as "whether the parsing should continue", and the code
just passed it on. For JSON text, parsing stopped anyway, but sax_parse()
could report success for invalid input. The binary readers read on after
the error, looping forever on a CBOR indefinite-length array without its
end.

Now false stops parsing, and true recovers from the error:

- JSON text is repaired with the smallest local edit (insert a missing
  ',' or ':', remove a stray token, keep the readable part of a broken
  string or number, null for a value that cannot be read, close the
  innermost container at a wrong closing bracket and all of them at the
  end of the input), and parsing continues. The SAX events stay balanced,
  every key is followed by exactly one value, and each token is reported
  at most once.
- The binary formats cannot resynchronize, so they stop, but complete
  the value read so far.

sax_parse() returns false after any error. parse(), accept(), and the
from_*() functions never recover and compile to the same code as before.

Supersedes #4522.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

.github/labeler.yml

@@ -37,13 +37,13 @@ labels:
   files:
     - "include/nlohmann/detail/input/binary_reader\\.hpp"
     - "include/nlohmann/detail/output/binary_writer\\.hpp"
-    - "tests/src/unit-(bson|cbor|msgpack|ubjson|bjdata|binary_formats)"
-    - "tests/src/fuzzer-parse_(bson|cbor|msgpack|ubjson|bjdata)"
+    - "tests/src/unit-(bson|cbor|msgpack|ubjson|bjdata|bon8|binary_formats)"
+    - "tests/src/fuzzer-parse_(bson|cbor|msgpack|ubjson|bjdata|bon8)"
     - "docs/mkdocs/docs/features/binary_formats/"
-    - "docs/mkdocs/docs/(api/basic_json|examples)/(to|from)_(bson|cbor|msgpack|ubjson|bjdata)"
+    - "docs/mkdocs/docs/(api/basic_json|examples)/(to|from)_(bson|cbor|msgpack|ubjson|bjdata|bon8)"
 
 - label: "aspect: binary formats"
-  title: "(?i)(bson|cbor|msgpack|messagepack|ubjson|bjdata|binary format)"
+  title: "(?i)(bson|cbor|msgpack|messagepack|ubjson|bjdata|bon8|binary format)"
 
 - label: "python"
   files:

Makefile

@@ -36,6 +36,7 @@ all:
 	@echo "clean - remove built files"
 	@echo "doctest - compile example files and check their output"
 	@echo "fuzz_testing - prepare fuzz testing of the JSON parser"
+	@echo "fuzz_testing_bon8 - prepare fuzz testing of the BON8 parser"
 	@echo "fuzz_testing_bson - prepare fuzz testing of the BSON parser"
 	@echo "fuzz_testing_cbor - prepare fuzz testing of the CBOR parser"
 	@echo "fuzz_testing_msgpack - prepare fuzz testing of the MessagePack parser"
@@ -71,6 +72,14 @@ fuzz_testing:
 	find tests/data/json_tests -size -5k -name *json | xargs -I{} cp "{}" fuzz-testing/testcases
 	@echo "Execute: afl-fuzz -i fuzz-testing/testcases -o fuzz-testing/out fuzz-testing/fuzzer"
 
+fuzz_testing_bon8:
+	rm -fr fuzz-testing
+	mkdir -p fuzz-testing fuzz-testing/testcases fuzz-testing/out
+	$(MAKE) parse_bon8_fuzzer -C tests CXX=afl-clang++
+	mv tests/parse_bon8_fuzzer fuzz-testing/fuzzer
+	find tests/data -size -5k -name *.bon8 | xargs -I{} cp "{}" fuzz-testing/testcases
+	@echo "Execute: afl-fuzz -i fuzz-testing/testcases -o fuzz-testing/out fuzz-testing/fuzzer"
+
 fuzz_testing_bson:
 	rm -fr fuzz-testing
 	mkdir -p fuzz-testing fuzz-testing/testcases fuzz-testing/out

README.md

@@ -40,7 +40,7 @@
   - [Implicit conversions](#implicit-conversions)
   - [Conversions to/from arbitrary types](#arbitrary-types-conversions)
   - [Specializing enum conversion](#specializing-enum-conversion)
-  - [Binary formats (BSON, CBOR, MessagePack, UBJSON, and BJData)](#binary-formats-bson-cbor-messagepack-ubjson-and-bjdata)
+  - [Binary formats (BSON, CBOR, MessagePack, UBJSON, BJData, and BON8)](#binary-formats-bson-cbor-messagepack-ubjson-bjdata-and-bon8)
 - [Customers](#customers)
 - [Ecosystem](#ecosystem)
 - [Supported compilers](#supported-compilers)
@@ -128,7 +128,7 @@ There is also a [**docset**](https://github.com/Kapeli/Dash-User-Contributions/t
 - **JSON Pointer functions**: [flatten](https://json.nlohmann.me/api/basic_json/flatten), [unflatten](https://json.nlohmann.me/api/basic_json/unflatten)
 - **JSON Patch functions**: [patch](https://json.nlohmann.me/api/basic_json/patch), [patch_inplace](https://json.nlohmann.me/api/basic_json/patch_inplace), [diff](https://json.nlohmann.me/api/basic_json/diff), [merge_patch](https://json.nlohmann.me/api/basic_json/merge_patch)
 - **Static functions**: [meta](https://json.nlohmann.me/api/basic_json/meta), [get_allocator](https://json.nlohmann.me/api/basic_json/get_allocator)
-- **Binary formats**: [from_bjdata](https://json.nlohmann.me/api/basic_json/from_bjdata), [from_bson](https://json.nlohmann.me/api/basic_json/from_bson), [from_cbor](https://json.nlohmann.me/api/basic_json/from_cbor), [from_msgpack](https://json.nlohmann.me/api/basic_json/from_msgpack), [from_ubjson](https://json.nlohmann.me/api/basic_json/from_ubjson), [to_bjdata](https://json.nlohmann.me/api/basic_json/to_bjdata), [to_bson](https://json.nlohmann.me/api/basic_json/to_bson), [to_cbor](https://json.nlohmann.me/api/basic_json/to_cbor), [to_msgpack](https://json.nlohmann.me/api/basic_json/to_msgpack), [to_ubjson](https://json.nlohmann.me/api/basic_json/to_ubjson)
+- **Binary formats**: [from_bjdata](https://json.nlohmann.me/api/basic_json/from_bjdata), [from_bon8](https://json.nlohmann.me/api/basic_json/from_bon8), [from_bson](https://json.nlohmann.me/api/basic_json/from_bson), [from_cbor](https://json.nlohmann.me/api/basic_json/from_cbor), [from_msgpack](https://json.nlohmann.me/api/basic_json/from_msgpack), [from_ubjson](https://json.nlohmann.me/api/basic_json/from_ubjson), [to_bjdata](https://json.nlohmann.me/api/basic_json/to_bjdata), [to_bon8](https://json.nlohmann.me/api/basic_json/to_bon8), [to_bson](https://json.nlohmann.me/api/basic_json/to_bson), [to_cbor](https://json.nlohmann.me/api/basic_json/to_cbor), [to_msgpack](https://json.nlohmann.me/api/basic_json/to_msgpack), [to_ubjson](https://json.nlohmann.me/api/basic_json/to_ubjson)
 - **Non-member functions**: [operator<<](https://json.nlohmann.me/api/operator_ltlt/), [operator>>](https://json.nlohmann.me/api/operator_gtgt/), [to_string](https://json.nlohmann.me/api/basic_json/to_string)
 - **Literals**: [operator""_json](https://json.nlohmann.me/api/operator_literal_json)
 - **Helper classes**: [std::hash&lt;basic_json&gt;](https://json.nlohmann.me/api/basic_json/std_hash), [std::swap&lt;basic_json&gt;](https://json.nlohmann.me/api/basic_json/std_swap)
@@ -496,7 +496,7 @@ bool key(string_t& val);
 bool parse_error(std::size_t position, const std::string& last_token, const detail::exception& ex);
 ```
 
-The return value of each function determines whether parsing should proceed.
+The return value of each function determines whether parsing should proceed. For `parse_error`, returning `true` [recovers from the error](https://json.nlohmann.me/features/parsing/error_recovery/): the parser repairs the input and continues.
 
 To implement your own SAX handler, proceed as follows:
 
@@ -504,7 +504,7 @@ To implement your own SAX handler, proceed as follows:
 2. Create an object of your SAX interface class, e.g. `my_sax`.
 3. Call `bool json::sax_parse(input, &my_sax)`; where the first parameter can be any input like a string or an input stream and the second parameter is a pointer to your SAX interface.
 
-Note the `sax_parse` function only returns a `bool` indicating the result of the last executed SAX event. It does not return a  `json` value - it is up to you to decide what to do with the SAX events. Furthermore, no exceptions are thrown in case of a parse error -- it is up to you what to do with the exception object passed to your `parse_error` implementation. Internally, the SAX interface is used for the DOM parser (class `json_sax_dom_parser`) as well as the acceptor (`json_sax_acceptor`), see file [`json_sax.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/json_sax.hpp).
+Note the `sax_parse` function only returns a `bool` indicating whether the input was parsed without errors and no SAX event returned `false`. It does not return a  `json` value - it is up to you to decide what to do with the SAX events. Furthermore, no exceptions are thrown in case of a parse error -- it is up to you what to do with the exception object passed to your `parse_error` implementation. Internally, the SAX interface is used for the DOM parser (class `json_sax_dom_parser`) as well as the acceptor (`json_sax_acceptor`), see file [`json_sax.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/json_sax.hpp).
 
 ### STL-like access
 
@@ -1110,9 +1110,9 @@ Other Important points:
 - When using `get<ENUM_TYPE>()`, undefined JSON values will default to the first pair specified in your map. Select this default pair carefully. If you desire an exception in this circumstance use `NLOHMANN_JSON_SERIALIZE_ENUM_STRICT()` which behaves identically except for throwing an exception on unrecognized values.
 - If an enum or JSON value is specified more than once in your map, the first matching occurrence from the top of the map will be returned when converting to or from JSON.
 
-### Binary formats (BSON, CBOR, MessagePack, UBJSON, and BJData)
+### Binary formats (BSON, CBOR, MessagePack, UBJSON, BJData, and BON8)
 
-Though JSON is a ubiquitous data format, it is not a very compact format suitable for data exchange, for instance over a network. Hence, the library supports [BSON](https://bsonspec.org) (Binary JSON), [CBOR](https://cbor.io) (Concise Binary Object Representation), [MessagePack](https://msgpack.org), [UBJSON](https://ubjson.org) (Universal Binary JSON Specification) and [BJData](https://neurojson.org/bjdata) (Binary JData) to efficiently encode JSON values to byte vectors and to decode such vectors.
+Though JSON is a ubiquitous data format, it is not a very compact format suitable for data exchange, for instance over a network. Hence, the library supports [BSON](https://bsonspec.org) (Binary JSON), [CBOR](https://cbor.io) (Concise Binary Object Representation), [MessagePack](https://msgpack.org), [UBJSON](https://ubjson.org) (Universal Binary JSON Specification), [BJData](https://neurojson.org/bjdata) (Binary JData), and [BON8](https://github.com/hikoworks/hikogui/blob/main/docs/BON8.md) (Binary Object Notation 8) to efficiently encode JSON values to byte vectors and to decode such vectors.
 
 ```cpp
 // create a JSON value
@@ -1149,6 +1149,14 @@ std::vector<std::uint8_t> v_ubjson = json::to_ubjson(j);
 
 // roundtrip
 json j_from_ubjson = json::from_ubjson(v_ubjson);
+
+// serialize to BON8
+std::vector<std::uint8_t> v_bon8 = json::to_bon8(j);
+
+// 0x88, 0x63, 0x6F, 0x6D, 0x70, 0x61, 0x63, 0x74, 0xF9, 0x73, 0x63, 0x68, 0x65, 0x6D, 0x61, 0x90
+
+// roundtrip
+json j_from_bon8 = json::from_bon8(v_bon8);
 ```
 
 The library also supports binary types from BSON, CBOR (byte strings), and MessagePack (bin, ext, fixext). They are stored by default as `std::vector<std::uint8_t>` to be processed outside the library.

cmake/ci.cmake

@@ -542,7 +542,7 @@ add_custom_target(ci_infer
 
 add_custom_target(ci_offline_testdata
     COMMAND mkdir -p ${PROJECT_BINARY_DIR}/build_offline_testdata/test_data
-    COMMAND cd ${PROJECT_BINARY_DIR}/build_offline_testdata/test_data && ${GIT_TOOL} clone -c advice.detachedHead=false --branch v3.1.0 https://github.com/nlohmann/json_test_data.git --quiet --depth 1
+    COMMAND cd ${PROJECT_BINARY_DIR}/build_offline_testdata/test_data && ${GIT_TOOL} clone -c advice.detachedHead=false --branch v3.2.0 https://github.com/nlohmann/json_test_data.git --quiet --depth 1
     COMMAND ${CMAKE_COMMAND}
         -DCMAKE_BUILD_TYPE=Debug -GNinja
         -DJSON_BuildTests=ON -DJSON_FastTests=ON -DJSON_TestDataDirectory=${PROJECT_BINARY_DIR}/build_offline_testdata/test_data/json_test_data

cmake/download_test_data.cmake

@@ -1,5 +1,5 @@
 set(JSON_TEST_DATA_URL     https://github.com/nlohmann/json_test_data)
-set(JSON_TEST_DATA_VERSION 3.1.0)
+set(JSON_TEST_DATA_VERSION 3.2.0)
 
 include(ExternalProject)
 

docs/docset/docSet.sql

@@ -48,6 +48,7 @@ INSERT INTO searchIndex(name, type, path) VALUES ('basic_json::from_bjdata', 'Fu
 INSERT INTO searchIndex(name, type, path) VALUES ('basic_json::from_bson', 'Function', 'api/basic_json/from_bson/index.html');
 INSERT INTO searchIndex(name, type, path) VALUES ('basic_json::from_cbor', 'Function', 'api/basic_json/from_cbor/index.html');
 INSERT INTO searchIndex(name, type, path) VALUES ('basic_json::from_msgpack', 'Function', 'api/basic_json/from_msgpack/index.html');
+INSERT INTO searchIndex(name, type, path) VALUES ('basic_json::from_bon8', 'Function', 'api/basic_json/from_bon8/index.html');
 INSERT INTO searchIndex(name, type, path) VALUES ('basic_json::from_ubjson', 'Function', 'api/basic_json/from_ubjson/index.html');
 INSERT INTO searchIndex(name, type, path) VALUES ('basic_json::front', 'Method', 'api/basic_json/front/index.html');
 INSERT INTO searchIndex(name, type, path) VALUES ('basic_json::get', 'Method', 'api/basic_json/get/index.html');
@@ -121,6 +122,7 @@ INSERT INTO searchIndex(name, type, path) VALUES ('basic_json::to_bjdata', 'Func
 INSERT INTO searchIndex(name, type, path) VALUES ('basic_json::to_bson', 'Function', 'api/basic_json/to_bson/index.html');
 INSERT INTO searchIndex(name, type, path) VALUES ('basic_json::to_cbor', 'Function', 'api/basic_json/to_cbor/index.html');
 INSERT INTO searchIndex(name, type, path) VALUES ('basic_json::to_msgpack', 'Function', 'api/basic_json/to_msgpack/index.html');
+INSERT INTO searchIndex(name, type, path) VALUES ('basic_json::to_bon8', 'Function', 'api/basic_json/to_bon8/index.html');
 INSERT INTO searchIndex(name, type, path) VALUES ('basic_json::to_string', 'Method', 'api/basic_json/to_string/index.html');
 INSERT INTO searchIndex(name, type, path) VALUES ('basic_json::to_ubjson', 'Function', 'api/basic_json/to_ubjson/index.html');
 INSERT INTO searchIndex(name, type, path) VALUES ('basic_json::value', 'Method', 'api/basic_json/value/index.html');
@@ -171,6 +173,7 @@ INSERT INTO searchIndex(name, type, path) VALUES ('Binary Formats: BJData', 'Gui
 INSERT INTO searchIndex(name, type, path) VALUES ('Binary Formats: BSON', 'Guide', 'features/binary_formats/bson/index.html');
 INSERT INTO searchIndex(name, type, path) VALUES ('Binary Formats: CBOR', 'Guide', 'features/binary_formats/cbor/index.html');
 INSERT INTO searchIndex(name, type, path) VALUES ('Binary Formats: MessagePack', 'Guide', 'features/binary_formats/messagepack/index.html');
+INSERT INTO searchIndex(name, type, path) VALUES ('Binary Formats: BON8', 'Guide', 'features/binary_formats/bon8/index.html');
 INSERT INTO searchIndex(name, type, path) VALUES ('Binary Formats: UBJSON', 'Guide', 'features/binary_formats/ubjson/index.html');
 INSERT INTO searchIndex(name, type, path) VALUES ('Binary Values', 'Guide', 'features/binary_values/index.html');
 INSERT INTO searchIndex(name, type, path) VALUES ('Comments', 'Guide', 'features/comments/index.html');

docs/mkdocs/docs/api/basic_json/basic_json.md

@@ -272,15 +272,6 @@ basic_json(basic_json&& other) noexcept;
         When used without parentheses around an empty initializer list, `basic_json()` is called instead of this
         function, yielding the JSON `#!json null` value.
 
-- Overload 4:
-
-    !!! info "Implicit conversion"
-
-        The conversion is implicit unless [`JSON_USE_IMPLICIT_CONVERSIONS`](../macros/json_use_implicit_conversions.md)
-        is defined to `0` and `BasicJsonType::string_t` differs from `string_t`. In that case, the constructor is
-        `explicit`, so a JSON value with a different string type is no longer silently converted, for example when it is
-        passed to a function taking `#!cpp const json&`. Write `#!cpp json(other)` or `#!cpp other.get<json>()` instead.
-
 - Overload 7:
 
     !!! info "Preconditions"
@@ -429,8 +420,7 @@ basic_json(basic_json&& other) noexcept;
 1. Since version 1.0.0.
 2. Since version 1.0.0.
 3. Since version 2.1.0.
-4. Since version 3.2.0. Explicit for different string types if `JSON_USE_IMPLICIT_CONVERSIONS` is `0` since
-   version 3.13.0.
+4. Since version 3.2.0.
 5. Since version 1.0.0.
 6. Since version 1.0.0.
 7. Since version 1.0.0.

docs/mkdocs/docs/api/basic_json/from_bjdata.md

@@ -104,6 +104,7 @@ Linear in the size of the input.
 - [from_msgpack](from_msgpack.md) create a JSON value from an input in MessagePack format
 - [from_bson](from_bson.md) create a JSON value from an input in BSON format
 - [from_ubjson](from_ubjson.md) create a JSON value from an input in UBJSON format
+- [from_bon8](from_bon8.md) create a JSON value from an input in BON8 format
 
 ## Version history
 

docs/mkdocs/docs/api/basic_json/from_bon8.md

@@ -0,0 +1,108 @@
+# <small>nlohmann::basic_json::</small>from_bon8
+
+```cpp
+// (1)
+template<typename InputType>
+static basic_json from_bon8(InputType&& i,
+                            const bool strict = true,
+                            const bool allow_exceptions = true);
+// (2)
+template<typename IteratorType, typename SentinelType = IteratorType>
+static basic_json from_bon8(IteratorType first, SentinelType last,
+                            const bool strict = true,
+                            const bool allow_exceptions = true);
+```
+
+Deserializes a given input to a JSON value using the BON8 (Binary Object Notation 8) serialization format.
+
+1. Reads from a compatible input.
+2. Reads from an iterator range, or an iterator and a sentinel of a different type (C++20 ranges support).
+
+The exact mapping and its limitations are described on a [dedicated page](../../features/binary_formats/bon8.md).
+
+## Template parameters
+
+`InputType`
+:   A compatible input, for instance:
+
+    - an `std::istream` object
+    - a `FILE` pointer
+    - a C-style array of characters
+    - a pointer to a null-terminated string of single byte characters
+    - a container `obj` for which `begin(obj)` and `end(obj)` produce a valid pair of iterators
+      (as found via ADL or member functions, with semantics compatible to `std::begin` and `std::end`)
+
+`IteratorType`
+:   a compatible iterator type
+
+`SentinelType`
+:   defaults to `IteratorType`; may be a different type comparable to `IteratorType` via `operator!=`, for instance.
+
+    - a custom sentinel type for C++20 ranges
+    - `std::default_sentinel_t`, when `IteratorType` is `std::counted_iterator`
+
+## Parameters
+
+`i` (in)
+:   an input in BON8 format convertible to an input adapter
+
+`first` (in)
+:   iterator to the start of the input
+
+`last` (in)
+:   iterator to the end of the input, or a sentinel value that compares equal to the end iterator with `operator!=`
+
+`strict` (in)
+:   whether to expect the input to be consumed until EOF (`#!cpp true` by default)
+
+`allow_exceptions` (in)
+:   whether to throw exceptions in case of a parse error (optional, `#!cpp true` by default)
+
+## Return value
+
+deserialized JSON value; in case of a parse error and `allow_exceptions` set to `#!cpp false`, the return value will be
+`value_t::discarded`. The latter can be checked with [`is_discarded`](is_discarded.md).
+
+## Exception safety
+
+Strong guarantee: if an exception is thrown, there are no changes in the JSON value.
+
+## Exceptions
+
+- Throws [parse_error.110](../../home/exceptions.md#jsonexceptionparse_error110) if the given input ends prematurely or
+  the end of the file was not reached when `strict` was set to true
+- Throws [parse_error.112](../../home/exceptions.md#jsonexceptionparse_error112) if a parse error occurs, for instance
+  an invalid byte, a string that is not valid UTF-8, or an object key that is not a string
+
+## Complexity
+
+Linear in the size of the input.
+
+## Examples
+
+??? example
+
+    The example shows the deserialization of a byte vector in BON8 format to a JSON value.
+     
+    ```cpp
+    --8<-- "examples/from_bon8.cpp"
+    ```
+    
+    Output:
+    
+    ```json
+    --8<-- "examples/from_bon8.output"
+    ```
+
+## See also
+
+- [to_bon8](to_bon8.md) create a BON8 serialization of a JSON value
+- [from_cbor](from_cbor.md) create a JSON value from an input in CBOR format
+- [from_msgpack](from_msgpack.md) create a JSON value from an input in MessagePack format
+- [from_bson](from_bson.md) create a JSON value from an input in BSON format
+- [from_ubjson](from_ubjson.md) create a JSON value from an input in UBJSON format
+- [from_bjdata](from_bjdata.md) create a JSON value from an input in BJData format
+
+## Version history
+
+- Added in version 3.13.0.

docs/mkdocs/docs/api/basic_json/from_bson.md

@@ -104,6 +104,7 @@ Linear in the size of the input.
 - [from_msgpack](from_msgpack.md) for the related MessagePack format
 - [from_ubjson](from_ubjson.md) for the related UBJSON format
 - [from_bjdata](from_bjdata.md) for the related BJData format
+- [from_bon8](from_bon8.md) for the related BON8 format
 
 ## Version history
 

docs/mkdocs/docs/api/basic_json/from_cbor.md

@@ -110,6 +110,7 @@ Linear in the size of the input.
 - [from_bson](from_bson.md) create a JSON value from an input in BSON format
 - [from_ubjson](from_ubjson.md) create a JSON value from an input in UBJSON format
 - [from_bjdata](from_bjdata.md) create a JSON value from an input in BJData format
+- [from_bon8](from_bon8.md) create a JSON value from an input in BON8 format
 
 ## Version history
 

docs/mkdocs/docs/api/basic_json/from_msgpack.md

@@ -103,6 +103,7 @@ Linear in the size of the input.
 - [from_bson](from_bson.md) create a JSON value from an input in BSON format
 - [from_ubjson](from_ubjson.md) create a JSON value from an input in UBJSON format
 - [from_bjdata](from_bjdata.md) create a JSON value from an input in BJData format
+- [from_bon8](from_bon8.md) create a JSON value from an input in BON8 format
 
 ## Version history
 

docs/mkdocs/docs/api/basic_json/from_ubjson.md

@@ -104,6 +104,7 @@ Linear in the size of the input.
 - [from_msgpack](from_msgpack.md) create a JSON value from an input in MessagePack format
 - [from_bson](from_bson.md) create a JSON value from an input in BSON format
 - [from_bjdata](from_bjdata.md) create a JSON value from an input in BJData format
+- [from_bon8](from_bon8.md) create a JSON value from an input in BON8 format
 
 ## Version history
 

docs/mkdocs/docs/api/basic_json/index.md

@@ -290,11 +290,13 @@ Access to the JSON value
 ### Binary formats
 
 - [**from_bjdata**](from_bjdata.md) (_static_) - create a JSON value from an input in BJData format
+- [**from_bon8**](from_bon8.md) (_static_) - create a JSON value from an input in BON8 format
 - [**from_bson**](from_bson.md) (_static_) - create a JSON value from an input in BSON format
 - [**from_cbor**](from_cbor.md) (_static_) - create a JSON value from an input in CBOR format
 - [**from_msgpack**](from_msgpack.md) (_static_) - create a JSON value from an input in MessagePack format
 - [**from_ubjson**](from_ubjson.md) (_static_) - create a JSON value from an input in UBJSON format
 - [**to_bjdata**](to_bjdata.md) (_static_) - create a BJData serialization of a given JSON value
+- [**to_bon8**](to_bon8.md) (_static_) - create a BON8 serialization of a given JSON value
 - [**to_bson**](to_bson.md) (_static_) - create a BSON serialization of a given JSON value
 - [**to_cbor**](to_cbor.md) (_static_) - create a CBOR serialization of a given JSON value
 - [**to_msgpack**](to_msgpack.md) (_static_) - create a MessagePack serialization of a given JSON value

docs/mkdocs/docs/api/basic_json/input_format_t.md

@@ -7,7 +7,8 @@ enum class input_format_t {
     msgpack,
     ubjson,
     bson,
-    bjdata
+    bjdata,
+    bon8
 };
 ```
 
@@ -31,6 +32,9 @@ bson
 bjdata
 :   BJData (Binary JData)
 
+bon8
+:   BON8 (Binary Object Notation 8)
+
 ## Examples
 
 ??? example

docs/mkdocs/docs/api/basic_json/parse_error.md

@@ -5,7 +5,7 @@ class parse_error : public exception;
 ```
 
 The library throws this exception when a parse error occurs. Parse errors can occur during the deserialization of
-JSON text, BSON, CBOR, MessagePack, UBJSON, as well as when using JSON Patch.
+JSON text, BJData, BON8, BSON, CBOR, MessagePack, UBJSON, as well as when using JSON Patch.
 
 Member `byte` holds the byte index of the last read character in the input file (see note below).
 

docs/mkdocs/docs/api/basic_json/sax_parse.md

@@ -65,7 +65,8 @@ The SAX event lister must follow the interface of [`json_sax`](../json_sax/index
 :   SAX event listener (must not be null)
 
 `format` (in)
-:    the format to parse (JSON, CBOR, MessagePack, or UBJSON) (optional, `input_format_t::json` by default), see
+:    the format to parse (JSON, BJData, BON8, BSON, CBOR, MessagePack, or UBJSON) (optional, `input_format_t::json` by
+     default), see
      [`input_format_t`](input_format_t.md) for more information
 
 `strict` (in)
@@ -89,7 +90,9 @@ The SAX event lister must follow the interface of [`json_sax`](../json_sax/index
 
 ## Return value
 
-return value of the last processed SAX event
+`#!cpp true` if the input was parsed without errors and no SAX event returned `#!cpp false`; `#!cpp false` otherwise.
+In particular, the result is `#!cpp false` for input with errors, even if the SAX parser recovered from all of them
+(see [error recovery](../../features/parsing/error_recovery.md)).
 
 ## Exception safety
 
@@ -137,6 +140,7 @@ A UTF-8 byte order mark is silently ignored.
 - Ignoring comments via `ignore_comments` added in version 3.9.0.
 - Added `ignore_trailing_commas` in version 3.13.0.
 - Extended container support (1) to include types with lvalue-only ADL `begin`/`end` (matching `std::begin`/`std::end` semantics) in version 3.13.0.
+- Recovering from parse errors (see [`parse_error`](../json_sax/parse_error.md)) added in version 3.13.0.
 - Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
 - `JSON_PRECISE_STREAM_POSITION` added in version 3.13.0 to optionally leave a `#!cpp std::istream` positioned right
   after the parsed value when `strict` is `#!cpp false`.

docs/mkdocs/docs/api/basic_json/to_bjdata.md

@@ -84,6 +84,7 @@ Linear in the size of the JSON value `j`.
 - [to_msgpack](to_msgpack.md) create a MessagePack serialization of a JSON value
 - [to_bson](to_bson.md) create a BSON serialization of a JSON value
 - [to_ubjson](to_ubjson.md) create a UBJSON serialization of a JSON value
+- [to_bon8](to_bon8.md) create a BON8 serialization of a JSON value
 
 ## Version history
 

docs/mkdocs/docs/api/basic_json/to_bon8.md

@@ -0,0 +1,76 @@
+# <small>nlohmann::basic_json::</small>to_bon8
+
+```cpp
+// (1)
+static std::vector<std::uint8_t> to_bon8(const basic_json& j);
+
+// (2)
+static void to_bon8(const basic_json& j, detail::output_adapter<std::uint8_t> o);
+static void to_bon8(const basic_json& j, detail::output_adapter<char> o);
+```
+
+Serializes a given JSON value `j` to a byte vector using the BON8 (Binary Object Notation 8) serialization format. BON8
+is a compact binary serialization format that stores strings as UTF-8 without a length prefix.
+
+1. Returns a byte vector containing the BON8 serialization.
+2. Writes the BON8 serialization to an output adapter.
+
+The exact mapping and its limitations are described on a [dedicated page](../../features/binary_formats/bon8.md).
+
+## Parameters
+
+`j` (in)
+:   JSON value to serialize
+
+`o` (in)
+:   output adapter to write serialization to
+
+## Return value
+
+1. BON8 serialization as a byte vector
+2. (none)
+
+## Exception safety
+
+Strong guarantee: if an exception is thrown, there are no changes in the JSON value `j`, which is never modified.
+With (2), the bytes written before the exception remain in the output adapter.
+
+## Exceptions
+
+- Throws [out_of_range.407](../../home/exceptions.md#jsonexceptionout_of_range407) if `j` contains an unsigned integer
+  above 9223372036854775807, which BON8 cannot represent
+- Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if `j` contains a string that is not
+  valid UTF-8
+
+## Complexity
+
+Linear in the size of the JSON value `j`.
+
+## Examples
+
+??? example
+
+    The example shows the serialization of a JSON value to a byte vector in BON8 format.
+     
+    ```cpp
+    --8<-- "examples/to_bon8.cpp"
+    ```
+    
+    Output:
+    
+    ```json
+    --8<-- "examples/to_bon8.output"
+    ```
+
+## See also
+
+- [from_bon8](from_bon8.md) create a JSON value from an input in BON8 format
+- [to_cbor](to_cbor.md) create a CBOR serialization of a JSON value
+- [to_msgpack](to_msgpack.md) create a MessagePack serialization of a JSON value
+- [to_bson](to_bson.md) create a BSON serialization of a JSON value
+- [to_ubjson](to_ubjson.md) create a UBJSON serialization of a JSON value
+- [to_bjdata](to_bjdata.md) create a BJData serialization of a JSON value
+
+## Version history
+
+- Added in version 3.13.0.

docs/mkdocs/docs/api/basic_json/to_bson.md

@@ -72,6 +72,7 @@ pass before anything is written.
 - [to_msgpack](to_msgpack.md) create a MessagePack serialization of a JSON value
 - [to_ubjson](to_ubjson.md) create a UBJSON serialization of a JSON value
 - [to_bjdata](to_bjdata.md) create a BJData serialization of a JSON value
+- [to_bon8](to_bon8.md) create a BON8 serialization of a JSON value
 
 ## Version history
 

docs/mkdocs/docs/api/basic_json/to_cbor.md

@@ -62,6 +62,7 @@ Linear in the size of the JSON value `j`.
 - [to_bson](to_bson.md) create a BSON serialization of a JSON value
 - [to_ubjson](to_ubjson.md) create a UBJSON serialization of a JSON value
 - [to_bjdata](to_bjdata.md) create a BJData serialization of a JSON value
+- [to_bon8](to_bon8.md) create a BON8 serialization of a JSON value
 
 ## Version history
 

docs/mkdocs/docs/api/basic_json/to_msgpack.md

@@ -70,6 +70,7 @@ Linear in the size of the JSON value `j`.
 - [to_bson](to_bson.md) create a BSON serialization of a JSON value
 - [to_ubjson](to_ubjson.md) create a UBJSON serialization of a JSON value
 - [to_bjdata](to_bjdata.md) create a BJData serialization of a JSON value
+- [to_bon8](to_bon8.md) create a BON8 serialization of a JSON value
 
 ## Version history
 

docs/mkdocs/docs/api/basic_json/to_ubjson.md

@@ -77,6 +77,7 @@ Linear in the size of the JSON value `j`.
 - [to_msgpack](to_msgpack.md) create a MessagePack serialization of a JSON value
 - [to_bson](to_bson.md) create a BSON serialization of a JSON value
 - [to_bjdata](to_bjdata.md) create a BJData serialization of a JSON value
+- [to_bon8](to_bon8.md) create a BON8 serialization of a JSON value
 
 ## Version history
 

docs/mkdocs/docs/api/json_sax/index.md

@@ -7,7 +7,8 @@ struct json_sax;
 
 This class describes the SAX interface used by [sax_parse](../basic_json/sax_parse.md). Each function is called in
 different situations while the input is parsed. The boolean return value informs the parser whether to continue
-processing the input.
+processing the input; for [`parse_error`](parse_error.md), it decides whether to
+[recover from the error](../../features/parsing/error_recovery.md).
 
 ## Template parameters
 

docs/mkdocs/docs/api/json_sax/parse_error.md

@@ -21,7 +21,14 @@ A parse error occurred.
 
 ## Return value
 
-Whether parsing should proceed (**must return `#!cpp false`**).
+Whether to recover from the error:
+
+- `#!cpp false` stops parsing.
+- `#!cpp true` recovers from the error: JSON text is repaired and parsing continues; for the binary formats, the value
+  read so far is completed and parsing stops. See [error recovery](../../features/parsing/error_recovery.md) for how
+  errors are repaired.
+
+Either way, [`sax_parse`](../basic_json/sax_parse.md) returns `#!cpp false`.
 
 ## Examples
 
@@ -39,6 +46,22 @@ Whether parsing should proceed (**must return `#!cpp false`**).
     --8<-- "examples/sax_parse.output"
     ```
 
+??? example
+
+    The example below shows how a SAX parser recovers from errors.
+
+    ```cpp
+    --8<-- "examples/sax_parse__error_recovery.cpp"
+    ```
+    
+    Output:
+    
+    ```
+    --8<-- "examples/sax_parse__error_recovery.output"
+    ```
+
 ## Version history
 
 - Added in version 3.2.0.
+- Returning `#!cpp true` recovers from the error since version 3.13.0; before, parsing stopped, but the result of
+  [`sax_parse`](../basic_json/sax_parse.md) could be wrong.

docs/mkdocs/docs/api/macros/json_strict_nul_handling.md

@@ -11,9 +11,9 @@ The macro only affects the JSON text parser ([`parse`](../basic_json/parse.md),
 [`sax_parse`](../basic_json/sax_parse.md), and [`operator>>`](../operator_gtgt.md)). There are three cases where a NUL
 byte is still not rejected:
 
-- The binary formats ([`from_bjdata`](../basic_json/from_bjdata.md), [`from_bson`](../basic_json/from_bson.md),
-  [`from_cbor`](../basic_json/from_cbor.md), [`from_msgpack`](../basic_json/from_msgpack.md),
-  [`from_ubjson`](../basic_json/from_ubjson.md)) are never affected: there, `0x00` is ordinary data.
+- The binary formats ([`from_bjdata`](../basic_json/from_bjdata.md), [`from_bon8`](../basic_json/from_bon8.md),
+  [`from_bson`](../basic_json/from_bson.md), [`from_cbor`](../basic_json/from_cbor.md),
+  [`from_msgpack`](../basic_json/from_msgpack.md), [`from_ubjson`](../basic_json/from_ubjson.md)) are never affected: there, `0x00` is ordinary data.
 - A bare `const char*` pointer has no length of its own, so its length is still determined with `strlen()`. The first
   NUL byte therefore still marks the end of the input, and nothing after it is read.
 - One trailing `'\0'` at the end of a `char` array (e.g., a string literal) is trimmed; see the warning below.

docs/mkdocs/docs/api/macros/json_use_implicit_conversions.md

@@ -5,9 +5,7 @@
 ```
 
 When defined to `0`, implicit conversions are switched off. By default, implicit conversions are switched on. The
-value directly affects [`operator ValueType`](../basic_json/operator_ValueType.md) and the
-[converting constructor](../basic_json/basic_json.md) from a `basic_json` specialization with a different string
-type (overload 4).
+value directly affects [`operator ValueType`](../basic_json/operator_ValueType.md).
 
 ## Default definition
 
@@ -59,25 +57,6 @@ By default, implicit conversions are enabled.
     auto s = j.get<std::string>();
     ```
 
-??? example "Conversion between `basic_json` specializations"
-
-    A `basic_json` specialization with a different string type is also no longer converted implicitly when
-    `JSON_USE_IMPLICIT_CONVERSIONS` is defined to `0`:
-
-    ```cpp
-    using wjson = nlohmann::basic_json<std::map, std::vector, std::wstring>;
-
-    void load(const nlohmann::json& j);
-
-    wjson wj = /* ... */;
-    load(wj);                            // error: no implicit conversion
-    load(nlohmann::json(wj));            // OK: explicit conversion
-    load(wj.get<nlohmann::json>());      // OK: explicit conversion
-    ```
-
-    Specializations that share the same string type, such as `json` and `ordered_json`, remain implicitly
-    convertible.
-
 ## See also
 
 - [**operator ValueType**](../basic_json/operator_ValueType.md) - get a value (implicit)
@@ -87,4 +66,3 @@ By default, implicit conversions are enabled.
 ## Version history
 
 - Added in version 3.9.0.
-- Also affects the conversion between `basic_json` specializations with different string types since version 3.13.0.

docs/mkdocs/docs/examples/from_bon8.cpp

@@ -0,0 +1,21 @@
+#include <iostream>
+#include <iomanip>
+#include <nlohmann/json.hpp>
+
+using json = nlohmann::json;
+
+int main()
+{
+    // create byte vector
+    std::vector<std::uint8_t> v = {0x89, 0x63, 0x6f, 0x6d, 0x70, 0x61, 0x63, 0x74,
+                                   0xf9, 0x66, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0xff,
+                                   0x42, 0x4f, 0x4e, 0x38, 0xff, 0x73, 0x63, 0x68,
+                                   0x65, 0x6d, 0x61, 0x90
+                                  };
+
+    // deserialize it with BON8
+    json j = json::from_bon8(v);
+
+    // print the deserialized JSON value
+    std::cout << std::setw(2) << j << std::endl;
+}

docs/mkdocs/docs/examples/from_bon8.output

@@ -0,0 +1,5 @@
+{
+  "compact": true,
+  "format": "BON8",
+  "schema": 0
+}

docs/mkdocs/docs/examples/sax_parse__error_recovery.cpp

@@ -0,0 +1,43 @@
+#include <iostream>
+#include <iomanip>
+#include <nlohmann/json.hpp>
+
+using json = nlohmann::json;
+
+// a SAX parser that creates a JSON value like json::parse does, but that
+// recovers from parse errors instead of stopping at the first one
+class recovering_parser : public nlohmann::detail::json_sax_dom_parser<json>
+{
+  public:
+    explicit recovering_parser(json& result)
+        : nlohmann::detail::json_sax_dom_parser<json>(result, false)
+    {}
+
+    bool parse_error(std::size_t position,
+                     const std::string& /*last_token*/,
+                     const json::exception& ex)
+    {
+        std::cout << "byte " << position << ": " << ex.what() << '\n';
+
+        // repair the input and continue
+        return true;
+    }
+};
+
+int main()
+{
+    // JSON text with several mistakes that ends too early
+    const std::string text = R"({
+    "name": "Hello World",
+    "tags": ["a" "b",],
+    "valid": tru,
+    "size": 1.,
+    "nested": {"x": 1)";
+
+    json result;
+    recovering_parser sax(result);
+    const bool valid = json::sax_parse(text, &sax);
+
+    std::cout << "\nvalid JSON: " << std::boolalpha << valid << '\n'
+              << std::setw(4) << result << std::endl;
+}

docs/mkdocs/docs/examples/sax_parse__error_recovery.output

@@ -0,0 +1,19 @@
+byte 49: [json.exception.parse_error.101] parse error at line 3, column 20: syntax error while parsing array - unexpected string literal; expected ']'
+byte 51: [json.exception.parse_error.101] parse error at line 3, column 22: syntax error while parsing value - unexpected ']'; expected '[', '{', or a literal
+byte 70: [json.exception.parse_error.101] parse error at line 4, column 17: syntax error while parsing value - invalid literal; last read: '"valid": tru,'
+byte 86: [json.exception.parse_error.101] parse error at line 5, column 15: syntax error while parsing value - invalid number; expected digit after '.'; last read: '1.,'
+byte 109: [json.exception.parse_error.101] parse error at line 6, column 22: syntax error while parsing object - unexpected end of input; expected '}'
+
+valid JSON: false
+{
+    "name": "Hello World",
+    "nested": {
+        "x": 1
+    },
+    "size": 1,
+    "tags": [
+        "a",
+        "b"
+    ],
+    "valid": null
+}

docs/mkdocs/docs/examples/to_bon8.cpp

@@ -0,0 +1,22 @@
+#include <iostream>
+#include <iomanip>
+#include <nlohmann/json.hpp>
+
+using json = nlohmann::json;
+using namespace nlohmann::literals;
+
+int main()
+{
+    // create a JSON value
+    json j = R"({"compact": true, "format": "BON8", "schema": 0})"_json;
+
+    // serialize it to BON8
+    std::vector<std::uint8_t> v = json::to_bon8(j);
+
+    // print the vector content
+    for (auto& byte : v)
+    {
+        std::cout << "0x" << std::hex << std::setw(2) << std::setfill('0') << (int)byte << " ";
+    }
+    std::cout << std::endl;
+}

docs/mkdocs/docs/examples/to_bon8.output

@@ -0,0 +1 @@
+0x89 0x63 0x6f 0x6d 0x70 0x61 0x63 0x74 0xf9 0x66 0x6f 0x72 0x6d 0x61 0x74 0xff 0x42 0x4f 0x4e 0x38 0xff 0x73 0x63 0x68 0x65 0x6d 0x61 0x90 

docs/mkdocs/docs/features/binary_formats/bon8.md

@@ -0,0 +1,159 @@
+# BON8
+
+BON8 (Binary Object Notation 8) is a compact binary serialization format for JSON values. It uses the byte values that
+cannot begin a UTF-8 character as type markers, so strings are stored as plain UTF-8 without a length prefix: a string
+ends at the first byte that cannot continue it. Integers from -10 to 39, `true`, `false`, `null`, and the floating-point
+values -1.0, 0.0, and 1.0 take a single byte, and arrays and objects with up to four elements need no terminator.
+
+!!! abstract "References"
+
+    - [BON8 specification](https://github.com/hikoworks/hikogui/blob/main/docs/BON8.md)
+    - [Reference implementation](https://github.com/hikoworks/hikogui/blob/main/src/hikogui/codec/BON8.hpp) in HikoGUI
+
+## Serialization
+
+The library uses the following mapping from JSON values types to BON8 types according to the BON8 specification:
+
+| JSON value type | value/range                                  | BON8 type                     | first byte |
+|-----------------|----------------------------------------------|-------------------------------|------------|
+| null            | `null`                                       | null                          | 0xFA       |
+| boolean         | `true`                                       | true                          | 0xF9       |
+| boolean         | `false`                                      | false                         | 0xF8       |
+| number_integer  | -9223372036854775808..-2147483649            | int64                         | 0x8D       |
+| number_integer  | -2147483648..-33818507                       | int32                         | 0x8C       |
+| number_integer  | -33818506..-264075                           | 4-byte negative integer       | 0xF0..0xF7 |
+| number_integer  | -264074..-1931                               | 3-byte negative integer       | 0xE0..0xEF |
+| number_integer  | -1930..-11                                   | 2-byte negative integer       | 0xC2..0xDF |
+| number_integer  | -10..-1                                      | 1-byte negative integer       | 0xB8..0xC1 |
+| number_integer  | 0..39                                        | 1-byte positive integer       | 0x90..0xB7 |
+| number_integer  | 40..3879                                     | 2-byte positive integer       | 0xC2..0xDF |
+| number_integer  | 3880..528167                                 | 3-byte positive integer       | 0xE0..0xEF |
+| number_integer  | 528168..67637031                             | 4-byte positive integer       | 0xF0..0xF7 |
+| number_integer  | 67637032..2147483647                         | int32                         | 0x8C       |
+| number_integer  | 2147483648..9223372036854775807              | int64                         | 0x8D       |
+| number_unsigned | 0..39                                        | 1-byte positive integer       | 0x90..0xB7 |
+| number_unsigned | 40..3879                                     | 2-byte positive integer       | 0xC2..0xDF |
+| number_unsigned | 3880..528167                                 | 3-byte positive integer       | 0xE0..0xEF |
+| number_unsigned | 528168..67637031                             | 4-byte positive integer       | 0xF0..0xF7 |
+| number_unsigned | 67637032..2147483647                         | int32                         | 0x8C       |
+| number_unsigned | 2147483648..9223372036854775807              | int64                         | 0x8D       |
+| number_float    | `-1.0`                                       | -1.0                          | 0xFB       |
+| number_float    | `0.0`                                        | 0.0                           | 0xFC       |
+| number_float    | `1.0`                                        | 1.0                           | 0xFD       |
+| number_float    | *any other value representable by a float*   | binary32                      | 0x8E       |
+| number_float    | *any value NOT representable by a float*     | binary64                      | 0x8F       |
+| string          | *empty*                                      | end of string                 | 0xFF       |
+| string          | *non-empty*                                  | UTF-8 string                  | 0x00..0x7F, 0xC2..0xF4 |
+| array           | *size*: 0..4                                 | array with count              | 0x80..0x84 |
+| array           | *size*: 5 or more                            | array (terminated by 0xFE)    | 0x85       |
+| object          | *size*: 0..4                                 | object with count             | 0x86..0x8A |
+| object          | *size*: 5 or more                            | object (terminated by 0xFE)   | 0x8B       |
+| binary          | *size*: 0..4                                 | array with count              | 0x80..0x84 |
+| binary          | *size*: 5 or more                            | array (terminated by 0xFE)    | 0x85       |
+
+An integer that takes 2 to 4 bytes starts with a UTF-8 lead byte (0xC2..0xF7) that is followed by a byte that cannot
+continue a UTF-8 character: 0x00..0x7F for positive and 0xC0..0xFF for negative integers. A string is terminated by
+0xFF only if it is empty, if another string follows it, or if it is the last value of the message; otherwise, the first
+byte of the next value ends it.
+
+!!! success "Complete mapping"
+
+    Except for the values listed below, any JSON value can be converted to a BON8 value.
+
+    Any BON8 output created by `to_bon8` can be successfully parsed by `from_bon8`.
+
+!!! warning "Unsupported values"
+
+    The following values can **not** be converted to a BON8 value:
+
+      - unsigned integers above 9223372036854775807, because BON8 has no unsigned 64-bit integer type
+        ([out_of_range.407](../../home/exceptions.md#jsonexceptionout_of_range407))
+      - strings that are not valid UTF-8, because the end of a string is determined from its encoding
+        ([type_error.316](../../home/exceptions.md#jsonexceptiontype_error316))
+
+!!! info "NaN/infinity handling"
+
+    `-0.0`, `Infinity`, and `-Infinity` are serialized as binary32 (type 0x8E, 5 bytes total). `NaN` is serialized as
+    the binary32 value 0x7F800001 that the specification recommends. This is in contrast to the
+    [dump](../../api/basic_json/dump.md) function which serializes NaN or Infinity to `null`.
+
+!!! warning "Binary values"
+
+    BON8 has no binary type. Binary values are serialized as arrays of integers (0..255), so they are read back as
+    arrays. The subtype is not serialized.
+
+!!! info "Canonical representation"
+
+    The output follows the specification's canonical representation rules: every value uses the shortest encoding,
+    floating-point numbers use binary32 whenever that loses no precision, and object keys are sorted by their UTF-8
+    code units. There are two exceptions:
+
+    - Strings are not normalized to Unicode Normalization Form C (NFC).
+    - Object keys are written in the order of the object type, which is sorted for `json`, but not for
+      [`ordered_json`](../../api/ordered_json.md).
+
+??? example
+
+    ```cpp
+    --8<-- "examples/to_bon8.cpp"
+    ```
+
+    Output:
+
+    ```c
+    --8<-- "examples/to_bon8.output"
+    ```
+
+## Deserialization
+
+The library maps BON8 types to JSON value types as follows:
+
+| BON8 type                     | JSON value type | first byte             |
+|-------------------------------|-----------------|------------------------|
+| UTF-8 string                  | string          | 0x00..0x7F             |
+| array with count              | array           | 0x80..0x84             |
+| array (terminated by 0xFE)    | array           | 0x85                   |
+| object with count             | object          | 0x86..0x8A             |
+| object (terminated by 0xFE)   | object          | 0x8B                   |
+| int32                         | number_unsigned or number_integer | 0x8C |
+| int64                         | number_unsigned or number_integer | 0x8D |
+| binary32                      | number_float    | 0x8E                   |
+| binary64                      | number_float    | 0x8F                   |
+| 1-byte positive integer       | number_unsigned | 0x90..0xB7             |
+| 1-byte negative integer       | number_integer  | 0xB8..0xC1             |
+| UTF-8 string                  | string          | 0xC2..0xF4, followed by 0x80..0xBF |
+| 2- to 4-byte positive integer | number_unsigned | 0xC2..0xF7, followed by 0x00..0x7F |
+| 2- to 4-byte negative integer | number_integer  | 0xC2..0xF7, followed by 0xC0..0xFF |
+| false                         | `false`         | 0xF8                   |
+| true                          | `true`          | 0xF9                   |
+| null                          | `null`          | 0xFA                   |
+| -1.0                          | number_float    | 0xFB                   |
+| 0.0                           | number_float    | 0xFC                   |
+| 1.0                           | number_float    | 0xFD                   |
+| empty string                  | string          | 0xFF                   |
+
+Non-negative integers are read as number_unsigned, negative integers as number_integer.
+
+!!! info
+
+    Values that do not use the canonical representation, such as integers with a longer encoding than necessary,
+    arrays and objects with up to four elements that are terminated by 0xFE, unsorted object keys, or a 0xFF after a
+    string that would also end without it, are accepted. A second 0xFF is not a terminator but an empty string.
+
+    Strings must be valid UTF-8, and the last string of a message must be terminated by 0xFF.
+
+!!! info
+
+    Any BON8 output created by `to_bon8` can be successfully parsed by `from_bon8`.
+
+??? example
+
+    ```cpp
+    --8<-- "examples/from_bon8.cpp"
+    ```
+
+    Output:
+
+    ```json
+    --8<-- "examples/from_bon8.output"
+    ```

docs/mkdocs/docs/features/binary_formats/index.md

@@ -4,6 +4,7 @@ Though JSON is a ubiquitous data format, it is not a very compact format suitabl
 a network. Hence, the library supports
 
 - [BJData](bjdata.md) (Binary JData),
+- [BON8](bon8.md) (Binary Object Notation 8),
 - [BSON](bson.md) (Binary JSON),
 - [CBOR](cbor.md) (Concise Binary Object Representation),
 - [MessagePack](messagepack.md), and
@@ -18,6 +19,7 @@ to efficiently encode JSON values to byte vectors and to decode such vectors.
 | Format      | Serialization                                 | Deserialization                              |
 |-------------|-----------------------------------------------|----------------------------------------------|
 | BJData      | complete                                      | complete                                     |
+| BON8        | incomplete: no unsigned integers above int64  | complete                                     |
 | BSON        | incomplete: top-level value must be an object | incomplete, but all JSON types are supported |
 | CBOR        | complete                                      | incomplete, but all JSON types are supported |
 | MessagePack | complete                                      | complete                                     |
@@ -28,6 +30,7 @@ to efficiently encode JSON values to byte vectors and to decode such vectors.
 | Format      | Binary values | Binary subtypes |
 |-------------|---------------|-----------------|
 | BJData      | not supported | not supported   |
+| BON8        | not supported | not supported   |
 | BSON        | supported     | supported       |
 | CBOR        | supported     | supported       |
 | MessagePack | supported     | supported       |
@@ -42,6 +45,7 @@ See [binary values](../binary_values.md) for more information.
 | BJData             | 53.2 %      | 91.1 %       | 78.1 %            | 96.6 %        |
 | BJData (size)      | 58.6 %      | 92.1 %       | 86.7 %            | 97.4 %        |
 | BJData (size+type) | 58.6 %      | 92.1 %       | 86.5 %            | 97.4 %        |
+| BON8               | 50.5 %      | 83.8 %       | 63.5 %            | 87.5 %        |
 | BSON               | 85.8 %      | 95.2 %       | 95.8 %            | 106.7 %       |
 | CBOR               | 50.5 %      | 86.3 %       | 68.4 %            | 88.0 %        |
 | MessagePack        | 50.5 %      | 86.0 %       | 68.5 %            | 87.9 %        |

docs/mkdocs/docs/features/binary_values.md

@@ -187,6 +187,41 @@ as an array of uint8 values. The library implements this translation.
     }
     ```
 
+### BON8
+
+[BON8](binary_formats/bon8.md) neither supports binary values nor subtypes. The library serializes binary values as an
+array of integers.
+
+??? example
+
+    Code:
+
+    ```cpp
+    // create a binary value of subtype 42 (will be ignored in BON8)
+    json j;
+    j["binary"] = json::binary({0xCA, 0xFE, 0xBA, 0xBE}, 42);
+
+    // convert to BON8
+    auto v = json::to_bon8(j);
+    ```
+
+    `v` is a `std::vector<std::uint8_t>` with the following 16 elements:
+
+    ```c
+    0x87                                     // object with 1 member
+        0x62 0x69 0x6E 0x61 0x72 0x79        // "binary"
+        0x84                                 // array with 4 elements
+            0xC3 0x22 0xC3 0x56 0xC3 0x12 0xC3 0x16  // content (each byte as a 2-byte integer)
+    ```
+
+    Note that the subtype is lost, and deserializing `v` would yield the following value:
+
+    ```json
+    {
+      "binary": [202, 254, 186, 190]
+    }
+    ```
+
 ### BSON
 
 [BSON](binary_formats/bson.md) supports binary values and subtypes. If a subtype is given, it is used and added as an

docs/mkdocs/docs/features/index.md

@@ -35,8 +35,8 @@ C++ types, and finally serialize it again.
 - [Serialization](serialization.md) — turn a value back into JSON text with [`dump`](../api/basic_json/dump.md),
   including pretty-printing and handling of non-ASCII and invalid UTF-8.
 - [Binary formats](binary_formats/index.md) — encode values more compactly as
-  [BJData](binary_formats/bjdata.md), [BSON](binary_formats/bson.md), [CBOR](binary_formats/cbor.md),
-  [MessagePack](binary_formats/messagepack.md), or [UBJSON](binary_formats/ubjson.md).
+  [BJData](binary_formats/bjdata.md), [BON8](binary_formats/bon8.md), [BSON](binary_formats/bson.md),
+  [CBOR](binary_formats/cbor.md), [MessagePack](binary_formats/messagepack.md), or [UBJSON](binary_formats/ubjson.md).
 - [Binary values](binary_values.md) — store and exchange raw byte sequences.
 
 ## How values are stored and configured

docs/mkdocs/docs/features/parsing/error_recovery.md

@@ -0,0 +1,94 @@
+# Error Recovery
+
+By default, parsing stops at the first error. With the [SAX interface](sax_interface.md), you can instead ask the
+parser to *recover*: to repair the error and continue, so that you get as much as possible out of malformed input, for
+instance a file that was cut off, JSON edited by hand, or the output of a language model.
+
+## Recovering from errors
+
+The SAX parser's [`parse_error`](../../api/json_sax/parse_error.md) function is called for every error. Its return value
+decides what happens next:
+
+- `#!cpp false` stops parsing. This is what the SAX parsers of the library do, so [`parse`](../../api/basic_json/parse.md)
+  and [`accept`](../../api/basic_json/accept.md) never recover.
+- `#!cpp true` repairs the error and continues parsing.
+
+When recovering, the SAX parser still receives well-formed events: every `start_object` or `start_array` is followed by
+the matching `end_object` or `end_array`, and every `key` is followed by exactly one value. A SAX parser that creates a
+JSON value, such as the one in the example below, therefore gets a complete value. Parsing always ends, and
+[`sax_parse`](../../api/basic_json/sax_parse.md) returns `#!cpp false` for input that is not valid JSON, even if every
+error was repaired. Each token is reported at most once, and the SAX parser can stop at any error by returning
+`#!cpp false`.
+
+!!! example
+
+    The example below derives a SAX parser from the library's parser for `json` values (`json_sax_dom_parser`),
+    and recovers from all errors.
+
+    ```cpp
+    --8<-- "examples/sax_parse__error_recovery.cpp"
+    ```
+    
+    Output:
+    
+    ```
+    --8<-- "examples/sax_parse__error_recovery.output"
+    ```
+
+## How errors are repaired
+
+Each error is repaired with the smallest local edit: a missing separator is inserted, a stray token is removed, what can
+be read of a broken string or number is kept, and a value that cannot be read at all becomes `#!json null`.
+
+| Mistake                   | Repair                                                                         | Example                                  | Result                     |
+|---------------------------|--------------------------------------------------------------------------------|------------------------------------------|----------------------------|
+| missing `,` or `:`        | inserted                                                                       | `#!json [1 2]`, `#!json {"a" 1}`         | `[1,2]`, `{"a":1}`         |
+| missing value             | `#!json null` for an object key or between commas in an array                  | `#!json {"a":}`, `#!json [1,,2]`         | `{"a":null}`, `[1,null,2]` |
+| trailing comma            | removed                                                                        | `#!json [1,2,]`                          | `[1,2]`                    |
+| broken string             | invalid escapes and bytes are replaced (see below); a line break ends the string | `#!json ["a\qb"]`                        | `["aqb"]`                  |
+| broken number             | the longest valid beginning is kept                                            | `#!json [1., 2e+]`                       | `[1,2]`                    |
+| unreadable value          | `#!json null`                                                                  | `#!json [1, NaN, tru]`                   | `[1,null,null]`            |
+| number too large          | passed as infinity, together with its text                                     | `#!json [1e999]`                         | infinity (see below)       |
+| stray `:`                 | removed                                                                        | `#!json ["a":1]`                         | `["a",1]`                  |
+| member without a key      | skipped up to the next `,` or `}`                                              | `#!json {1:2, "b":3}`                    | `{"b":3}`                  |
+| wrong closing bracket     | closes the innermost array or object                                           | `#!json {"a":[1,2}, "b":3}`              | `{"a":[1,2],"b":3}`        |
+| input ends too early      | all open arrays and objects are closed                                         | `#!json {"a":[1,2`                       | `{"a":[1,2]}`              |
+| text before the value     | skipped                                                                        | `#!json )]}'{"a":1}`                     | `{"a":1}`                  |
+
+In a string, an unknown escape like `\q` stands for the escaped character (`q`), as in JavaScript. An invalid `\u`
+escape, a lone surrogate, and ill-formed UTF-8 are each replaced by U+FFFD (REPLACEMENT CHARACTER), and control
+characters are kept. A string without its closing quote ends at the next line break or at the end of the input.
+
+The input after the top-level value is not repaired: as without recovery, it is reported as an error, and parsing stops.
+
+## Binary formats
+
+The binary formats ([BJData](../binary_formats/bjdata.md), [BON8](../binary_formats/bon8.md),
+[BSON](../binary_formats/bson.md), [CBOR](../binary_formats/cbor.md), [MessagePack](../binary_formats/messagepack.md),
+and [UBJSON](../binary_formats/ubjson.md)) cannot be repaired: a value's size is stored before its content, and every
+byte is a valid type marker, so after an error there is no way to tell where the next value begins. Parsing therefore
+always stops at the first error. If `parse_error` returns `#!cpp true`, the value read so far is completed before
+parsing stops: a key that waits for its value gets `#!json null`, and all open arrays and objects are closed. This keeps
+everything before the error of an input that was cut off.
+
+## Limitations
+
+- A repair is a guess. For example, `#!json {"a" "b": 1}` could be meant as `#!json {"a": "b"}` or as
+  `#!json {"a": null, "b": 1}`; it is repaired to the former. Treat recovered values as a best effort, and check the
+  reported errors.
+- A closing bracket always closes the innermost array or object. If a bracket is missing rather than wrong, the
+  repair differs from the intention: `#!json {"a": {"b": [1, 2}, "c": 3}` is repaired to
+  `#!json {"a": {"b": [1, 2], "c": 3}}`, although `#!json {"a": {"b": [1, 2]}, "c": 3}` may have been meant.
+- Keys without quotes, and strings in single quotes, are not supported; such members are skipped.
+- A number that is too large for `number_float_t` is passed as positive or negative infinity. The SAX parser's
+  `number_float` also gets the number's text, but a JSON value cannot store it, and
+  [`dump`](../../api/basic_json/dump.md) serializes infinity as `#!json null`.
+- When parsing is not strict (see [`sax_parse`](../../api/basic_json/sax_parse.md)), a repair may read parts of the
+  input after the value, for instance of the next value in a stream of concatenated values.
+
+## See also
+
+- [SAX interface](sax_interface.md) - implement a custom SAX handler
+- [`parse_error`](../../api/json_sax/parse_error.md) - the SAX event for parse errors
+- [`sax_parse`](../../api/basic_json/sax_parse.md) - generate SAX events
+- [parsing and exceptions](parse_exceptions.md) - control error handling

docs/mkdocs/docs/features/parsing/index.md

@@ -65,7 +65,7 @@ You can influence a DOM parse without switching to the SAX interface by passing
 When the input is not valid JSON, the `parse` function throws an exception by default. If exceptions are undesired or
 unavailable, the parser can instead return a discarded value, or [`accept`](../../api/basic_json/accept.md) can be used
 to only check whether an input is valid JSON. See [parsing and exceptions](parse_exceptions.md) for the available
-options.
+options. To get as much as possible out of malformed input, a SAX parser can [recover from errors](error_recovery.md).
 
 ## See also
 
@@ -76,3 +76,4 @@ options.
 - [parser callbacks](parser_callbacks.md) - influence the parsing by a callback function
 - [SAX interface](sax_interface.md) - implement a custom SAX handler
 - [parsing and exceptions](parse_exceptions.md) - control error handling
+- [error recovery](error_recovery.md) - get as much as possible out of malformed input

docs/mkdocs/docs/features/parsing/parse_exceptions.md

@@ -64,7 +64,8 @@ bool parse_error(std::size_t position,
                  const json::exception& ex);
 ```
 
-The return value indicates whether the parsing should continue, so the function should usually return `#!cpp false`.
+The return value decides whether to stop parsing (`#!cpp false`) or to repair the error and continue
+(`#!cpp true`); see [error recovery](error_recovery.md) for the latter.
 
 ??? example
 

docs/mkdocs/docs/features/parsing/sax_interface.md

@@ -60,7 +60,8 @@ bool key(string_t& val);
 bool parse_error(std::size_t position, const std::string& last_token, const json::exception& ex);
 ```
 
-The return value of each function determines whether parsing should proceed.
+The return value of each function determines whether parsing should proceed. For `parse_error`, returning
+`#!cpp true` [recovers from the error](error_recovery.md).
 
 To implement your own SAX handler, proceed as follows:
 
@@ -68,7 +69,7 @@ To implement your own SAX handler, proceed as follows:
 2. Create an object of your SAX interface class, e.g. `my_sax`.
 3. Call `#!cpp bool json::sax_parse(input, &my_sax);` where the first parameter can be any input like a string or an input stream and the second parameter is a pointer to your SAX interface.
 
-Note the `sax_parse` function only returns a `#!cpp bool` indicating the result of the last executed SAX event. It does not return `json` value - it is up to you to decide what to do with the SAX events. Furthermore, no exceptions are thrown in case of a parse error - it is up to you what to do with the exception object passed to your `parse_error` implementation. Internally, the SAX interface is used for the DOM parser (class `json_sax_dom_parser`) as well as the acceptor (`json_sax_acceptor`), see file `json_sax.hpp`.
+Note the `sax_parse` function only returns a `#!cpp bool` indicating whether the input was parsed without errors and no SAX event returned `#!cpp false`. It does not return `json` value - it is up to you to decide what to do with the SAX events. Furthermore, no exceptions are thrown in case of a parse error - it is up to you what to do with the exception object passed to your `parse_error` implementation. Internally, the SAX interface is used for the DOM parser (class `json_sax_dom_parser`) as well as the acceptor (`json_sax_acceptor`), see file `json_sax.hpp`.
 
 ## See also
 

docs/mkdocs/docs/features/serialization.md

@@ -117,7 +117,7 @@ For the [{fmt}](https://github.com/fmtlib/fmt) library, the library ships a
 ## Serializing to other formats
 
 Besides JSON text, a value can also be serialized to the more compact [binary formats](binary_formats/index.md)
-(BJData, BSON, CBOR, MessagePack, UBJSON).
+(BJData, BON8, BSON, CBOR, MessagePack, UBJSON).
 
 ## See also
 

docs/mkdocs/docs/features/types/template_parameters.md

@@ -547,7 +547,7 @@ Grisu2 algorithm, which produces the shortest representation that round-trips. O
 ### Required for the binary formats
 
 `NumberFloatType` must be `#!cpp float` or `#!cpp double`. The writers for
-[CBOR, MessagePack, UBJSON, BJData, and BSON](../binary_formats/index.md) map a floating-point value onto an IEEE 754
+[CBOR, MessagePack, UBJSON, BJData, BON8, and BSON](../binary_formats/index.md) map a floating-point value onto an IEEE 754
 binary32 or binary64 field and have no encoding for `#!cpp long double`.
 
 ### Compatible types

docs/mkdocs/includes/glossary.md

@@ -5,6 +5,7 @@
 *[ASCII]: American Standard Code for Information Interchange
 *[BDFL]: Benevolent Dictator for Life
 *[BJData]: Binary JData
+*[BON8]: Binary Object Notation 8
 *[BSON]: Binary JSON
 *[CBOR]: Concise Binary Object Representation
 *[CC0]: Creative Commons Zero

docs/mkdocs/mkdocs.yml

@@ -63,6 +63,7 @@ nav:
       - Binary Formats:
           - features/binary_formats/index.md
           - features/binary_formats/bjdata.md
+          - features/binary_formats/bon8.md
           - features/binary_formats/bson.md
           - features/binary_formats/cbor.md
           - features/binary_formats/messagepack.md
@@ -86,6 +87,7 @@ nav:
       - features/object_order.md
       - Parsing:
           - features/parsing/index.md
+          - features/parsing/error_recovery.md
           - features/parsing/json_lines.md
           - features/parsing/parse_exceptions.md
           - features/parsing/parser_callbacks.md
@@ -142,6 +144,7 @@ nav:
           - 'flatten': api/basic_json/flatten.md
           - 'format_as': api/basic_json/format_as.md
           - 'from_bjdata': api/basic_json/from_bjdata.md
+          - 'from_bon8': api/basic_json/from_bon8.md
           - 'from_bson': api/basic_json/from_bson.md
           - 'from_cbor': api/basic_json/from_cbor.md
           - 'from_msgpack': api/basic_json/from_msgpack.md
@@ -213,6 +216,7 @@ nav:
           - 'swap': api/basic_json/swap.md
           - 'std::swap<basic_json>': api/basic_json/std_swap.md
           - 'to_bjdata': api/basic_json/to_bjdata.md
+          - 'to_bon8': api/basic_json/to_bon8.md
           - 'to_bson': api/basic_json/to_bson.md
           - 'to_cbor': api/basic_json/to_cbor.md
           - 'to_msgpack': api/basic_json/to_msgpack.md
@@ -412,7 +416,7 @@ plugins:
       markdown_description: >
         JSON for Modern C++ is a C++11 header-only library implementing a JSON
         value type with an STL-like API, JSON Pointer/Patch, CBOR/MessagePack/
-        BSON/UBJSON/BJData binary format support, and a SAX-style parser interface.
+        BSON/UBJSON/BJData/BON8 binary format support, and a SAX-style parser interface.
       sections:
         Home:
           - index.md

include/nlohmann/detail/conversions/to_json.hpp

@@ -291,9 +291,7 @@ void to_json(BasicJsonType& j, const std::optional<T>& opt) noexcept
 {
     if (opt.has_value())
     {
-        // explicit construction, as the conversion from a basic_json with a different
-        // string type is explicit if JSON_USE_IMPLICIT_CONVERSIONS is 0 (#2649)
-        j = BasicJsonType(*opt);
+        j = *opt;
     }
     else
     {

include/nlohmann/detail/input/input_adapters.hpp

@@ -34,7 +34,7 @@ namespace detail
 {
 
 /// the supported input formats
-enum class input_format_t { json, cbor, msgpack, ubjson, bson, bjdata };
+enum class input_format_t { json, cbor, msgpack, ubjson, bson, bjdata, bon8 };
 
 ////////////////////
 // input adapters //

include/nlohmann/detail/input/json_sax.hpp

@@ -131,7 +131,9 @@ struct json_sax
     @param[in] position    the position in the input where the error occurs
     @param[in] last_token  the last read token
     @param[in] ex          an exception object describing the error
-    @return whether parsing should proceed (must return false)
+    @return whether to recover from the error: false stops parsing; true
+            repairs JSON text and continues, or, for the binary formats, stops
+            after closing the containers read so far
     */
     virtual bool parse_error(std::size_t position,
                              const std::string& last_token,
@@ -186,9 +188,12 @@ a pointer to the respective array or object for each recursion depth.
 After successful parsing, the value that is passed by reference to the
 constructor contains the parsed value.
 
-@tparam BasicJsonType  the JSON type
+@tparam BasicJsonType     the JSON type
+@tparam InputAdapterType  the input adapter of the lexer that can be passed to
+                          the constructor to record diagnostic positions; it
+                          does not matter if no lexer is passed
 */
-template<typename BasicJsonType, typename InputAdapterType>
+template<typename BasicJsonType, typename InputAdapterType = string_input_adapter_type>
 class json_sax_dom_parser
 {
   public:
@@ -505,7 +510,7 @@ class json_sax_dom_parser
     lexer_t* m_lexer_ref = nullptr;
 };
 
-template<typename BasicJsonType, typename InputAdapterType>
+template<typename BasicJsonType, typename InputAdapterType = string_input_adapter_type>
 class json_sax_dom_callback_parser
 {
   public:
@@ -1207,5 +1212,176 @@ class json_sax_acceptor
     }
 };
 
+/*!
+@brief SAX proxy that lets the binary readers keep what was read before an error
+
+The binary formats cannot continue after an error: a value's size is given
+before its payload, and every byte value is a valid type marker, so there is no
+way to find where the next value begins. When the SAX parser's parse_error()
+returns true to ask for error recovery, the best the binary readers can offer is
+the value read up to the error.
+
+This proxy forwards every event to the SAX parser and records which containers
+are open and whether a key still waits for its value. After an error the SAX
+parser asked to recover from, @ref close_open_containers then completes the
+value with null for a pending key and the missing end events, so the SAX parser
+sees balanced events (see #3989).
+
+@tparam BasicJsonType  the JSON type
+@tparam SAX            the SAX parser to forward the events to
+*/
+template<typename BasicJsonType, typename SAX>
+class json_sax_salvager
+{
+  public:
+    using number_integer_t = typename BasicJsonType::number_integer_t;
+    using number_unsigned_t = typename BasicJsonType::number_unsigned_t;
+    using number_float_t = typename BasicJsonType::number_float_t;
+    using string_t = typename BasicJsonType::string_t;
+    using binary_t = typename BasicJsonType::binary_t;
+
+    explicit json_sax_salvager(SAX* sax_) noexcept
+        : sax(sax_)
+    {}
+
+    bool null()
+    {
+        key_pending = false;
+        return sax->null();
+    }
+
+    bool boolean(bool val)
+    {
+        key_pending = false;
+        return sax->boolean(val);
+    }
+
+    bool number_integer(number_integer_t val)
+    {
+        key_pending = false;
+        return sax->number_integer(val);
+    }
+
+    bool number_unsigned(number_unsigned_t val)
+    {
+        key_pending = false;
+        return sax->number_unsigned(val);
+    }
+
+    bool number_float(number_float_t val, const string_t& s)
+    {
+        key_pending = false;
+        return sax->number_float(val, s);
+    }
+
+    bool string(string_t& val)
+    {
+        key_pending = false;
+        return sax->string(val);
+    }
+
+    bool binary(binary_t& val)
+    {
+        key_pending = false;
+        return sax->binary(val);
+    }
+
+    bool start_object(std::size_t len)
+    {
+        key_pending = false;
+        if (JSON_HEDLEY_UNLIKELY(!sax->start_object(len)))
+        {
+            return false;
+        }
+        open_containers.push_back(true);
+        return true;
+    }
+
+    bool key(string_t& val)
+    {
+        key_pending = true;
+        return sax->key(val);
+    }
+
+    bool end_object()
+    {
+        JSON_ASSERT(!open_containers.empty() && open_containers.back());
+        open_containers.pop_back();
+        return sax->end_object();
+    }
+
+    bool start_array(std::size_t len)
+    {
+        key_pending = false;
+        if (JSON_HEDLEY_UNLIKELY(!sax->start_array(len)))
+        {
+            return false;
+        }
+        open_containers.push_back(false);
+        return true;
+    }
+
+    bool end_array()
+    {
+        JSON_ASSERT(!open_containers.empty() && !open_containers.back());
+        open_containers.pop_back();
+        return sax->end_array();
+    }
+
+    template<class Exception>
+    bool parse_error(std::size_t position, const std::string& last_token,
+                     const Exception& ex)
+    {
+        recovery_requested = sax->parse_error(position, last_token, ex);
+        // the binary readers stop after an error anyway
+        return false;
+    }
+
+    /*!
+    @brief complete the value read before an error
+
+    Does nothing unless the SAX parser's parse_error() returned true. Otherwise
+    passes null for a key that waits for its value and closes the containers
+    that are still open, innermost first, until an event returns false.
+    */
+    void close_open_containers()
+    {
+        if (!recovery_requested)
+        {
+            return;
+        }
+        recovery_requested = false;
+
+        if (key_pending)
+        {
+            key_pending = false;
+            if (JSON_HEDLEY_UNLIKELY(!sax->null()))
+            {
+                return;
+            }
+        }
+
+        while (!open_containers.empty())
+        {
+            const bool is_object = open_containers.back();
+            open_containers.pop_back();
+            if (JSON_HEDLEY_UNLIKELY(is_object ? !sax->end_object() : !sax->end_array()))
+            {
+                return;
+            }
+        }
+    }
+
+  private:
+    /// the SAX parser the events are forwarded to
+    SAX* sax = nullptr;
+    /// the containers that are open, innermost last; true for an object
+    std::vector<bool> open_containers {}; // NOLINT(readability-redundant-member-init)
+    /// whether a key was passed whose value has not been passed yet
+    bool key_pending = false;
+    /// whether the SAX parser's parse_error() asked to recover from the error
+    bool recovery_requested = false;
+};
+
 }  // namespace detail
 NLOHMANN_JSON_NAMESPACE_END

include/nlohmann/detail/input/lexer.hpp

@@ -11,6 +11,7 @@
 #include <array> // array
 #include <clocale> // localeconv
 #include <cstddef> // size_t
+#include <cstdint> // uint8_t
 #include <cstdio> // snprintf
 #include <cstdlib> // strtof, strtod, strtold, strtoll, strtoull
 #include <initializer_list> // initializer_list
@@ -453,8 +454,16 @@ class lexer : public lexer_base<BasicJsonType>
                             if (0xD800 <= codepoint1 && codepoint1 <= 0xDBFF)
                             {
                                 // expect next \uxxxx entry
-                                if (JSON_HEDLEY_LIKELY(get() == '\\' && get() == 'u'))
+                                if (JSON_HEDLEY_LIKELY(get() == '\\'))
                                 {
+                                    if (JSON_HEDLEY_UNLIKELY(get() != 'u'))
+                                    {
+                                        // current is the character escaped by the backslash
+                                        error_message = "invalid string: surrogate U+D800..U+DBFF must be followed by U+DC00..U+DFFF";
+                                        string_error_resume = resume_kind::escaped_character;
+                                        return token_type::parse_error;
+                                    }
+
                                     const int codepoint2 = get_codepoint();
 
                                     if (JSON_HEDLEY_UNLIKELY(codepoint2 == -1))
@@ -479,7 +488,11 @@ class lexer : public lexer_base<BasicJsonType>
                                     }
                                     else
                                     {
+                                        // the second escape was read completely and is a
+                                        // code point of its own
                                         error_message = "invalid string: surrogate U+D800..U+DBFF must be followed by U+DC00..U+DFFF";
+                                        string_error_resume = resume_kind::after_escape;
+                                        string_error_codepoint = codepoint2;
                                         return token_type::parse_error;
                                     }
                                 }
@@ -493,7 +506,9 @@ class lexer : public lexer_base<BasicJsonType>
                             {
                                 if (JSON_HEDLEY_UNLIKELY(0xDC00 <= codepoint1 && codepoint1 <= 0xDFFF))
                                 {
+                                    // the escape was read completely
                                     error_message = "invalid string: surrogate U+DC00..U+DFFF must follow U+D800..U+DBFF";
+                                    string_error_resume = resume_kind::after_escape;
                                     return token_type::parse_error;
                                 }
                             }
@@ -2238,6 +2253,552 @@ scan_number_done:
         }
     }
 
+    /////////////////////
+    // error recovery
+    /////////////////////
+
+    /*!
+    @brief make the best of the token that scan() rejected
+
+    Called by the parser after scan() returned token_type::parse_error and the
+    SAX parser asked to recover from the error (see #3989). Keeps what can be
+    read of the token and skips the rest:
+
+    - A string keeps its characters. An unknown escape stands for the escaped
+      character itself (as in JavaScript), an invalid `\u` escape and ill-formed
+      UTF-8 become U+FFFD, and a control character is kept. A line break or the
+      end of the input ends a string that lacks its closing quote.
+    - A number keeps its longest valid prefix, e.g. `1` for `1.` or `1e+`.
+    - A block comment that is not closed runs to the end of the input.
+    - Anything else is skipped.
+
+    The rest of an invalid token is skipped up to the next delimiter
+    (whitespace, a structural character, or a quote). A delimiter that the
+    invalid token consumed is returned to the input, so that the next scan()
+    reads it.
+
+    @return token_type::value_string or a number token type if a string or a
+            number could be read, token_type::end_of_input for a block comment
+            that is not closed, token_type::uninitialized otherwise
+    */
+    token_type recover_token()
+    {
+        const resume_kind resume = string_error_resume;
+        const int codepoint = string_error_codepoint;
+        string_error_resume = resume_kind::character;
+        string_error_codepoint = -1;
+
+        if (error_message_starts_with("invalid string"))
+        {
+            return recover_string(resume, codepoint);
+        }
+
+        if (error_message_starts_with("invalid number"))
+        {
+            return recover_number();
+        }
+
+        if (error_message_starts_with("invalid comment; missing"))
+        {
+            // the comment runs to the end of the input
+            return token_type::end_of_input;
+        }
+
+        skip_to_delimiter();
+        return token_type::uninitialized;
+    }
+
+    /*!
+    @brief return the token that scan() read last to the input, so that the
+           next scan() reads it again
+
+    Called by the parser when recovering from an error. The token must be a
+    single character (',', ':', '[', ']', '{', or '}') or the end of the
+    input, and scan() must have read it last.
+    */
+    void unget_token()
+    {
+        JSON_ASSERT(!next_unget);
+        unget();
+    }
+
+    /*!
+    @brief let the token string for the next error begin at the current character
+
+    The token string of an error reaches back to the beginning of the last
+    string or number. After an error, the parser calls this function so that
+    the next error does not report (and, with many errors, copy) everything
+    read since then.
+    */
+    void restart_token_string()
+    {
+        restart_token_string_impl(std::integral_constant<bool, lazy_token_string> {});
+    }
+
+  private:
+    /// how recover_string() continues after the error scan_string() reported
+    enum class resume_kind : std::uint8_t
+    {
+        /// current is the next character of the string (or the end of input)
+        character,
+        /// current is the character escaped by the preceding backslash
+        escaped_character,
+        /// current is the last character of a complete escape
+        after_escape
+    };
+
+    /// whether error_message begins with @a prefix
+    bool error_message_starts_with(const char* prefix) const noexcept
+    {
+        const char* message = error_message;
+        while (*prefix != '\0')
+        {
+            if (*message++ != *prefix++)
+            {
+                return false;
+            }
+        }
+        return true;
+    }
+
+    /// whether current ends an invalid token (see recover_token())
+    bool current_is_delimiter() const noexcept
+    {
+        switch (current)
+        {
+            case ' ':
+            case '\t':
+            case '\n':
+            case '\r':
+            case '[':
+            case ']':
+            case '{':
+            case '}':
+            case ',':
+            case ':':
+            case '\"':
+#if !JSON_STRICT_NUL_HANDLING
+            case '\0':
+#endif
+            case char_traits<char_type>::eof():
+                return true;
+
+            case '/':
+                return ignore_comments;
+
+            default:
+                return false;
+        }
+    }
+
+    /// skip the rest of an invalid token and return its delimiter to the input
+    void skip_to_delimiter()
+    {
+        while (!current_is_delimiter())
+        {
+            get();
+        }
+
+        if (current != char_traits<char_type>::eof())
+        {
+            unget();
+        }
+    }
+
+    /// append U+FFFD REPLACEMENT CHARACTER to token_buffer
+    void add_replacement_character()
+    {
+        add(0xEF);
+        add(0xBF);
+        add(0xBD);
+    }
+
+    /// append the UTF-8 encoding of @a codepoint (not a surrogate) to token_buffer
+    void add_codepoint(const int codepoint)
+    {
+        JSON_ASSERT(0x00 <= codepoint && codepoint <= 0x10FFFF);
+        const auto cp = static_cast<unsigned int>(codepoint);
+        if (cp < 0x80)
+        {
+            add(static_cast<char_int_type>(cp));
+        }
+        else if (cp <= 0x7FF)
+        {
+            add(static_cast<char_int_type>(0xC0u | (cp >> 6u)));
+            add(static_cast<char_int_type>(0x80u | (cp & 0x3Fu)));
+        }
+        else if (cp <= 0xFFFF)
+        {
+            add(static_cast<char_int_type>(0xE0u | (cp >> 12u)));
+            add(static_cast<char_int_type>(0x80u | ((cp >> 6u) & 0x3Fu)));
+            add(static_cast<char_int_type>(0x80u | (cp & 0x3Fu)));
+        }
+        else
+        {
+            add(static_cast<char_int_type>(0xF0u | (cp >> 18u)));
+            add(static_cast<char_int_type>(0x80u | ((cp >> 12u) & 0x3Fu)));
+            add(static_cast<char_int_type>(0x80u | ((cp >> 6u) & 0x3Fu)));
+            add(static_cast<char_int_type>(0x80u | (cp & 0x3Fu)));
+        }
+    }
+
+    /// append a code point read from a `\u` escape; a surrogate becomes U+FFFD
+    void add_escaped_codepoint(const int codepoint)
+    {
+        if (0xD800 <= codepoint && codepoint <= 0xDFFF)
+        {
+            add_replacement_character();
+        }
+        else
+        {
+            add_codepoint(codepoint);
+        }
+    }
+
+    /*!
+    @brief remove an incomplete UTF-8 sequence from the end of token_buffer
+
+    next_byte_in_range() adds the bytes of a sequence as it checks them, so
+    when it rejects a byte, the beginning of the sequence is already in
+    token_buffer, which otherwise holds only complete sequences.
+
+    @return whether an incomplete sequence was removed
+    */
+    bool remove_incomplete_utf8_sequence()
+    {
+        std::size_t lead = token_buffer.size();
+        std::size_t continuation_bytes = 0;
+        while (lead > 0 && continuation_bytes < 3
+                && (static_cast<unsigned char>(token_buffer[lead - 1]) & 0xC0u) == 0x80u)
+        {
+            --lead;
+            ++continuation_bytes;
+        }
+        if (lead == 0)
+        {
+            return false;
+        }
+
+        const auto lead_byte = static_cast<unsigned char>(token_buffer[lead - 1]);
+        const std::size_t expected = (lead_byte >= 0xF0) ? 3 : (lead_byte >= 0xE0) ? 2 : (lead_byte >= 0xC0) ? 1 : 0;
+        if (continuation_bytes >= expected)
+        {
+            return false;
+        }
+
+        token_buffer.resize(lead - 1);
+        return true;
+    }
+
+    /*!
+    @brief read the UTF-8 sequence that begins with current, which is not ASCII
+    @return whether the next character must be read; false if current still
+            needs to be handled, because it does not belong to the sequence
+    */
+    bool recover_utf8_sequence()
+    {
+        // the number of continuation bytes and the range of the first one;
+        // see the ranges in scan_string()
+        std::size_t count = 0;
+        char_int_type low = 0x80;
+        char_int_type high = 0xBF;
+        if (current >= 0xC2 && current <= 0xDF)
+        {
+            count = 1;
+        }
+        else if (current >= 0xE0 && current <= 0xEF)
+        {
+            count = 2;
+            low = (current == 0xE0) ? 0xA0 : 0x80;
+            high = (current == 0xED) ? 0x9F : 0xBF;
+        }
+        else if (current >= 0xF0 && current <= 0xF4)
+        {
+            count = 3;
+            low = (current == 0xF0) ? 0x90 : 0x80;
+            high = (current == 0xF4) ? 0x8F : 0xBF;
+        }
+        else
+        {
+            // an ill-formed byte
+            add_replacement_character();
+            return true;
+        }
+
+        const std::size_t start = token_buffer.size();
+        add(current);
+        for (std::size_t i = 0; i < count; ++i)
+        {
+            get();
+            if (current < low || current > high)
+            {
+                token_buffer.resize(start);
+                add_replacement_character();
+                return false;
+            }
+            add(current);
+            low = 0x80;
+            high = 0xBF;
+        }
+        return true;
+    }
+
+    /*!
+    @brief read the low surrogate that must follow the high surrogate @a high
+    @return whether the next character must be read; false if current still
+            needs to be handled
+    */
+    bool recover_low_surrogate(int high)
+    {
+        while (true)
+        {
+            if (get() != '\\')
+            {
+                add_replacement_character();
+                return false;
+            }
+            if (get() != 'u')
+            {
+                add_replacement_character();
+                // not 'u', so this does not come back here
+                return recover_escape();
+            }
+
+            const int low = get_codepoint();
+            if (low == -1)
+            {
+                add_replacement_character();
+                return false;
+            }
+            if (0xDC00 <= low && low <= 0xDFFF)
+            {
+                add_codepoint(static_cast<int>((static_cast<unsigned int>(high) << 10u)
+                                               + static_cast<unsigned int>(low) - 0x35FDC00u));
+                return true;
+            }
+
+            // high has no low surrogate
+            add_replacement_character();
+            if (low < 0xD800 || low > 0xDBFF)
+            {
+                add_codepoint(low);
+                return true;
+            }
+            // another high surrogate
+            high = low;
+        }
+    }
+
+    /*!
+    @brief read the escape whose backslash was read; current is the escaped character
+    @return whether the next character must be read; false if current still
+            needs to be handled
+    */
+    bool recover_escape()
+    {
+        switch (current)
+        {
+            case '\"':
+                add('\"');
+                return true;
+            case '\\':
+                add('\\');
+                return true;
+            case '/':
+                add('/');
+                return true;
+            case 'b':
+                add('\b');
+                return true;
+            case 'f':
+                add('\f');
+                return true;
+            case 'n':
+                add('\n');
+                return true;
+            case 'r':
+                add('\r');
+                return true;
+            case 't':
+                add('\t');
+                return true;
+
+            case 'u':
+            {
+                const int codepoint = get_codepoint();
+                if (codepoint == -1)
+                {
+                    add_replacement_character();
+                    return false;
+                }
+                if (0xD800 <= codepoint && codepoint <= 0xDBFF)
+                {
+                    return recover_low_surrogate(codepoint);
+                }
+                add_escaped_codepoint(codepoint);
+                return true;
+            }
+
+            // an unknown escape stands for the escaped character
+            default:
+                return false;
+        }
+    }
+
+    /*!
+    @brief read the rest of a string after scan_string() rejected it
+
+    token_buffer holds what scan_string() read before the error. See
+    recover_token() for how errors are repaired.
+
+    @param[in] resume     how to continue, see resume_kind
+    @param[in] codepoint  for a high surrogate followed by an escape of another
+                          code point: that code point; -1 otherwise
+    */
+    token_type recover_string(const resume_kind resume, const int codepoint)
+    {
+        // whether the next character must be read before it can be handled
+        bool fetch = false;
+
+        if (error_message_starts_with("invalid string: surrogate")
+                || error_message_starts_with("invalid string: '\\u'"))
+        {
+            add_replacement_character();
+        }
+        else if (error_message_starts_with("invalid string: ill-formed UTF-8")
+                 && remove_incomplete_utf8_sequence())
+        {
+            add_replacement_character();
+        }
+
+        switch (resume)
+        {
+            case resume_kind::escaped_character:
+                fetch = recover_escape();
+                break;
+            case resume_kind::after_escape:
+                if (0xD800 <= codepoint && codepoint <= 0xDBFF)
+                {
+                    fetch = recover_low_surrogate(codepoint);
+                }
+                else
+                {
+                    if (codepoint != -1)
+                    {
+                        add_escaped_codepoint(codepoint);
+                    }
+                    fetch = true;
+                }
+                break;
+            case resume_kind::character:
+            default:
+                break;
+        }
+
+        while (true)
+        {
+            if (fetch)
+            {
+                get();
+            }
+            fetch = true;
+
+            switch (current)
+            {
+                case '\"':
+                // a line break or the end of the input ends a string that
+                // lacks its closing quote
+                case '\n':
+                case '\r':
+                case char_traits<char_type>::eof():
+                    return token_type::value_string;
+
+#if !JSON_STRICT_NUL_HANDLING
+                case '\0':
+                    // the end of the input, see scan()
+                    unget();
+                    return token_type::value_string;
+#endif
+
+                case '\\':
+                    get();
+                    fetch = recover_escape();
+                    break;
+
+                default:
+                    if (current < 0x80)
+                    {
+                        // including control characters
+                        add(current);
+                    }
+                    else
+                    {
+                        fetch = recover_utf8_sequence();
+                    }
+                    break;
+            }
+        }
+    }
+
+    /*!
+    @brief keep the longest valid prefix of a number that scan_number() rejected
+
+    token_buffer holds the characters scan_number() accepted before the error,
+    so the prefix ends at its last digit.
+    */
+    token_type recover_number()
+    {
+        while (!token_buffer.empty() && (token_buffer.back() < '0' || token_buffer.back() > '9'))
+        {
+            token_buffer.pop_back();
+        }
+
+        if (token_buffer.empty())
+        {
+            skip_to_delimiter();
+            return token_type::uninitialized;
+        }
+
+        if (decimal_point_position >= token_buffer.size())
+        {
+            decimal_point_position = std::string::npos;
+        }
+
+        const std::size_t exponent = token_buffer.find_first_of("eE");
+        const std::size_t mantissa_end = (exponent == std::string::npos) ? token_buffer.size() : exponent;
+        token_type number_type = token_type::value_unsigned;
+        if (decimal_point_position != std::string::npos || exponent != std::string::npos)
+        {
+            number_type = token_type::value_float;
+        }
+        else if (token_buffer.front() == '-')
+        {
+            number_type = token_type::value_integer;
+        }
+
+        const token_type result = convert_number(number_type, mantissa_end);
+        skip_to_delimiter();
+        return result;
+    }
+
+    /// seekable adapter: the token string begins at current, which was consumed
+    void restart_token_string_impl(std::true_type /*lazy*/) noexcept
+    {
+        const std::size_t consumed = ia.get_consumed_count();
+        token_string_start = (consumed > 0 && current != char_traits<char_type>::eof()) ? consumed - 1 : consumed;
+    }
+
+    /// streaming adapter: the token string begins at current; a character
+    /// that was put back is copied again when it is read again
+    void restart_token_string_impl(std::false_type /*lazy*/)
+    {
+        token_string.clear();
+        if (!next_unget && current != char_traits<char_type>::eof())
+        {
+            token_string.push_back(char_traits<char_type>::to_char_type(current));
+        }
+    }
+
   private:
     /// input adapter
     InputAdapterType ia;
@@ -2278,6 +2839,13 @@ scan_number_done:
     /// a description of occurred lexer errors
     const char* error_message = "";
 
+    /// how recover_token() continues a string that scan_string() rejected;
+    /// set only on the error paths that need more than error_message
+    resume_kind string_error_resume = resume_kind::character;
+    /// the code point of the second escape when a high surrogate is followed
+    /// by an escape that is not a low surrogate; -1 otherwise
+    int string_error_codepoint = -1;
+
     // number values
     number_integer_t value_integer = 0;
     number_unsigned_t value_unsigned = 0;

include/nlohmann/detail/input/parser.hpp

@@ -98,7 +98,7 @@ class parser
         if (callback)
         {
             json_sax_dom_callback_parser<BasicJsonType, InputAdapterType> sdp(result, callback, allow_exceptions, &m_lexer);
-            sax_parse_internal(&sdp);
+            sax_parse_internal<false>(&sdp);
 
             if (strict)
             {
@@ -135,7 +135,7 @@ class parser
         else
         {
             json_sax_dom_parser<BasicJsonType, InputAdapterType> sdp(result, allow_exceptions, &m_lexer);
-            sax_parse_internal(&sdp);
+            sax_parse_internal<false>(&sdp);
 
             if (strict)
             {
@@ -173,26 +173,59 @@ class parser
     bool accept(const bool strict = true)
     {
         json_sax_acceptor<BasicJsonType> sax_acceptor;
-        return sax_parse(&sax_acceptor, strict);
+        return sax_parse_impl<false>(&sax_acceptor, strict);
     }
 
+    /*!
+    @brief public SAX interface
+
+    If the SAX parser's parse_error() returns true, the parser recovers from
+    the error: it repairs the input and continues (see #3989).
+
+    @param[in] sax     the SAX parser
+    @param[in] strict  whether to expect the last token to be EOF
+    @return whether the input was parsed without errors and no SAX event
+            returned false
+    */
     template<typename SAX>
     JSON_HEDLEY_NON_NULL(2)
     bool sax_parse(SAX* sax, const bool strict = true)
+    {
+        return sax_parse_impl<true>(sax, strict);
+    }
+
+  private:
+    /// what sax_parse_internal() does after an object key was expected
+    enum class next_step : std::uint8_t
+    {
+        /// stop parsing
+        stop,
+        /// parse a value that begins with last_token
+        parse_value,
+        /// evaluate the state of the innermost container, which reads
+        /// last_token again
+        evaluate_state
+    };
+
+    template<bool AllowRecovery, typename SAX>
+    JSON_HEDLEY_NON_NULL(2)
+    bool sax_parse_impl(SAX* sax, const bool strict)
     {
         (void)detail::is_sax_static_asserts<SAX, BasicJsonType> {};
-        const bool result = sax_parse_internal(sax);
+        const bool result = sax_parse_internal<AllowRecovery>(sax);
 
         if (result)
         {
             if (strict)
             {
-                // strict mode: next byte must be EOF
-                if (get_token() != token_type::end_of_input)
+                // strict mode: next byte must be EOF; after recovering from an
+                // error, the end of the input may already have been read
+                if (last_token != token_type::end_of_input && get_token() != token_type::end_of_input)
                 {
-                    return sax->parse_error(m_lexer.get_position(),
-                                            m_lexer.get_token_string(),
-                                            parse_error::create(101, m_lexer.get_position(), exception_message(token_type::end_of_input, "value"), nullptr));
+                    // the value is complete, so there is nothing to recover
+                    static_cast<void>(report_error(sax, parse_error::create(101, m_lexer.get_position(), exception_message(token_type::end_of_input, "value"), nullptr),
+                                                   std::integral_constant<bool, AllowRecovery> {}));
+                    return false;
                 }
             }
             else
@@ -203,14 +236,23 @@ class parser
             }
         }
 
-        return result;
+        return result && !error_reported;
     }
 
-  private:
-    template<typename SAX>
+    /*!
+    @brief parse a JSON value and pass it to a SAX parser
+
+    @tparam AllowRecovery  whether to recover from an error if the SAX parser's
+                           parse_error() returns true; false for the SAX parsers
+                           of parse() and accept(), which never do, so that no
+                           code for recovering is generated for them
+    */
+    template<bool AllowRecovery, typename SAX>
     JSON_HEDLEY_NON_NULL(2)
     bool sax_parse_internal(SAX* sax)
     {
+        const std::integral_constant<bool, AllowRecovery> allow_recovery{};
+
         // stack to remember the hierarchy of structured values we are parsing
         // true = array; false = object
         std::vector<bool> states;
@@ -241,12 +283,18 @@ class parser
                             break;
                         }
 
-                        // parse key
+                        // remember we are now inside an object
+                        states.push_back(false);
+
+                        // parse key (the steps of parse_key(), which are
+                        // repeated here and below for speed)
                         if (JSON_HEDLEY_UNLIKELY(last_token != token_type::value_string))
                         {
-                            return sax->parse_error(m_lexer.get_position(),
-                                                    m_lexer.get_token_string(),
-                                                    parse_error::create(101, m_lexer.get_position(), exception_message(token_type::value_string, "object key"), nullptr));
+                            if (!continue_after(key_error(sax, allow_recovery, false), skip_to_state_evaluation))
+                            {
+                                return false;
+                            }
+                            continue;
                         }
                         if (JSON_HEDLEY_UNLIKELY(!sax->key(m_lexer.get_string())))
                         {
@@ -256,14 +304,13 @@ class parser
                         // parse separator (:)
                         if (JSON_HEDLEY_UNLIKELY(get_token() != token_type::name_separator))
                         {
-                            return sax->parse_error(m_lexer.get_position(),
-                                                    m_lexer.get_token_string(),
-                                                    parse_error::create(101, m_lexer.get_position(), exception_message(token_type::name_separator, "object separator"), nullptr));
+                            if (!continue_after(key_error(sax, allow_recovery, true), skip_to_state_evaluation))
+                            {
+                                return false;
+                            }
+                            continue;
                         }
 
-                        // remember we are now inside an object
-                        states.push_back(false);
-
                         // parse values
                         get_token();
                         continue;
@@ -299,9 +346,11 @@ class parser
 
                         if (JSON_HEDLEY_UNLIKELY(!std::isfinite(res)))
                         {
-                            return sax->parse_error(m_lexer.get_position(),
-                                                    m_lexer.get_token_string(),
-                                                    out_of_range::create(406, concat("number overflow parsing '", m_lexer.get_token_string(), '\''), nullptr));
+                            if (!overflow_error(sax, res, allow_recovery))
+                            {
+                                return false;
+                            }
+                            break;
                         }
 
                         if (JSON_HEDLEY_UNLIKELY(!sax->number_float(res, m_lexer.get_string())))
@@ -369,23 +418,63 @@ class parser
                     case token_type::parse_error:
                     {
                         // using "uninitialized" to avoid an "expected" message
-                        return sax->parse_error(m_lexer.get_position(),
-                                                m_lexer.get_token_string(),
-                                                parse_error::create(101, m_lexer.get_position(), exception_message(token_type::uninitialized, "value"), nullptr));
+                        if (!report_error(sax, parse_error::create(101, m_lexer.get_position(), exception_message(token_type::uninitialized, "value"), nullptr), allow_recovery))
+                        {
+                            return false;
+                        }
+
+                        // recover: keep what can be read of the token
+                        recover_token();
+                        if (last_token != token_type::uninitialized)
+                        {
+                            // a string or a number
+                            continue;
+                        }
+                        if (states.empty())
+                        {
+                            // look for the value after the garbage
+                            if (!skip_to_value())
+                            {
+                                return false;
+                            }
+                            continue;
+                        }
+                        // nothing could be read
+                        if (JSON_HEDLEY_UNLIKELY(!sax->null()))
+                        {
+                            return false;
+                        }
+                        break;
                     }
                     case token_type::end_of_input:
                     {
                         if (JSON_HEDLEY_UNLIKELY(m_lexer.get_position().chars_read_total == 1))
                         {
-                            return sax->parse_error(m_lexer.get_position(),
-                                                    m_lexer.get_token_string(),
-                                                    parse_error::create(101, m_lexer.get_position(),
-                                                            "attempting to parse an empty input; check that your input string or stream contains the expected JSON", nullptr));
+                            // there is nothing to recover
+                            static_cast<void>(report_error(sax, parse_error::create(101, m_lexer.get_position(),
+                                                           "attempting to parse an empty input; check that your input string or stream contains the expected JSON", nullptr), allow_recovery));
+                            return false;
                         }
 
-                        return sax->parse_error(m_lexer.get_position(),
-                                                m_lexer.get_token_string(),
-                                                parse_error::create(101, m_lexer.get_position(), exception_message(token_type::literal_or_value, "value"), nullptr));
+                        if (!report_error(sax, parse_error::create(101, m_lexer.get_position(), exception_message(token_type::literal_or_value, "value"), nullptr), allow_recovery))
+                        {
+                            return false;
+                        }
+
+                        // recover: the input ends where a value is missing
+                        if (states.empty())
+                        {
+                            // there is no value
+                            return false;
+                        }
+                        if (!recover_missing_value(sax, states))
+                        {
+                            return false;
+                        }
+                        // the state evaluation reads the token again
+                        m_lexer.unget_token();
+                        skip_to_state_evaluation = true;
+                        continue;
                     }
                     case token_type::uninitialized:
                     case token_type::end_array:
@@ -395,9 +484,35 @@ class parser
                     case token_type::literal_or_value:
                     default: // the last token was unexpected
                     {
-                        return sax->parse_error(m_lexer.get_position(),
-                                                m_lexer.get_token_string(),
-                                                parse_error::create(101, m_lexer.get_position(), exception_message(token_type::literal_or_value, "value"), nullptr));
+                        if (!report_error(sax, parse_error::create(101, m_lexer.get_position(), exception_message(token_type::literal_or_value, "value"), nullptr), allow_recovery))
+                        {
+                            return false;
+                        }
+
+                        // recover
+                        if (states.empty())
+                        {
+                            // look for the value after the garbage
+                            if (!skip_to_value())
+                            {
+                                return false;
+                            }
+                            continue;
+                        }
+                        if (last_token == token_type::name_separator)
+                        {
+                            // a stray ':'; the value may follow
+                            get_token();
+                            continue;
+                        }
+                        if (!recover_missing_value(sax, states))
+                        {
+                            return false;
+                        }
+                        // the state evaluation reads the token again
+                        m_lexer.unget_token();
+                        skip_to_state_evaluation = true;
+                        continue;
                     }
                 }
             }
@@ -447,9 +562,30 @@ class parser
                     continue;
                 }
 
-                return sax->parse_error(m_lexer.get_position(),
-                                        m_lexer.get_token_string(),
-                                        parse_error::create(101, m_lexer.get_position(), exception_message(token_type::end_array, "array"), nullptr));
+                if (!report_error(sax, parse_error::create(101, m_lexer.get_position(), exception_message(token_type::end_array, "array"), nullptr), allow_recovery))
+                {
+                    return false;
+                }
+
+                // recover
+                if (last_token == token_type::end_of_input)
+                {
+                    // the input ends inside the array
+                    return close_containers(sax, states);
+                }
+                if (last_token == token_type::end_object)
+                {
+                    // a wrong closing bracket closes the innermost container
+                    if (JSON_HEDLEY_UNLIKELY(!sax->end_array()))
+                    {
+                        return false;
+                    }
+                    states.pop_back();
+                    skip_to_state_evaluation = true;
+                }
+                // otherwise, a missing ',' (or a stray ':', which value
+                // parsing drops): the next value begins here
+                continue;
             }
 
             // states.back() is false -> object
@@ -466,11 +602,12 @@ class parser
                     // parse key
                     if (JSON_HEDLEY_UNLIKELY(last_token != token_type::value_string))
                     {
-                        return sax->parse_error(m_lexer.get_position(),
-                                                m_lexer.get_token_string(),
-                                                parse_error::create(101, m_lexer.get_position(), exception_message(token_type::value_string, "object key"), nullptr));
+                        if (!continue_after(key_error(sax, allow_recovery, false), skip_to_state_evaluation))
+                        {
+                            return false;
+                        }
+                        continue;
                     }
-
                     if (JSON_HEDLEY_UNLIKELY(!sax->key(m_lexer.get_string())))
                     {
                         return false;
@@ -479,9 +616,11 @@ class parser
                     // parse separator (:)
                     if (JSON_HEDLEY_UNLIKELY(get_token() != token_type::name_separator))
                     {
-                        return sax->parse_error(m_lexer.get_position(),
-                                                m_lexer.get_token_string(),
-                                                parse_error::create(101, m_lexer.get_position(), exception_message(token_type::name_separator, "object separator"), nullptr));
+                        if (!continue_after(key_error(sax, allow_recovery, true), skip_to_state_evaluation))
+                        {
+                            return false;
+                        }
+                        continue;
                     }
 
                     // parse values
@@ -508,12 +647,479 @@ class parser
                 continue;
             }
 
-            return sax->parse_error(m_lexer.get_position(),
-                                    m_lexer.get_token_string(),
-                                    parse_error::create(101, m_lexer.get_position(), exception_message(token_type::end_object, "object"), nullptr));
+            if (!report_error(sax, parse_error::create(101, m_lexer.get_position(), exception_message(token_type::end_object, "object"), nullptr), allow_recovery))
+            {
+                return false;
+            }
+
+            // recover
+            if (last_token == token_type::end_of_input)
+            {
+                // the input ends inside the object
+                return close_containers(sax, states);
+            }
+            if (last_token == token_type::end_array)
+            {
+                // a wrong closing bracket closes the innermost container
+                if (JSON_HEDLEY_UNLIKELY(!sax->end_object()))
+                {
+                    return false;
+                }
+                states.pop_back();
+                skip_to_state_evaluation = true;
+                continue;
+            }
+            if (!continue_after(recover_member(sax, allow_recovery), skip_to_state_evaluation))
+            {
+                return false;
+            }
         }
     }
 
+    /*!
+    @brief continue sax_parse_internal() after a recovery
+    @return whether to continue parsing
+    */
+    bool continue_after(const next_step step, bool& skip_to_state_evaluation)
+    {
+        if (step == next_step::evaluate_state)
+        {
+            // the state evaluation reads the token again
+            m_lexer.unget_token();
+            skip_to_state_evaluation = true;
+        }
+        return step != next_step::stop;
+    }
+
+    /// the parser for parse() and accept() never recovers: stop parsing
+    static std::false_type continue_after(std::false_type /*step*/, bool& /*skip_to_state_evaluation*/) noexcept
+    {
+        return {};
+    }
+
+    /*!
+    @brief parse an object key and the name separator (:) after it
+
+    last_token is the token where the key is expected. sax_parse_internal()
+    repeats these steps rather than calling this function, which is used
+    when recovering from an error.
+
+    @return next_step::parse_value if the value follows, with last_token its
+            first token; next_step::evaluate_state if the object's state is
+            to be evaluated after recovering from an error; next_step::stop
+            to stop parsing
+    */
+    template<typename SAX>
+    next_step parse_key(SAX* sax)
+    {
+        const std::true_type allow_recovery{};
+
+        if (JSON_HEDLEY_UNLIKELY(last_token != token_type::value_string))
+        {
+            return key_error(sax, allow_recovery, false);
+        }
+
+        if (JSON_HEDLEY_UNLIKELY(!sax->key(m_lexer.get_string())))
+        {
+            return next_step::stop;
+        }
+
+        // parse separator (:)
+        if (JSON_HEDLEY_UNLIKELY(get_token() != token_type::name_separator))
+        {
+            return key_error(sax, allow_recovery, true);
+        }
+
+        // the value begins with the next token
+        get_token();
+        return next_step::parse_value;
+    }
+
+    /*!
+    @brief report a number that is too large for number_float_t, and recover
+           from the error by passing the value on; the SAX parser gets the
+           number's text as well
+
+    This is a separate function, as reading other numbers is measurably
+    slower if the error is handled where they are read.
+
+    @param[in] sax    the SAX parser
+    @param[in] value  the value that is not finite
+    @return whether to continue parsing
+    */
+    template<typename SAX, typename AllowRecovery>
+    bool overflow_error(SAX* sax, const number_float_t value, AllowRecovery allow_recovery)
+    {
+        if (!report_error(sax, out_of_range::create(406, concat("number overflow parsing '", m_lexer.get_token_string(), '\''), nullptr), allow_recovery))
+        {
+            return false;
+        }
+        return sax->number_float(value, m_lexer.get_string());
+    }
+
+    /*!
+    @brief report a missing key, or a missing name separator (:) after the
+           key; the parser for parse() and accept() never recovers
+
+    @param[in] key_read  whether the key was read, so that the name separator
+                         is missing
+    @return std::false_type, see report_error()
+    */
+    template<typename SAX>
+    std::false_type key_error(SAX* sax, std::false_type allow_recovery, const bool key_read)
+    {
+        return report_error(sax, parse_error::create(101, m_lexer.get_position(), key_read
+                            ? exception_message(token_type::name_separator, "object separator")
+                            : exception_message(token_type::value_string, "object key"), nullptr), allow_recovery);
+    }
+
+    /*!
+    @brief report a missing key, or a missing name separator (:) after the
+           key, and recover from it
+
+    @param[in] key_read  whether the key was read, so that the name separator
+                         is missing
+    */
+    template<typename SAX>
+    next_step key_error(SAX* sax, std::true_type allow_recovery, const bool key_read)
+    {
+        if (!key_read)
+        {
+            if (!report_error(sax, parse_error::create(101, m_lexer.get_position(), exception_message(token_type::value_string, "object key"), nullptr), allow_recovery))
+            {
+                return next_step::stop;
+            }
+            return recover_key(sax);
+        }
+
+        if (!report_error(sax, parse_error::create(101, m_lexer.get_position(), exception_message(token_type::name_separator, "object separator"), nullptr), allow_recovery))
+        {
+            return next_step::stop;
+        }
+        return recover_name_separator(sax);
+    }
+
+    /////////////////////
+    // error recovery
+    /////////////////////
+
+    /*
+    The functions below repair an error after the SAX parser's parse_error()
+    returned true (see #3989). Each mistake is repaired by the smallest local
+    edit: a missing ',' or ':' is inserted, a stray token is removed, what can
+    be read of an invalid string or number is kept (see
+    lexer::recover_token()), a missing value becomes null, a wrong closing
+    bracket closes the innermost container, and the end of the input closes
+    all of them. The events stay balanced, and every key() is followed by
+    exactly one value.
+
+    A repair hands a token to the state evaluation, by returning it to the
+    lexer (lexer::unget_token()) so that the state evaluation reads it again,
+    only if it is ',', ']', '}', or the end of the input. The state evaluation
+    hands a token to value or key parsing only if it is none of them, so a
+    token is never handed back and forth. Every other step reads a token or
+    closes a container, so parsing always ends.
+    */
+
+    /*!
+    @brief report an error to the SAX parser; the parser for parse() and
+           accept() never recovers
+
+    @return std::false_type rather than false: its value is known where the
+            function is called even if the call is not inlined, so the code
+            for recovering is not generated
+    */
+    template<typename SAX, typename Exception>
+    std::false_type report_error(SAX* sax, const Exception& ex, std::false_type /*allow_recovery*/)
+    {
+        error_reported = true;
+        static_cast<void>(sax->parse_error(m_lexer.get_position(), m_lexer.get_token_string(), ex));
+        return {};
+    }
+
+    /*!
+    @brief report an error to the SAX parser
+    @return whether to recover from the error
+    */
+    template<typename SAX, typename Exception>
+    bool report_error(SAX* sax, const Exception& ex, std::true_type /*allow_recovery*/)
+    {
+        const std::size_t position = m_lexer.get_position().chars_read_total;
+        if (error_reported && position == last_error_position && last_token == last_error_token)
+        {
+            // a repair handed on the token of the error it repaired; the
+            // token was reported already, and the SAX parser asked to recover
+            return true;
+        }
+
+        error_reported = true;
+        last_error_position = position;
+        last_error_token = last_token;
+
+        if (!sax->parse_error(m_lexer.get_position(), m_lexer.get_token_string(), ex))
+        {
+            return false;
+        }
+
+        // the token string of the next error begins here
+        m_lexer.restart_token_string();
+        return true;
+    }
+
+    /*!
+    @brief keep what can be read of the token that the lexer rejected
+
+    The error was reported for the rejected token, so it is not reported again
+    for the token it is repaired to (see lexer::recover_token()).
+    */
+    token_type recover_token()
+    {
+        last_token = m_lexer.recover_token();
+        last_error_position = m_lexer.get_position().chars_read_total;
+        last_error_token = last_token;
+        return last_token;
+    }
+
+    /// pass the end events of all open containers
+    template<typename SAX>
+    bool close_containers(SAX* sax, std::vector<bool>& states)
+    {
+        while (!states.empty())
+        {
+            const bool is_array = states.back();
+            states.pop_back();
+            if (JSON_HEDLEY_UNLIKELY(is_array ? !sax->end_array() : !sax->end_object()))
+            {
+                return false;
+            }
+        }
+        return true;
+    }
+
+    /*!
+    @brief read tokens until one begins a value, skipping everything before
+           the top-level value
+    @return whether a value begins with last_token
+    */
+    bool skip_to_value()
+    {
+        while (true)
+        {
+            switch (get_token())
+            {
+                case token_type::begin_array:
+                case token_type::begin_object:
+                case token_type::literal_false:
+                case token_type::literal_null:
+                case token_type::literal_true:
+                case token_type::value_float:
+                case token_type::value_integer:
+                case token_type::value_string:
+                case token_type::value_unsigned:
+                    return true;
+
+                case token_type::end_of_input:
+                    return false;
+
+                case token_type::parse_error:
+                    recover_token();
+                    if (last_token != token_type::uninitialized)
+                    {
+                        return true;
+                    }
+                    break;
+
+                case token_type::uninitialized:
+                case token_type::end_array:
+                case token_type::end_object:
+                case token_type::name_separator:
+                case token_type::value_separator:
+                case token_type::literal_or_value:
+                default:
+                    break;
+            }
+        }
+    }
+
+    /*!
+    @brief skip the rest of an object member that cannot be read
+
+    Reads tokens, beginning with last_token, until a ',', '}', or ']' that is
+    not inside a container that begins in the skipped tokens, or the end of
+    the input.
+    */
+    void skip_member()
+    {
+        std::size_t depth = 0;
+        while (true)
+        {
+            switch (last_token)
+            {
+                case token_type::begin_array:
+                case token_type::begin_object:
+                    ++depth;
+                    break;
+
+                case token_type::end_array:
+                case token_type::end_object:
+                    if (depth == 0)
+                    {
+                        return;
+                    }
+                    --depth;
+                    break;
+
+                case token_type::value_separator:
+                    if (depth == 0)
+                    {
+                        return;
+                    }
+                    break;
+
+                case token_type::end_of_input:
+                    return;
+
+                case token_type::parse_error:
+                    recover_token();
+                    break;
+
+                case token_type::uninitialized:
+                case token_type::literal_true:
+                case token_type::literal_false:
+                case token_type::literal_null:
+                case token_type::value_string:
+                case token_type::value_unsigned:
+                case token_type::value_integer:
+                case token_type::value_float:
+                case token_type::name_separator:
+                case token_type::literal_or_value:
+                default:
+                    break;
+            }
+            get_token();
+        }
+    }
+
+    /*!
+    @brief pass a value where it is missing
+
+    last_token is ',', ']', '}', or the end of the input, where a value was
+    expected. In an object, the key gets null; in an array, a ',' where a
+    value is missing stands for null (as in JavaScript), while an array that
+    ends there just ends.
+    */
+    template<typename SAX>
+    bool recover_missing_value(SAX* sax, const std::vector<bool>& states)
+    {
+        JSON_ASSERT(!states.empty());
+        if (!states.back() || last_token == token_type::value_separator)
+        {
+            return sax->null();
+        }
+        return true;
+    }
+
+    /// recover from a missing key; last_token is where it was expected
+    template<typename SAX>
+    next_step recover_key(SAX* sax)
+    {
+        switch (last_token)
+        {
+            case token_type::value_separator:
+            case token_type::end_object:
+            case token_type::end_array:
+            case token_type::end_of_input:
+                // no member: the object's state handles the token
+                return next_step::evaluate_state;
+
+            case token_type::parse_error:
+                recover_token();
+                if (last_token == token_type::value_string)
+                {
+                    // a key that could be repaired
+                    return parse_key(sax);
+                }
+                skip_member();
+                return next_step::evaluate_state;
+
+            case token_type::uninitialized:
+            case token_type::literal_true:
+            case token_type::literal_false:
+            case token_type::literal_null:
+            case token_type::value_string:
+            case token_type::value_unsigned:
+            case token_type::value_integer:
+            case token_type::value_float:
+            case token_type::begin_array:
+            case token_type::begin_object:
+            case token_type::name_separator:
+            case token_type::literal_or_value:
+            default:
+                // a member without a key
+                skip_member();
+                return next_step::evaluate_state;
+        }
+    }
+
+    /// recover from a missing name separator (:) after the key; last_token
+    /// is where it was expected
+    template<typename SAX>
+    next_step recover_name_separator(SAX* sax)
+    {
+        switch (last_token)
+        {
+            case token_type::value_separator:
+            case token_type::end_object:
+            case token_type::end_array:
+            case token_type::end_of_input:
+                // the value is missing as well
+                return sax->null() ? next_step::evaluate_state : next_step::stop;
+
+            case token_type::uninitialized:
+            case token_type::literal_true:
+            case token_type::literal_false:
+            case token_type::literal_null:
+            case token_type::value_string:
+            case token_type::value_unsigned:
+            case token_type::value_integer:
+            case token_type::value_float:
+            case token_type::begin_array:
+            case token_type::begin_object:
+            case token_type::name_separator:
+            case token_type::parse_error:
+            case token_type::literal_or_value:
+            default:
+                // a missing ':'; the value begins here
+                return next_step::parse_value;
+        }
+    }
+
+    /// recover from a token after an object member that is neither ',' nor
+    /// '}' (nor ']' or the end of the input, which the caller handles)
+    template<typename SAX>
+    next_step recover_member(SAX* sax, std::true_type /*allow_recovery*/)
+    {
+        if (last_token == token_type::parse_error)
+        {
+            recover_token();
+        }
+        if (last_token == token_type::value_string)
+        {
+            // a missing ','; the next key begins here
+            return parse_key(sax);
+        }
+        skip_member();
+        return next_step::evaluate_state;
+    }
+
+    /// the parser for parse() and accept() never recovers (and does not come
+    /// here, as report_error() returned false)
+    template<typename SAX>
+    std::false_type recover_member(SAX* /*sax*/, std::false_type /*allow_recovery*/) const noexcept
+    {
+        return {};
+    }
+
     /// get next token from lexer
     token_type get_token()
     {
@@ -560,6 +1166,12 @@ class parser
     const bool allow_exceptions = true;
     /// whether trailing commas in objects and arrays should be ignored (true) or signaled as errors (false)
     const bool ignore_trailing_commas = false;
+    /// whether an error was reported to the SAX parser
+    bool error_reported = false;
+    /// the position of the last reported error
+    std::size_t last_error_position = 0;
+    /// the token of the last reported error
+    token_type last_error_token = token_type::uninitialized;
 };
 
 }  // namespace detail

include/nlohmann/detail/input/string_scan.hpp

@@ -201,6 +201,43 @@ inline std::size_t validate_one_utf8(const unsigned char* data, std::size_t avai
     return 0; // invalid, incomplete, or must be diagnosed by the byte path
 }
 
+// Return the length of the longest prefix of [data, data+n) that consists of
+// ASCII characters and complete well-formed UTF-8 sequences; n if all of it is
+// valid UTF-8. Unlike scalar_string_bulk_run(), quotes, escapes, and control
+// characters are ordinary characters here. ASCII is skipped 8 bytes at a time.
+inline std::size_t valid_utf8_prefix(const unsigned char* data, std::size_t n) noexcept
+{
+    constexpr std::uint64_t high = 0x8080808080808080ull;
+    std::size_t pos = 0;
+    while (pos < n)
+    {
+        if (pos + 8 <= n)
+        {
+            std::uint64_t word = 0;
+            std::memcpy(&word, data + pos, sizeof(word));
+            if ((word & high) == 0)
+            {
+                pos += 8;
+                continue;
+            }
+        }
+
+        if (data[pos] < 0x80u)
+        {
+            ++pos;
+            continue;
+        }
+
+        const std::size_t seq = validate_one_utf8(data + pos, n - pos);
+        if (seq == 0)
+        {
+            break; // ill-formed or truncated
+        }
+        pos += seq;
+    }
+    return pos;
+}
+
 // Scalar (C++11) computation of the bulk run length: the number of leading
 // bytes in [data, data+n) that are ordinary ASCII or complete well-formed UTF-8
 // sequences, stopping before the first byte that needs individual handling (the

include/nlohmann/detail/output/binary_writer.hpp

@@ -25,6 +25,7 @@
 #endif
 
 #include <nlohmann/detail/input/binary_reader.hpp>
+#include <nlohmann/detail/input/string_scan.hpp>
 #include <nlohmann/detail/macro_scope.hpp>
 #include <nlohmann/detail/output/output_adapters.hpp>
 #include <nlohmann/detail/string_concat.hpp>
@@ -76,7 +77,7 @@ std::size_t binary_reserve_hint(const BasicJsonType& j)
 }
 
 /*!
-@brief serialization to CBOR and MessagePack values
+@brief serialization to BJData, BON8, BSON, CBOR, MessagePack, and UBJSON values
 */
 template<typename BasicJsonType, typename CharType, typename OutputSinkType = output_adapter_sink<CharType>>
 class binary_writer
@@ -873,6 +874,21 @@ class binary_writer
         }
     }
 
+    /*!
+    @param[in] j  JSON value to serialize
+    */
+    void write_bon8(const BasicJsonType& j)
+    {
+        bool string_open = false;
+        write_bon8_value(j, string_open);
+
+        // the last string of a message must be terminated
+        if (string_open)
+        {
+            oa.write_character(to_char_type(0xFF));
+        }
+    }
+
   private:
     //////////
     // BSON //
@@ -1431,6 +1447,28 @@ class binary_writer
         return to_char_type(0xCB);  // float 64
     }
 
+    /// @return the BON8 type marker for binary32 (float) or binary64 (double)
+    template<typename FloatType>
+    static constexpr CharType get_bon8_float_prefix()
+    {
+        return to_char_type(std::is_same<FloatType, float>::value ? 0x8E : 0x8F);
+    }
+
+    /// @return the type marker for a FloatType value in @a format (CBOR, MessagePack, or BON8)
+    template<typename FloatType>
+    static CharType get_compact_float_prefix(const detail::input_format_t format)
+    {
+        if (format == detail::input_format_t::cbor)
+        {
+            return get_cbor_float_prefix(FloatType{});
+        }
+        if (format == detail::input_format_t::bon8)
+        {
+            return get_bon8_float_prefix<FloatType>();
+        }
+        return get_msgpack_float_prefix(FloatType{});
+    }
+
     ////////////
     // UBJSON //
     ////////////
@@ -2050,6 +2088,322 @@ class binary_writer
         return false;
     }
 
+    //////////
+    // BON8 //
+    //////////
+
+    /*!
+    @brief write a BON8 value
+
+    A string is written without length or terminator: it ends at the first
+    byte that cannot continue it, which is the first byte of any non-string
+    value and of the end-of-container marker 0xFE. It only needs an explicit
+    end-of-string marker (0xFF) when it is empty, when another string follows,
+    or when it is the last thing in the message.
+
+    @param[in] j                JSON value to serialize
+    @param[in,out] string_open  whether the output ends with a non-empty
+                                string that has not been terminated with 0xFF
+    */
+    void write_bon8_value(const BasicJsonType& j, bool& string_open)
+    {
+        switch (j.type())
+        {
+            case value_t::null:
+            {
+                write_bon8_marker(0xFA, string_open);
+                break;
+            }
+
+            case value_t::boolean:
+            {
+                write_bon8_marker(j.m_data.m_value.boolean ? 0xF9 : 0xF8, string_open);
+                break;
+            }
+
+            case value_t::number_unsigned:
+            {
+                if (j.m_data.m_value.number_unsigned > static_cast<typename BasicJsonType::number_unsigned_t>((std::numeric_limits<std::int64_t>::max)()))
+                {
+                    JSON_THROW(out_of_range::create(407, concat("integer number ", std::to_string(j.m_data.m_value.number_unsigned), " cannot be represented by BON8 as it does not fit int64"), &j));
+                }
+                write_bon8_integer(static_cast<std::int64_t>(j.m_data.m_value.number_unsigned));
+                string_open = false;
+                break;
+            }
+
+            case value_t::number_integer:
+            {
+                write_bon8_integer(static_cast<std::int64_t>(j.m_data.m_value.number_integer));
+                string_open = false;
+                break;
+            }
+
+            case value_t::number_float:
+            {
+                write_bon8_float(j.m_data.m_value.number_float);
+                string_open = false;
+                break;
+            }
+
+            case value_t::string:
+            {
+                write_bon8_string(*j.m_data.m_value.string, string_open, j);
+                break;
+            }
+
+            case value_t::array:
+            {
+                const auto N = j.m_data.m_value.array->size();
+                // 0x80..0x84: array with 0..4 elements; 0x85: array ended by 0xFE
+                write_bon8_marker(static_cast<std::uint8_t>(N <= 4 ? 0x80 + N : 0x85), string_open);
+
+                for (const auto& el : *j.m_data.m_value.array)
+                {
+                    write_bon8_value(el, string_open);
+                }
+
+                if (N > 4)
+                {
+                    write_bon8_marker(0xFE, string_open);
+                }
+                break;
+            }
+
+            case value_t::object:
+            {
+                const auto N = j.m_data.m_value.object->size();
+                // 0x86..0x8A: object with 0..4 members; 0x8B: object ended by 0xFE
+                write_bon8_marker(static_cast<std::uint8_t>(N <= 4 ? 0x86 + N : 0x8B), string_open);
+
+                for (const auto& el : *j.m_data.m_value.object)
+                {
+                    write_bon8_string(el.first, string_open, j);
+                    write_bon8_value(el.second, string_open);
+                }
+
+                if (N > 4)
+                {
+                    write_bon8_marker(0xFE, string_open);
+                }
+                break;
+            }
+
+            case value_t::binary:
+            {
+                // BON8 has no binary type: write the bytes as an array of
+                // integers, like UBJSON and BJData do
+                const auto N = j.m_data.m_value.binary->size();
+                write_bon8_marker(static_cast<std::uint8_t>(N <= 4 ? 0x80 + N : 0x85), string_open);
+
+                for (std::size_t i = 0; i < N; ++i)
+                {
+                    // the cast is needed for binary types whose value type
+                    // is not an integer (e.g., std::byte)
+                    write_bon8_integer(static_cast<std::uint8_t>(j.m_data.m_value.binary->data()[i]));
+                }
+
+                if (N > 4)
+                {
+                    write_bon8_marker(0xFE, string_open);
+                }
+                break;
+            }
+
+            case value_t::discarded:
+            default:
+                break;
+        }
+    }
+
+    /*!
+    @brief write a single byte that is not part of a string
+
+    @param[in] marker        the byte to write
+    @param[out] string_open  set to false, because the output no longer ends
+                             with a string; see @ref write_bon8_value
+    */
+    void write_bon8_marker(const std::uint8_t marker, bool& string_open)
+    {
+        oa.write_character(to_char_type(marker));
+        string_open = false;
+    }
+
+    /*!
+    @brief write a string
+
+    @param[in] s                the string to write
+    @param[in,out] string_open  see @ref write_bon8_value
+    @param[in] context          the value the string belongs to (for diagnostics)
+
+    @throw type_error.316 if @a s is not valid UTF-8, because the end of a
+           string is determined from its encoding
+    */
+    void write_bon8_string(const string_t& s, bool& string_open, const BasicJsonType& context)
+    {
+        check_bon8_utf8(s, context);
+
+        // a string that follows another string terminates it
+        if (string_open)
+        {
+            oa.write_character(to_char_type(0xFF));
+        }
+
+        if (s.empty())
+        {
+            // the empty string is just the end-of-string marker
+            oa.write_character(to_char_type(0xFF));
+            string_open = false;
+        }
+        else
+        {
+            oa.write_characters(reinterpret_cast<const CharType*>(s.data()), s.size());
+            string_open = true;
+        }
+    }
+
+    /*!
+    @brief check that a string is valid UTF-8 (RFC 3629)
+
+    @param[in] s        the string to check
+    @param[in] context  the value the string belongs to (for diagnostics)
+
+    @throw type_error.316 if @a s is not valid UTF-8; the message names the
+           first byte of the first invalid or incomplete sequence
+    */
+    static void check_bon8_utf8(const string_t& s, const BasicJsonType& context)
+    {
+        static_cast<void>(context); // only used when exceptions are enabled
+        const auto* data = reinterpret_cast<const unsigned char*>(s.data());
+        const std::size_t valid = valid_utf8_prefix(data, s.size());
+        if (JSON_HEDLEY_UNLIKELY(valid != s.size()))
+        {
+            JSON_THROW(type_error::create(316, concat("invalid UTF-8 byte at index ", std::to_string(valid), ": 0x", hex_byte(data[valid])), &context));
+        }
+    }
+
+    /// @return a byte as two uppercase hexadecimal digits
+    static std::string hex_byte(const std::uint8_t byte)
+    {
+        std::string result = "00";
+        constexpr const char* nibble_to_hex = "0123456789ABCDEF";
+        result[0] = nibble_to_hex[byte / 16];
+        result[1] = nibble_to_hex[byte % 16];
+        return result;
+    }
+
+    /*!
+    @brief write an integer in the shortest encoding
+
+    Integers from -10 to 39 take one byte. Up to -33818506 and 67637031, an
+    integer takes 2 to 4 bytes that begin with a UTF-8 lead byte (0xC2..0xF7)
+    followed by a byte that is not a continuation byte: 0x00..0x7F for
+    positive and 0xC0..0xFF for negative integers. Each range starts where the
+    shorter one ends. Larger integers are written as int32 (0x8C) or int64
+    (0x8D) in big-endian byte order.
+
+    @param[in] value  the integer to write
+    */
+    void write_bon8_integer(std::int64_t value)
+    {
+        if (value < (std::numeric_limits<std::int32_t>::min)() || value > (std::numeric_limits<std::int32_t>::max)())
+        {
+            oa.write_character(to_char_type(0x8D));
+            write_number(value);
+        }
+        else if (value < -33818506 || value > 67637031)
+        {
+            oa.write_character(to_char_type(0x8C));
+            write_number(static_cast<std::int32_t>(value));
+        }
+        else if (value <= -264075)
+        {
+            value = -(value + 264075);
+            write_bon8_bytes(0xF0 + ((value >> 22) & 0x07), 0xC0 + ((value >> 16) & 0x3F), value >> 8, value);
+        }
+        else if (value <= -1931)
+        {
+            value = -(value + 1931);
+            write_bon8_bytes(0xE0 + ((value >> 14) & 0x0F), 0xC0 + ((value >> 8) & 0x3F), value);
+        }
+        else if (value <= -11)
+        {
+            value = -(value + 11);
+            write_bon8_bytes(0xC2 + ((value >> 6) & 0x1F), 0xC0 + (value & 0x3F));
+        }
+        else if (value <= -1)
+        {
+            write_bon8_bytes(0xB8 - (value + 1));
+        }
+        else if (value <= 39)
+        {
+            write_bon8_bytes(0x90 + value);
+        }
+        else if (value <= 3879)
+        {
+            value -= 40;
+            write_bon8_bytes(0xC2 + ((value >> 7) & 0x1F), value & 0x7F);
+        }
+        else if (value <= 528167)
+        {
+            value -= 3880;
+            write_bon8_bytes(0xE0 + ((value >> 15) & 0x0F), (value >> 8) & 0x7F, value);
+        }
+        else
+        {
+            value -= 528168;
+            write_bon8_bytes(0xF0 + ((value >> 23) & 0x07), (value >> 16) & 0x7F, value >> 8, value);
+        }
+    }
+
+    /// write the low byte of each argument
+    template<typename... Bytes>
+    void write_bon8_bytes(const Bytes... bytes)
+    {
+        const std::array<CharType, sizeof...(Bytes)> buffer{{to_char_type(static_cast<std::uint8_t>(bytes & 0xFF))...}};
+        oa.write_characters(buffer.data(), buffer.size());
+    }
+
+    /*!
+    @brief write a floating-point number
+
+    -1.0, +0.0, and 1.0 take one byte. Other numbers are written as binary32
+    (0x8E) if that loses no precision, and as binary64 (0x8F) otherwise; -0.0,
+    infinities, and NaN are always written as binary32, NaN as 0x7F800001.
+
+    @param[in] n  the number to write
+    */
+    void write_bon8_float(const number_float_t n)
+    {
+#ifdef __GNUC__
+        JSON_HEDLEY_DIAGNOSTIC_PUSH
+        JSON_HEDLEY_PRAGMA(GCC diagnostic ignored "-Wfloat-equal")
+#endif
+        if (n == static_cast<number_float_t>(-1))
+        {
+            oa.write_character(to_char_type(0xFB));
+        }
+        else if (n == static_cast<number_float_t>(0) && !std::signbit(n))
+        {
+            oa.write_character(to_char_type(0xFC));
+        }
+        else if (n == static_cast<number_float_t>(1))
+        {
+            oa.write_character(to_char_type(0xFD));
+        }
+        else if (std::isnan(n))
+        {
+            write_bon8_bytes(0x8E, 0x7F, 0x80, 0x00, 0x01);
+        }
+        else
+        {
+            write_compact_float(n, detail::input_format_t::bon8);
+        }
+#ifdef __GNUC__
+        JSON_HEDLEY_DIAGNOSTIC_POP
+#endif
+    }
+
     ///////////////////////
     // Utility functions //
     ///////////////////////
@@ -2184,16 +2538,12 @@ class binary_writer
                                    static_cast<double>(n) <= static_cast<double>((std::numeric_limits<float>::max)()) &&
                                    static_cast<double>(static_cast<float>(n)) == static_cast<double>(n))))
         {
-            oa.write_character(format == detail::input_format_t::cbor
-                               ? get_cbor_float_prefix(static_cast<float>(n))
-                               : get_msgpack_float_prefix(static_cast<float>(n)));
+            oa.write_character(get_compact_float_prefix<float>(format));
             write_number(static_cast<float>(n));
         }
         else
         {
-            oa.write_character(format == detail::input_format_t::cbor
-                               ? get_cbor_float_prefix(n)
-                               : get_msgpack_float_prefix(n));
+            oa.write_character(get_compact_float_prefix<number_float_t>(format));
             write_number(n);
         }
 #ifdef __GNUC__

include/nlohmann/json.hpp

@@ -1605,42 +1605,12 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         assert_invariant();
     }
 
-  private:
-    /// whether a basic_json specialization can be converted implicitly into this one;
-    /// with JSON_USE_IMPLICIT_CONVERSIONS set to 0, this is only the case if both share
-    /// the same string type (see https://github.com/nlohmann/json/issues/2649)
-    template<typename BasicJsonType>
-    using is_implicitly_convertible_basic_json = std::integral_constant < bool,
-          (JSON_USE_IMPLICIT_CONVERSIONS != 0)
-          || std::is_same<typename BasicJsonType::string_t, string_t>::value >;
-
-    /// tag to select the constructor that performs the conversion from another basic_json specialization
-    struct convert_basic_json_tag {};
-
-  public:
     /// @brief create a JSON value from an existing one
     /// @sa https://json.nlohmann.me/api/basic_json/basic_json/
     template < typename BasicJsonType,
                detail::enable_if_t <
-                   detail::is_basic_json<BasicJsonType>::value&& !std::is_same<basic_json, BasicJsonType>::value
-                   && is_implicitly_convertible_basic_json<BasicJsonType>::value, int > = 0 >
+                   detail::is_basic_json<BasicJsonType>::value&& !std::is_same<basic_json, BasicJsonType>::value, int > = 0 >
     basic_json(const BasicJsonType& val)
-        : basic_json(val, convert_basic_json_tag{})
-    {}
-
-    /// @brief create a JSON value from an existing one
-    /// @sa https://json.nlohmann.me/api/basic_json/basic_json/
-    template < typename BasicJsonType,
-               detail::enable_if_t <
-                   detail::is_basic_json<BasicJsonType>::value&& !std::is_same<basic_json, BasicJsonType>::value
-                   && !is_implicitly_convertible_basic_json<BasicJsonType>::value, int > = 0 >
-    explicit basic_json(const BasicJsonType& val)
-        : basic_json(val, convert_basic_json_tag{})
-    {}
-
-  private:
-    template<typename BasicJsonType>
-    basic_json(const BasicJsonType& val, convert_basic_json_tag /*unused*/)
 #if JSON_DIAGNOSTIC_POSITIONS
         : start_position(val.start_pos()),
           end_position(val.end_pos())
@@ -1696,7 +1666,6 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         assert_invariant();
     }
 
-  public:
     /// @brief create a container (array or object) from an initializer list
     /// @sa https://json.nlohmann.me/api/basic_json/basic_json/
     basic_json(initializer_list_t init,
@@ -2463,7 +2432,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
                    int > = 0 >
     BasicJsonType get_impl(detail::priority_tag<2> /*unused*/) const
     {
-        return BasicJsonType(*this);
+        return *this;
     }
 
     /*!
@@ -2602,7 +2571,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
                  int> = 0>
     ValueType & get_to(ValueType& v) const
     {
-        v = ValueType(*this);
+        v = *this;
         return v;
     }
 
@@ -5010,6 +4979,26 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         return parser(i.get(), nullptr, false, ignore_comments, ignore_trailing_commas, true).accept(true);
     }
 
+  private:
+    /// read a binary format and pass it to a SAX parser; if the SAX parser
+    /// asks to recover from an error, the value read so far is completed
+    /// (see detail::json_sax_salvager and #3989)
+    template<typename InputAdapterType, typename SAX>
+    static bool sax_parse_binary(InputAdapterType ia, SAX* sax,
+                                 const input_format_t format, const bool strict)
+    {
+        (void)detail::is_sax_static_asserts<SAX, basic_json> {};
+        using salvager_t = detail::json_sax_salvager<basic_json, SAX>;
+        salvager_t salvager(sax);
+        const bool result = detail::binary_reader<basic_json, InputAdapterType, salvager_t>(std::move(ia), format).sax_parse(format, &salvager, strict);
+        if (!result)
+        {
+            salvager.close_open_containers();
+        }
+        return result;
+    }
+
+  public:
     /// @brief generate SAX events
     /// @sa https://json.nlohmann.me/api/basic_json/sax_parse/
     template <typename InputType, typename SAX>
@@ -5023,7 +5012,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         auto ia = detail::input_adapter(std::forward<InputType>(i));
         return format == input_format_t::json
                ? parser(std::move(ia), nullptr, true, ignore_comments, ignore_trailing_commas).sax_parse(sax, strict)
-               : detail::binary_reader<basic_json, decltype(ia), SAX>(std::move(ia), format).sax_parse(format, sax, strict);
+               : sax_parse_binary(std::move(ia), sax, format, strict);
     }
 
     /// @brief generate SAX events (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -5040,7 +5029,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         auto ia = detail::input_adapter(std::move(first), std::move(last));
         return format == input_format_t::json
                ? parser(std::move(ia), nullptr, true, ignore_comments, ignore_trailing_commas).sax_parse(sax, strict)
-               : detail::binary_reader<basic_json, decltype(ia), SAX>(std::move(ia), format).sax_parse(format, sax, strict);
+               : sax_parse_binary(std::move(ia), sax, format, strict);
     }
 
     /// @brief generate SAX events
@@ -5062,7 +5051,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
                // NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg)
                ? parser(std::move(ia), nullptr, true, ignore_comments, ignore_trailing_commas).sax_parse(sax, strict)
                // NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg)
-               : detail::binary_reader<basic_json, decltype(ia), SAX>(std::move(ia), format).sax_parse(format, sax, strict);
+               : sax_parse_binary(std::move(ia), sax, format, strict);
     }
 #ifndef JSON_NO_IO
     /// @brief deserialize from stream
@@ -5321,6 +5310,30 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         binary_writer<char>(o).write_bson(j);
     }
 
+    /// @brief create a BON8 serialization of a given JSON value
+    /// @sa https://json.nlohmann.me/api/basic_json/to_bon8/
+    static std::vector<std::uint8_t> to_bon8(const basic_json& j)
+    {
+        std::vector<std::uint8_t> result;
+        result.reserve(detail::binary_reserve_hint(j));
+        vector_writer(result).write_bon8(j);
+        return result;
+    }
+
+    /// @brief create a BON8 serialization of a given JSON value
+    /// @sa https://json.nlohmann.me/api/basic_json/to_bon8/
+    static void to_bon8(const basic_json& j, detail::output_adapter<std::uint8_t> o)
+    {
+        binary_writer<std::uint8_t>(o).write_bon8(j);
+    }
+
+    /// @brief create a BON8 serialization of a given JSON value
+    /// @sa https://json.nlohmann.me/api/basic_json/to_bon8/
+    static void to_bon8(const basic_json& j, detail::output_adapter<char> o)
+    {
+        binary_writer<char>(o).write_bon8(j);
+    }
+
     /// @brief create a JSON value from an input in CBOR format
     /// @sa https://json.nlohmann.me/api/basic_json/from_cbor/
     template<typename InputType>
@@ -5554,6 +5567,43 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         return result;
     }
 
+    /// @brief create a JSON value from an input in BON8 format
+    /// @sa https://json.nlohmann.me/api/basic_json/from_bon8/
+    template<typename InputType>
+    JSON_HEDLEY_WARN_UNUSED_RESULT
+    static basic_json from_bon8(InputType&& i,
+                                const bool strict = true,
+                                const bool allow_exceptions = true)
+    {
+        basic_json result;
+        auto ia = detail::input_adapter(std::forward<InputType>(i));
+        detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
+        if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::bon8).sax_parse(input_format_t::bon8, &sdp, strict)) // cppcheck-suppress[accessMoved]
+        {
+            result = value_t::discarded;
+        }
+        return result;
+    }
+
+    /// @brief create a JSON value from an input in BON8 format (iterator pair, or iterator+sentinel pair for C++20 ranges support)
+    /// @sa https://json.nlohmann.me/api/basic_json/from_bon8/
+    template<typename IteratorType, typename SentinelType = IteratorType,
+             detail::enable_if_t<detail::can_compare_ne<IteratorType, SentinelType>::value, int> = 0>
+    JSON_HEDLEY_WARN_UNUSED_RESULT
+    static basic_json from_bon8(IteratorType first, SentinelType last,
+                                const bool strict = true,
+                                const bool allow_exceptions = true)
+    {
+        basic_json result;
+        auto ia = detail::input_adapter(std::move(first), std::move(last));
+        detail::json_sax_dom_parser<basic_json, decltype(ia)> sdp(result, allow_exceptions);
+        if (!binary_reader<decltype(ia)>(std::move(ia), input_format_t::bon8).sax_parse(input_format_t::bon8, &sdp, strict)) // cppcheck-suppress[accessMoved]
+        {
+            result = value_t::discarded;
+        }
+        return result;
+    }
+
     /// @brief create a JSON value from an input in BSON format
     /// @sa https://json.nlohmann.me/api/basic_json/from_bson/
     template<typename InputType>

src/modules/json.cppm

@@ -46,6 +46,7 @@ inline namespace json_literals
 namespace detail
 {
     using NLOHMANN_JSON_NAMESPACE::detail::json_sax_dom_callback_parser;
+    using NLOHMANN_JSON_NAMESPACE::detail::json_sax_dom_parser;
     using NLOHMANN_JSON_NAMESPACE::detail::unknown_size;
 } // namespace detail
 

tests/CMakeLists.txt

@@ -112,7 +112,7 @@ endif()
 
 if (CMAKE_CXX_COMPILER_ID STREQUAL "MSVC")
     # avoid stack overflow, see https://github.com/nlohmann/json/issues/2955
-    json_test_set_test_options("test-cbor;test-msgpack;test-ubjson;test-bjdata;test-binary_formats" LINK_OPTIONS /STACK:4000000)
+    json_test_set_test_options("test-bon8;test-cbor;test-msgpack;test-ubjson;test-bjdata;test-binary_formats" LINK_OPTIONS /STACK:4000000)
 endif()
 
 # disable exceptions for test-disabled_exceptions

tests/Makefile

@@ -10,7 +10,7 @@ CXXFLAGS += -std=c++11
 CPPFLAGS += -I ../single_include
 
 FUZZER_ENGINE = src/fuzzer-driver_afl.cpp
-FUZZERS = parse_afl_fuzzer parse_bson_fuzzer parse_cbor_fuzzer parse_msgpack_fuzzer parse_ubjson_fuzzer parse_bjdata_fuzzer
+FUZZERS = parse_afl_fuzzer parse_bson_fuzzer parse_cbor_fuzzer parse_msgpack_fuzzer parse_ubjson_fuzzer parse_bjdata_fuzzer parse_bon8_fuzzer
 fuzzers: $(FUZZERS)
 
 parse_afl_fuzzer:
@@ -30,3 +30,6 @@ parse_ubjson_fuzzer:
 
 parse_bjdata_fuzzer:
 	$(CXX) $(CXXFLAGS) $(CPPFLAGS) $(FUZZER_ENGINE) src/fuzzer-parse_bjdata.cpp -o $@
+
+parse_bon8_fuzzer:
+	$(CXX) $(CXXFLAGS) $(CPPFLAGS) $(FUZZER_ENGINE) src/fuzzer-parse_bon8.cpp -o $@

tests/benchmarks/src/benchmarks.cpp

@@ -274,7 +274,8 @@ enum class binary_format
     ubjson_optimized,
     bjdata,
     bjdata_optimized,
-    bson
+    bson,
+    bon8
 };
 
 static std::vector<std::uint8_t> to_binary(const json& j, const binary_format format)
@@ -293,6 +294,8 @@ static std::vector<std::uint8_t> to_binary(const json& j, const binary_format fo
             return json::to_bjdata(j);
         case binary_format::bjdata_optimized:
             return json::to_bjdata(j, true, true);
+        case binary_format::bon8:
+            return json::to_bon8(j);
         case binary_format::bson:
         default:
             return json::to_bson(j);
@@ -313,6 +316,8 @@ static json from_binary(const std::vector<std::uint8_t>& bytes, const binary_for
         case binary_format::bjdata:
         case binary_format::bjdata_optimized:
             return json::from_bjdata(bytes);
+        case binary_format::bon8:
+            return json::from_bon8(bytes);
         case binary_format::bson:
         default:
             return json::from_bson(bytes);
@@ -333,6 +338,8 @@ static json from_binary(std::FILE* file, const binary_format format)
         case binary_format::bjdata:
         case binary_format::bjdata_optimized:
             return json::from_bjdata(file);
+        case binary_format::bon8:
+            return json::from_bon8(file);
         case binary_format::bson:
         default:
             return json::from_bson(file);
@@ -407,6 +414,10 @@ BENCHMARK_CAPTURE(FromBinaryBuffer, bjdata / canada, TEST_DATA_DIRECTORY "/nativ
 BENCHMARK_CAPTURE(FromBinaryBuffer, bjdata / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::bjdata);
 BENCHMARK_CAPTURE(FromBinaryBuffer, bjdata_optimized / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::bjdata_optimized);
 BENCHMARK_CAPTURE(FromBinaryBuffer, bjdata_optimized / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::bjdata_optimized);
+BENCHMARK_CAPTURE(FromBinaryBuffer, bon8 / jeopardy, TEST_DATA_DIRECTORY "/jeopardy/jeopardy.json", binary_format::bon8);
+BENCHMARK_CAPTURE(FromBinaryBuffer, bon8 / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::bon8);
+BENCHMARK_CAPTURE(FromBinaryBuffer, bon8 / citm_catalog, TEST_DATA_DIRECTORY "/nativejson-benchmark/citm_catalog.json", binary_format::bon8);
+BENCHMARK_CAPTURE(FromBinaryBuffer, bon8 / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::bon8);
 // BSON requires an object at the top level, so the array-rooted test files
 // (jeopardy and the regression files) cannot be captured here
 BENCHMARK_CAPTURE(FromBinaryBuffer, bson / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::bson);
@@ -450,6 +461,8 @@ BENCHMARK_CAPTURE(FromBinaryFile, cbor / twitter, TEST_DATA_DIRECTORY "/nativejs
 BENCHMARK_CAPTURE(FromBinaryFile, ubjson / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::ubjson);
 BENCHMARK_CAPTURE(FromBinaryFile, ubjson / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::ubjson);
 BENCHMARK_CAPTURE(FromBinaryFile, bjdata / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::bjdata);
+BENCHMARK_CAPTURE(FromBinaryFile, bon8 / canada, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", binary_format::bon8);
+BENCHMARK_CAPTURE(FromBinaryFile, bon8 / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::bon8);
 BENCHMARK_CAPTURE(FromBinaryFile, bson / twitter, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", binary_format::bson);
 
 //////////////////////////////////////////////////////////////////////////////
@@ -530,18 +543,21 @@ BENCHMARK_CAPTURE(FromBinaryShape, nested / msgpack, make_nested, binary_format:
 BENCHMARK_CAPTURE(FromBinaryShape, nested / ubjson, make_nested, binary_format::ubjson);
 BENCHMARK_CAPTURE(FromBinaryShape, nested / bjdata, make_nested, binary_format::bjdata);
 BENCHMARK_CAPTURE(FromBinaryShape, nested / bson, make_nested, binary_format::bson);
+BENCHMARK_CAPTURE(FromBinaryShape, nested / bon8, make_nested, binary_format::bon8);
 BENCHMARK_CAPTURE(FromBinaryShape, containers / cbor, make_containers, binary_format::cbor);
 BENCHMARK_CAPTURE(FromBinaryShape, containers / msgpack, make_containers, binary_format::msgpack);
 BENCHMARK_CAPTURE(FromBinaryShape, containers / ubjson, make_containers, binary_format::ubjson);
 BENCHMARK_CAPTURE(FromBinaryShape, containers / ubjson_optimized, make_containers, binary_format::ubjson_optimized);
 BENCHMARK_CAPTURE(FromBinaryShape, containers / bjdata, make_containers, binary_format::bjdata);
 BENCHMARK_CAPTURE(FromBinaryShape, containers / bson, make_containers, binary_format::bson);
+BENCHMARK_CAPTURE(FromBinaryShape, containers / bon8, make_containers, binary_format::bon8);
 // BSON names every array element, so a large array measures key generation
 // rather than scalar decoding and is left out here
 BENCHMARK_CAPTURE(FromBinaryShape, scalars / cbor, make_scalars, binary_format::cbor);
 BENCHMARK_CAPTURE(FromBinaryShape, scalars / msgpack, make_scalars, binary_format::msgpack);
 BENCHMARK_CAPTURE(FromBinaryShape, scalars / ubjson, make_scalars, binary_format::ubjson);
 BENCHMARK_CAPTURE(FromBinaryShape, scalars / bjdata, make_scalars, binary_format::bjdata);
+BENCHMARK_CAPTURE(FromBinaryShape, scalars / bon8, make_scalars, binary_format::bon8);
 
 /*!
 @brief parse an indefinite-length CBOR string

tests/fuzzing.md

@@ -1,6 +1,6 @@
 # Fuzz testing
 
-Each parser of the library (JSON, BJData, BSON, CBOR, MessagePack, and UBJSON) can be fuzz tested. Currently,
+Each parser of the library (JSON, BJData, BON8, BSON, CBOR, MessagePack, and UBJSON) can be fuzz tested. Currently,
 [libFuzzer](https://llvm.org/docs/LibFuzzer.html) and [afl++](https://github.com/AFLplusplus/AFLplusplus) are supported.
 
 ## Corpus creation
@@ -10,11 +10,11 @@ directory with some simple input files that cover several features of the parser
 for mutations.
 
 ```shell
-TEST_DATA_VERSION=3.1.0
+TEST_DATA_VERSION=3.2.0
 wget https://github.com/nlohmann/json_test_data/archive/refs/tags/v$TEST_DATA_VERSION.zip
 unzip v$TEST_DATA_VERSION.zip
 rm v$TEST_DATA_VERSION.zip
-for FORMAT in json bjdata bson cbor msgpack ubjson
+for FORMAT in json bjdata bon8 bson cbor msgpack ubjson
 do
   rm -fr corpus_$FORMAT
   mkdir corpus_$FORMAT

tests/src/fuzzer-parse_bon8.cpp

@@ -0,0 +1,103 @@
+//     __ _____ _____ _____
+//  __|  |   __|     |   | |  JSON for Modern C++ (supporting code)
+// |  |  |__   |  |  | | | |  version 3.12.0
+// |_____|_____|_____|_|___|  https://github.com/nlohmann/json
+//
+// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
+// SPDX-License-Identifier: MIT
+
+/*
+This file implements a parser test suitable for fuzz testing. Given a byte
+array data, it performs the following steps:
+
+- j1 = from_bon8(data)
+- vec = to_bon8(j1)
+- j2 = from_bon8(vec)
+- assert(j1 == j2)
+
+It also checks that reading the data from a stream, which reads strings byte by
+byte, gives the same value or error as reading it from contiguous memory, which
+copies strings in bulk.
+
+The provided function `LLVMFuzzerTestOneInput` can be used in different fuzzer
+drivers.
+*/
+
+#include <cassert>
+#include <iostream>
+#include <sstream>
+#include <nlohmann/json.hpp>
+
+// the round-trip checks below are assertions; NDEBUG would compile them away
+#ifdef NDEBUG
+    #error "the fuzzer drivers must be built without NDEBUG"
+#endif
+
+using json = nlohmann::json;
+
+namespace
+{
+// the serialization of the value read from @a input, or the error message
+template<typename InputType>
+std::string read_bon8(InputType&& input)
+{
+    try
+    {
+        const auto vec = json::to_bon8(json::from_bon8(std::forward<InputType>(input)));
+        return {vec.begin(), vec.end()};
+    }
+    catch (const json::exception& e)
+    {
+        return e.what();
+    }
+}
+} // namespace
+
+// see http://llvm.org/docs/LibFuzzer.html
+extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
+{
+    // contiguous and stream input must be read alike
+    {
+        std::istringstream stream(std::string(reinterpret_cast<const char*>(data), size));
+        assert(read_bon8(std::vector<uint8_t>(data, data + size)) == read_bon8(stream));
+    }
+
+    try
+    {
+        // step 1: parse input
+        std::vector<uint8_t> const vec1(data, data + size);
+        json const j1 = json::from_bon8(vec1);
+
+        try
+        {
+            // step 2: round trip
+            std::vector<uint8_t> const vec2 = json::to_bon8(j1);
+
+            // parse serialization
+            json const j2 = json::from_bon8(vec2);
+
+            // serializations must match
+            assert(json::to_bon8(j2) == vec2);
+        }
+        catch (const json::parse_error&)
+        {
+            // parsing a BON8 serialization must not fail
+            assert(false);
+        }
+    }
+    catch (const json::parse_error&)
+    {
+        // parse errors are ok, because input may be random bytes
+    }
+    catch (const json::type_error&)
+    {
+        // type errors can occur during parsing, too
+    }
+    catch (const json::out_of_range&)
+    {
+        // out of range errors may happen if provided sizes are excessive
+    }
+
+    // return 0 - non-zero return values are reserved for future use
+    return 0;
+}

tests/src/fuzzer-parse_json.cpp

@@ -16,6 +16,10 @@ array data, it performs the following steps:
 - s2 = serialize(j2)
 - assert(s1 == s2)
 
+Furthermore, it parses data with a SAX parser that recovers from every error
+and checks that the events are balanced, that parsing ends, and that valid
+input is parsed without errors (see #3989).
+
 The provided function `LLVMFuzzerTestOneInput` can be used in different fuzzer
 drivers.
 */
@@ -23,6 +27,8 @@ drivers.
 #include <cassert>
 #include <iostream>
 #include <sstream>
+#include <string>
+#include <vector>
 #include <nlohmann/json.hpp>
 
 // the round-trip checks below are assertions; NDEBUG would compile them away
@@ -32,9 +38,143 @@ drivers.
 
 using json = nlohmann::json;
 
+namespace
+{
+// a SAX parser that recovers from every error and checks that the events are
+// balanced and that every key is followed by exactly one value
+class recovering_checker : public nlohmann::json_sax<json>
+{
+  public:
+    bool null() override
+    {
+        return value();
+    }
+
+    bool boolean(bool /*val*/) override
+    {
+        return value();
+    }
+
+    bool number_integer(number_integer_t /*val*/) override
+    {
+        return value();
+    }
+
+    bool number_unsigned(number_unsigned_t /*val*/) override
+    {
+        return value();
+    }
+
+    bool number_float(number_float_t /*val*/, const string_t& /*s*/) override
+    {
+        return value();
+    }
+
+    bool string(string_t& /*val*/) override
+    {
+        return value();
+    }
+
+    bool binary(binary_t& /*val*/) override
+    {
+        return value();
+    }
+
+    bool start_object(std::size_t /*elements*/) override
+    {
+        value();
+        stack.push_back('o');
+        return true;
+    }
+
+    bool key(string_t& /*val*/) override
+    {
+        ++events;
+        assert(!stack.empty() && stack.back() == 'o');
+        stack.back() = 'v';
+        return true;
+    }
+
+    bool end_object() override
+    {
+        ++events;
+        assert(!stack.empty() && stack.back() == 'o');
+        stack.pop_back();
+        return true;
+    }
+
+    bool start_array(std::size_t /*elements*/) override
+    {
+        value();
+        stack.push_back('a');
+        return true;
+    }
+
+    bool end_array() override
+    {
+        ++events;
+        assert(!stack.empty() && stack.back() == 'a');
+        stack.pop_back();
+        return true;
+    }
+
+    bool parse_error(std::size_t /*position*/, const std::string& /*last_token*/, const nlohmann::detail::exception& /*ex*/) override
+    {
+        ++errors;
+        return true;
+    }
+
+    bool complete() const
+    {
+        return stack.empty();
+    }
+
+    std::size_t events = 0;
+    std::size_t errors = 0;
+
+  private:
+    bool value()
+    {
+        ++events;
+        if (!stack.empty())
+        {
+            // an array element, or the value of a key
+            assert(stack.back() != 'o');
+            if (stack.back() == 'v')
+            {
+                stack.back() = 'o';
+            }
+        }
+        return true;
+    }
+
+    // 'a' for an array, 'o' for an object that expects a key, 'v' for an
+    // object that expects the value of a key
+    std::vector<char> stack;
+};
+} // namespace
+
 // see http://llvm.org/docs/LibFuzzer.html
 extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
 {
+    // step 0: recover from all errors, reading from memory and from a stream
+    {
+        recovering_checker checker;
+        const bool ok = json::sax_parse(data, data + size, &checker);
+        assert(checker.complete());
+        assert(checker.errors <= size + 1);
+        assert(checker.events <= (4 * size) + 4);
+        assert(ok == json::accept(data, data + size));
+        assert(ok == (checker.errors == 0));
+
+        std::istringstream stream(std::string(reinterpret_cast<const char*>(data), size));
+        recovering_checker stream_checker;
+        assert(json::sax_parse(stream, &stream_checker) == ok);
+        assert(stream_checker.complete());
+        assert(stream_checker.events == checker.events);
+        assert(stream_checker.errors == checker.errors);
+    }
+
     try
     {
         // step 1: parse input

tests/src/unit-alt-string.cpp

@@ -13,7 +13,6 @@
 
 #include <cstdint>
 #include <string>
-#include <type_traits>
 #include <utility>
 #include <vector>
 
@@ -186,6 +185,7 @@ TEST_CASE("alternative string type")
 
         CHECK(alt_json::from_cbor(alt_json::to_cbor(doc)) == doc);
         CHECK(alt_json::from_msgpack(alt_json::to_msgpack(doc)) == doc);
+        CHECK(alt_json::from_bon8(alt_json::to_bon8(doc)) == doc);
         // BSON is not covered: it additionally needs string_t::find(value_type),
         // which alt_string does not provide
         CHECK(alt_json::from_ubjson(alt_json::to_ubjson(doc)) == doc);
@@ -374,39 +374,4 @@ TEST_CASE("alternative string type")
         const auto j2 = j.flatten();
         CHECK(j2.dump() == R"({"/foo/0":"bar","/foo/1":"baz"})");
     }
-
-    SECTION("conversion between basic_json specializations (#2649)")
-    {
-        // explicit conversions are always possible
-        CHECK(std::is_constructible<nlohmann::json, alt_json>::value);
-        CHECK(std::is_constructible<alt_json, nlohmann::json>::value);
-        CHECK(std::is_constructible<nlohmann::json, nlohmann::ordered_json>::value);
-        CHECK(std::is_constructible<nlohmann::ordered_json, nlohmann::json>::value);
-
-        // specializations with the same string type are implicitly convertible
-        CHECK(std::is_convertible<nlohmann::ordered_json, nlohmann::json>::value);
-        CHECK(std::is_convertible<nlohmann::json, nlohmann::ordered_json>::value);
-
-        // specializations with different string types are only implicitly convertible
-        // if implicit conversions are enabled
-#if JSON_USE_IMPLICIT_CONVERSIONS
-        CHECK(std::is_convertible<alt_json, nlohmann::json>::value);
-        CHECK(std::is_convertible<nlohmann::json, alt_json>::value);
-#else
-        CHECK_FALSE(std::is_convertible<alt_json, nlohmann::json>::value);
-        CHECK_FALSE(std::is_convertible<nlohmann::json, alt_json>::value);
-#endif
-
-        // get<BasicJsonType>() works in either case
-        const nlohmann::json j = {{"foo", 1}, {"bar", true}};
-        CHECK(j.get<nlohmann::ordered_json>() == nlohmann::ordered_json(j));
-        // (only a number is converted here, as objects and strings are affected by #3425)
-        CHECK(nlohmann::json(42).get<alt_json>() == 42);
-        CHECK(alt_json(nlohmann::json(42)) == 42);
-
-        // get_to() also works in either case
-        alt_json a;
-        nlohmann::json(42).get_to(a);
-        CHECK(a == 42);
-    }
 }

tests/src/unit-binary_formats.cpp

@@ -25,6 +25,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
         const auto bjdata_1_size = json::to_bjdata(j).size();
         const auto bjdata_2_size = json::to_bjdata(j, true).size();
         const auto bjdata_3_size = json::to_bjdata(j, true, true).size();
+        const auto bon8_size = json::to_bon8(j).size();
         const auto bson_size = json::to_bson(j).size();
         const auto cbor_size = json::to_cbor(j).size();
         const auto msgpack_size = json::to_msgpack(j).size();
@@ -36,6 +37,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
         CHECK(bjdata_1_size == 1112030);
         CHECK(bjdata_2_size == 1224148);
         CHECK(bjdata_3_size == 1224148);
+        CHECK(bon8_size == 1055792);
         CHECK(bson_size == 1794522);
         CHECK(cbor_size == 1055552);
         CHECK(msgpack_size == 1056145);
@@ -47,6 +49,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
         CHECK((100.0 * double(bjdata_1_size) / double(json_size)) == Approx(53.199));
         CHECK((100.0 * double(bjdata_2_size) / double(json_size)) == Approx(58.563));
         CHECK((100.0 * double(bjdata_3_size) / double(json_size)) == Approx(58.563));
+        CHECK((100.0 * double(bon8_size) / double(json_size)) == Approx(50.509));
         CHECK((100.0 * double(bson_size) / double(json_size)) == Approx(85.849));
         CHECK((100.0 * double(cbor_size) / double(json_size)) == Approx(50.497));
         CHECK((100.0 * double(msgpack_size) / double(json_size)) == Approx(50.526));
@@ -64,6 +67,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
         const auto bjdata_1_size = json::to_bjdata(j).size();
         const auto bjdata_2_size = json::to_bjdata(j, true).size();
         const auto bjdata_3_size = json::to_bjdata(j, true, true).size();
+        const auto bon8_size = json::to_bon8(j).size();
         const auto bson_size = json::to_bson(j).size();
         const auto cbor_size = json::to_cbor(j).size();
         const auto msgpack_size = json::to_msgpack(j).size();
@@ -75,6 +79,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
         CHECK(bjdata_1_size == 425342);
         CHECK(bjdata_2_size == 429970);
         CHECK(bjdata_3_size == 429970);
+        CHECK(bon8_size == 391396);
         CHECK(bson_size == 444568);
         CHECK(cbor_size == 402814);
         CHECK(msgpack_size == 401510);
@@ -86,6 +91,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
         CHECK((100.0 * double(bjdata_1_size) / double(json_size)) == Approx(91.097));
         CHECK((100.0 * double(bjdata_2_size) / double(json_size)) == Approx(92.089));
         CHECK((100.0 * double(bjdata_3_size) / double(json_size)) == Approx(92.089));
+        CHECK((100.0 * double(bon8_size) / double(json_size)) == Approx(83.828));
         CHECK((100.0 * double(bson_size) / double(json_size)) == Approx(95.215));
         CHECK((100.0 * double(cbor_size) / double(json_size)) == Approx(86.273));
         CHECK((100.0 * double(msgpack_size) / double(json_size)) == Approx(85.993));
@@ -103,6 +109,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
         const auto bjdata_1_size = json::to_bjdata(j).size();
         const auto bjdata_2_size = json::to_bjdata(j, true).size();
         const auto bjdata_3_size = json::to_bjdata(j, true, true).size();
+        const auto bon8_size = json::to_bon8(j).size();
         const auto bson_size = json::to_bson(j).size();
         const auto cbor_size = json::to_cbor(j).size();
         const auto msgpack_size = json::to_msgpack(j).size();
@@ -114,6 +121,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
         CHECK(bjdata_1_size == 390781);
         CHECK(bjdata_2_size == 433557);
         CHECK(bjdata_3_size == 432964);
+        CHECK(bon8_size == 317879);
         CHECK(bson_size == 479430);
         CHECK(cbor_size == 342373);
         CHECK(msgpack_size == 342473);
@@ -125,6 +133,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
         CHECK((100.0 * double(bjdata_1_size) / double(json_size)) == Approx(78.109));
         CHECK((100.0 * double(bjdata_2_size) / double(json_size)) == Approx(86.659));
         CHECK((100.0 * double(bjdata_3_size) / double(json_size)) == Approx(86.541));
+        CHECK((100.0 * double(bon8_size) / double(json_size)) == Approx(63.538));
         CHECK((100.0 * double(bson_size) / double(json_size)) == Approx(95.828));
         CHECK((100.0 * double(cbor_size) / double(json_size)) == Approx(68.433));
         CHECK((100.0 * double(msgpack_size) / double(json_size)) == Approx(68.453));
@@ -142,6 +151,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
         const auto bjdata_1_size = json::to_bjdata(j).size();
         const auto bjdata_2_size = json::to_bjdata(j, true).size();
         const auto bjdata_3_size = json::to_bjdata(j, true, true).size();
+        const auto bon8_size = json::to_bon8(j).size();
         const auto bson_size = json::to_bson({{"", j}}).size(); // wrap array in object for BSON
         const auto cbor_size = json::to_cbor(j).size();
         const auto msgpack_size = json::to_msgpack(j).size();
@@ -153,6 +163,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
         CHECK(bjdata_1_size == 50710965);
         CHECK(bjdata_2_size == 51144830);
         CHECK(bjdata_3_size == 51144830);
+        CHECK(bon8_size == 45942080);
         CHECK(bson_size == 56008520);
         CHECK(cbor_size == 46187320);
         CHECK(msgpack_size == 46158575);
@@ -164,6 +175,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
         CHECK((100.0 * double(bjdata_1_size) / double(json_size)) == Approx(96.576));
         CHECK((100.0 * double(bjdata_2_size) / double(json_size)) == Approx(97.402));
         CHECK((100.0 * double(bjdata_3_size) / double(json_size)) == Approx(97.402));
+        CHECK((100.0 * double(bon8_size) / double(json_size)) == Approx(87.494));
         CHECK((100.0 * double(bson_size) / double(json_size)) == Approx(106.665));
         CHECK((100.0 * double(cbor_size) / double(json_size)) == Approx(87.961));
         CHECK((100.0 * double(msgpack_size) / double(json_size)) == Approx(87.906));
@@ -181,6 +193,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
         const auto bjdata_1_size = json::to_bjdata(j).size();
         const auto bjdata_2_size = json::to_bjdata(j, true).size();
         const auto bjdata_3_size = json::to_bjdata(j, true, true).size();
+        const auto bon8_size = json::to_bon8(j).size();
         // BSON cannot process the file as it contains code point  U+0000
         const auto cbor_size = json::to_cbor(j).size();
         const auto msgpack_size = json::to_msgpack(j).size();
@@ -192,6 +205,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
         CHECK(bjdata_1_size == 148695);
         CHECK(bjdata_2_size == 150569);
         CHECK(bjdata_3_size == 150569);
+        CHECK(bon8_size == 144477);
         CHECK(cbor_size == 147095);
         CHECK(msgpack_size == 147017);
         CHECK(ubjson_1_size == 148695);
@@ -202,6 +216,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
         CHECK((100.0 * double(bjdata_1_size) / double(json_size)) == Approx(88.153));
         CHECK((100.0 * double(bjdata_2_size) / double(json_size)) == Approx(89.264));
         CHECK((100.0 * double(bjdata_3_size) / double(json_size)) == Approx(89.264));
+        CHECK((100.0 * double(bon8_size) / double(json_size)) == Approx(85.653));
         CHECK((100.0 * double(cbor_size) / double(json_size)) == Approx(87.205));
         CHECK((100.0 * double(msgpack_size) / double(json_size)) == Approx(87.158));
         CHECK((100.0 * double(ubjson_1_size) / double(json_size)) == Approx(88.153));

tests/src/unit-binary_writer_sinks.cpp

@@ -12,6 +12,7 @@
 using nlohmann::json;
 
 #include <cstdint>
+#include <limits>
 #include <string>
 #include <vector>
 
@@ -49,6 +50,12 @@ std::vector<json> test_values()
     };
 }
 
+// BON8 has no integers above the int64 range, so to_bon8() rejects them
+bool bon8_representable(const json& j)
+{
+    return !j.is_number_unsigned() || j.get<std::uint64_t>() <= static_cast<std::uint64_t>((std::numeric_limits<std::int64_t>::max)());
+}
+
 // values to_bson() accepts: the document must be an object
 std::vector<json> bson_values()
 {
@@ -92,6 +99,13 @@ TEST_CASE("binary writer output sinks")
             json::to_msgpack(j, msgpack);
             CHECK(json::to_msgpack(j) == msgpack);
 
+            if (bon8_representable(j))
+            {
+                std::vector<std::uint8_t> bon8;
+                json::to_bon8(j, bon8);
+                CHECK(json::to_bon8(j) == bon8);
+            }
+
             for (const bool use_size :
                     {
                         false, true
@@ -172,6 +186,10 @@ TEST_CASE("binary_reserve_hint never over-reserves")
         CHECK(hint <= json::to_ubjson(j).size());
         CHECK(hint <= json::to_ubjson(j, true, true).size());
         CHECK(hint <= json::to_bjdata(j).size());
+        if (bon8_representable(j))
+        {
+            CHECK(hint <= json::to_bon8(j).size());
+        }
     }
 
     for (const auto& j : bson_values())

tests/src/unit-bson.cpp

@@ -1066,6 +1066,45 @@ TEST_CASE("Incomplete BSON Input")
     }
 }
 
+// the test catches the exceptions of invalid input
+#if !defined(JSON_NOEXCEPTION)
+TEST_CASE("BSON keys from contiguous and stream input")
+{
+    // contiguous input reads a key up to its \x00-byte in one step, a stream
+    // reads it byte by byte; both must give the same value or error for the
+    // complete document and for every truncation of it
+    const json j = {{"", true}, {"k", {1, 2, 3}}, {std::string(40, 'x'), {{"nested key", "value"}}}};
+    const std::vector<std::uint8_t> bson = json::to_bson(j);
+    CHECK(json::from_bson(bson) == j);
+
+    for (std::size_t length = 0; length <= bson.size(); ++length)
+    {
+        CAPTURE(length)
+        const std::vector<std::uint8_t> input(bson.begin(), bson.begin() + static_cast<std::ptrdiff_t>(length));
+        std::string from_vector;
+        std::string from_stream;
+        try
+        {
+            from_vector = json::from_bson(input).dump();
+        }
+        catch (const json::parse_error& e)
+        {
+            from_vector = e.what();
+        }
+        try
+        {
+            std::istringstream stream(std::string(input.begin(), input.end()));
+            from_stream = json::from_bson(stream).dump();
+        }
+        catch (const json::parse_error& e)
+        {
+            from_stream = e.what();
+        }
+        CHECK(from_vector == from_stream);
+    }
+}
+#endif
+
 TEST_CASE("Negative size of binary value")
 {
     // invalid BSON: the size of the binary value is -1

tests/src/unit-class_parser.cpp

@@ -2761,7 +2761,7 @@ TEST_CASE("diagnostic positions: value lifetime, input adapters, and SAX")
 
         SECTION("binary formats have no text positions")
         {
-            // binary formats (CBOR, MessagePack, UBJSON, BSON, BJData) are
+            // binary formats (BJData, BON8, BSON, CBOR, MessagePack, UBJSON) are
             // parsed via detail::binary_reader, which never sets
             // start_position/end_position on the values it produces (they
             // have no notion of a text offset), so every value's position
@@ -2778,6 +2778,10 @@ TEST_CASE("diagnostic positions: value lifetime, input adapters, and SAX")
             CHECK(from_msgpack.start_pos() == std::string::npos);
             CHECK(from_msgpack.end_pos() == std::string::npos);
 
+            const json from_bon8 = json::from_bon8(json::to_bon8(src));
+            CHECK(from_bon8.start_pos() == std::string::npos);
+            CHECK(from_bon8.end_pos() == std::string::npos);
+
             const json from_ubjson = json::from_ubjson(json::to_ubjson(src));
             CHECK(from_ubjson.start_pos() == std::string::npos);
             CHECK(from_ubjson.end_pos() == std::string::npos);
@@ -2813,3 +2817,586 @@ TEST_CASE("diagnostic positions: value lifetime, input adapters, and SAX")
     }
 }
 #endif
+
+namespace
+{
+/// builds a value like json::parse(), but asks the parser to recover from
+/// errors (see #3989), and checks that the events it receives are balanced
+class RecoveringDomParser : public nlohmann::detail::json_sax_dom_parser<json>
+{
+    using base = nlohmann::detail::json_sax_dom_parser<json>;
+
+  public:
+    explicit RecoveringDomParser(json& j, std::size_t max_errors_ = static_cast<std::size_t>(-1))
+        : base(j, false)
+        , max_errors(max_errors_)
+    {}
+
+    bool null()
+    {
+        value();
+        return base::null();
+    }
+
+    bool boolean(bool val)
+    {
+        value();
+        return base::boolean(val);
+    }
+
+    bool number_integer(json::number_integer_t val)
+    {
+        value();
+        return base::number_integer(val);
+    }
+
+    bool number_unsigned(json::number_unsigned_t val)
+    {
+        value();
+        return base::number_unsigned(val);
+    }
+
+    bool number_float(json::number_float_t val, const std::string& s)
+    {
+        value();
+        return base::number_float(val, s);
+    }
+
+    bool string(std::string& val)
+    {
+        value();
+        return base::string(val);
+    }
+
+    bool start_object(std::size_t elements)
+    {
+        value();
+        stack.push_back('o');
+        return base::start_object(elements);
+    }
+
+    bool key(std::string& val)
+    {
+        ++events;
+        if (stack.empty() || stack.back() != 'o')
+        {
+            well_formed = false;
+            return false;
+        }
+        stack.back() = 'v';
+        return base::key(val);
+    }
+
+    bool end_object()
+    {
+        ++events;
+        if (stack.empty() || stack.back() != 'o')
+        {
+            well_formed = false;
+            return false;
+        }
+        stack.pop_back();
+        return base::end_object();
+    }
+
+    bool start_array(std::size_t elements)
+    {
+        value();
+        stack.push_back('a');
+        return base::start_array(elements);
+    }
+
+    bool end_array()
+    {
+        ++events;
+        if (stack.empty() || stack.back() != 'a')
+        {
+            well_formed = false;
+            return false;
+        }
+        stack.pop_back();
+        return base::end_array();
+    }
+
+    bool parse_error(std::size_t /*unused*/, const std::string& /*unused*/, const json::exception& ex)
+    {
+        errors.emplace_back(ex.what());
+        return errors.size() < max_errors;
+    }
+
+    /// whether the events were balanced and every key was followed by a value
+    bool balanced() const
+    {
+        return well_formed && stack.empty();
+    }
+
+    std::vector<std::string> errors {}; // NOLINT(readability-redundant-member-init)
+    std::size_t events = 0;
+    /// the open containers: 'a' for an array, 'o' for an object that expects
+    /// a key, 'v' for an object that expects the value of a key
+    std::vector<char> stack {}; // NOLINT(readability-redundant-member-init)
+    bool well_formed = true;
+    std::size_t max_errors;
+
+  private:
+    /// a value is passed: it is an array element, or the value of a key
+    void value()
+    {
+        ++events;
+        if (!stack.empty())
+        {
+            if (stack.back() == 'v')
+            {
+                stack.back() = 'o';
+            }
+            else if (stack.back() == 'o')
+            {
+                // a value without a key
+                well_formed = false;
+            }
+        }
+    }
+
+};
+
+struct RecoveryResult
+{
+    json value;
+    std::vector<std::string> errors;
+    std::size_t events;
+    bool ok;
+    bool balanced;
+};
+
+template<typename InputType>
+RecoveryResult parse_recovering(InputType&& input, const bool strict = true,
+                                const bool ignore_comments = false, const bool ignore_trailing_commas = false)
+{
+    json j;
+    RecoveringDomParser sax(j);
+    const bool ok = json::sax_parse(std::forward<InputType>(input), &sax, json::input_format_t::json,
+                                    strict, ignore_comments, ignore_trailing_commas);
+    return {j, sax.errors, sax.events, ok, sax.balanced()};
+}
+
+/// logs the events as strings and recovers from errors
+class RecoveringEventLogger : public SaxEventLogger
+{
+  public:
+    bool parse_error(std::size_t position, const std::string& /*unused*/, const json::exception& /*unused*/)
+    {
+        events.push_back("parse_error(" + std::to_string(position) + ")");
+        return true;
+    }
+};
+
+/// stops after a number of events, but recovers from errors
+class RecoveringCountdown : public SaxCountdown
+{
+  public:
+    using SaxCountdown::SaxCountdown;
+
+    bool parse_error(std::size_t /*position*/, const std::string& /*last_token*/, const json::exception& /*ex*/) override
+    {
+        return true;
+    }
+};
+
+/// a repaired input: the value it is repaired to, and the number of errors
+struct Repair
+{
+    const char* input;
+    const char* expected;
+    std::size_t errors;
+};
+} // namespace
+
+TEST_CASE("parser error recovery (#3989)")
+{
+    SECTION("repairs")
+    {
+        const std::vector<Repair> repairs =
+        {
+            // a missing separator is inserted
+            {"[1 2]", "[1,2]", 1},
+            {R"({"a":1 "b":2})", R"({"a":1,"b":2})", 1},
+            {R"({"a" 1})", R"({"a":1})", 1},
+            {"[1 tru 2]", "[1,null,2]", 2},
+            {R"({"a" "b": 1})", R"({"a":"b"})", 2},
+
+            // a missing value is null in an object; in an array, a ',' stands
+            // for null, while an array that ends there just ends
+            {R"({"a":})", R"({"a":null})", 1},
+            {R"({"a"})", R"({"a":null})", 1},
+            {R"({"a","b":1})", R"({"a":null,"b":1})", 1},
+            {"[1,,2]", "[1,null,2]", 1},
+            {"[,1]", "[null,1]", 1},
+            {"[1,]", "[1]", 1},
+            {"[1,2,3,]", "[1,2,3]", 1},
+            {R"({"a":1,})", R"({"a":1})", 1},
+
+            // a broken string keeps what can be read
+            {R"(["a\qb"])", R"(["aqb"])", 1},
+            {R"({"na\me":1})", R"({"name":1})", 1},
+            {"[\"\xFF\"]", R"(["\uFFFD"])", 1},
+            {"[\"a\xC3(\"]", R"(["a\uFFFD("])", 1},
+            {"[\"\xE2\x82\"]", R"(["\uFFFD"])", 1},
+            {"[\"\xC3\\\\\", 1]", R"(["\uFFFD\\",1])", 1},
+            {R"(["\u12"])", R"(["\uFFFD"])", 1},
+            {R"(["\u12G4"])", R"(["\uFFFDG4"])", 1},
+            {R"(["\uDC00x"])", R"(["\uFFFDx"])", 1},
+            {R"(["\uD800x"])", R"(["\uFFFDx"])", 1},
+            {R"(["\uD800\u0041"])", R"(["\uFFFDA"])", 1},
+            {R"(["\uD800\uD800\uDC00"])", R"(["\uFFFD\uD800\uDC00"])", 1},
+            {R"(["\uD800\uD800\uD800x"])", R"(["\uFFFD\uFFFD\uFFFDx"])", 1},
+            {
+                R"(["\uD800\"x", 1])", R"(["\uFFFD\"x",1])", 1
+            },
+            {R"(["\uD800\q"])", R"(["\uFFFDq"])", 1},
+            {"[\"a\tb\"]", R"(["a\tb"])", 1},
+            {R"(["a\qb\u0041\x"])", R"(["aqbAx"])", 1},
+
+            // a broken number keeps its longest valid prefix
+            {"[1.]", "[1]", 1},
+            {"[-2.]", "[-2]", 1},
+            {"[1.5e]", "[1.5]", 1},
+            {"[1e+]", "[1]", 1},
+            {"[1.x2, 3]", "[1,3]", 1},
+
+            // what cannot be read at all is null
+            {"[1,NaN,3]", "[1,null,3]", 1},
+            {"[tru]", "[null]", 1},
+            {"[-]", "[null]", 1},
+            {R"({"a":Infinity})", R"({"a":null})", 1},
+
+            // a stray token is dropped
+            {"[:1]", "[1]", 1},
+            {R"(["a":1])", R"(["a",1])", 1},
+            {R"({"a"::1})", R"({"a":1})", 1},
+
+            // a member that cannot be read is skipped
+            {R"({1:2,"b":3})", R"({"b":3})", 1},
+            {R"({"a":1 2})", R"({"a":1})", 1},
+            {R"({,"a":1})", R"({"a":1})", 1},
+            {R"({"a":1,,"b":2})", R"({"a":1,"b":2})", 1},
+            {"{a:1}", "{}", 1},
+            {R"({"a":1 [1,{"b":2}], "c":3})", R"({"a":1,"c":3})", 1},
+            {R"([{1}, "a"])", R"([{},"a"])", 1},
+
+            // a wrong closing bracket closes the innermost container
+            {R"({"a":[1,2}, "b":3})", R"({"a":[1,2],"b":3})", 1},
+            {R"([{"a":1], 2])", R"([{"a":1},2])", 1},
+            {"{]", "{}", 1},
+            {"[}", "[]", 1},
+
+            // the end of the input closes all containers
+            {R"({"a":[1,2)", R"({"a":[1,2]})", 1},
+            {"[", "[]", 1},
+            {"{", "{}", 1},
+            {R"({"a")", R"({"a":null})", 1},
+            {R"({"a":)", R"({"a":null})", 1},
+            {"[1,", "[1]", 1},
+            {"[[[1", "[[[1]]]", 1},
+            {
+                R"(["abc)", R"(["abc"])", 2
+            },
+            {"[1,tr", "[1,null]", 2},
+            {"\"abc", "\"abc\"", 1},
+            {"[\"ab\ncd\"]", R"(["ab",null,"]"])", 4},
+
+            // what comes before the top-level value is skipped
+            {")]}'\n{\"a\":1}", R"({"a":1})", 1},
+            {R"(data: {"a":1})", R"({"a":1})", 1},
+            {"\xEF\xBB[1]", "[1]", 1},
+
+            // what comes after it is an error that ends parsing
+            {R"({"a":1}})", R"({"a":1})", 1},
+            {"[1}]", "[1]", 2},
+            {"[1] [2]", "[1]", 1},
+        };
+
+        for (const auto& repair : repairs)
+        {
+            CAPTURE(repair.input);
+            const auto result = parse_recovering(std::string(repair.input));
+            CHECK(!result.ok);
+            CHECK(result.balanced);
+            CHECK(result.value == json::parse(repair.expected));
+            CHECK(result.errors.size() == repair.errors);
+        }
+    }
+
+    SECTION("number overflow")
+    {
+        const auto result = parse_recovering(std::string("[1e999,-1e999]"));
+        CHECK(!result.ok);
+        CHECK(result.balanced);
+        CHECK(result.errors.size() == 2);
+        CHECK(result.errors[0] == "[json.exception.out_of_range.406] number overflow parsing '1e999'");
+        REQUIRE(result.value.size() == 2);
+        CHECK(result.value[0].is_number_float());
+        CHECK(result.value[0].get<double>() == std::numeric_limits<double>::infinity());
+        CHECK(result.value[1].get<double>() == -std::numeric_limits<double>::infinity());
+
+        // the SAX parser gets the number's text
+        RecoveringEventLogger logger;
+        CHECK(!json::sax_parse("1e999", &logger));
+        CHECK(logger.events == std::vector<std::string>({"parse_error(5)", "number_float(1e999)"}));
+    }
+
+    SECTION("nothing to recover")
+    {
+        for (const std::string s :
+                {
+                    "", "   ", "]", "tru", "NaN", ",:", "/* comment"
+                })
+        {
+            CAPTURE(s);
+            const auto result = parse_recovering(s, true, true);
+            CHECK(!result.ok);
+            CHECK(result.balanced);
+            CHECK(result.events == 0);
+            CHECK(result.value == nullptr);
+            CHECK(result.errors.size() == 1);
+        }
+    }
+
+    SECTION("error messages")
+    {
+        // the first error is reported as without recovery
+        for (const std::string s :
+                {
+                    "[1 2]", R"({"a":1 "b":2})", R"({"a" 1})", R"({"a":})", "[1,]", "[1.]",
+                    R"(["a\qb"])", "[1e999]", "{1:2}", R"({"a":[1,2}})", "[1,", "[1] [2]", "{a:1}"
+                })
+        {
+            CAPTURE(s);
+            const auto result = parse_recovering(s);
+            REQUIRE(!result.errors.empty());
+            json _;
+            CHECK_THROWS_WITH_STD_STR(_ = json::parse(s), result.errors.front());
+        }
+
+        // the token of an error begins where the previous error was
+        const auto result = parse_recovering(std::string("[tru, fals, nul]"));
+        CHECK(result.errors == std::vector<std::string>(
+        {
+            "[json.exception.parse_error.101] parse error at line 1, column 5: syntax error while parsing value - invalid literal; last read: '[tru,'",
+            "[json.exception.parse_error.101] parse error at line 1, column 11: syntax error while parsing value - invalid literal; last read: ', fals,'",
+            "[json.exception.parse_error.101] parse error at line 1, column 16: syntax error while parsing value - invalid literal; last read: ', nul]'"
+        }));
+        CHECK(result.value == json::parse("[null,null,null]"));
+    }
+
+    SECTION("events")
+    {
+        // see #4522
+        RecoveringEventLogger logger;
+        CHECK(!json::sax_parse(R"([{1}, "a"])", &logger));
+        CHECK(logger.events == std::vector<std::string>(
+        {
+            "start_array()", "start_object()", "parse_error(3)", "end_object()", "string(a)", "end_array()"
+        }));
+    }
+
+    SECTION("options")
+    {
+        SECTION("strict")
+        {
+            const auto result = parse_recovering(std::string("[1 2] [3]"), false);
+            CHECK(!result.ok);
+            CHECK(result.value == json::parse("[1,2]"));
+            CHECK(result.errors.size() == 1);
+        }
+
+        SECTION("ignore_trailing_commas")
+        {
+            for (const std::string s :
+                    {
+                        "[1,]", R"({"a":1,})", "[[1,],]"
+                    })
+            {
+                CAPTURE(s);
+                const auto result = parse_recovering(s, true, false, true);
+                CHECK(result.ok);
+                CHECK(result.errors.empty());
+            }
+
+            auto result = parse_recovering(std::string("[1,,]"), true, false, true);
+            CHECK(result.value == json::parse("[1,null]"));
+            CHECK(result.errors.size() == 1);
+
+            result = parse_recovering(std::string(R"({"a":1,,})"), true, false, true);
+            CHECK(result.value == json::parse(R"({"a":1})"));
+            CHECK(result.errors.size() == 1);
+        }
+
+        SECTION("ignore_comments")
+        {
+            auto result = parse_recovering(std::string("[1 /* one */ 2]"), true, true);
+            CHECK(result.value == json::parse("[1,2]"));
+            CHECK(result.errors.size() == 1);
+
+            // a comment that is not closed runs to the end of the input, which
+            // is not reported again
+            result = parse_recovering(std::string("[1, 2 /* unterminated"), true, true);
+            CHECK(result.balanced);
+            CHECK(result.value == json::parse("[1,2]"));
+            CHECK(result.errors.size() == 1);
+
+            // a '/' that does not begin a comment is garbage
+            result = parse_recovering(std::string("[1, /x, 2]"), true, true);
+            CHECK(result.balanced);
+            CHECK(result.value == json::parse("[1,null,2]"));
+            CHECK(result.errors.size() == 1);
+        }
+    }
+
+    SECTION("null bytes")
+    {
+        // a null byte ends the input, unless JSON_STRICT_NUL_HANDLING is set
+        const auto result = parse_recovering(std::string("[1,\0x", 5));
+        CHECK(result.balanced);
+        CHECK(!result.ok);
+#ifdef JSON_TEST_STRICT_NUL_HANDLING_ENABLED
+        CHECK(result.value == json::parse("[1,null]"));
+#else
+        CHECK(result.value == json::parse("[1]"));
+        CHECK(result.errors.size() == 1);
+#endif
+
+        const auto in_string = parse_recovering(std::string("[\"a\0b\"]", 7));
+        CHECK(in_string.balanced);
+#ifdef JSON_TEST_STRICT_NUL_HANDLING_ENABLED
+        CHECK(in_string.value == json::parse(R"(["a\u0000b"])"));
+#else
+        CHECK(in_string.value == json::parse(R"(["a"])"));
+#endif
+    }
+
+    SECTION("the SAX parser stops recovering")
+    {
+        json j;
+        RecoveringDomParser sax(j, 2);
+        CHECK(!json::sax_parse("[1 2 3 4 5]", &sax));
+        CHECK(sax.errors.size() == 2);
+
+        // an error at a delimiter that an invalid token consumed is reported
+        // to the SAX parser, too
+        json j2;
+        RecoveringDomParser sax2(j2, 2);
+        CHECK(!json::sax_parse("[tru}, 1]", &sax2));
+        CHECK(sax2.errors.size() == 2);
+    }
+
+    SECTION("an event stops parsing during a repair")
+    {
+        // start_object() and key() are passed, then null() for the missing
+        // value returns false
+        RecoveringCountdown countdown(2);
+        CHECK(!json::sax_parse(R"({"a":})", &countdown));
+
+        // the end of the input: end_array() for the second array returns false
+        RecoveringCountdown countdown2(4);
+        CHECK(!json::sax_parse("[[1", &countdown2));
+    }
+
+    SECTION("input adapters")
+    {
+        // the lexer reads contiguous and streaming input differently, and it
+        // puts back a character that ended an invalid token
+        for (const std::string s :
+                {
+                    "[1 2]", "[tru}, 1]", R"({"a" "b\q", "c":[1.x, 2}})", "[\"\xFF\xC3(\", -, 1e+]", "{a:1,\"b\":2", ")]}' [1]"
+                })
+        {
+            CAPTURE(s);
+            const auto reference = parse_recovering(s);
+            CHECK(reference.balanced);
+
+            const auto from_c_string = parse_recovering(s.c_str());
+            CHECK(from_c_string.value == reference.value);
+            CHECK(from_c_string.errors == reference.errors);
+
+            const std::list<char> l(s.begin(), s.end());
+            json j;
+            RecoveringDomParser sax(j);
+            CHECK(!json::sax_parse(l.begin(), l.end(), &sax));
+            CHECK(j == reference.value);
+            CHECK(sax.errors == reference.errors);
+
+            std::istringstream ss(s);
+            const auto from_stream = parse_recovering(ss);
+            CHECK(from_stream.value == reference.value);
+            CHECK(from_stream.errors == reference.errors);
+        }
+    }
+
+    SECTION("long runs of errors")
+    {
+        // no error may copy all the input read before it
+        const auto closing = parse_recovering("[" + std::string(100000, '}'));
+        CHECK(closing.balanced);
+        CHECK(closing.value == json::array());
+
+        const auto garbage = parse_recovering("[" + std::string(100000, 'x') + "]");
+        CHECK(garbage.balanced);
+        CHECK(garbage.errors.size() == 1);
+
+        const auto commas = parse_recovering("{" + std::string(100000, ',') + "}");
+        CHECK(commas.balanced);
+        CHECK(commas.value == json::object());
+    }
+
+    SECTION("mutations of valid input")
+    {
+        // whatever the input, the events are balanced, every error is reported
+        // at most once, and valid input is parsed as usual
+        const std::vector<std::string> documents =
+        {
+            R"({"name": "value", "list": [1, -2.5, true, null, {"x": [[]]}], "e": "\u00e9"})",
+            R"([{"a": [1, 2, {"b": "c"}]}, [], {}, "\ud83d\ude00", 1e10])",
+            "{\"\xC3\xA9\": \"\xF0\x9F\x98\x80\"}",
+            R"(  {"k" : [ "v" , 0 ] }  )",
+        };
+        // each character that can be inserted, including a null byte
+        const std::string insertions("[]{},:\"x\\\0\xFF", 11);
+
+        std::vector<std::string> inputs;
+        for (const auto& doc : documents)
+        {
+            for (std::size_t i = 0; i <= doc.size(); ++i)
+            {
+                inputs.push_back(doc.substr(0, i));
+                if (i < doc.size())
+                {
+                    inputs.push_back(doc.substr(0, i) + doc.substr(i + 1));
+                }
+                for (const char c : insertions)
+                {
+                    inputs.push_back(doc.substr(0, i) + c + doc.substr(i));
+                }
+            }
+        }
+
+        for (const auto& s : inputs)
+        {
+            CAPTURE(s);
+            const auto result = parse_recovering(s);
+            CHECK(result.balanced);
+            CHECK(result.errors.size() <= s.size() + 1);
+            CHECK(result.events <= (4 * s.size()) + 4);
+            if (json::accept(s))
+            {
+                CHECK(result.ok);
+                CHECK(result.errors.empty());
+                CHECK(result.value == json::parse(s));
+            }
+            else
+            {
+                CHECK(!result.ok);
+                CHECK(!result.errors.empty());
+            }
+        }
+    }
+}

tests/src/unit-custom-binary-type.cpp

@@ -84,6 +84,8 @@ TEST_CASE("binary type whose value type is not std::uint8_t")
 
         // UBJSON has no binary type, so binary values are written as an array
         CHECK(byte_binary_json::from_ubjson(byte_binary_json::to_ubjson(j)) == byte_binary_json({0, 1, 255}));
+        // the same holds for BON8
+        CHECK(byte_binary_json::from_bon8(byte_binary_json::to_bon8(j)) == byte_binary_json({0, 1, 255}));
     }
 #endif
 }

tests/src/unit-custom-object-type.cpp

@@ -297,6 +297,7 @@ TEST_CASE("object type without key_compare")
         const auto j = no_key_compare_json::parse(R"({"a":[1,2,3],"b":"x"})");
         CHECK(no_key_compare_json::from_cbor(no_key_compare_json::to_cbor(j)) == j);
         CHECK(no_key_compare_json::from_msgpack(no_key_compare_json::to_msgpack(j)) == j);
+        CHECK(no_key_compare_json::from_bon8(no_key_compare_json::to_bon8(j)) == j);
     }
 
     SECTION("flatten and unflatten")

tests/src/unit-explicit_instantiation.cpp

@@ -0,0 +1,37 @@
+//     __ _____ _____ _____
+//  __|  |   __|     |   | |  JSON for Modern C++ (supporting code)
+// |  |  |__   |  |  | | | |  version 3.12.0
+// |_____|_____|_____|_|___|  https://github.com/nlohmann/json
+//
+// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
+// SPDX-License-Identifier: MIT
+
+// cmake/test.cmake selects the C++ standard versions with which to build a
+// unit test based on the presence of JSON_HAS_CPP_<VERSION> macros.
+// The regression below only showed on C++17, so build this file for every
+// standard like the other regression tests:
+// JSON_HAS_CPP_17 JSON_HAS_CPP_20 (do not remove; see note at top of file)
+
+#include "doctest_compatibility.h"
+
+#include <nlohmann/json.hpp>
+using json = nlohmann::json;
+
+/////////////////////////////////////////////////////////////////////
+// for #4825 - explicitly instantiating basic_json must compile; this
+// forces instantiation of binary_writer::write_bjdata_ndarray, whose
+// static_cast<string_t> was ambiguous under explicit instantiation on
+// C++17. Merely compiling this translation unit is the regression test.
+//
+// The instantiation compiles every member function, so it has a file of its
+// own: in unit-regression3.cpp it made the object too large for the MinGW
+// linker to relocate (see #5511).
+/////////////////////////////////////////////////////////////////////
+template class nlohmann::basic_json<>;
+
+TEST_CASE("explicit instantiation of basic_json (#4825)")
+{
+    const json j = {1, "two", 3.0};
+    CHECK(j.size() == 3);
+    CHECK(json::from_bjdata(json::to_bjdata(j)) == j);
+}

tests/src/unit-msgpack.cpp

@@ -2165,6 +2165,13 @@ TEST_CASE("MessagePack with std::byte")
 #endif
 
 // the fake sizes below do not fit into a 32-bit std::size_t
+// with clang and libstdc++ 10, the std::filesystem::path conversion that
+// C++17 builds consider for every string type is ambiguous for a class
+// derived from std::string, so the string case is not tested there
+#if !(defined(__clang__) && defined(_GLIBCXX_RELEASE) && _GLIBCXX_RELEASE < 11)
+    #define JSON_TEST_BEYOND_UINT32_STRING 1
+#endif
+
 #if SIZE_MAX > UINT32_MAX
 template<typename T, typename A = std::allocator<T>>
 struct huge_array : std::vector<T, A>
@@ -2262,11 +2269,12 @@ TEST_CASE("MessagePack Size above uint32 for object")
     object.fake_size = false;
 }
 
+#ifdef JSON_TEST_BEYOND_UINT32_STRING
 struct huge_string : std::string
 {
     using std::string::string;
 
-    std::size_t size() const noexcept
+    std::size_t size() const noexcept // NOLINT(readability-convert-member-functions-to-static)
     {
         return static_cast<std::size_t>(UINT32_MAX) + 1ULL;
     }
@@ -2287,20 +2295,20 @@ using huge_string_json = nlohmann::basic_json <
 
 TEST_CASE("MessagePack Size above uint32 for string")
 {
-
-    huge_string_json j = "hello";
+    const huge_string_json j = "hello";
 
     CHECK_THROWS_WITH_AS(
         huge_string_json::to_msgpack(j),
         "[json.exception.out_of_range.412] MessagePack length 4294967296 exceeds maximum of 4294967295",
         json::out_of_range&);
 }
+#endif
 
 struct huge_binary : std::vector<std::uint8_t>
 {
     using std::vector<std::uint8_t>::vector;
 
-    std::size_t size() const noexcept
+    std::size_t size() const noexcept // NOLINT(readability-convert-member-functions-to-static)
     {
         return static_cast<std::size_t>(UINT32_MAX) + 1ULL;
     }
@@ -2355,13 +2363,6 @@ class beyond_uint32_binary_t : public std::vector<std::uint8_t>
     }
 };
 
-// with clang and libstdc++ 10, the std::filesystem::path conversion that
-// C++17 builds consider for every string type is ambiguous for a class
-// derived from std::string, so the string case is not tested there
-#if !(defined(__clang__) && defined(_GLIBCXX_RELEASE) && _GLIBCXX_RELEASE < 11)
-    #define JSON_TEST_BEYOND_UINT32_STRING 1
-#endif
-
 #ifdef JSON_TEST_BEYOND_UINT32_STRING
 class beyond_uint32_string_t : public std::string
 {

tests/src/unit-ordered_json2.cpp

@@ -326,6 +326,15 @@ TEST_CASE("ordered_json across binary formats")
         CHECK(collect_keys(restored) == original_keys);
         CHECK(collect_keys(restored["mango"]) == original_mango_keys);
     }
+
+    SECTION("BON8")
+    {
+        const auto bytes = ordered_json::to_bon8(original);
+        const auto restored = ordered_json::from_bon8(bytes);
+        CHECK(restored == original);
+        CHECK(collect_keys(restored) == original_keys);
+        CHECK(collect_keys(restored["mango"]) == original_mango_keys);
+    }
 }
 
 TEST_CASE("alt_json (custom string_t) across binary formats")
@@ -353,6 +362,13 @@ TEST_CASE("alt_json (custom string_t) across binary formats")
         CHECK(restored == original);
     }
 
+    SECTION("BON8")
+    {
+        const auto bytes = alt_json::to_bon8(original);
+        const auto restored = alt_json::from_bon8(bytes);
+        CHECK(restored == original);
+    }
+
     SECTION("BSON")
     {
         const auto bytes = alt_json::to_bson(original);

tests/src/unit-regression2.cpp

@@ -332,6 +332,7 @@ TEST_CASE("regression tests 2")
         CHECK(float_json::from_cbor(float_json::to_cbor(j)) == j);
         CHECK(float_json::from_msgpack(float_json::to_msgpack(j)) == j);
         CHECK(float_json::from_ubjson(float_json::to_ubjson(j)) == j);
+        CHECK(float_json::from_bon8(float_json::to_bon8(j)) == j);
 
         float_json j2 = {1000.0, 2000.0, 3000.0};
         CHECK(float_json::from_ubjson(float_json::to_ubjson(j2, true, true)) == j2);
@@ -869,4 +870,262 @@ TEST_CASE("regression test - excessive binary container size honors allow_except
     CHECK(json::from_cbor(std::vector<std::uint8_t> {0x9b, 0, 0, 0, 0, 0, 0, 0, 0x02}, true, false).is_discarded());
 }
 
+namespace
+{
+/// builds a value from SAX events, asks the parser to recover from its first
+/// 100 errors, and checks that the events are balanced (see #3989)
+class RecoveringParser : public nlohmann::detail::json_sax_dom_parser<json>
+{
+    using base = nlohmann::detail::json_sax_dom_parser<json>;
+
+  public:
+    explicit RecoveringParser(json& j)
+        : base(j, false)
+    {}
+
+    bool null()
+    {
+        value();
+        return base::null();
+    }
+
+    bool boolean(bool val)
+    {
+        value();
+        return base::boolean(val);
+    }
+
+    bool number_integer(json::number_integer_t val)
+    {
+        value();
+        return base::number_integer(val);
+    }
+
+    bool number_unsigned(json::number_unsigned_t val)
+    {
+        value();
+        return base::number_unsigned(val);
+    }
+
+    bool number_float(json::number_float_t val, const std::string& s)
+    {
+        value();
+        return base::number_float(val, s);
+    }
+
+    bool string(std::string& val)
+    {
+        value();
+        return base::string(val);
+    }
+
+    bool binary(json::binary_t& val)
+    {
+        value();
+        return base::binary(val);
+    }
+
+    bool start_object(std::size_t elements)
+    {
+        value();
+        stack.push_back('o');
+        return base::start_object(elements);
+    }
+
+    bool key(std::string& val)
+    {
+        if (stack.empty() || stack.back() != 'o')
+        {
+            well_formed = false;
+            return false;
+        }
+        stack.back() = 'v';
+        return base::key(val);
+    }
+
+    bool end_object()
+    {
+        if (stack.empty() || stack.back() != 'o')
+        {
+            well_formed = false;
+            return false;
+        }
+        stack.pop_back();
+        return base::end_object();
+    }
+
+    bool start_array(std::size_t elements)
+    {
+        value();
+        stack.push_back('a');
+        return base::start_array(elements);
+    }
+
+    bool end_array()
+    {
+        if (stack.empty() || stack.back() != 'a')
+        {
+            well_formed = false;
+            return false;
+        }
+        stack.pop_back();
+        return base::end_array();
+    }
+
+    bool parse_error(std::size_t /*unused*/, const std::string& /*unused*/, const json::exception& /*unused*/)
+    {
+        // a limit, so that a reader that does not stop fails the test
+        // instead of making it hang
+        return ++errors < 100;
+    }
+
+    /// whether the events were balanced and every key was followed by a value
+    bool balanced() const
+    {
+        return well_formed && stack.empty();
+    }
+
+    std::size_t errors = 0;
+    std::vector<char> stack {}; // NOLINT(readability-redundant-member-init)
+    bool well_formed = true;
+
+  private:
+    void value()
+    {
+        if (!stack.empty())
+        {
+            if (stack.back() == 'v')
+            {
+                stack.back() = 'o';
+            }
+            else if (stack.back() == 'o')
+            {
+                well_formed = false;
+            }
+        }
+    }
+
+};
+
+struct BinaryParseResult
+{
+    json value;
+    std::size_t errors;
+    bool ok;
+    bool balanced;
+};
+
+BinaryParseResult parse_binary_recovering(const std::vector<std::uint8_t>& input, const json::input_format_t format)
+{
+    json j;
+    RecoveringParser sax(j);
+    const bool ok = json::sax_parse(input, &sax, format);
+    return {j, sax.errors, ok, sax.balanced()};
+}
+}  // namespace
+
+TEST_CASE("regression test - #3989 SAX parse_error() returning true")
+{
+    SECTION("binary formats stop after an error and complete what was read")
+    {
+        const json j = {{"a", {1, -2, {{"b", "c"}}, json::array()}}, {"d", {{"e", nullptr}, {"f", true}}}, {"g", 1.5}, {"h", json::binary({1, 2, 3})}};
+
+        const std::vector<std::pair<json::input_format_t, std::vector<std::uint8_t>>> encodings =
+        {
+            {json::input_format_t::cbor, json::to_cbor(j)},
+            {json::input_format_t::msgpack, json::to_msgpack(j)},
+            {json::input_format_t::ubjson, json::to_ubjson(j)},
+            {json::input_format_t::ubjson, json::to_ubjson(j, true, true)},
+            {json::input_format_t::bjdata, json::to_bjdata(j)},
+            {json::input_format_t::bjdata, json::to_bjdata(j, true, true)},
+            {json::input_format_t::bson, json::to_bson(j)},
+            {json::input_format_t::bon8, json::to_bon8(j)},
+        };
+
+        for (const auto& encoding : encodings)
+        {
+            const auto format = encoding.first;
+            const auto& bytes = encoding.second;
+            CAPTURE(format);
+
+            // every prefix is truncated input
+            for (std::size_t length = 0; length < bytes.size(); ++length)
+            {
+                CAPTURE(length);
+                const auto result = parse_binary_recovering(std::vector<std::uint8_t>(bytes.begin(), bytes.begin() + static_cast<std::ptrdiff_t>(length)), format);
+                CHECK(!result.ok);
+                CHECK(result.errors == 1);
+                CHECK(result.balanced);
+            }
+
+            // the complete input is read as usual (binary values do not
+            // round-trip through every format, so compare with a plain parse)
+            json expected;
+            nlohmann::detail::json_sax_dom_parser<json> dom(expected);
+            CHECK(json::sax_parse(bytes, &dom, format));
+            const auto complete = parse_binary_recovering(bytes, format);
+            CHECK(complete.ok);
+            CHECK(complete.errors == 0);
+            CHECK(complete.value == expected);
+
+            // a byte after the value
+            auto trailing_bytes = bytes;
+            trailing_bytes.push_back(0x01);
+            const auto trailing = parse_binary_recovering(trailing_bytes, format);
+            CHECK(!trailing.ok);
+            CHECK(trailing.errors == 1);
+            CHECK(trailing.value == expected);
+        }
+    }
+
+    SECTION("containers without an end")
+    {
+        // these made the readers loop, or read on, after the error
+        const auto cbor_array = parse_binary_recovering({0x9F}, json::input_format_t::cbor);
+        CHECK(cbor_array.errors == 1);
+        CHECK(cbor_array.value == json::array());
+
+        const auto cbor_map = parse_binary_recovering({0xBF, 0x61, 'a'}, json::input_format_t::cbor);
+        CHECK(cbor_map.errors == 1);
+        CHECK(cbor_map.value == json({{"a", nullptr}}));
+
+        const auto msgpack_array = parse_binary_recovering({0xDD, 0xFF, 0xFF, 0xFF, 0xFF}, json::input_format_t::msgpack);
+        CHECK(msgpack_array.errors == 1);
+        CHECK(msgpack_array.value == json::array());
+
+        const auto msgpack_map = parse_binary_recovering({0x81, 0xA1, 'a', 0x92, 0x01}, json::input_format_t::msgpack);
+        CHECK(msgpack_map.errors == 1);
+        CHECK(msgpack_map.value == json({{"a", {1}}}));
+    }
+
+    SECTION("BJData ndarray")
+    {
+        // a 2x3 int8 array with two of its six elements; the annotated array
+        // format opens an object and two arrays of its own
+        const auto result = parse_binary_recovering({'[', '$', 'i', '#', '[', '$', 'i', '#', 'i', 2, 2, 3, 1, 2}, json::input_format_t::bjdata);
+        CHECK(result.errors == 1);
+        CHECK(result.balanced);
+        CHECK(result.value == json({{"_ArrayType_", "int8"}, {"_ArraySize_", {2, 3}}, {"_ArrayData_", {1, 2}}}));
+    }
+
+    SECTION("JSON text")
+    {
+        // the parser stopped, but reported success
+        json j;
+        RecoveringParser sax(j);
+        CHECK(!json::sax_parse("[1,2,3,]", &sax));
+        CHECK(sax.errors == 1);
+        CHECK(j == json({1, 2, 3}));
+    }
+
+    SECTION("the SAX parsers of the library stop")
+    {
+        json _;
+        CHECK(json::from_cbor(std::vector<std::uint8_t> {0x9F}, true, false).is_discarded());
+        CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<std::uint8_t> {0x9F}), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing CBOR value: unexpected end of input", json::parse_error&);
+        CHECK(json::parse("[1,2,3,]", nullptr, false).is_discarded());
+        CHECK(!json::accept("[1,2,3,]"));
+    }
+}
+
 DOCTEST_CLANG_SUPPRESS_WARNING_POP

tests/src/unit-regression3.cpp

@@ -74,13 +74,7 @@ using ordered_json = nlohmann::ordered_json;
     #endif
 #endif
 
-/////////////////////////////////////////////////////////////////////
-// for #4825 - explicitly instantiating basic_json must compile; this
-// forces instantiation of binary_writer::write_bjdata_ndarray, whose
-// static_cast<string_t> was ambiguous under explicit instantiation on
-// C++17. Merely compiling this translation unit is the regression test.
-/////////////////////////////////////////////////////////////////////
-template class nlohmann::basic_json<>;
+// the explicit instantiation for #4825 is in unit-explicit_instantiation.cpp
 
 /////////////////////////////////////////////////////////////////////
 // for #4440
@@ -894,6 +888,7 @@ TEST_CASE("regression test #5476 - array type without reserve()")
         // the binary formats pass a definite length to start_array()
         CHECK(deque_json::from_cbor(deque_json::to_cbor(j)) == j);
         CHECK(deque_json::from_msgpack(deque_json::to_msgpack(j)) == j);
+        CHECK(deque_json::from_bon8(deque_json::to_bon8(j)) == j);
 
         // parse() instantiates the callback parser as well, which reserves too
         const auto with_callback = deque_json::parse(R"([1,2,3])", [](int /*depth*/, deque_json::parse_event_t /*event*/, deque_json& /*parsed*/) noexcept