Keep writing ill-formed UTF-8 unchanged unless JSON_STRICT_BINARY_UTF8 is set

Throwing type_error.316 from to_cbor(), to_ubjson(), to_bjdata(), and
to_bson() by default would break code that works with 3.12.0, for example
code that stores ISO 8859-1 in a string and only writes it to a binary
format. Following the plan for 4.0.0, the check is now opt-in via the new
macro JSON_STRICT_BINARY_UTF8 (default 0), which is planned to become the
default in 4.0.0:

- abi_macros.hpp: default 0 and ABI tag _sbu8; macro_unscope.hpp undefines
  it; CMake option JSON_StrictBinaryUTF8; ABI namespace tests updated.
- binary_writer: the four writers call check_text_utf8(), which checks
  only if the macro is enabled. BON8 always checks, MessagePack never.
- Tests: the strict checks move to unit-binary_utf8_strict.cpp, which
  defines the macro; the per-format tests now check that the default
  writes the bytes unchanged (from_X(to_X(j)) == j).
- Docs: new macro page, macro lists, CMake option, namespace tags, and the
  writer/format pages describe the default and the opt-in.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/to_bjdata.md

@@ -56,8 +56,9 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
 
 - Throws [`other_error.502`](../../home/exceptions.md#jsonexceptionother_error502) if `use_type` is true and `use_size`
   is false.
-- Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if a string or object key in `j` is
-  not valid UTF-8
+- Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if a string or object key in `j` is not
+  valid UTF-8 and [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled; otherwise, the bytes are
+  written unchanged
 
 ## Complexity
 
@@ -92,4 +93,5 @@ Linear in the size of the JSON value `j`.
 
 - Added in version 3.11.0.
 - BJData version parameter (for draft3 binary encoding) added in version 3.12.0.
-- Throwing `type_error.316` for a string or object key that is not valid UTF-8 added in version 3.13.0.
+- Throwing `type_error.316` for a string or object key that is not valid UTF-8 if
+  [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled added in version 3.13.0.

docs/mkdocs/docs/api/basic_json/to_bson.md

@@ -46,8 +46,9 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
 - Throws [`out_of_range.415`](../../home/exceptions.md#jsonexceptionout_of_range415) if the subtype of a binary value
   exceeds 255, the maximum of the BSON binary subtype; example:
   `"subtype 70000 is too large for the BSON binary subtype (max 255)"`
-- Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if a string or object key is
-  not valid UTF-8
+- Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if a string or object key is not valid
+  UTF-8 and [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled; otherwise, the bytes are
+  written unchanged
 
 ## Complexity
 
@@ -84,5 +85,6 @@ pass before anything is written.
 - Added in version 3.4.0.
 - Linear in the size of `j`, and no longer limited by the call stack for deeply nested values, since version 3.13.0.
 - `out_of_range.415` is now detected before anything is written, like the other exceptions above, since version 3.13.0.
-- Throwing `type_error.316` for a string value or object key that is not valid UTF-8, detected before anything is
-  written, added in version 3.13.0.
+- Throwing `type_error.316` for a string value or object key that is not valid UTF-8 if
+  [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled, detected before anything is written,
+  added in version 3.13.0.

docs/mkdocs/docs/api/basic_json/to_cbor.md

@@ -37,8 +37,9 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
 
 ## Exceptions
 
-- Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if a string or object key in `j` is
-  not valid UTF-8
+- Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if a string or object key in `j` is not
+  valid UTF-8 and [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled; otherwise, the bytes are
+  written unchanged
 
 ## Complexity
 
@@ -73,4 +74,5 @@ Linear in the size of the JSON value `j`.
 
 - Added in version 2.0.9.
 - Compact representation of floating-point numbers added in version 3.8.0.
-- Throwing `type_error.316` for a string or object key that is not valid UTF-8 added in version 3.13.0.
+- Throwing `type_error.316` for a string or object key that is not valid UTF-8 if
+  [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled added in version 3.13.0.

docs/mkdocs/docs/api/basic_json/to_ubjson.md

@@ -49,8 +49,9 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
 
 - Throws [`other_error.502`](../../home/exceptions.md#jsonexceptionother_error502) if `use_type` is true and `use_size`
   is false.
-- Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if a string or object key in `j` is
-  not valid UTF-8
+- Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if a string or object key in `j` is not
+  valid UTF-8 and [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled; otherwise, the bytes are
+  written unchanged
 
 ## Complexity
 
@@ -84,4 +85,5 @@ Linear in the size of the JSON value `j`.
 ## Version history
 
 - Added in version 3.1.0.
-- Throwing `type_error.316` for a string or object key that is not valid UTF-8 added in version 3.13.0.
+- Throwing `type_error.316` for a string or object key that is not valid UTF-8 if
+  [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled added in version 3.13.0.

docs/mkdocs/docs/api/macros/index.md

@@ -18,6 +18,8 @@ header. See also the [macro overview page](../../features/macros.md).
 
 - [**JSON_PRECISE_STREAM_POSITION**](json_precise_stream_position.md) - opt in to leaving an input stream positioned
   right after a parsed number
+- [**JSON_STRICT_BINARY_UTF8**](json_strict_binary_utf8.md) - opt in to checking strings for valid UTF-8 in the CBOR,
+  UBJSON, BJData, and BSON writers
 - [**JSON_STRICT_NUL_HANDLING**](json_strict_nul_handling.md) - opt in to rejecting a NUL byte in the input instead of
   treating it as end of input
 

docs/mkdocs/docs/api/macros/json_strict_binary_utf8.md

@@ -0,0 +1,97 @@
+# JSON_STRICT_BINARY_UTF8
+
+```cpp
+#define JSON_STRICT_BINARY_UTF8 /* value */
+```
+
+When defined to `1`, the binary writers [`to_cbor`](../basic_json/to_cbor.md), [`to_ubjson`](../basic_json/to_ubjson.md),
+[`to_bjdata`](../basic_json/to_bjdata.md), and [`to_bson`](../basic_json/to_bson.md) check every string value and
+object key for valid UTF-8 and throw [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for
+ill-formed UTF-8, like [`dump`](../basic_json/dump.md) does. Without it, they write the bytes unchanged.
+
+The macro does not affect:
+
+- [`to_msgpack`](../basic_json/to_msgpack.md): the MessagePack specification allows a `str` value to contain bytes that
+  are not valid UTF-8, so it always writes them unchanged.
+- [`to_bon8`](../basic_json/to_bon8.md): BON8 always checks, because the UTF-8 lead bytes mark where a string ends.
+- The binary readers ([`from_cbor`](../basic_json/from_cbor.md), [`from_msgpack`](../basic_json/from_msgpack.md),
+  [`from_ubjson`](../basic_json/from_ubjson.md), [`from_bjdata`](../basic_json/from_bjdata.md),
+  [`from_bson`](../basic_json/from_bson.md)): none of these formats requires a decoder to reject ill-formed UTF-8, so
+  they always return the bytes unchanged.
+
+## Default definition
+
+The default value is `0` (disabled, the behavior of version 3.12.0 and earlier is preserved).
+
+```cpp
+#define JSON_STRICT_BINARY_UTF8 0
+```
+
+## Notes
+
+!!! note "Background"
+
+    CBOR, UBJSON, BJData, and BSON all require strings to be UTF-8. Up to version 3.12.0, the writers did not check
+    this, so they could produce output that other decoders reject. Checking by default would break code that stores
+    other encodings (for instance ISO 8859-1) in a string and only ever writes it to a binary format, so this macro
+    offers the check as an opt-in ahead of version 4.0.0, where it is planned to become the default (see
+    [#5529](https://github.com/nlohmann/json/issues/5529) and [#5651](https://github.com/nlohmann/json/issues/5651)).
+
+!!! warning "Opt-in only"
+
+    This macro must be defined **before** including `<nlohmann/json.hpp>`. Defining it after the include has no
+    effect.
+
+!!! note "ABI compatibility"
+
+    The value of this macro is encoded in the [namespace](../../features/namespace.md) (tag `_sbu8`), resulting in
+    distinct symbol names. Translation units compiled with and without it can therefore be linked into the same program
+    without One Definition Rule (ODR) violations, but they cannot exchange instances of library types.
+
+## Examples
+
+??? example "Default behavior (macro not defined)"
+
+    Without the macro, the bytes are written unchanged:
+
+    ```cpp
+    #include <nlohmann/json.hpp>
+
+    using json = nlohmann::json;
+
+    int main()
+    {
+        auto v = json::to_cbor(json("\xFF"));
+        // v is {0x61, 0xFF}
+    }
+    ```
+
+??? example "Opt-in check (macro defined to 1)"
+
+    With the macro, ill-formed UTF-8 is rejected:
+
+    ```cpp
+    #define JSON_STRICT_BINARY_UTF8 1
+    #include <nlohmann/json.hpp>
+
+    using json = nlohmann::json;
+
+    int main()
+    {
+        auto v = json::to_cbor(json("\xFF"));
+        // throws type_error.316: invalid UTF-8 byte at index 0: 0xFF
+    }
+    ```
+
+## See also
+
+- [**to_cbor**](../basic_json/to_cbor.md) - create a CBOR serialization of a JSON value
+- [**to_ubjson**](../basic_json/to_ubjson.md) - create a UBJSON serialization of a JSON value
+- [**to_bjdata**](../basic_json/to_bjdata.md) - create a BJData serialization of a JSON value
+- [**to_bson**](../basic_json/to_bson.md) - create a BSON serialization of a JSON value
+- [**error_handler_t**](../basic_json/error_handler_t.md) - how [`dump`](../basic_json/dump.md) treats ill-formed UTF-8
+
+## Version history
+
+- Added in version 3.13.0.
+- Planned to become the default (with the macro removed) in version 4.0.0.

docs/mkdocs/docs/features/binary_formats/bjdata.md

@@ -65,9 +65,10 @@ The library uses the following mapping from JSON values types to BJData types ac
 
 !!! warning "UTF-8 validation of string values and object keys"
 
-    BJData strings must use UTF-8 encoding. `to_bjdata()` validates the bytes of every string value and object key
-    and throws [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for ill-formed UTF-8, so a
-    value with such a string cannot be serialized in the first place.
+    BJData strings must use UTF-8 encoding. By default, `to_bjdata()` writes the bytes of string values and object keys
+    unchanged, even if they are not valid UTF-8. If
+    [`JSON_STRICT_BINARY_UTF8`](../../api/macros/json_strict_binary_utf8.md) is enabled, it throws
+    [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for ill-formed UTF-8 instead.
 
 !!! info "Unused BJData markers"
 
@@ -220,8 +221,8 @@ The library maps BJData types to JSON value types as follows:
     value or object key whose bytes are not valid UTF-8 and hands them back unchanged. However,
     [`dump()`](../../api/basic_json/dump.md) still requires valid UTF-8 and throws
     [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for such a value, unless an error
-    handler is passed that replaces or ignores the ill-formed bytes. `to_bjdata()` is strict as well (see above), so
-    a value read this way cannot be written back to BJData.
+    handler is passed that replaces or ignores the ill-formed bytes. By default, `to_bjdata()` writes such a value
+    back unchanged (see above).
 
 !!! info "Round trips"
 

docs/mkdocs/docs/features/binary_formats/bson.md

@@ -114,12 +114,11 @@ The library maps BSON record types to JSON value types as follows:
     The BSON specification requires `string` values (type `0x02`) to be valid UTF-8, but this is not required of a
     decoder. `from_bson()` accepts a `string` value whose bytes are not valid UTF-8 and hands them back unchanged.
     However, [`dump()`](../../api/basic_json/dump.md) still requires valid UTF-8 and throws
-    [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for such a value, unless an error
-    handler is passed that replaces or ignores the ill-formed bytes. `to_bson()` is strict as well and throws the
-    same exception for a string value or element (key) name that is not valid UTF-8, so an object with such a key
-    or value cannot be produced in the first place, even though `from_bson()` would accept it from another source.
-    Element (key) names are never validated on read, since they are read byte-by-byte as a C string. `binary`
-    values (type `0x05`) are unaffected, since they are not required to hold text.
+    [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for such a value, unless an error handler is
+    passed that replaces or ignores the ill-formed bytes. By default, `to_bson()` writes such a string value or element
+    (key) name unchanged; if [`JSON_STRICT_BINARY_UTF8`](../../api/macros/json_strict_binary_utf8.md) is enabled, it
+    throws the same exception instead. Element (key) names are never validated on read, since they are read byte-by-byte
+    as a C string. `binary` values (type `0x05`) are unaffected, since they are not required to hold text.
 
 ??? example
 

docs/mkdocs/docs/features/binary_formats/cbor.md

@@ -191,14 +191,14 @@ The library maps CBOR types to JSON value types as follows:
 
 !!! warning "Ill-formed UTF-8 in text strings"
 
-    [RFC 8949, Section 3.1](https://www.rfc-editor.org/rfc/rfc8949.html#section-3.1) requires CBOR text strings
-    (major type 3) to be valid UTF-8, but leaves it up to the decoder whether to enforce this. This library does
-    not: `from_cbor()` accepts a text string (object keys included) whose bytes are not valid UTF-8 and hands them
-    back unchanged. However, [`dump()`](../../api/basic_json/dump.md) still requires valid UTF-8 and throws
-    [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for such a value, unless an error
-    handler is passed that replaces or ignores the ill-formed bytes. `to_cbor()` is strict as well and throws the
-    same exception for a string value or object key that is not valid UTF-8, so such a value cannot be written back
-    to CBOR. Byte strings (major type 2) are unaffected, since they are not required to hold text.
+    [RFC 8949, Section 3.1](https://www.rfc-editor.org/rfc/rfc8949.html#section-3.1) requires CBOR text strings (major
+    type 3) to be valid UTF-8, but leaves it up to the decoder whether to enforce this. This library does not:
+    `from_cbor()` accepts a text string (object keys included) whose bytes are not valid UTF-8 and hands them back
+    unchanged. However, [`dump()`](../../api/basic_json/dump.md) still requires valid UTF-8 and throws
+    [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for such a value, unless an error handler is
+    passed that replaces or ignores the ill-formed bytes. By default, `to_cbor()` writes such a value back unchanged; if
+    [`JSON_STRICT_BINARY_UTF8`](../../api/macros/json_strict_binary_utf8.md) is enabled, it throws the same exception
+    instead. Byte strings (major type 2) are unaffected, since they are not required to hold text.
 
 !!! warning "Tagged items"
 

docs/mkdocs/docs/features/binary_formats/ubjson.md

@@ -49,9 +49,10 @@ The library uses the following mapping from JSON values types to UBJSON types ac
 
 !!! warning "UTF-8 validation of string values and object keys"
 
-    UBJSON's required string encoding is UTF-8. `to_ubjson()` validates the bytes of every string value and object
-    key and throws [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for ill-formed UTF-8, so
-    a value with such a string cannot be serialized in the first place.
+    UBJSON's required string encoding is UTF-8. By default, `to_ubjson()` writes the bytes of string values and object
+    keys unchanged, even if they are not valid UTF-8. If
+    [`JSON_STRICT_BINARY_UTF8`](../../api/macros/json_strict_binary_utf8.md) is enabled, it throws
+    [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for ill-formed UTF-8 instead.
 
 !!! info "Unused UBJSON markers"
 
@@ -132,8 +133,8 @@ The library maps UBJSON types to JSON value types as follows:
     value or object key whose bytes are not valid UTF-8 and hands them back unchanged. However,
     [`dump()`](../../api/basic_json/dump.md) still requires valid UTF-8 and throws
     [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for such a value, unless an error
-    handler is passed that replaces or ignores the ill-formed bytes. `to_ubjson()` is strict as well (see above), so
-    a value read this way cannot be written back to UBJSON.
+    handler is passed that replaces or ignores the ill-formed bytes. By default, `to_ubjson()` writes such a value
+    back unchanged (see above).
 
 ??? example
 

docs/mkdocs/docs/features/macros.md

@@ -138,6 +138,20 @@ using the library with compilers that do not fully support C++11 and may only wo
 
 See [full documentation of `JSON_SKIP_UNSUPPORTED_COMPILER_CHECK`](../api/macros/json_skip_unsupported_compiler_check.md).
 
+## `JSON_STRICT_BINARY_UTF8`
+
+When defined to `1`, [`to_cbor`](../api/basic_json/to_cbor.md), [`to_ubjson`](../api/basic_json/to_ubjson.md),
+[`to_bjdata`](../api/basic_json/to_bjdata.md), and [`to_bson`](../api/basic_json/to_bson.md) throw
+[`type_error.316`](../home/exceptions.md#jsonexceptiontype_error316) for a string value or object key that is not
+valid UTF-8. The default value is `0`, which writes the bytes unchanged as before version 3.13.0; this is planned to
+become the default in version 4.0.0.
+
+The check can also be enabled with the CMake option
+[`JSON_StrictBinaryUTF8`](../integration/cmake.md#json_strictbinaryutf8) (`OFF` by default) which sets
+`JSON_STRICT_BINARY_UTF8` accordingly.
+
+See [full documentation of `JSON_STRICT_BINARY_UTF8`](../api/macros/json_strict_binary_utf8.md).
+
 ## `JSON_STRICT_NUL_HANDLING`
 
 When defined to `1`, a `'\0'` (NUL) byte anywhere in the input is rejected with `parse_error.101`, like any other

docs/mkdocs/docs/features/namespace.md

@@ -20,6 +20,7 @@ The complete default namespace name is derived as follows:
       `_bics`.
     - [`JSON_PRECISE_STREAM_POSITION`](../api/macros/json_precise_stream_position.md) defined non-zero appends `_psp`.
     - [`JSON_STRICT_NUL_HANDLING`](../api/macros/json_strict_nul_handling.md) defined non-zero appends `_snul`.
+    - [`JSON_STRICT_BINARY_UTF8`](../api/macros/json_strict_binary_utf8.md) defined non-zero appends `_sbu8`.
 - The inline namespace ends with the suffix `_v` followed by the 3 components of the version number separated by
   underscores. To omit the version component, see [Disabling the version component](#disabling-the-version-component)
   below.

docs/mkdocs/docs/home/exceptions.md

@@ -747,6 +747,11 @@ The `unflatten()` function only works for an object whose keys are JSON Pointers
 
 The `dump()` function only works with UTF-8 encoded strings; that is, if you assign a `std::string` to a JSON value, make sure it is UTF-8 encoded.
 
+If [`JSON_STRICT_BINARY_UTF8`](../api/macros/json_strict_binary_utf8.md) is enabled, the binary writers
+[`to_cbor()`](../api/basic_json/to_cbor.md), [`to_ubjson()`](../api/basic_json/to_ubjson.md),
+[`to_bjdata()`](../api/basic_json/to_bjdata.md), and [`to_bson()`](../api/basic_json/to_bson.md) throw this exception
+for a string value or object key that is not valid UTF-8 as well.
+
 !!! failure "Example message"
 
     Calling `dump()` on a JSON value containing an ISO 8859-1 encoded string:

docs/mkdocs/docs/integration/cmake.md

@@ -204,6 +204,11 @@ Use the non-amalgamated version of the library. This option is `ON` by default.
 
 Treat the library headers like system headers (i.e., adding `SYSTEM` to the [`target_include_directories`](https://cmake.org/cmake/help/latest/command/target_include_directories.html) call) to check for this library by tools like Clang-Tidy. This option is `OFF` by default.
 
+### `JSON_StrictBinaryUTF8`
+
+Check string values and object keys for valid UTF-8 in the CBOR, UBJSON, BJData, and BSON writers, by defining the
+macro [`JSON_STRICT_BINARY_UTF8`](../api/macros/json_strict_binary_utf8.md). This option is `OFF` by default.
+
 ### `JSON_StrictNulHandling`
 
 Reject a `'\0'` (NUL) byte in the input instead of treating it as end of input, by defining the macro

docs/mkdocs/mkdocs.yml

@@ -301,6 +301,7 @@ nav:
           - 'JSON_PRECISE_STREAM_POSITION': api/macros/json_precise_stream_position.md
           - 'JSON_SKIP_LIBRARY_VERSION_CHECK': api/macros/json_skip_library_version_check.md
           - 'JSON_SKIP_UNSUPPORTED_COMPILER_CHECK': api/macros/json_skip_unsupported_compiler_check.md
+          - 'JSON_STRICT_BINARY_UTF8': api/macros/json_strict_binary_utf8.md
           - 'JSON_STRICT_NUL_HANDLING': api/macros/json_strict_nul_handling.md
           - 'JSON_USE_GLOBAL_UDLS': api/macros/json_use_global_udls.md
           - 'JSON_USE_IMPLICIT_CONVERSIONS': api/macros/json_use_implicit_conversions.md