amorfo77/json
1
0
Fork 0
mirror of https://github.com/nlohmann/json.git synced 2026-10-02 20:50:32 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge #5741 and make keep the default error_handler of the binary writers

Browse Source 
Merges claude/binary-utf8-roundtrip-5651, where the writers now only check
UTF-8 if JSON_STRICT_BINARY_UTF8 is enabled, and applies the same rule to
the error_handler parameter added here: to_cbor(), to_ubjson(),
to_bjdata(), and to_bson() default to error_handler_t::keep, the behavior
of 3.12.0, or to error_handler_t::strict if JSON_STRICT_BINARY_UTF8 is
enabled (detail::binary_writer_default_error_handler()). Passing a handler
explicitly always takes precedence.

- Conflicts in binary_writer.hpp resolved in favor of this branch's
  error_handler, which replaces #5741's check_text_utf8().
- Tests: the "default parameters" section checks that the default equals
  keep; unit-binary_utf8_strict.cpp checks that an explicit handler
  overrides the macro's default.
- Docs: signatures show keep; parameter, exception, and format pages
  describe keep as the default and the macro; the two 3.13.0 version
  history entries of each writer are merged into one.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann
2026-10-02 08:48:59 +02:00
parent fff221c58b 809ace26b3
commit cf55a6cb3a
33 changed files with 503 additions and 167 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CMakeLists.txt
+6
View File
@@ -61,6 +61,7 @@ option(JSON_Install "Install CMake targets during install
option(JSON_MultipleHeaders "Use non-amalgamated version of the library." ON)
option(JSON_SystemInclude "Include as system headers (skip for clang-tidy)." OFF)
option(JSON_StrictNulHandling "Build with strict NUL-byte handling enabled." OFF)
option(JSON_StrictBinaryUTF8 "Build with UTF-8 checks in the CBOR, UBJSON, BJData, and BSON writers enabled." OFF)
if (JSON_CI)
include(ci)
@@ -118,6 +119,10 @@ if (JSON_StrictNulHandling)
message(STATUS "Strict NUL-byte handling enabled (JSON_STRICT_NUL_HANDLING=1)")
endif()
if (JSON_StrictBinaryUTF8)
message(STATUS "Strict UTF-8 checks in binary writers enabled (JSON_STRICT_BINARY_UTF8=1)")
endif()
if (JSON_Diagnostic_Positions)
message(STATUS "Diagnostic positions enabled (JSON_DIAGNOSTIC_POSITIONS=1)")
endif()
@@ -153,6 +158,7 @@ target_compile_definitions(
$<$<BOOL:${JSON_Diagnostic_Positions}>:JSON_DIAGNOSTIC_POSITIONS=1>
$<$<BOOL:${JSON_LegacyDiscardedValueComparison}>:JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON=1>
$<$<BOOL:${JSON_StrictNulHandling}>:JSON_STRICT_NUL_HANDLING=1>
$<$<BOOL:${JSON_StrictBinaryUTF8}>:JSON_STRICT_BINARY_UTF8=1>
)
target_include_directories(
cmake/ci.cmake
+1 -1
View File
@@ -691,7 +691,7 @@ ci_get_cmake(4.0.0 CMAKE_4_0_0_BINARY)
# the tests require CMake 3.13 or later, so they are excluded for CMake 3.5.0
set(JSON_CMAKE_FLAGS_3_5_0 JSON_Diagnostics JSON_Diagnostic_Positions JSON_GlobalUDLs JSON_ImplicitConversions JSON_DisableEnumSerialization
JSON_LegacyDiscardedValueComparison JSON_Install JSON_MultipleHeaders JSON_SystemInclude JSON_Valgrind
JSON_StrictNulHandling)
JSON_StrictNulHandling JSON_StrictBinaryUTF8)
set(JSON_CMAKE_FLAGS_3_31_6 JSON_BuildTests ${JSON_CMAKE_FLAGS_3_5_0})
set(JSON_CMAKE_FLAGS_4_0_0 JSON_BuildTests ${JSON_CMAKE_FLAGS_3_5_0})
docs/mkdocs/docs/api/basic_json/error_handler_t.md
+3 -1
View File
@@ -14,7 +14,9 @@ This enumeration is used to choose how to treat ill-formed UTF-8 in a string val
- [`dump`](dump.md) uses it while serializing a `basic_json` value to text.
- [`to_cbor`](to_cbor.md), [`to_ubjson`](to_ubjson.md), [`to_bjdata`](to_bjdata.md), and [`to_bson`](to_bson.md) use it
while serializing a `basic_json` value to that binary format; none of CBOR, UBJSON, BJData, or BSON requires a
decoder to reject ill-formed UTF-8, so by default (`strict`) the library checks on write instead. `to_msgpack` and
decoder to reject ill-formed UTF-8, so the library can check on write instead. Their default is `keep`, as no
binary writer checked before this parameter was added, or `strict` if
[`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled. `to_msgpack` and
`to_bon8` do not take this parameter: MessagePack's specification explicitly allows a string to contain ill-formed
UTF-8, so `to_msgpack` always passes it through, while BON8 always validates, since UTF-8 lead bytes are structural
to that format.
docs/mkdocs/docs/api/basic_json/to_bjdata.md
+11 -9
View File
@@ -6,17 +6,17 @@ static std::vector<std::uint8_t> to_bjdata(const basic_json& j,
const bool use_size = false,
const bool use_type = false,
const bjdata_version_t version = bjdata_version_t::draft2,
const error_handler_t error_handler = error_handler_t::strict);
const error_handler_t error_handler = error_handler_t::keep);
// (2)
static void to_bjdata(const basic_json& j, detail::output_adapter<std::uint8_t> o,
const bool use_size = false, const bool use_type = false,
const bjdata_version_t version = bjdata_version_t::draft2,
const error_handler_t error_handler = error_handler_t::strict);
const error_handler_t error_handler = error_handler_t::keep);
static void to_bjdata(const basic_json& j, detail::output_adapter<char> o,
const bool use_size = false, const bool use_type = false,
const bjdata_version_t version = bjdata_version_t::draft2,
const error_handler_t error_handler = error_handler_t::strict);
const error_handler_t error_handler = error_handler_t::keep);
```
Serializes a given JSON value `j` to a byte vector using the BJData (Binary JData) serialization format. BJData aims to
@@ -48,9 +48,9 @@ optional, `#!cpp bjdata_version_t::draft2` by default.
`error_handler` (in)
: how to treat a string or object key in `j` that is not valid UTF-8; see [`error_handler_t`](error_handler_t.md).
The default, `strict`, throws; `keep` writes the ill-formed bytes to the output as is, as every version of
`to_bjdata` did before this parameter was added; `replace`/`ignore` sanitize it the same way
[`dump`](dump.md) would
The default, `keep`, writes the ill-formed bytes to the output as is, as every version of `to_bjdata` did before
this parameter was added; `strict` throws; `replace`/`ignore` sanitize it the same way [`dump`](dump.md) would.
If [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled, the default is `strict` instead.
## Return value
@@ -66,7 +66,8 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
- Throws [`other_error.502`](../../home/exceptions.md#jsonexceptionother_error502) if `use_type` is true and `use_size`
is false.
- Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if a string or object key in `j` is
not valid UTF-8 and `error_handler` is `strict` (the default)
not valid UTF-8 and `error_handler` is `strict` (the default only if
[`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled)
## Complexity
@@ -101,5 +102,6 @@ Linear in the size of the JSON value `j`.
- Added in version 3.11.0.
- BJData version parameter (for draft3 binary encoding) added in version 3.12.0.
- Throwing `type_error.316` for a string or object key that is not valid UTF-8 added in version 3.13.0.
- Added `error_handler` parameter in version 3.13.0.
- Added `error_handler` parameter in version 3.13.0. Its default, `keep`, writes the bytes of a string or object key
that is not valid UTF-8 unchanged, as before; `strict` (the default if
[`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled) throws `type_error.316`.
docs/mkdocs/docs/api/basic_json/to_bson.md
+12 -10
View File
@@ -3,13 +3,13 @@
```cpp
// (1)
static std::vector<std::uint8_t> to_bson(const basic_json& j,
const error_handler_t error_handler = error_handler_t::strict);
const error_handler_t error_handler = error_handler_t::keep);
// (2)
static void to_bson(const basic_json& j, detail::output_adapter<std::uint8_t> o,
const error_handler_t error_handler = error_handler_t::strict);
const error_handler_t error_handler = error_handler_t::keep);
static void to_bson(const basic_json& j, detail::output_adapter<char> o,
const error_handler_t error_handler = error_handler_t::strict);
const error_handler_t error_handler = error_handler_t::keep);
```
BSON (Binary JSON) is a binary format in which zero or more ordered key/value pairs are stored as a single entity (a
@@ -30,9 +30,9 @@ The exact mapping and its limitations are described on a [dedicated page](../../
`error_handler` (in)
: how to treat a string or object key in `j` that is not valid UTF-8; see [`error_handler_t`](error_handler_t.md).
The default, `strict`, throws; `keep` writes the ill-formed bytes to the output as is, as every version of
`to_bson` did before this parameter was added; `replace`/`ignore` sanitize it the same way
[`dump`](dump.md) would
The default, `keep`, writes the ill-formed bytes to the output as is, as every version of `to_bson` did before
this parameter was added; `strict` throws; `replace`/`ignore` sanitize it the same way [`dump`](dump.md) would.
If [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled, the default is `strict` instead.
## Return value
@@ -56,7 +56,8 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
exceeds 255, the maximum of the BSON binary subtype; example:
`"subtype 70000 is too large for the BSON binary subtype (max 255)"`
- Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if a string or object key is
not valid UTF-8 and `error_handler` is `strict` (the default)
not valid UTF-8 and `error_handler` is `strict` (the default only if
[`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled)
## Complexity
@@ -93,6 +94,7 @@ pass before anything is written.
- Added in version 3.4.0.
- Linear in the size of `j`, and no longer limited by the call stack for deeply nested values, since version 3.13.0.
- `out_of_range.415` is now detected before anything is written, like the other exceptions above, since version 3.13.0.
- Throwing `type_error.316` for a string value or object key that is not valid UTF-8, detected before anything is
written, added in version 3.13.0.
- Added `error_handler` parameter in version 3.13.0.
- Added `error_handler` parameter in version 3.13.0. Its default, `keep`, writes the bytes of a string or object key
that is not valid UTF-8 unchanged, as before; `strict` (the default if
[`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled) throws `type_error.316` before anything
is written.
docs/mkdocs/docs/api/basic_json/to_cbor.md
+11 -9
View File
@@ -3,13 +3,13 @@
```cpp
// (1)
static std::vector<std::uint8_t> to_cbor(const basic_json& j,
const error_handler_t error_handler = error_handler_t::strict);
const error_handler_t error_handler = error_handler_t::keep);
// (2)
static void to_cbor(const basic_json& j, detail::output_adapter<std::uint8_t> o,
const error_handler_t error_handler = error_handler_t::strict);
const error_handler_t error_handler = error_handler_t::keep);
static void to_cbor(const basic_json& j, detail::output_adapter<char> o,
const error_handler_t error_handler = error_handler_t::strict);
const error_handler_t error_handler = error_handler_t::keep);
```
Serializes a given JSON value `j` to a byte vector using the CBOR (Concise Binary Object Representation) serialization
@@ -31,9 +31,9 @@ The exact mapping and its limitations are described on a [dedicated page](../../
`error_handler` (in)
: how to treat a string or object key in `j` that is not valid UTF-8; see [`error_handler_t`](error_handler_t.md).
The default, `strict`, throws; `keep` writes the ill-formed bytes to the output as is, as every version of
`to_cbor` did before this parameter was added; `replace`/`ignore` sanitize it the same way
[`dump`](dump.md) would
The default, `keep`, writes the ill-formed bytes to the output as is, as every version of `to_cbor` did before
this parameter was added; `strict` throws; `replace`/`ignore` sanitize it the same way [`dump`](dump.md) would.
If [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled, the default is `strict` instead.
## Return value
@@ -47,7 +47,8 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
## Exceptions
- Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if a string or object key in `j` is
not valid UTF-8 and `error_handler` is `strict` (the default)
not valid UTF-8 and `error_handler` is `strict` (the default only if
[`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled)
## Complexity
@@ -82,5 +83,6 @@ Linear in the size of the JSON value `j`.
- Added in version 2.0.9.
- Compact representation of floating-point numbers added in version 3.8.0.
- Throwing `type_error.316` for a string or object key that is not valid UTF-8 added in version 3.13.0.
- Added `error_handler` parameter in version 3.13.0.
- Added `error_handler` parameter in version 3.13.0. Its default, `keep`, writes the bytes of a string or object key
that is not valid UTF-8 unchanged, as before; `strict` (the default if
[`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled) throws `type_error.316`.
docs/mkdocs/docs/api/basic_json/to_ubjson.md
+11 -9
View File
@@ -5,15 +5,15 @@
static std::vector<std::uint8_t> to_ubjson(const basic_json& j,
const bool use_size = false,
const bool use_type = false,
const error_handler_t error_handler = error_handler_t::strict);
const error_handler_t error_handler = error_handler_t::keep);
// (2)
static void to_ubjson(const basic_json& j, detail::output_adapter<std::uint8_t> o,
const bool use_size = false, const bool use_type = false,
const error_handler_t error_handler = error_handler_t::strict);
const error_handler_t error_handler = error_handler_t::keep);
static void to_ubjson(const basic_json& j, detail::output_adapter<char> o,
const bool use_size = false, const bool use_type = false,
const error_handler_t error_handler = error_handler_t::strict);
const error_handler_t error_handler = error_handler_t::keep);
```
Serializes a given JSON value `j` to a byte vector using the UBJSON (Universal Binary JSON) serialization format. UBJSON
@@ -41,9 +41,9 @@ The exact mapping and its limitations are described on a [dedicated page](../../
`error_handler` (in)
: how to treat a string or object key in `j` that is not valid UTF-8; see [`error_handler_t`](error_handler_t.md).
The default, `strict`, throws; `keep` writes the ill-formed bytes to the output as is, as every version of
`to_ubjson` did before this parameter was added; `replace`/`ignore` sanitize it the same way
[`dump`](dump.md) would
The default, `keep`, writes the ill-formed bytes to the output as is, as every version of `to_ubjson` did before
this parameter was added; `strict` throws; `replace`/`ignore` sanitize it the same way [`dump`](dump.md) would.
If [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled, the default is `strict` instead.
## Return value
@@ -59,7 +59,8 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
- Throws [`other_error.502`](../../home/exceptions.md#jsonexceptionother_error502) if `use_type` is true and `use_size`
is false.
- Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if a string or object key in `j` is
not valid UTF-8 and `error_handler` is `strict` (the default)
not valid UTF-8 and `error_handler` is `strict` (the default only if
[`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled)
## Complexity
@@ -93,5 +94,6 @@ Linear in the size of the JSON value `j`.
## Version history
- Added in version 3.1.0.
- Throwing `type_error.316` for a string or object key that is not valid UTF-8 added in version 3.13.0.
- Added `error_handler` parameter in version 3.13.0.
- Added `error_handler` parameter in version 3.13.0. Its default, `keep`, writes the bytes of a string or object key
that is not valid UTF-8 unchanged, as before; `strict` (the default if
[`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled) throws `type_error.316`.
docs/mkdocs/docs/api/macros/index.md
+2
View File
@@ -18,6 +18,8 @@ header. See also the [macro overview page](../../features/macros.md).
- [**JSON_PRECISE_STREAM_POSITION**](json_precise_stream_position.md) - opt in to leaving an input stream positioned
right after a parsed number
- [**JSON_STRICT_BINARY_UTF8**](json_strict_binary_utf8.md) - opt in to checking strings for valid UTF-8 in the CBOR,
UBJSON, BJData, and BSON writers
- [**JSON_STRICT_NUL_HANDLING**](json_strict_nul_handling.md) - opt in to rejecting a NUL byte in the input instead of
treating it as end of input
docs/mkdocs/docs/api/macros/json_strict_binary_utf8.md
+101
View File
@@ -0,0 +1,101 @@
# JSON_STRICT_BINARY_UTF8
```cpp
#define JSON_STRICT_BINARY_UTF8 /* value */
```
When defined to `1`, the `error_handler` parameter of the binary writers [`to_cbor`](../basic_json/to_cbor.md),
[`to_ubjson`](../basic_json/to_ubjson.md), [`to_bjdata`](../basic_json/to_bjdata.md), and
[`to_bson`](../basic_json/to_bson.md) defaults to [`error_handler_t::strict`](../basic_json/error_handler_t.md) instead
of `error_handler_t::keep`. These writers then check every string value and object key for valid UTF-8 and throw
[`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for ill-formed UTF-8, like
[`dump`](../basic_json/dump.md) does. Without it, they write the bytes unchanged. An `error_handler` passed explicitly
always takes precedence.
The macro does not affect:
- [`to_msgpack`](../basic_json/to_msgpack.md): the MessagePack specification allows a `str` value to contain bytes that
are not valid UTF-8, so it always writes them unchanged.
- [`to_bon8`](../basic_json/to_bon8.md): BON8 always checks, because the UTF-8 lead bytes mark where a string ends.
- The binary readers ([`from_cbor`](../basic_json/from_cbor.md), [`from_msgpack`](../basic_json/from_msgpack.md),
[`from_ubjson`](../basic_json/from_ubjson.md), [`from_bjdata`](../basic_json/from_bjdata.md),
[`from_bson`](../basic_json/from_bson.md)): none of these formats requires a decoder to reject ill-formed UTF-8, so
they always return the bytes unchanged.
## Default definition
The default value is `0` (disabled, the behavior of version 3.12.0 and earlier is preserved).
```cpp
#define JSON_STRICT_BINARY_UTF8 0
```
## Notes
!!! note "Background"
CBOR, UBJSON, BJData, and BSON all require strings to be UTF-8. Up to version 3.12.0, the writers did not check
this, so they could produce output that other decoders reject. Checking by default would break code that stores
other encodings (for instance ISO 8859-1) in a string and only ever writes it to a binary format. You can pass
`error_handler_t::strict` to each call, or use this macro to check by default ahead of version 4.0.0, where
`strict` is planned to become the default (see
[#5529](https://github.com/nlohmann/json/issues/5529) and [#5651](https://github.com/nlohmann/json/issues/5651)).
!!! warning "Opt-in only"
This macro must be defined **before** including `<nlohmann/json.hpp>`. Defining it after the include has no
effect.
!!! note "ABI compatibility"
The value of this macro is encoded in the [namespace](../../features/namespace.md) (tag `_sbu8`), resulting in
distinct symbol names. Translation units compiled with and without it can therefore be linked into the same program
without One Definition Rule (ODR) violations, but they cannot exchange instances of library types.
## Examples
??? example "Default behavior (macro not defined)"
Without the macro, the bytes are written unchanged:
```cpp
#include <nlohmann/json.hpp>
using json = nlohmann::json;
int main()
{
auto v = json::to_cbor(json("\xFF"));
// v is {0x61, 0xFF}
}
```
??? example "Opt-in check (macro defined to 1)"
With the macro, ill-formed UTF-8 is rejected:
```cpp
#define JSON_STRICT_BINARY_UTF8 1
#include <nlohmann/json.hpp>
using json = nlohmann::json;
int main()
{
auto v = json::to_cbor(json("\xFF"));
// throws type_error.316: invalid UTF-8 byte at index 0: 0xFF
}
```
## See also
- [**to_cbor**](../basic_json/to_cbor.md) - create a CBOR serialization of a JSON value
- [**to_ubjson**](../basic_json/to_ubjson.md) - create a UBJSON serialization of a JSON value
- [**to_bjdata**](../basic_json/to_bjdata.md) - create a BJData serialization of a JSON value
- [**to_bson**](../basic_json/to_bson.md) - create a BSON serialization of a JSON value
- [**error_handler_t**](../basic_json/error_handler_t.md) - how [`dump`](../basic_json/dump.md) treats ill-formed UTF-8
## Version history
- Added in version 3.13.0.
- Planned to become the default (with the macro removed) in version 4.0.0.
docs/mkdocs/docs/features/binary_formats/bjdata.md
+7 -5
View File
@@ -65,9 +65,12 @@ The library uses the following mapping from JSON values types to BJData types ac
!!! warning "UTF-8 validation of string values and object keys"
BJData strings must use UTF-8 encoding. `to_bjdata()` validates the bytes of every string value and object key
and throws [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for ill-formed UTF-8, so a
value with such a string cannot be serialized in the first place.
BJData strings must use UTF-8 encoding. By default (the [`error_handler`](../../api/basic_json/to_bjdata.md)
parameter left at `keep`), `to_bjdata()` writes the bytes of string values and object keys unchanged, even if they
are not valid UTF-8. With `error_handler_t::strict`, it throws
[`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for ill-formed UTF-8 instead;
`replace`/`ignore` sanitize the string. [`JSON_STRICT_BINARY_UTF8`](../../api/macros/json_strict_binary_utf8.md)
makes `strict` the default.
!!! info "Unused BJData markers"
@@ -225,8 +228,7 @@ The library maps BJData types to JSON value types as follows:
[`dump()`](../../api/basic_json/dump.md) still requires valid UTF-8 and throws
[`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for a value read with the default
`keep` handler, unless an error handler is passed that replaces or ignores the ill-formed bytes. `to_bjdata()`'s
own `error_handler` parameter defaults to `strict` (see above), so a value read this way cannot be written back
to BJData unless a non-strict handler is passed there too.
own `error_handler` parameter defaults to `keep` (see above), so such a value is written back unchanged.
!!! info "Round trips"
docs/mkdocs/docs/features/binary_formats/bson.md
+7 -8
View File
@@ -117,14 +117,13 @@ The library maps BSON record types to JSON value types as follows:
unchanged. Passing `error_handler_t::strict` makes `from_bson()` check and throw
[`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113) for ill-formed UTF-8, and
`replace`/`ignore` sanitize the string instead of keeping it. However, [`dump()`](../../api/basic_json/dump.md)
still requires valid UTF-8 and throws [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316)
for a value read with the default `keep` handler, unless an error handler is passed that replaces or ignores
the ill-formed bytes. `to_bson()`'s own `error_handler` parameter defaults to `strict` and throws the same
exception for a string value or element (key) name that is not valid UTF-8, so an object with such a key or
value cannot be produced in the first place unless a non-strict handler is passed there, even though
`from_bson()` would accept it from another source with the default `keep` handler. Element (key) names are
never validated on read, since they are read byte-by-byte as a C string. `binary` values (type `0x05`) are
unaffected, since they are not required to hold text.
still requires valid UTF-8 and throws [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for a
value read with the default `keep` handler, unless an error handler is passed that replaces or ignores the
ill-formed bytes. `to_bson()`'s own `error_handler` parameter defaults to `keep`, so such a string value or element
(key) name is written unchanged; with `strict` (the default if
[`JSON_STRICT_BINARY_UTF8`](../../api/macros/json_strict_binary_utf8.md) is enabled), it throws the same exception
instead. Element (key) names are never validated on read, since they are read byte-by-byte as a C string. `binary`
values (type `0x05`) are unaffected, since they are not required to hold text.
??? example
docs/mkdocs/docs/features/binary_formats/cbor.md
+11 -11
View File
@@ -191,19 +191,19 @@ The library maps CBOR types to JSON value types as follows:
!!! warning "Ill-formed UTF-8 in text strings"
[RFC 8949, Section 3.1](https://www.rfc-editor.org/rfc/rfc8949.html#section-3.1) requires CBOR text strings
(major type 3) to be valid UTF-8, but leaves it up to the decoder whether to enforce this, so checking is
opt-in: with the [`error_handler`](../../api/basic_json/from_cbor.md) parameter left at `keep` (the default),
`from_cbor()` accepts a text string (object keys included) whose bytes are not valid UTF-8 and hands them back
unchanged. Passing `error_handler_t::strict` makes `from_cbor()` check and throw
[RFC 8949, Section 3.1](https://www.rfc-editor.org/rfc/rfc8949.html#section-3.1) requires CBOR text strings (major
type 3) to be valid UTF-8, but leaves it up to the decoder whether to enforce this, so checking is opt-in: with the
[`error_handler`](../../api/basic_json/from_cbor.md) parameter left at `keep` (the default), `from_cbor()` accepts a
text string (object keys included) whose bytes are not valid UTF-8 and hands them back unchanged. Passing
`error_handler_t::strict` makes `from_cbor()` check and throw
[`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113) for ill-formed UTF-8, and
`replace`/`ignore` sanitize the string instead of keeping it. However, [`dump()`](../../api/basic_json/dump.md)
still requires valid UTF-8 and throws [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316)
for a value read with the default `keep` handler, unless an error handler is passed that replaces or ignores
the ill-formed bytes. `to_cbor()`'s own [`error_handler`](../../api/basic_json/to_cbor.md) parameter defaults
to `strict` and throws the same exception for a string value or object key that is not valid UTF-8, so such a
value cannot be written back to CBOR unless a non-strict handler is passed there too. Byte strings (major
type 2) are unaffected, since they are not required to hold text.
still requires valid UTF-8 and throws [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for a
value read with the default `keep` handler, unless an error handler is passed that replaces or ignores the
ill-formed bytes. `to_cbor()`'s own [`error_handler`](../../api/basic_json/to_cbor.md) parameter defaults to `keep`,
so such a value is written back unchanged; with `strict` (the default if
[`JSON_STRICT_BINARY_UTF8`](../../api/macros/json_strict_binary_utf8.md) is enabled), it throws the same exception
instead. Byte strings (major type 2) are unaffected, since they are not required to hold text.
!!! warning "Tagged items"
docs/mkdocs/docs/features/binary_formats/ubjson.md
+7 -5
View File
@@ -49,9 +49,12 @@ The library uses the following mapping from JSON values types to UBJSON types ac
!!! warning "UTF-8 validation of string values and object keys"
UBJSON's required string encoding is UTF-8. `to_ubjson()` validates the bytes of every string value and object
key and throws [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for ill-formed UTF-8, so
a value with such a string cannot be serialized in the first place.
UBJSON's required string encoding is UTF-8. By default (the [`error_handler`](../../api/basic_json/to_ubjson.md)
parameter left at `keep`), `to_ubjson()` writes the bytes of string values and object keys unchanged, even if they
are not valid UTF-8. With `error_handler_t::strict`, it throws
[`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for ill-formed UTF-8 instead;
`replace`/`ignore` sanitize the string. [`JSON_STRICT_BINARY_UTF8`](../../api/macros/json_strict_binary_utf8.md)
makes `strict` the default.
!!! info "Unused UBJSON markers"
@@ -137,8 +140,7 @@ The library maps UBJSON types to JSON value types as follows:
[`dump()`](../../api/basic_json/dump.md) still requires valid UTF-8 and throws
[`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for a value read with the default
`keep` handler, unless an error handler is passed that replaces or ignores the ill-formed bytes. `to_ubjson()`'s
own `error_handler` parameter defaults to `strict` (see above), so a value read this way cannot be written back
to UBJSON unless a non-strict handler is passed there too.
own `error_handler` parameter defaults to `keep` (see above), so such a value is written back unchanged.
??? example
docs/mkdocs/docs/features/macros.md
+14
View File
@@ -138,6 +138,20 @@ using the library with compilers that do not fully support C++11 and may only wo
See [full documentation of `JSON_SKIP_UNSUPPORTED_COMPILER_CHECK`](../api/macros/json_skip_unsupported_compiler_check.md).
## `JSON_STRICT_BINARY_UTF8`
When defined to `1`, [`to_cbor`](../api/basic_json/to_cbor.md), [`to_ubjson`](../api/basic_json/to_ubjson.md),
[`to_bjdata`](../api/basic_json/to_bjdata.md), and [`to_bson`](../api/basic_json/to_bson.md) throw
[`type_error.316`](../home/exceptions.md#jsonexceptiontype_error316) for a string value or object key that is not
valid UTF-8. The default value is `0`, which writes the bytes unchanged as before version 3.13.0; this is planned to
become the default in version 4.0.0.
The check can also be enabled with the CMake option
[`JSON_StrictBinaryUTF8`](../integration/cmake.md#json_strictbinaryutf8) (`OFF` by default) which sets
`JSON_STRICT_BINARY_UTF8` accordingly.
See [full documentation of `JSON_STRICT_BINARY_UTF8`](../api/macros/json_strict_binary_utf8.md).
## `JSON_STRICT_NUL_HANDLING`
When defined to `1`, a `'\0'` (NUL) byte anywhere in the input is rejected with `parse_error.101`, like any other
docs/mkdocs/docs/features/namespace.md
+1
View File
@@ -20,6 +20,7 @@ The complete default namespace name is derived as follows:
`_bics`.
- [`JSON_PRECISE_STREAM_POSITION`](../api/macros/json_precise_stream_position.md) defined non-zero appends `_psp`.
- [`JSON_STRICT_NUL_HANDLING`](../api/macros/json_strict_nul_handling.md) defined non-zero appends `_snul`.
- [`JSON_STRICT_BINARY_UTF8`](../api/macros/json_strict_binary_utf8.md) defined non-zero appends `_sbu8`.
- The inline namespace ends with the suffix `_v` followed by the 3 components of the version number separated by
underscores. To omit the version component, see [Disabling the version component](#disabling-the-version-component)
below.
docs/mkdocs/docs/home/exceptions.md
+5
View File
@@ -752,6 +752,11 @@ The `unflatten()` function only works for an object whose keys are JSON Pointers
The `dump()` function only works with UTF-8 encoded strings; that is, if you assign a `std::string` to a JSON value, make sure it is UTF-8 encoded.
The binary writers [`to_cbor()`](../api/basic_json/to_cbor.md), [`to_ubjson()`](../api/basic_json/to_ubjson.md),
[`to_bjdata()`](../api/basic_json/to_bjdata.md), and [`to_bson()`](../api/basic_json/to_bson.md) throw this exception
as well for a string value or object key that is not valid UTF-8 if their `error_handler` is `strict` (the default if
[`JSON_STRICT_BINARY_UTF8`](../api/macros/json_strict_binary_utf8.md) is enabled).
!!! failure "Example message"
Calling `dump()` on a JSON value containing an ISO 8859-1 encoded string:
docs/mkdocs/docs/integration/cmake.md
+5
View File
@@ -204,6 +204,11 @@ Use the non-amalgamated version of the library. This option is `ON` by default.
Treat the library headers like system headers (i.e., adding `SYSTEM` to the [`target_include_directories`](https://cmake.org/cmake/help/latest/command/target_include_directories.html) call) to check for this library by tools like Clang-Tidy. This option is `OFF` by default.
### `JSON_StrictBinaryUTF8`
Check string values and object keys for valid UTF-8 in the CBOR, UBJSON, BJData, and BSON writers, by defining the
macro [`JSON_STRICT_BINARY_UTF8`](../api/macros/json_strict_binary_utf8.md). This option is `OFF` by default.
### `JSON_StrictNulHandling`
Reject a `'\0'` (NUL) byte in the input instead of treating it as end of input, by defining the macro
docs/mkdocs/mkdocs.yml
+1
View File
@@ -301,6 +301,7 @@ nav:
- 'JSON_PRECISE_STREAM_POSITION': api/macros/json_precise_stream_position.md
- 'JSON_SKIP_LIBRARY_VERSION_CHECK': api/macros/json_skip_library_version_check.md
- 'JSON_SKIP_UNSUPPORTED_COMPILER_CHECK': api/macros/json_skip_unsupported_compiler_check.md
- 'JSON_STRICT_BINARY_UTF8': api/macros/json_strict_binary_utf8.md
- 'JSON_STRICT_NUL_HANDLING': api/macros/json_strict_nul_handling.md
- 'JSON_USE_GLOBAL_UDLS': api/macros/json_use_global_udls.md
- 'JSON_USE_IMPLICIT_CONVERSIONS': api/macros/json_use_implicit_conversions.md
include/nlohmann/detail/abi_macros.hpp
+15 -4
View File
@@ -46,6 +46,10 @@
#define JSON_STRICT_NUL_HANDLING 0
#endif
#ifndef JSON_STRICT_BINARY_UTF8
#define JSON_STRICT_BINARY_UTF8 0
#endif
#if JSON_DIAGNOSTICS
#define NLOHMANN_JSON_ABI_TAG_DIAGNOSTICS _diag
#else
@@ -82,14 +86,20 @@
#define NLOHMANN_JSON_ABI_TAG_STRICT_NUL_HANDLING
#endif
#if JSON_STRICT_BINARY_UTF8
#define NLOHMANN_JSON_ABI_TAG_STRICT_BINARY_UTF8 _sbu8
#else
#define NLOHMANN_JSON_ABI_TAG_STRICT_BINARY_UTF8
#endif
#ifndef NLOHMANN_JSON_NAMESPACE_NO_VERSION
#define NLOHMANN_JSON_NAMESPACE_NO_VERSION 0
#endif
// Construct the namespace ABI tags component
#define NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b, c, d, e, f) json_abi ## a ## b ## c ## d ## e ## f
#define NLOHMANN_JSON_ABI_TAGS_CONCAT(a, b, c, d, e, f) \
NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b, c, d, e, f)
#define NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b, c, d, e, f, g) json_abi ## a ## b ## c ## d ## e ## f ## g
#define NLOHMANN_JSON_ABI_TAGS_CONCAT(a, b, c, d, e, f, g) \
NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b, c, d, e, f, g)
#define NLOHMANN_JSON_ABI_TAGS \
NLOHMANN_JSON_ABI_TAGS_CONCAT( \
@@ -98,7 +108,8 @@
NLOHMANN_JSON_ABI_TAG_DIAGNOSTIC_POSITIONS, \
NLOHMANN_JSON_ABI_TAG_BRACE_INIT_COPY_SEMANTICS, \
NLOHMANN_JSON_ABI_TAG_PRECISE_STREAM_POSITION, \
NLOHMANN_JSON_ABI_TAG_STRICT_NUL_HANDLING)
NLOHMANN_JSON_ABI_TAG_STRICT_NUL_HANDLING, \
NLOHMANN_JSON_ABI_TAG_STRICT_BINARY_UTF8)
// Construct the namespace version component
#define NLOHMANN_JSON_NAMESPACE_VERSION_CONCAT_EX(major, minor, patch) \
include/nlohmann/detail/macro_unscope.hpp
+1
View File
@@ -41,6 +41,7 @@
#undef JSON_BRACE_INIT_COPY_SEMANTICS
#undef JSON_PRECISE_STREAM_POSITION
#undef JSON_STRICT_NUL_HANDLING
#undef JSON_STRICT_BINARY_UTF8
#endif
#include <nlohmann/thirdparty/hedley/hedley_undef.hpp>
include/nlohmann/detail/output/binary_writer.hpp
+3 -3
View File
@@ -98,7 +98,7 @@ class binary_writer
is not valid UTF-8 (CBOR, UBJSON, BJData, and BSON only; never
consulted by @ref write_msgpack or @ref write_bon8)
*/
explicit binary_writer(OutputSinkType sink, const error_handler_t error_handler_ = error_handler_t::strict)
explicit binary_writer(OutputSinkType sink, const error_handler_t error_handler_ = binary_writer_default_error_handler())
: oa(std::move(sink)), error_handler(error_handler_)
{}
@@ -118,7 +118,7 @@ class binary_writer
*/
template < typename SinkType = OutputSinkType,
typename std::enable_if < std::is_constructible<SinkType, output_adapter_t<CharType>>::value, int >::type = 0 >
explicit binary_writer(output_adapter_t<CharType> adapter, const error_handler_t error_handler_ = error_handler_t::strict)
explicit binary_writer(output_adapter_t<CharType> adapter, const error_handler_t error_handler_ = binary_writer_default_error_handler())
: oa(SinkType(std::move(adapter))), error_handler(error_handler_)
{}
@@ -2601,7 +2601,7 @@ class binary_writer
/// how to treat a string value or object key that is not valid UTF-8
/// (CBOR, UBJSON, BJData, and BSON only)
const error_handler_t error_handler = error_handler_t::strict;
const error_handler_t error_handler = binary_writer_default_error_handler();
};
} // namespace detail
include/nlohmann/detail/output/error_handler.hpp
+12
View File
@@ -34,5 +34,17 @@ enum class error_handler_t
keep ///< keep invalid UTF-8 sequences unchanged
};
/// the default error handler of the CBOR, UBJSON, BJData, and BSON writers:
/// error_handler_t::strict if JSON_STRICT_BINARY_UTF8 is enabled, otherwise
/// error_handler_t::keep (the behavior before version 3.13.0)
constexpr error_handler_t binary_writer_default_error_handler() noexcept
{
#if JSON_STRICT_BINARY_UTF8
return error_handler_t::strict;
#else
return error_handler_t::keep;
#endif
}
} // namespace detail
NLOHMANN_JSON_NAMESPACE_END
include/nlohmann/json.hpp
+13 -13
View File
@@ -202,7 +202,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
template<typename CharType> using vector_binary_writer =
::nlohmann::detail::binary_writer<basic_json, CharType, ::nlohmann::detail::output_vector_sink<CharType>>;
template<typename CharType> static vector_binary_writer<CharType> vector_writer(
std::vector<CharType>& v, const ::nlohmann::detail::error_handler_t error_handler = ::nlohmann::detail::error_handler_t::strict)
std::vector<CharType>& v, const ::nlohmann::detail::error_handler_t error_handler = ::nlohmann::detail::binary_writer_default_error_handler())
{
return vector_binary_writer<CharType>(::nlohmann::detail::output_vector_sink<CharType>(v), error_handler);
}
@@ -5447,7 +5447,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
/// @brief create a CBOR serialization of a given JSON value
/// @sa https://json.nlohmann.me/api/basic_json/to_cbor/
static std::vector<std::uint8_t> to_cbor(const basic_json& j,
const error_handler_t error_handler = error_handler_t::strict)
const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{
std::vector<std::uint8_t> result;
result.reserve(detail::binary_reserve_hint(j));
@@ -5458,7 +5458,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
/// @brief create a CBOR serialization of a given JSON value
/// @sa https://json.nlohmann.me/api/basic_json/to_cbor/
static void to_cbor(const basic_json& j, detail::output_adapter<std::uint8_t> o,
const error_handler_t error_handler = error_handler_t::strict)
const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{
binary_writer<std::uint8_t>(o, error_handler).write_cbor(j);
}
@@ -5466,7 +5466,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
/// @brief create a CBOR serialization of a given JSON value
/// @sa https://json.nlohmann.me/api/basic_json/to_cbor/
static void to_cbor(const basic_json& j, detail::output_adapter<char> o,
const error_handler_t error_handler = error_handler_t::strict)
const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{
binary_writer<char>(o, error_handler).write_cbor(j);
}
@@ -5500,7 +5500,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
static std::vector<std::uint8_t> to_ubjson(const basic_json& j,
const bool use_size = false,
const bool use_type = false,
const error_handler_t error_handler = error_handler_t::strict)
const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{
std::vector<std::uint8_t> result;
result.reserve(detail::binary_reserve_hint(j));
@@ -5512,7 +5512,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
/// @sa https://json.nlohmann.me/api/basic_json/to_ubjson/
static void to_ubjson(const basic_json& j, detail::output_adapter<std::uint8_t> o,
const bool use_size = false, const bool use_type = false,
const error_handler_t error_handler = error_handler_t::strict)
const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{
binary_writer<std::uint8_t>(o, error_handler).write_ubjson(j, use_size, use_type);
}
@@ -5521,7 +5521,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
/// @sa https://json.nlohmann.me/api/basic_json/to_ubjson/
static void to_ubjson(const basic_json& j, detail::output_adapter<char> o,
const bool use_size = false, const bool use_type = false,
const error_handler_t error_handler = error_handler_t::strict)
const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{
binary_writer<char>(o, error_handler).write_ubjson(j, use_size, use_type);
}
@@ -5532,7 +5532,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
const bool use_size = false,
const bool use_type = false,
const bjdata_version_t version = bjdata_version_t::draft2,
const error_handler_t error_handler = error_handler_t::strict)
const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{
std::vector<std::uint8_t> result;
result.reserve(detail::binary_reserve_hint(j));
@@ -5545,7 +5545,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
static void to_bjdata(const basic_json& j, detail::output_adapter<std::uint8_t> o,
const bool use_size = false, const bool use_type = false,
const bjdata_version_t version = bjdata_version_t::draft2,
const error_handler_t error_handler = error_handler_t::strict)
const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{
binary_writer<std::uint8_t>(o, error_handler).write_ubjson(j, use_size, use_type, true, true, version);
}
@@ -5555,7 +5555,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
static void to_bjdata(const basic_json& j, detail::output_adapter<char> o,
const bool use_size = false, const bool use_type = false,
const bjdata_version_t version = bjdata_version_t::draft2,
const error_handler_t error_handler = error_handler_t::strict)
const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{
binary_writer<char>(o, error_handler).write_ubjson(j, use_size, use_type, true, true, version);
}
@@ -5563,7 +5563,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
/// @brief create a BSON serialization of a given JSON value
/// @sa https://json.nlohmann.me/api/basic_json/to_bson/
static std::vector<std::uint8_t> to_bson(const basic_json& j,
const error_handler_t error_handler = error_handler_t::strict)
const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{
std::vector<std::uint8_t> result;
result.reserve(detail::binary_reserve_hint(j));
@@ -5574,7 +5574,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
/// @brief create a BSON serialization of a given JSON value
/// @sa https://json.nlohmann.me/api/basic_json/to_bson/
static void to_bson(const basic_json& j, detail::output_adapter<std::uint8_t> o,
const error_handler_t error_handler = error_handler_t::strict)
const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{
binary_writer<std::uint8_t>(o, error_handler).write_bson(j);
}
@@ -5582,7 +5582,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
/// @brief create a BSON serialization of a given JSON value
/// @sa https://json.nlohmann.me/api/basic_json/to_bson/
static void to_bson(const basic_json& j, detail::output_adapter<char> o,
const error_handler_t error_handler = error_handler_t::strict)
const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{
binary_writer<char>(o, error_handler).write_bson(j);
}
single_include/nlohmann/json.hpp
+44 -20
View File
@@ -104,6 +104,10 @@
#define JSON_STRICT_NUL_HANDLING 0
#endif
#ifndef JSON_STRICT_BINARY_UTF8
#define JSON_STRICT_BINARY_UTF8 0
#endif
#if JSON_DIAGNOSTICS
#define NLOHMANN_JSON_ABI_TAG_DIAGNOSTICS _diag
#else
@@ -140,14 +144,20 @@
#define NLOHMANN_JSON_ABI_TAG_STRICT_NUL_HANDLING
#endif
#if JSON_STRICT_BINARY_UTF8
#define NLOHMANN_JSON_ABI_TAG_STRICT_BINARY_UTF8 _sbu8
#else
#define NLOHMANN_JSON_ABI_TAG_STRICT_BINARY_UTF8
#endif
#ifndef NLOHMANN_JSON_NAMESPACE_NO_VERSION
#define NLOHMANN_JSON_NAMESPACE_NO_VERSION 0
#endif
// Construct the namespace ABI tags component
#define NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b, c, d, e, f) json_abi ## a ## b ## c ## d ## e ## f
#define NLOHMANN_JSON_ABI_TAGS_CONCAT(a, b, c, d, e, f) \
NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b, c, d, e, f)
#define NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b, c, d, e, f, g) json_abi ## a ## b ## c ## d ## e ## f ## g
#define NLOHMANN_JSON_ABI_TAGS_CONCAT(a, b, c, d, e, f, g) \
NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b, c, d, e, f, g)
#define NLOHMANN_JSON_ABI_TAGS \
NLOHMANN_JSON_ABI_TAGS_CONCAT( \
@@ -156,7 +166,8 @@
NLOHMANN_JSON_ABI_TAG_DIAGNOSTIC_POSITIONS, \
NLOHMANN_JSON_ABI_TAG_BRACE_INIT_COPY_SEMANTICS, \
NLOHMANN_JSON_ABI_TAG_PRECISE_STREAM_POSITION, \
NLOHMANN_JSON_ABI_TAG_STRICT_NUL_HANDLING)
NLOHMANN_JSON_ABI_TAG_STRICT_NUL_HANDLING, \
NLOHMANN_JSON_ABI_TAG_STRICT_BINARY_UTF8)
// Construct the namespace version component
#define NLOHMANN_JSON_NAMESPACE_VERSION_CONCAT_EX(major, minor, patch) \
@@ -6258,6 +6269,18 @@ enum class error_handler_t
keep ///< keep invalid UTF-8 sequences unchanged
};
/// the default error handler of the CBOR, UBJSON, BJData, and BSON writers:
/// error_handler_t::strict if JSON_STRICT_BINARY_UTF8 is enabled, otherwise
/// error_handler_t::keep (the behavior before version 3.13.0)
constexpr error_handler_t binary_writer_default_error_handler() noexcept
{
#if JSON_STRICT_BINARY_UTF8
return error_handler_t::strict;
#else
return error_handler_t::keep;
#endif
}
} // namespace detail
NLOHMANN_JSON_NAMESPACE_END
@@ -21398,7 +21421,7 @@ class binary_writer
is not valid UTF-8 (CBOR, UBJSON, BJData, and BSON only; never
consulted by @ref write_msgpack or @ref write_bon8)
*/
explicit binary_writer(OutputSinkType sink, const error_handler_t error_handler_ = error_handler_t::strict)
explicit binary_writer(OutputSinkType sink, const error_handler_t error_handler_ = binary_writer_default_error_handler())
: oa(std::move(sink)), error_handler(error_handler_)
{}
@@ -21418,7 +21441,7 @@ class binary_writer
*/
template < typename SinkType = OutputSinkType,
typename std::enable_if < std::is_constructible<SinkType, output_adapter_t<CharType>>::value, int >::type = 0 >
explicit binary_writer(output_adapter_t<CharType> adapter, const error_handler_t error_handler_ = error_handler_t::strict)
explicit binary_writer(output_adapter_t<CharType> adapter, const error_handler_t error_handler_ = binary_writer_default_error_handler())
: oa(SinkType(std::move(adapter))), error_handler(error_handler_)
{}
@@ -23901,7 +23924,7 @@ class binary_writer
/// how to treat a string value or object key that is not valid UTF-8
/// (CBOR, UBJSON, BJData, and BSON only)
const error_handler_t error_handler = error_handler_t::strict;
const error_handler_t error_handler = binary_writer_default_error_handler();
};
} // namespace detail
@@ -27249,7 +27272,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
template<typename CharType> using vector_binary_writer =
::nlohmann::detail::binary_writer<basic_json, CharType, ::nlohmann::detail::output_vector_sink<CharType>>;
template<typename CharType> static vector_binary_writer<CharType> vector_writer(
std::vector<CharType>& v, const ::nlohmann::detail::error_handler_t error_handler = ::nlohmann::detail::error_handler_t::strict)
std::vector<CharType>& v, const ::nlohmann::detail::error_handler_t error_handler = ::nlohmann::detail::binary_writer_default_error_handler())
{
return vector_binary_writer<CharType>(::nlohmann::detail::output_vector_sink<CharType>(v), error_handler);
}
@@ -32494,7 +32517,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
/// @brief create a CBOR serialization of a given JSON value
/// @sa https://json.nlohmann.me/api/basic_json/to_cbor/
static std::vector<std::uint8_t> to_cbor(const basic_json& j,
const error_handler_t error_handler = error_handler_t::strict)
const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{
std::vector<std::uint8_t> result;
result.reserve(detail::binary_reserve_hint(j));
@@ -32505,7 +32528,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
/// @brief create a CBOR serialization of a given JSON value
/// @sa https://json.nlohmann.me/api/basic_json/to_cbor/
static void to_cbor(const basic_json& j, detail::output_adapter<std::uint8_t> o,
const error_handler_t error_handler = error_handler_t::strict)
const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{
binary_writer<std::uint8_t>(o, error_handler).write_cbor(j);
}
@@ -32513,7 +32536,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
/// @brief create a CBOR serialization of a given JSON value
/// @sa https://json.nlohmann.me/api/basic_json/to_cbor/
static void to_cbor(const basic_json& j, detail::output_adapter<char> o,
const error_handler_t error_handler = error_handler_t::strict)
const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{
binary_writer<char>(o, error_handler).write_cbor(j);
}
@@ -32547,7 +32570,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
static std::vector<std::uint8_t> to_ubjson(const basic_json& j,
const bool use_size = false,
const bool use_type = false,
const error_handler_t error_handler = error_handler_t::strict)
const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{
std::vector<std::uint8_t> result;
result.reserve(detail::binary_reserve_hint(j));
@@ -32559,7 +32582,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
/// @sa https://json.nlohmann.me/api/basic_json/to_ubjson/
static void to_ubjson(const basic_json& j, detail::output_adapter<std::uint8_t> o,
const bool use_size = false, const bool use_type = false,
const error_handler_t error_handler = error_handler_t::strict)
const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{
binary_writer<std::uint8_t>(o, error_handler).write_ubjson(j, use_size, use_type);
}
@@ -32568,7 +32591,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
/// @sa https://json.nlohmann.me/api/basic_json/to_ubjson/
static void to_ubjson(const basic_json& j, detail::output_adapter<char> o,
const bool use_size = false, const bool use_type = false,
const error_handler_t error_handler = error_handler_t::strict)
const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{
binary_writer<char>(o, error_handler).write_ubjson(j, use_size, use_type);
}
@@ -32579,7 +32602,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
const bool use_size = false,
const bool use_type = false,
const bjdata_version_t version = bjdata_version_t::draft2,
const error_handler_t error_handler = error_handler_t::strict)
const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{
std::vector<std::uint8_t> result;
result.reserve(detail::binary_reserve_hint(j));
@@ -32592,7 +32615,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
static void to_bjdata(const basic_json& j, detail::output_adapter<std::uint8_t> o,
const bool use_size = false, const bool use_type = false,
const bjdata_version_t version = bjdata_version_t::draft2,
const error_handler_t error_handler = error_handler_t::strict)
const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{
binary_writer<std::uint8_t>(o, error_handler).write_ubjson(j, use_size, use_type, true, true, version);
}
@@ -32602,7 +32625,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
static void to_bjdata(const basic_json& j, detail::output_adapter<char> o,
const bool use_size = false, const bool use_type = false,
const bjdata_version_t version = bjdata_version_t::draft2,
const error_handler_t error_handler = error_handler_t::strict)
const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{
binary_writer<char>(o, error_handler).write_ubjson(j, use_size, use_type, true, true, version);
}
@@ -32610,7 +32633,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
/// @brief create a BSON serialization of a given JSON value
/// @sa https://json.nlohmann.me/api/basic_json/to_bson/
static std::vector<std::uint8_t> to_bson(const basic_json& j,
const error_handler_t error_handler = error_handler_t::strict)
const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{
std::vector<std::uint8_t> result;
result.reserve(detail::binary_reserve_hint(j));
@@ -32621,7 +32644,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
/// @brief create a BSON serialization of a given JSON value
/// @sa https://json.nlohmann.me/api/basic_json/to_bson/
static void to_bson(const basic_json& j, detail::output_adapter<std::uint8_t> o,
const error_handler_t error_handler = error_handler_t::strict)
const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{
binary_writer<std::uint8_t>(o, error_handler).write_bson(j);
}
@@ -32629,7 +32652,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
/// @brief create a BSON serialization of a given JSON value
/// @sa https://json.nlohmann.me/api/basic_json/to_bson/
static void to_bson(const basic_json& j, detail::output_adapter<char> o,
const error_handler_t error_handler = error_handler_t::strict)
const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{
binary_writer<char>(o, error_handler).write_bson(j);
}
@@ -34215,6 +34238,7 @@ struct formatter<nlohmann::NLOHMANN_BASIC_JSON_TPL, char> // NOLINT(cert-dcl58-c
#undef JSON_BRACE_INIT_COPY_SEMANTICS
#undef JSON_PRECISE_STREAM_POSITION
#undef JSON_STRICT_NUL_HANDLING
#undef JSON_STRICT_BINARY_UTF8
#endif
// #include <nlohmann/thirdparty/hedley/hedley_undef.hpp>
single_include/nlohmann/json_fwd.hpp
+15 -4
View File
@@ -63,6 +63,10 @@
#define JSON_STRICT_NUL_HANDLING 0
#endif
#ifndef JSON_STRICT_BINARY_UTF8
#define JSON_STRICT_BINARY_UTF8 0
#endif
#if JSON_DIAGNOSTICS
#define NLOHMANN_JSON_ABI_TAG_DIAGNOSTICS _diag
#else
@@ -99,14 +103,20 @@
#define NLOHMANN_JSON_ABI_TAG_STRICT_NUL_HANDLING
#endif
#if JSON_STRICT_BINARY_UTF8
#define NLOHMANN_JSON_ABI_TAG_STRICT_BINARY_UTF8 _sbu8
#else
#define NLOHMANN_JSON_ABI_TAG_STRICT_BINARY_UTF8
#endif
#ifndef NLOHMANN_JSON_NAMESPACE_NO_VERSION
#define NLOHMANN_JSON_NAMESPACE_NO_VERSION 0
#endif
// Construct the namespace ABI tags component
#define NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b, c, d, e, f) json_abi ## a ## b ## c ## d ## e ## f
#define NLOHMANN_JSON_ABI_TAGS_CONCAT(a, b, c, d, e, f) \
NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b, c, d, e, f)
#define NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b, c, d, e, f, g) json_abi ## a ## b ## c ## d ## e ## f ## g
#define NLOHMANN_JSON_ABI_TAGS_CONCAT(a, b, c, d, e, f, g) \
NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b, c, d, e, f, g)
#define NLOHMANN_JSON_ABI_TAGS \
NLOHMANN_JSON_ABI_TAGS_CONCAT( \
@@ -115,7 +125,8 @@
NLOHMANN_JSON_ABI_TAG_DIAGNOSTIC_POSITIONS, \
NLOHMANN_JSON_ABI_TAG_BRACE_INIT_COPY_SEMANTICS, \
NLOHMANN_JSON_ABI_TAG_PRECISE_STREAM_POSITION, \
NLOHMANN_JSON_ABI_TAG_STRICT_NUL_HANDLING)
NLOHMANN_JSON_ABI_TAG_STRICT_NUL_HANDLING, \
NLOHMANN_JSON_ABI_TAG_STRICT_BINARY_UTF8)
// Construct the namespace version component
#define NLOHMANN_JSON_NAMESPACE_VERSION_CONCAT_EX(major, minor, patch) \
tests/abi/config/default.cpp
+4
View File
@@ -44,6 +44,10 @@ TEST_CASE("default namespace")
expected += "_snul";
#endif
#if JSON_STRICT_BINARY_UTF8
expected += "_sbu8";
#endif
expected += "_v" STRINGIZE(NLOHMANN_JSON_VERSION_MAJOR);
expected += "_" STRINGIZE(NLOHMANN_JSON_VERSION_MINOR);
expected += "_" STRINGIZE(NLOHMANN_JSON_VERSION_PATCH) "::basic_json";
tests/abi/config/noversion.cpp
+4
View File
@@ -45,6 +45,10 @@ TEST_CASE("default namespace without version component")
expected += "_snul";
#endif
#if JSON_STRICT_BINARY_UTF8
expected += "_sbu8";
#endif
expected += "::basic_json";
// fallback for Clang
tests/src/unit-binary_utf8_error_handler.cpp
+8 -5
View File
@@ -318,14 +318,17 @@ TEST_CASE("UTF-8 error_handler for the binary readers and writers")
{
const json jval = ill_formed_cases[0].bytes;
// to_*: the default error_handler is strict, so ill-formed input still throws
CHECK_THROWS_AS(json::to_cbor(jval), json::type_error&);
CHECK_THROWS_AS(json::to_ubjson(jval), json::type_error&);
CHECK_THROWS_AS(json::to_bjdata(jval), json::type_error&);
// to_*: the default error_handler is keep, so ill-formed bytes are
// written unchanged, exactly as in release 3.12.0 (it is strict only
// if JSON_STRICT_BINARY_UTF8 is enabled, see
// unit-binary_utf8_strict.cpp)
CHECK(json::to_cbor(jval) == json::to_cbor(jval, eh::keep));
CHECK(json::to_ubjson(jval) == json::to_ubjson(jval, false, false, eh::keep));
CHECK(json::to_bjdata(jval) == json::to_bjdata(jval, false, false, json::bjdata_version_t::draft2, eh::keep));
{
json jobj;
jobj["k"] = jval;
CHECK_THROWS_AS(json::to_bson(jobj), json::type_error&);
CHECK(json::to_bson(jobj) == json::to_bson(jobj, eh::keep));
}
// from_*: the default error_handler is keep, so ill-formed bytes are
tests/src/unit-binary_utf8_strict.cpp
+120
View File
@@ -0,0 +1,120 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++ (supporting code)
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
#include "doctest_compatibility.h"
// The binary writers check strings and object keys for valid UTF-8 only if
// JSON_STRICT_BINARY_UTF8 is enabled (planned to be the default in 4.0.0).
// Without it, they write the bytes unchanged, as before version 3.13.0; the
// tests for that are next to the other tests of each format.
#ifdef JSON_STRICT_BINARY_UTF8
#undef JSON_STRICT_BINARY_UTF8
#endif
#define JSON_STRICT_BINARY_UTF8 1
#include <nlohmann/json.hpp>
using nlohmann::json;
#include <cstdint>
#include <vector>
TEST_CASE("JSON_STRICT_BINARY_UTF8 (see #5529, #5651)")
{
SECTION("CBOR")
{
// a string value with ill-formed UTF-8 is rejected
CHECK_THROWS_WITH_AS(json::to_cbor(json("\xFF")), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xFF", json::type_error&);
// a truncated multi-byte sequence
CHECK_THROWS_WITH_AS(json::to_cbor(json("\xC3")), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xC3", json::type_error&);
// an encoded surrogate half (U+D800)
CHECK_THROWS_WITH_AS(json::to_cbor(json("\xED\xA0\x80")), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xED", json::type_error&);
// an overlong encoding of '.'
CHECK_THROWS_WITH_AS(json::to_cbor(json("\xC0\xAF")), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xC0", json::type_error&);
// an object key with ill-formed UTF-8 is rejected the same way
CHECK_THROWS_WITH_AS(json::to_cbor(json{{"\xFF", 1}}), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xFF", json::type_error&);
// binary values are not text and are unaffected
CHECK_NOTHROW(json::to_cbor(json::binary(std::vector<std::uint8_t>({0xFF}))));
// a value read back from CBOR with ill-formed bytes cannot be written
// back either (the reader is lenient regardless of the macro)
const json j = json::from_cbor(std::vector<std::uint8_t>({0x62, 0xc0, 0xae}));
CHECK_THROWS_WITH_AS(json::to_cbor(j), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xC0", json::type_error&);
}
SECTION("UBJSON")
{
CHECK_THROWS_WITH_AS(json::to_ubjson(json("\xFF")), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xFF", json::type_error&);
// a truncated multi-byte sequence
CHECK_THROWS_WITH_AS(json::to_ubjson(json("\xC3")), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xC3", json::type_error&);
// an encoded surrogate half (U+D800)
CHECK_THROWS_WITH_AS(json::to_ubjson(json("\xED\xA0\x80")), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xED", json::type_error&);
// an overlong encoding of '.'
CHECK_THROWS_WITH_AS(json::to_ubjson(json("\xC0\xAF")), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xC0", json::type_error&);
// an object key with ill-formed UTF-8 is rejected the same way
CHECK_THROWS_WITH_AS(json::to_ubjson(json{{"\xFF", 1}}), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xFF", json::type_error&);
}
SECTION("BJData")
{
CHECK_THROWS_WITH_AS(json::to_bjdata(json("\xFF")), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xFF", json::type_error&);
// a truncated multi-byte sequence
CHECK_THROWS_WITH_AS(json::to_bjdata(json("\xC3")), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xC3", json::type_error&);
// an encoded surrogate half (U+D800)
CHECK_THROWS_WITH_AS(json::to_bjdata(json("\xED\xA0\x80")), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xED", json::type_error&);
// an overlong encoding of '.'
CHECK_THROWS_WITH_AS(json::to_bjdata(json("\xC0\xAF")), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xC0", json::type_error&);
// an object key with ill-formed UTF-8 is rejected the same way
CHECK_THROWS_WITH_AS(json::to_bjdata(json{{"\xFF", 1}}), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xFF", json::type_error&);
}
SECTION("BSON")
{
// to_bson() rejects the same kind of ill-formed string value, before
// any bytes reach the output adapter (the BSON document length
// prefix must be known up front, so nothing is written incrementally)
std::vector<std::uint8_t> out{0x42}; // a sentinel byte the writer must not touch
CHECK_THROWS_WITH_AS(json::to_bson(json{{"s", "\xFF"}}, nlohmann::detail::output_adapter<std::uint8_t>(out)), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xFF", json::type_error&);
CHECK(out == std::vector<std::uint8_t> {0x42});
CHECK_THROWS_WITH_AS(json::to_bson(json{{"s", "\xFF"}}), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xFF", json::type_error&);
// a truncated multi-byte sequence
CHECK_THROWS_WITH_AS(json::to_bson(json{{"s", "\xC3"}}), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xC3", json::type_error&);
// an encoded surrogate half (U+D800)
CHECK_THROWS_WITH_AS(json::to_bson(json{{"s", "\xED\xA0\x80"}}), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xED", json::type_error&);
// an overlong encoding of '.'
CHECK_THROWS_WITH_AS(json::to_bson(json{{"s", "\xC0\xAF"}}), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xC0", json::type_error&);
// an object key with ill-formed UTF-8 is rejected as well; unlike
// the reader (which never validates element names), the writer
// checks both string values and object keys
CHECK_THROWS_WITH_AS(json::to_bson(json{{"\xFF", 1}}), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xFF", json::type_error&);
}
SECTION("an explicit error_handler overrides the default")
{
// the macro only changes the default of the error_handler parameter
CHECK(json::to_cbor(json("\xFF"), json::error_handler_t::keep) == std::vector<std::uint8_t>({0x61, 0xff}));
CHECK(json::to_ubjson(json("\xFF"), false, false, json::error_handler_t::keep) == std::vector<std::uint8_t>({'S', 'i', 1, 0xff}));
CHECK(json::to_bjdata(json("\xFF"), false, false, json::bjdata_version_t::draft2, json::error_handler_t::keep) == std::vector<std::uint8_t>({'S', 'i', 1, 0xff}));
CHECK(json::from_bson(json::to_bson(json{{"s", "\xFF"}}, json::error_handler_t::keep)) == json{{"s", "\xFF"}});
CHECK(json::to_cbor(json("\xFF"), json::error_handler_t::replace) == std::vector<std::uint8_t>({0x63, 0xef, 0xbf, 0xbd}));
}
SECTION("MessagePack and BON8 are unaffected")
{
// MessagePack allows any bytes in a str, so to_msgpack() writes them as
// is; BON8 always checks, because the lead bytes mark where strings end
CHECK(json::to_msgpack(json("\xFF")) == std::vector<std::uint8_t>({0xa1, 0xff}));
CHECK_THROWS_AS(json::to_bon8(json("\xFF")), json::type_error&);
}
}
tests/src/unit-bjdata.cpp
+12 -10
View File
@@ -3913,15 +3913,17 @@ TEST_CASE("Universal Binary JSON Specification Examples 1")
// none of the binary format specs requires a decoder to reject
// ill-formed UTF-8 in a text string, so a value whose bytes are
// not valid UTF-8 (0xC0 0xAE is an overlong encoding of '.')
// round-trips byte for byte as a string value; to_bjdata() is
// strict, so such a value cannot be written back
// round-trips byte for byte as a string value; to_bjdata() writes
// the bytes unchanged, as before 3.13.0, unless
// JSON_STRICT_BINARY_UTF8 is enabled (see
// unit-binary_utf8_strict.cpp)
const std::vector<uint8_t> v = {'S', 'i', 2, 0xc0, 0xae};
json j;
CHECK_NOTHROW(j = json::from_bjdata(v));
REQUIRE(j.is_string());
CHECK(j.get_ref<const json::string_t&>() == std::string("\xc0\xae"));
CHECK_THROWS_AS(j.dump(), json::type_error&);
CHECK_THROWS_AS(json::to_bjdata(j), json::type_error&);
CHECK(json::from_bjdata(json::to_bjdata(j)) == j);
// the same bytes as an object key round-trip as well
const std::vector<uint8_t> v_key = {'{', 'i', 2, 0xc0, 0xae, 'i', 1, '}'};
@@ -3929,18 +3931,18 @@ TEST_CASE("Universal Binary JSON Specification Examples 1")
CHECK_NOTHROW(j_key = json::from_bjdata(v_key));
REQUIRE(j_key.is_object());
CHECK(j_key.contains(std::string("\xc0\xae")));
CHECK_THROWS_AS(json::to_bjdata(j_key), json::type_error&);
CHECK(json::from_bjdata(json::to_bjdata(j_key)) == j_key);
CHECK_THROWS_WITH_AS(json::to_bjdata(json("\xFF")), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xFF", json::type_error&);
CHECK(json::from_bjdata(json::to_bjdata(json("\xFF"))) == json("\xFF"));
// a truncated multi-byte sequence
CHECK_THROWS_WITH_AS(json::to_bjdata(json("\xC3")), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xC3", json::type_error&);
CHECK(json::from_bjdata(json::to_bjdata(json("\xC3"))) == json("\xC3"));
// an encoded surrogate half (U+D800)
CHECK_THROWS_WITH_AS(json::to_bjdata(json("\xED\xA0\x80")), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xED", json::type_error&);
CHECK(json::from_bjdata(json::to_bjdata(json("\xED\xA0\x80"))) == json("\xED\xA0\x80"));
// an overlong encoding of '.'
CHECK_THROWS_WITH_AS(json::to_bjdata(json("\xC0\xAF")), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xC0", json::type_error&);
CHECK(json::from_bjdata(json::to_bjdata(json("\xC0\xAF"))) == json("\xC0\xAF"));
// an object key with ill-formed UTF-8 is rejected the same way
CHECK_THROWS_WITH_AS(json::to_bjdata(json{{"\xFF", 1}}), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xFF", json::type_error&);
// an object key with ill-formed UTF-8 is kept the same way
CHECK(json::from_bjdata(json::to_bjdata(json{{"\xFF", 1}})) == json{{"\xFF", 1}});
}
}
tests/src/unit-bson.cpp
+9 -17
View File
@@ -175,28 +175,20 @@ TEST_CASE("BSON")
CHECK(j["s"].get_ref<const json::string_t&>() == std::string("\xc0\xae"));
// dump() still requires valid UTF-8 and throws for such a value
CHECK_THROWS_AS(j.dump(), json::type_error&);
// to_bson() is strict as well, so the value cannot be written back
CHECK_THROWS_AS(json::to_bson(j), json::type_error&);
// to_bson() writes the bytes back unchanged, as before 3.13.0,
// unless JSON_STRICT_BINARY_UTF8 is enabled (see unit-binary_utf8_strict.cpp)
CHECK(json::from_bson(json::to_bson(j)) == j);
// to_bson() rejects the same kind of ill-formed string value, before
// any bytes reach the output adapter (the BSON document length
// prefix must be known up front, so nothing is written incrementally)
std::vector<std::uint8_t> out{0x42}; // a sentinel byte the writer must not touch
CHECK_THROWS_WITH_AS(json::to_bson(json{{"s", "\xFF"}}, nlohmann::detail::output_adapter<std::uint8_t>(out)), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xFF", json::type_error&);
CHECK(out == std::vector<std::uint8_t> {0x42});
CHECK_THROWS_WITH_AS(json::to_bson(json{{"s", "\xFF"}}), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xFF", json::type_error&);
CHECK(json::from_bson(json::to_bson(json{{"s", "\xFF"}})) == json{{"s", "\xFF"}});
// a truncated multi-byte sequence
CHECK_THROWS_WITH_AS(json::to_bson(json{{"s", "\xC3"}}), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xC3", json::type_error&);
CHECK(json::from_bson(json::to_bson(json{{"s", "\xC3"}})) == json{{"s", "\xC3"}});
// an encoded surrogate half (U+D800)
CHECK_THROWS_WITH_AS(json::to_bson(json{{"s", "\xED\xA0\x80"}}), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xED", json::type_error&);
CHECK(json::from_bson(json::to_bson(json{{"s", "\xED\xA0\x80"}})) == json{{"s", "\xED\xA0\x80"}});
// an overlong encoding of '.'
CHECK_THROWS_WITH_AS(json::to_bson(json{{"s", "\xC0\xAF"}}), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xC0", json::type_error&);
CHECK(json::from_bson(json::to_bson(json{{"s", "\xC0\xAF"}})) == json{{"s", "\xC0\xAF"}});
// an object key with ill-formed UTF-8 is rejected as well; unlike
// the reader (which never validates element names), the writer
// checks both string values and object keys
CHECK_THROWS_WITH_AS(json::to_bson(json{{"\xFF", 1}}), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xFF", json::type_error&);
// an object key with ill-formed UTF-8 is kept as well
CHECK(json::from_bson(json::to_bson(json{{"\xFF", 1}})) == json{{"\xFF", 1}});
}
SECTION("lengths exceeding INT32_MAX cannot be serialized to BSON")
tests/src/unit-cbor.cpp
+15 -13
View File
@@ -1821,8 +1821,9 @@ TEST_CASE("CBOR")
// unless an error handler that replaces or ignores the bytes is
// passed
CHECK_THROWS_AS(j_value.dump(), json::type_error&);
// to_cbor() is strict as well, so the value cannot be written back
CHECK_THROWS_AS(json::to_cbor(j_value), json::type_error&);
// to_cbor() writes the bytes back unchanged, as before 3.13.0,
// unless JSON_STRICT_BINARY_UTF8 is enabled (see unit-binary_utf8_strict.cpp)
CHECK(json::from_cbor(json::to_cbor(j_value)) == j_value);
// the same bytes as an object key round-trip as well
const std::vector<uint8_t> ill_formed_key = {0xa1, 0x62, 0xc0, 0xae, 0x01};
@@ -1830,7 +1831,7 @@ TEST_CASE("CBOR")
CHECK_NOTHROW(j_key = json::from_cbor(ill_formed_key));
REQUIRE(j_key.is_object());
CHECK(j_key.contains(std::string("\xc0\xae")));
CHECK_THROWS_AS(json::to_cbor(j_key), json::type_error&);
CHECK(json::from_cbor(json::to_cbor(j_key)) == j_key);
// a CBOR byte string (major type 2) with the very same bytes is
// NOT text and must still be accepted as-is
@@ -1843,20 +1844,21 @@ TEST_CASE("CBOR")
CHECK(json::from_cbor(json::to_cbor(j)) == j);
}
SECTION("to_cbor rejects ill-formed UTF-8 (see #5651)")
SECTION("to_cbor keeps ill-formed UTF-8 (see #5651)")
{
// to_cbor() must reject the same ill-formed strings from_cbor()
// rejects, so a value it accepts can always be read back
CHECK_THROWS_WITH_AS(json::to_cbor(json("\xFF")), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xFF", json::type_error&);
// to_cbor() writes the bytes unchanged, as before 3.13.0, unless
// JSON_STRICT_BINARY_UTF8 is enabled (see
// unit-binary_utf8_strict.cpp); from_cbor() reads them back as is
CHECK(json::from_cbor(json::to_cbor(json("\xFF"))) == json("\xFF"));
// a truncated multi-byte sequence
CHECK_THROWS_WITH_AS(json::to_cbor(json("\xC3")), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xC3", json::type_error&);
CHECK(json::from_cbor(json::to_cbor(json("\xC3"))) == json("\xC3"));
// an encoded surrogate half (U+D800)
CHECK_THROWS_WITH_AS(json::to_cbor(json("\xED\xA0\x80")), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xED", json::type_error&);
CHECK(json::from_cbor(json::to_cbor(json("\xED\xA0\x80"))) == json("\xED\xA0\x80"));
// an overlong encoding of '.'
CHECK_THROWS_WITH_AS(json::to_cbor(json("\xC0\xAF")), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xC0", json::type_error&);
CHECK(json::from_cbor(json::to_cbor(json("\xC0\xAF"))) == json("\xC0\xAF"));
// an object key with ill-formed UTF-8 is rejected the same way
CHECK_THROWS_WITH_AS(json::to_cbor(json{{"\xFF", 1}}), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xFF", json::type_error&);
// an object key with ill-formed UTF-8 is kept the same way
CHECK(json::from_cbor(json::to_cbor(json{{"\xFF", 1}})) == json{{"\xFF", 1}});
// binary values are not text and are unaffected
CHECK_NOTHROW(json::to_cbor(json::binary(std::vector<std::uint8_t>({0xFF}))));
@@ -1877,7 +1879,7 @@ TEST_CASE("CBOR")
CHECK_NOTHROW(_ = json::from_cbor(std::vector<uint8_t>({0x7f, 0x61, 0xc3, 0xff})));
CHECK(_ == "\xc3");
CHECK_THROWS_AS(_.dump(), json::type_error&);
CHECK_THROWS_AS(json::to_cbor(_), json::type_error&);
CHECK(json::from_cbor(json::to_cbor(_)) == _);
// an ill-formed later chunk is kept after valid ones
CHECK_NOTHROW(_ = json::from_cbor(std::vector<uint8_t>({0x7f, 0x62, 0xc3, 0xa9, 0x62, 0xc0, 0xae, 0xff})));
tests/src/unit-ubjson.cpp
+12 -10
View File
@@ -2511,15 +2511,17 @@ TEST_CASE("Universal Binary JSON Specification Examples 1")
// none of the binary format specs requires a decoder to reject
// ill-formed UTF-8 in a text string, so a value whose bytes are
// not valid UTF-8 (0xC0 0xAE is an overlong encoding of '.')
// round-trips byte for byte as a string value; to_ubjson() is
// strict, so such a value cannot be written back
// round-trips byte for byte as a string value; to_ubjson() writes
// the bytes unchanged, as before 3.13.0, unless
// JSON_STRICT_BINARY_UTF8 is enabled (see
// unit-binary_utf8_strict.cpp)
const std::vector<uint8_t> v = {'S', 'i', 2, 0xc0, 0xae};
json j;
CHECK_NOTHROW(j = json::from_ubjson(v));
REQUIRE(j.is_string());
CHECK(j.get_ref<const json::string_t&>() == std::string("\xc0\xae"));
CHECK_THROWS_AS(j.dump(), json::type_error&);
CHECK_THROWS_AS(json::to_ubjson(j), json::type_error&);
CHECK(json::from_ubjson(json::to_ubjson(j)) == j);
// the same bytes as an object key round-trip as well
const std::vector<uint8_t> v_key = {'{', 'i', 2, 0xc0, 0xae, 'i', 1, '}'};
@@ -2527,18 +2529,18 @@ TEST_CASE("Universal Binary JSON Specification Examples 1")
CHECK_NOTHROW(j_key = json::from_ubjson(v_key));
REQUIRE(j_key.is_object());
CHECK(j_key.contains(std::string("\xc0\xae")));
CHECK_THROWS_AS(json::to_ubjson(j_key), json::type_error&);
CHECK(json::from_ubjson(json::to_ubjson(j_key)) == j_key);
CHECK_THROWS_WITH_AS(json::to_ubjson(json("\xFF")), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xFF", json::type_error&);
CHECK(json::from_ubjson(json::to_ubjson(json("\xFF"))) == json("\xFF"));
// a truncated multi-byte sequence
CHECK_THROWS_WITH_AS(json::to_ubjson(json("\xC3")), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xC3", json::type_error&);
CHECK(json::from_ubjson(json::to_ubjson(json("\xC3"))) == json("\xC3"));
// an encoded surrogate half (U+D800)
CHECK_THROWS_WITH_AS(json::to_ubjson(json("\xED\xA0\x80")), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xED", json::type_error&);
CHECK(json::from_ubjson(json::to_ubjson(json("\xED\xA0\x80"))) == json("\xED\xA0\x80"));
// an overlong encoding of '.'
CHECK_THROWS_WITH_AS(json::to_ubjson(json("\xC0\xAF")), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xC0", json::type_error&);
CHECK(json::from_ubjson(json::to_ubjson(json("\xC0\xAF"))) == json("\xC0\xAF"));
// an object key with ill-formed UTF-8 is rejected the same way
CHECK_THROWS_WITH_AS(json::to_ubjson(json{{"\xFF", 1}}), "[json.exception.type_error.316] invalid UTF-8 byte at index 0: 0xFF", json::type_error&);
// an object key with ill-formed UTF-8 is kept the same way
CHECK(json::from_ubjson(json::to_ubjson(json{{"\xFF", 1}})) == json{{"\xFF", 1}});
}
}