mirror of
https://github.com/nlohmann/json.git
synced 2026-10-03 13:10:33 +00:00
Merge #5741 and make keep the default error_handler of the binary writers
Merges claude/binary-utf8-roundtrip-5651, where the writers now only check UTF-8 if JSON_STRICT_BINARY_UTF8 is enabled, and applies the same rule to the error_handler parameter added here: to_cbor(), to_ubjson(), to_bjdata(), and to_bson() default to error_handler_t::keep, the behavior of 3.12.0, or to error_handler_t::strict if JSON_STRICT_BINARY_UTF8 is enabled (detail::binary_writer_default_error_handler()). Passing a handler explicitly always takes precedence. - Conflicts in binary_writer.hpp resolved in favor of this branch's error_handler, which replaces #5741's check_text_utf8(). - Tests: the "default parameters" section checks that the default equals keep; unit-binary_utf8_strict.cpp checks that an explicit handler overrides the macro's default. - Docs: signatures show keep; parameter, exception, and format pages describe keep as the default and the macro; the two 3.13.0 version history entries of each writer are merged into one. Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann
@@ -14,7 +14,9 @@ This enumeration is used to choose how to treat ill-formed UTF-8 in a string val
|
||||
- [`dump`](dump.md) uses it while serializing a `basic_json` value to text.
|
||||
- [`to_cbor`](to_cbor.md), [`to_ubjson`](to_ubjson.md), [`to_bjdata`](to_bjdata.md), and [`to_bson`](to_bson.md) use it
|
||||
while serializing a `basic_json` value to that binary format; none of CBOR, UBJSON, BJData, or BSON requires a
|
||||
decoder to reject ill-formed UTF-8, so by default (`strict`) the library checks on write instead. `to_msgpack` and
|
||||
decoder to reject ill-formed UTF-8, so the library can check on write instead. Their default is `keep`, as no
|
||||
binary writer checked before this parameter was added, or `strict` if
|
||||
[`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled. `to_msgpack` and
|
||||
`to_bon8` do not take this parameter: MessagePack's specification explicitly allows a string to contain ill-formed
|
||||
UTF-8, so `to_msgpack` always passes it through, while BON8 always validates, since UTF-8 lead bytes are structural
|
||||
to that format.
|
||||
|
||||
@@ -6,17 +6,17 @@ static std::vector<std::uint8_t> to_bjdata(const basic_json& j,
|
||||
const bool use_size = false,
|
||||
const bool use_type = false,
|
||||
const bjdata_version_t version = bjdata_version_t::draft2,
|
||||
const error_handler_t error_handler = error_handler_t::strict);
|
||||
const error_handler_t error_handler = error_handler_t::keep);
|
||||
|
||||
// (2)
|
||||
static void to_bjdata(const basic_json& j, detail::output_adapter<std::uint8_t> o,
|
||||
const bool use_size = false, const bool use_type = false,
|
||||
const bjdata_version_t version = bjdata_version_t::draft2,
|
||||
const error_handler_t error_handler = error_handler_t::strict);
|
||||
const error_handler_t error_handler = error_handler_t::keep);
|
||||
static void to_bjdata(const basic_json& j, detail::output_adapter<char> o,
|
||||
const bool use_size = false, const bool use_type = false,
|
||||
const bjdata_version_t version = bjdata_version_t::draft2,
|
||||
const error_handler_t error_handler = error_handler_t::strict);
|
||||
const error_handler_t error_handler = error_handler_t::keep);
|
||||
```
|
||||
|
||||
Serializes a given JSON value `j` to a byte vector using the BJData (Binary JData) serialization format. BJData aims to
|
||||
@@ -48,9 +48,9 @@ optional, `#!cpp bjdata_version_t::draft2` by default.
|
||||
|
||||
`error_handler` (in)
|
||||
: how to treat a string or object key in `j` that is not valid UTF-8; see [`error_handler_t`](error_handler_t.md).
|
||||
The default, `strict`, throws; `keep` writes the ill-formed bytes to the output as is, as every version of
|
||||
`to_bjdata` did before this parameter was added; `replace`/`ignore` sanitize it the same way
|
||||
[`dump`](dump.md) would
|
||||
The default, `keep`, writes the ill-formed bytes to the output as is, as every version of `to_bjdata` did before
|
||||
this parameter was added; `strict` throws; `replace`/`ignore` sanitize it the same way [`dump`](dump.md) would.
|
||||
If [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled, the default is `strict` instead.
|
||||
|
||||
## Return value
|
||||
|
||||
@@ -66,7 +66,8 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
|
||||
- Throws [`other_error.502`](../../home/exceptions.md#jsonexceptionother_error502) if `use_type` is true and `use_size`
|
||||
is false.
|
||||
- Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if a string or object key in `j` is
|
||||
not valid UTF-8 and `error_handler` is `strict` (the default)
|
||||
not valid UTF-8 and `error_handler` is `strict` (the default only if
|
||||
[`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled)
|
||||
|
||||
## Complexity
|
||||
|
||||
@@ -101,5 +102,6 @@ Linear in the size of the JSON value `j`.
|
||||
|
||||
- Added in version 3.11.0.
|
||||
- BJData version parameter (for draft3 binary encoding) added in version 3.12.0.
|
||||
- Throwing `type_error.316` for a string or object key that is not valid UTF-8 added in version 3.13.0.
|
||||
- Added `error_handler` parameter in version 3.13.0.
|
||||
- Added `error_handler` parameter in version 3.13.0. Its default, `keep`, writes the bytes of a string or object key
|
||||
that is not valid UTF-8 unchanged, as before; `strict` (the default if
|
||||
[`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled) throws `type_error.316`.
|
||||
@@ -3,13 +3,13 @@
|
||||
```cpp
|
||||
// (1)
|
||||
static std::vector<std::uint8_t> to_bson(const basic_json& j,
|
||||
const error_handler_t error_handler = error_handler_t::strict);
|
||||
const error_handler_t error_handler = error_handler_t::keep);
|
||||
|
||||
// (2)
|
||||
static void to_bson(const basic_json& j, detail::output_adapter<std::uint8_t> o,
|
||||
const error_handler_t error_handler = error_handler_t::strict);
|
||||
const error_handler_t error_handler = error_handler_t::keep);
|
||||
static void to_bson(const basic_json& j, detail::output_adapter<char> o,
|
||||
const error_handler_t error_handler = error_handler_t::strict);
|
||||
const error_handler_t error_handler = error_handler_t::keep);
|
||||
```
|
||||
|
||||
BSON (Binary JSON) is a binary format in which zero or more ordered key/value pairs are stored as a single entity (a
|
||||
@@ -30,9 +30,9 @@ The exact mapping and its limitations are described on a [dedicated page](../../
|
||||
|
||||
`error_handler` (in)
|
||||
: how to treat a string or object key in `j` that is not valid UTF-8; see [`error_handler_t`](error_handler_t.md).
|
||||
The default, `strict`, throws; `keep` writes the ill-formed bytes to the output as is, as every version of
|
||||
`to_bson` did before this parameter was added; `replace`/`ignore` sanitize it the same way
|
||||
[`dump`](dump.md) would
|
||||
The default, `keep`, writes the ill-formed bytes to the output as is, as every version of `to_bson` did before
|
||||
this parameter was added; `strict` throws; `replace`/`ignore` sanitize it the same way [`dump`](dump.md) would.
|
||||
If [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled, the default is `strict` instead.
|
||||
|
||||
## Return value
|
||||
|
||||
@@ -56,7 +56,8 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
|
||||
exceeds 255, the maximum of the BSON binary subtype; example:
|
||||
`"subtype 70000 is too large for the BSON binary subtype (max 255)"`
|
||||
- Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if a string or object key is
|
||||
not valid UTF-8 and `error_handler` is `strict` (the default)
|
||||
not valid UTF-8 and `error_handler` is `strict` (the default only if
|
||||
[`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled)
|
||||
|
||||
## Complexity
|
||||
|
||||
@@ -93,6 +94,7 @@ pass before anything is written.
|
||||
- Added in version 3.4.0.
|
||||
- Linear in the size of `j`, and no longer limited by the call stack for deeply nested values, since version 3.13.0.
|
||||
- `out_of_range.415` is now detected before anything is written, like the other exceptions above, since version 3.13.0.
|
||||
- Throwing `type_error.316` for a string value or object key that is not valid UTF-8, detected before anything is
|
||||
written, added in version 3.13.0.
|
||||
- Added `error_handler` parameter in version 3.13.0.
|
||||
- Added `error_handler` parameter in version 3.13.0. Its default, `keep`, writes the bytes of a string or object key
|
||||
that is not valid UTF-8 unchanged, as before; `strict` (the default if
|
||||
[`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled) throws `type_error.316` before anything
|
||||
is written.
|
||||
|
||||
@@ -3,13 +3,13 @@
|
||||
```cpp
|
||||
// (1)
|
||||
static std::vector<std::uint8_t> to_cbor(const basic_json& j,
|
||||
const error_handler_t error_handler = error_handler_t::strict);
|
||||
const error_handler_t error_handler = error_handler_t::keep);
|
||||
|
||||
// (2)
|
||||
static void to_cbor(const basic_json& j, detail::output_adapter<std::uint8_t> o,
|
||||
const error_handler_t error_handler = error_handler_t::strict);
|
||||
const error_handler_t error_handler = error_handler_t::keep);
|
||||
static void to_cbor(const basic_json& j, detail::output_adapter<char> o,
|
||||
const error_handler_t error_handler = error_handler_t::strict);
|
||||
const error_handler_t error_handler = error_handler_t::keep);
|
||||
```
|
||||
|
||||
Serializes a given JSON value `j` to a byte vector using the CBOR (Concise Binary Object Representation) serialization
|
||||
@@ -31,9 +31,9 @@ The exact mapping and its limitations are described on a [dedicated page](../../
|
||||
|
||||
`error_handler` (in)
|
||||
: how to treat a string or object key in `j` that is not valid UTF-8; see [`error_handler_t`](error_handler_t.md).
|
||||
The default, `strict`, throws; `keep` writes the ill-formed bytes to the output as is, as every version of
|
||||
`to_cbor` did before this parameter was added; `replace`/`ignore` sanitize it the same way
|
||||
[`dump`](dump.md) would
|
||||
The default, `keep`, writes the ill-formed bytes to the output as is, as every version of `to_cbor` did before
|
||||
this parameter was added; `strict` throws; `replace`/`ignore` sanitize it the same way [`dump`](dump.md) would.
|
||||
If [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled, the default is `strict` instead.
|
||||
|
||||
## Return value
|
||||
|
||||
@@ -47,7 +47,8 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
|
||||
## Exceptions
|
||||
|
||||
- Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if a string or object key in `j` is
|
||||
not valid UTF-8 and `error_handler` is `strict` (the default)
|
||||
not valid UTF-8 and `error_handler` is `strict` (the default only if
|
||||
[`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled)
|
||||
|
||||
## Complexity
|
||||
|
||||
@@ -82,5 +83,6 @@ Linear in the size of the JSON value `j`.
|
||||
|
||||
- Added in version 2.0.9.
|
||||
- Compact representation of floating-point numbers added in version 3.8.0.
|
||||
- Throwing `type_error.316` for a string or object key that is not valid UTF-8 added in version 3.13.0.
|
||||
- Added `error_handler` parameter in version 3.13.0.
|
||||
- Added `error_handler` parameter in version 3.13.0. Its default, `keep`, writes the bytes of a string or object key
|
||||
that is not valid UTF-8 unchanged, as before; `strict` (the default if
|
||||
[`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled) throws `type_error.316`.
|
||||
|
||||
@@ -5,15 +5,15 @@
|
||||
static std::vector<std::uint8_t> to_ubjson(const basic_json& j,
|
||||
const bool use_size = false,
|
||||
const bool use_type = false,
|
||||
const error_handler_t error_handler = error_handler_t::strict);
|
||||
const error_handler_t error_handler = error_handler_t::keep);
|
||||
|
||||
// (2)
|
||||
static void to_ubjson(const basic_json& j, detail::output_adapter<std::uint8_t> o,
|
||||
const bool use_size = false, const bool use_type = false,
|
||||
const error_handler_t error_handler = error_handler_t::strict);
|
||||
const error_handler_t error_handler = error_handler_t::keep);
|
||||
static void to_ubjson(const basic_json& j, detail::output_adapter<char> o,
|
||||
const bool use_size = false, const bool use_type = false,
|
||||
const error_handler_t error_handler = error_handler_t::strict);
|
||||
const error_handler_t error_handler = error_handler_t::keep);
|
||||
```
|
||||
|
||||
Serializes a given JSON value `j` to a byte vector using the UBJSON (Universal Binary JSON) serialization format. UBJSON
|
||||
@@ -41,9 +41,9 @@ The exact mapping and its limitations are described on a [dedicated page](../../
|
||||
|
||||
`error_handler` (in)
|
||||
: how to treat a string or object key in `j` that is not valid UTF-8; see [`error_handler_t`](error_handler_t.md).
|
||||
The default, `strict`, throws; `keep` writes the ill-formed bytes to the output as is, as every version of
|
||||
`to_ubjson` did before this parameter was added; `replace`/`ignore` sanitize it the same way
|
||||
[`dump`](dump.md) would
|
||||
The default, `keep`, writes the ill-formed bytes to the output as is, as every version of `to_ubjson` did before
|
||||
this parameter was added; `strict` throws; `replace`/`ignore` sanitize it the same way [`dump`](dump.md) would.
|
||||
If [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled, the default is `strict` instead.
|
||||
|
||||
## Return value
|
||||
|
||||
@@ -59,7 +59,8 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
|
||||
- Throws [`other_error.502`](../../home/exceptions.md#jsonexceptionother_error502) if `use_type` is true and `use_size`
|
||||
is false.
|
||||
- Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if a string or object key in `j` is
|
||||
not valid UTF-8 and `error_handler` is `strict` (the default)
|
||||
not valid UTF-8 and `error_handler` is `strict` (the default only if
|
||||
[`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled)
|
||||
|
||||
## Complexity
|
||||
|
||||
@@ -93,5 +94,6 @@ Linear in the size of the JSON value `j`.
|
||||
## Version history
|
||||
|
||||
- Added in version 3.1.0.
|
||||
- Throwing `type_error.316` for a string or object key that is not valid UTF-8 added in version 3.13.0.
|
||||
- Added `error_handler` parameter in version 3.13.0.
|
||||
- Added `error_handler` parameter in version 3.13.0. Its default, `keep`, writes the bytes of a string or object key
|
||||
that is not valid UTF-8 unchanged, as before; `strict` (the default if
|
||||
[`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled) throws `type_error.316`.
|
||||
|
||||
@@ -18,6 +18,8 @@ header. See also the [macro overview page](../../features/macros.md).
|
||||
|
||||
- [**JSON_PRECISE_STREAM_POSITION**](json_precise_stream_position.md) - opt in to leaving an input stream positioned
|
||||
right after a parsed number
|
||||
- [**JSON_STRICT_BINARY_UTF8**](json_strict_binary_utf8.md) - opt in to checking strings for valid UTF-8 in the CBOR,
|
||||
UBJSON, BJData, and BSON writers
|
||||
- [**JSON_STRICT_NUL_HANDLING**](json_strict_nul_handling.md) - opt in to rejecting a NUL byte in the input instead of
|
||||
treating it as end of input
|
||||
|
||||
|
||||
@@ -0,0 +1,101 @@
|
||||
# JSON_STRICT_BINARY_UTF8
|
||||
|
||||
```cpp
|
||||
#define JSON_STRICT_BINARY_UTF8 /* value */
|
||||
```
|
||||
|
||||
When defined to `1`, the `error_handler` parameter of the binary writers [`to_cbor`](../basic_json/to_cbor.md),
|
||||
[`to_ubjson`](../basic_json/to_ubjson.md), [`to_bjdata`](../basic_json/to_bjdata.md), and
|
||||
[`to_bson`](../basic_json/to_bson.md) defaults to [`error_handler_t::strict`](../basic_json/error_handler_t.md) instead
|
||||
of `error_handler_t::keep`. These writers then check every string value and object key for valid UTF-8 and throw
|
||||
[`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for ill-formed UTF-8, like
|
||||
[`dump`](../basic_json/dump.md) does. Without it, they write the bytes unchanged. An `error_handler` passed explicitly
|
||||
always takes precedence.
|
||||
|
||||
The macro does not affect:
|
||||
|
||||
- [`to_msgpack`](../basic_json/to_msgpack.md): the MessagePack specification allows a `str` value to contain bytes that
|
||||
are not valid UTF-8, so it always writes them unchanged.
|
||||
- [`to_bon8`](../basic_json/to_bon8.md): BON8 always checks, because the UTF-8 lead bytes mark where a string ends.
|
||||
- The binary readers ([`from_cbor`](../basic_json/from_cbor.md), [`from_msgpack`](../basic_json/from_msgpack.md),
|
||||
[`from_ubjson`](../basic_json/from_ubjson.md), [`from_bjdata`](../basic_json/from_bjdata.md),
|
||||
[`from_bson`](../basic_json/from_bson.md)): none of these formats requires a decoder to reject ill-formed UTF-8, so
|
||||
they always return the bytes unchanged.
|
||||
|
||||
## Default definition
|
||||
|
||||
The default value is `0` (disabled, the behavior of version 3.12.0 and earlier is preserved).
|
||||
|
||||
```cpp
|
||||
#define JSON_STRICT_BINARY_UTF8 0
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
!!! note "Background"
|
||||
|
||||
CBOR, UBJSON, BJData, and BSON all require strings to be UTF-8. Up to version 3.12.0, the writers did not check
|
||||
this, so they could produce output that other decoders reject. Checking by default would break code that stores
|
||||
other encodings (for instance ISO 8859-1) in a string and only ever writes it to a binary format. You can pass
|
||||
`error_handler_t::strict` to each call, or use this macro to check by default ahead of version 4.0.0, where
|
||||
`strict` is planned to become the default (see
|
||||
[#5529](https://github.com/nlohmann/json/issues/5529) and [#5651](https://github.com/nlohmann/json/issues/5651)).
|
||||
|
||||
!!! warning "Opt-in only"
|
||||
|
||||
This macro must be defined **before** including `<nlohmann/json.hpp>`. Defining it after the include has no
|
||||
effect.
|
||||
|
||||
!!! note "ABI compatibility"
|
||||
|
||||
The value of this macro is encoded in the [namespace](../../features/namespace.md) (tag `_sbu8`), resulting in
|
||||
distinct symbol names. Translation units compiled with and without it can therefore be linked into the same program
|
||||
without One Definition Rule (ODR) violations, but they cannot exchange instances of library types.
|
||||
|
||||
## Examples
|
||||
|
||||
??? example "Default behavior (macro not defined)"
|
||||
|
||||
Without the macro, the bytes are written unchanged:
|
||||
|
||||
```cpp
|
||||
#include <nlohmann/json.hpp>
|
||||
|
||||
using json = nlohmann::json;
|
||||
|
||||
int main()
|
||||
{
|
||||
auto v = json::to_cbor(json("\xFF"));
|
||||
// v is {0x61, 0xFF}
|
||||
}
|
||||
```
|
||||
|
||||
??? example "Opt-in check (macro defined to 1)"
|
||||
|
||||
With the macro, ill-formed UTF-8 is rejected:
|
||||
|
||||
```cpp
|
||||
#define JSON_STRICT_BINARY_UTF8 1
|
||||
#include <nlohmann/json.hpp>
|
||||
|
||||
using json = nlohmann::json;
|
||||
|
||||
int main()
|
||||
{
|
||||
auto v = json::to_cbor(json("\xFF"));
|
||||
// throws type_error.316: invalid UTF-8 byte at index 0: 0xFF
|
||||
}
|
||||
```
|
||||
|
||||
## See also
|
||||
|
||||
- [**to_cbor**](../basic_json/to_cbor.md) - create a CBOR serialization of a JSON value
|
||||
- [**to_ubjson**](../basic_json/to_ubjson.md) - create a UBJSON serialization of a JSON value
|
||||
- [**to_bjdata**](../basic_json/to_bjdata.md) - create a BJData serialization of a JSON value
|
||||
- [**to_bson**](../basic_json/to_bson.md) - create a BSON serialization of a JSON value
|
||||
- [**error_handler_t**](../basic_json/error_handler_t.md) - how [`dump`](../basic_json/dump.md) treats ill-formed UTF-8
|
||||
|
||||
## Version history
|
||||
|
||||
- Added in version 3.13.0.
|
||||
- Planned to become the default (with the macro removed) in version 4.0.0.
|
||||
@@ -65,9 +65,12 @@ The library uses the following mapping from JSON values types to BJData types ac
|
||||
|
||||
!!! warning "UTF-8 validation of string values and object keys"
|
||||
|
||||
BJData strings must use UTF-8 encoding. `to_bjdata()` validates the bytes of every string value and object key
|
||||
and throws [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for ill-formed UTF-8, so a
|
||||
value with such a string cannot be serialized in the first place.
|
||||
BJData strings must use UTF-8 encoding. By default (the [`error_handler`](../../api/basic_json/to_bjdata.md)
|
||||
parameter left at `keep`), `to_bjdata()` writes the bytes of string values and object keys unchanged, even if they
|
||||
are not valid UTF-8. With `error_handler_t::strict`, it throws
|
||||
[`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for ill-formed UTF-8 instead;
|
||||
`replace`/`ignore` sanitize the string. [`JSON_STRICT_BINARY_UTF8`](../../api/macros/json_strict_binary_utf8.md)
|
||||
makes `strict` the default.
|
||||
|
||||
!!! info "Unused BJData markers"
|
||||
|
||||
@@ -225,8 +228,7 @@ The library maps BJData types to JSON value types as follows:
|
||||
[`dump()`](../../api/basic_json/dump.md) still requires valid UTF-8 and throws
|
||||
[`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for a value read with the default
|
||||
`keep` handler, unless an error handler is passed that replaces or ignores the ill-formed bytes. `to_bjdata()`'s
|
||||
own `error_handler` parameter defaults to `strict` (see above), so a value read this way cannot be written back
|
||||
to BJData unless a non-strict handler is passed there too.
|
||||
own `error_handler` parameter defaults to `keep` (see above), so such a value is written back unchanged.
|
||||
|
||||
!!! info "Round trips"
|
||||
|
||||
|
||||
@@ -117,14 +117,13 @@ The library maps BSON record types to JSON value types as follows:
|
||||
unchanged. Passing `error_handler_t::strict` makes `from_bson()` check and throw
|
||||
[`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113) for ill-formed UTF-8, and
|
||||
`replace`/`ignore` sanitize the string instead of keeping it. However, [`dump()`](../../api/basic_json/dump.md)
|
||||
still requires valid UTF-8 and throws [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316)
|
||||
for a value read with the default `keep` handler, unless an error handler is passed that replaces or ignores
|
||||
the ill-formed bytes. `to_bson()`'s own `error_handler` parameter defaults to `strict` and throws the same
|
||||
exception for a string value or element (key) name that is not valid UTF-8, so an object with such a key or
|
||||
value cannot be produced in the first place unless a non-strict handler is passed there, even though
|
||||
`from_bson()` would accept it from another source with the default `keep` handler. Element (key) names are
|
||||
never validated on read, since they are read byte-by-byte as a C string. `binary` values (type `0x05`) are
|
||||
unaffected, since they are not required to hold text.
|
||||
still requires valid UTF-8 and throws [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for a
|
||||
value read with the default `keep` handler, unless an error handler is passed that replaces or ignores the
|
||||
ill-formed bytes. `to_bson()`'s own `error_handler` parameter defaults to `keep`, so such a string value or element
|
||||
(key) name is written unchanged; with `strict` (the default if
|
||||
[`JSON_STRICT_BINARY_UTF8`](../../api/macros/json_strict_binary_utf8.md) is enabled), it throws the same exception
|
||||
instead. Element (key) names are never validated on read, since they are read byte-by-byte as a C string. `binary`
|
||||
values (type `0x05`) are unaffected, since they are not required to hold text.
|
||||
|
||||
??? example
|
||||
|
||||
|
||||
@@ -191,19 +191,19 @@ The library maps CBOR types to JSON value types as follows:
|
||||
|
||||
!!! warning "Ill-formed UTF-8 in text strings"
|
||||
|
||||
[RFC 8949, Section 3.1](https://www.rfc-editor.org/rfc/rfc8949.html#section-3.1) requires CBOR text strings
|
||||
(major type 3) to be valid UTF-8, but leaves it up to the decoder whether to enforce this, so checking is
|
||||
opt-in: with the [`error_handler`](../../api/basic_json/from_cbor.md) parameter left at `keep` (the default),
|
||||
`from_cbor()` accepts a text string (object keys included) whose bytes are not valid UTF-8 and hands them back
|
||||
unchanged. Passing `error_handler_t::strict` makes `from_cbor()` check and throw
|
||||
[RFC 8949, Section 3.1](https://www.rfc-editor.org/rfc/rfc8949.html#section-3.1) requires CBOR text strings (major
|
||||
type 3) to be valid UTF-8, but leaves it up to the decoder whether to enforce this, so checking is opt-in: with the
|
||||
[`error_handler`](../../api/basic_json/from_cbor.md) parameter left at `keep` (the default), `from_cbor()` accepts a
|
||||
text string (object keys included) whose bytes are not valid UTF-8 and hands them back unchanged. Passing
|
||||
`error_handler_t::strict` makes `from_cbor()` check and throw
|
||||
[`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113) for ill-formed UTF-8, and
|
||||
`replace`/`ignore` sanitize the string instead of keeping it. However, [`dump()`](../../api/basic_json/dump.md)
|
||||
still requires valid UTF-8 and throws [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316)
|
||||
for a value read with the default `keep` handler, unless an error handler is passed that replaces or ignores
|
||||
the ill-formed bytes. `to_cbor()`'s own [`error_handler`](../../api/basic_json/to_cbor.md) parameter defaults
|
||||
to `strict` and throws the same exception for a string value or object key that is not valid UTF-8, so such a
|
||||
value cannot be written back to CBOR unless a non-strict handler is passed there too. Byte strings (major
|
||||
type 2) are unaffected, since they are not required to hold text.
|
||||
still requires valid UTF-8 and throws [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for a
|
||||
value read with the default `keep` handler, unless an error handler is passed that replaces or ignores the
|
||||
ill-formed bytes. `to_cbor()`'s own [`error_handler`](../../api/basic_json/to_cbor.md) parameter defaults to `keep`,
|
||||
so such a value is written back unchanged; with `strict` (the default if
|
||||
[`JSON_STRICT_BINARY_UTF8`](../../api/macros/json_strict_binary_utf8.md) is enabled), it throws the same exception
|
||||
instead. Byte strings (major type 2) are unaffected, since they are not required to hold text.
|
||||
|
||||
!!! warning "Tagged items"
|
||||
|
||||
|
||||
@@ -49,9 +49,12 @@ The library uses the following mapping from JSON values types to UBJSON types ac
|
||||
|
||||
!!! warning "UTF-8 validation of string values and object keys"
|
||||
|
||||
UBJSON's required string encoding is UTF-8. `to_ubjson()` validates the bytes of every string value and object
|
||||
key and throws [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for ill-formed UTF-8, so
|
||||
a value with such a string cannot be serialized in the first place.
|
||||
UBJSON's required string encoding is UTF-8. By default (the [`error_handler`](../../api/basic_json/to_ubjson.md)
|
||||
parameter left at `keep`), `to_ubjson()` writes the bytes of string values and object keys unchanged, even if they
|
||||
are not valid UTF-8. With `error_handler_t::strict`, it throws
|
||||
[`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for ill-formed UTF-8 instead;
|
||||
`replace`/`ignore` sanitize the string. [`JSON_STRICT_BINARY_UTF8`](../../api/macros/json_strict_binary_utf8.md)
|
||||
makes `strict` the default.
|
||||
|
||||
!!! info "Unused UBJSON markers"
|
||||
|
||||
@@ -137,8 +140,7 @@ The library maps UBJSON types to JSON value types as follows:
|
||||
[`dump()`](../../api/basic_json/dump.md) still requires valid UTF-8 and throws
|
||||
[`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for a value read with the default
|
||||
`keep` handler, unless an error handler is passed that replaces or ignores the ill-formed bytes. `to_ubjson()`'s
|
||||
own `error_handler` parameter defaults to `strict` (see above), so a value read this way cannot be written back
|
||||
to UBJSON unless a non-strict handler is passed there too.
|
||||
own `error_handler` parameter defaults to `keep` (see above), so such a value is written back unchanged.
|
||||
|
||||
??? example
|
||||
|
||||
|
||||
@@ -138,6 +138,20 @@ using the library with compilers that do not fully support C++11 and may only wo
|
||||
|
||||
See [full documentation of `JSON_SKIP_UNSUPPORTED_COMPILER_CHECK`](../api/macros/json_skip_unsupported_compiler_check.md).
|
||||
|
||||
## `JSON_STRICT_BINARY_UTF8`
|
||||
|
||||
When defined to `1`, [`to_cbor`](../api/basic_json/to_cbor.md), [`to_ubjson`](../api/basic_json/to_ubjson.md),
|
||||
[`to_bjdata`](../api/basic_json/to_bjdata.md), and [`to_bson`](../api/basic_json/to_bson.md) throw
|
||||
[`type_error.316`](../home/exceptions.md#jsonexceptiontype_error316) for a string value or object key that is not
|
||||
valid UTF-8. The default value is `0`, which writes the bytes unchanged as before version 3.13.0; this is planned to
|
||||
become the default in version 4.0.0.
|
||||
|
||||
The check can also be enabled with the CMake option
|
||||
[`JSON_StrictBinaryUTF8`](../integration/cmake.md#json_strictbinaryutf8) (`OFF` by default) which sets
|
||||
`JSON_STRICT_BINARY_UTF8` accordingly.
|
||||
|
||||
See [full documentation of `JSON_STRICT_BINARY_UTF8`](../api/macros/json_strict_binary_utf8.md).
|
||||
|
||||
## `JSON_STRICT_NUL_HANDLING`
|
||||
|
||||
When defined to `1`, a `'\0'` (NUL) byte anywhere in the input is rejected with `parse_error.101`, like any other
|
||||
|
||||
@@ -20,6 +20,7 @@ The complete default namespace name is derived as follows:
|
||||
`_bics`.
|
||||
- [`JSON_PRECISE_STREAM_POSITION`](../api/macros/json_precise_stream_position.md) defined non-zero appends `_psp`.
|
||||
- [`JSON_STRICT_NUL_HANDLING`](../api/macros/json_strict_nul_handling.md) defined non-zero appends `_snul`.
|
||||
- [`JSON_STRICT_BINARY_UTF8`](../api/macros/json_strict_binary_utf8.md) defined non-zero appends `_sbu8`.
|
||||
- The inline namespace ends with the suffix `_v` followed by the 3 components of the version number separated by
|
||||
underscores. To omit the version component, see [Disabling the version component](#disabling-the-version-component)
|
||||
below.
|
||||
|
||||
@@ -752,6 +752,11 @@ The `unflatten()` function only works for an object whose keys are JSON Pointers
|
||||
|
||||
The `dump()` function only works with UTF-8 encoded strings; that is, if you assign a `std::string` to a JSON value, make sure it is UTF-8 encoded.
|
||||
|
||||
The binary writers [`to_cbor()`](../api/basic_json/to_cbor.md), [`to_ubjson()`](../api/basic_json/to_ubjson.md),
|
||||
[`to_bjdata()`](../api/basic_json/to_bjdata.md), and [`to_bson()`](../api/basic_json/to_bson.md) throw this exception
|
||||
as well for a string value or object key that is not valid UTF-8 if their `error_handler` is `strict` (the default if
|
||||
[`JSON_STRICT_BINARY_UTF8`](../api/macros/json_strict_binary_utf8.md) is enabled).
|
||||
|
||||
!!! failure "Example message"
|
||||
|
||||
Calling `dump()` on a JSON value containing an ISO 8859-1 encoded string:
|
||||
|
||||
@@ -204,6 +204,11 @@ Use the non-amalgamated version of the library. This option is `ON` by default.
|
||||
|
||||
Treat the library headers like system headers (i.e., adding `SYSTEM` to the [`target_include_directories`](https://cmake.org/cmake/help/latest/command/target_include_directories.html) call) to check for this library by tools like Clang-Tidy. This option is `OFF` by default.
|
||||
|
||||
### `JSON_StrictBinaryUTF8`
|
||||
|
||||
Check string values and object keys for valid UTF-8 in the CBOR, UBJSON, BJData, and BSON writers, by defining the
|
||||
macro [`JSON_STRICT_BINARY_UTF8`](../api/macros/json_strict_binary_utf8.md). This option is `OFF` by default.
|
||||
|
||||
### `JSON_StrictNulHandling`
|
||||
|
||||
Reject a `'\0'` (NUL) byte in the input instead of treating it as end of input, by defining the macro
|
||||
|
||||
@@ -301,6 +301,7 @@ nav:
|
||||
- 'JSON_PRECISE_STREAM_POSITION': api/macros/json_precise_stream_position.md
|
||||
- 'JSON_SKIP_LIBRARY_VERSION_CHECK': api/macros/json_skip_library_version_check.md
|
||||
- 'JSON_SKIP_UNSUPPORTED_COMPILER_CHECK': api/macros/json_skip_unsupported_compiler_check.md
|
||||
- 'JSON_STRICT_BINARY_UTF8': api/macros/json_strict_binary_utf8.md
|
||||
- 'JSON_STRICT_NUL_HANDLING': api/macros/json_strict_nul_handling.md
|
||||
- 'JSON_USE_GLOBAL_UDLS': api/macros/json_use_global_udls.md
|
||||
- 'JSON_USE_IMPLICIT_CONVERSIONS': api/macros/json_use_implicit_conversions.md
|
||||
|
||||
Reference in New Issue
Block a user