Merges claude/binary-utf8-roundtrip-5651, where the writers now only check
UTF-8 if JSON_STRICT_BINARY_UTF8 is enabled, and applies the same rule to
the error_handler parameter added here: to_cbor(), to_ubjson(),
to_bjdata(), and to_bson() default to error_handler_t::keep, the behavior
of 3.12.0, or to error_handler_t::strict if JSON_STRICT_BINARY_UTF8 is
enabled (detail::binary_writer_default_error_handler()). Passing a handler
explicitly always takes precedence.
- Conflicts in binary_writer.hpp resolved in favor of this branch's
error_handler, which replaces #5741's check_text_utf8().
- Tests: the "default parameters" section checks that the default equals
keep; unit-binary_utf8_strict.cpp checks that an explicit handler
overrides the macro's default.
- Docs: signatures show keep; parameter, exception, and format pages
describe keep as the default and the macro; the two 3.13.0 version
history entries of each writer are merged into one.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Adds error_handler_t::keep (invalid UTF-8 sequences are left unchanged) as
a fourth error_handler_t value, and threads an error_handler parameter
through the binary writers and readers:
- to_cbor/to_ubjson/to_bjdata/to_bson gain a trailing error_handler
parameter (default strict, matching their existing type_error.316
behavior); keep writes a string value or object key's bytes as is
instead of throwing, and replace/ignore sanitize it exactly like
dump() would, including for the BSON length prefix. to_msgpack and
to_bon8 are unchanged.
- from_cbor/from_msgpack/from_ubjson/from_bjdata/from_bson gain a
trailing error_handler parameter (default keep, i.e. the lenient
behavior every binary reader had in 3.12.0 and still has after
#5741); strict checks every string value and object key and raises
parse_error.113 for ill-formed UTF-8, honoring allow_exceptions;
replace/ignore sanitize it like dump() would. from_bon8 is
unchanged, since UTF-8 lead bytes are structural there.
dump()'s own keep support writes ill-formed bytes as is, even with
ensure_ascii, while still \u-escaping well-formed characters around
them as usual.
The UTF-8 validity check (is_valid_utf8) and the replace/ignore
sanitizing logic (sanitize_utf8) now live in string_utils.hpp, shared
by the serializer and the binary reader/writer; error_handler_t itself
moved to its own header (detail/output/error_handler.hpp) so that
string_utils.hpp does not need to depend on serializer.hpp.
See #5529 and #5741.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Niels Lohmann