Add an error_handler parameter to to_msgpack

The MessagePack spec allows ill-formed UTF-8 in a str, so the default stays
keep (also if JSON_STRICT_BINARY_UTF8 is enabled), but decoders such as
msgpack-python reject it by default. With the parameter, strict catches
such strings and replace/ignore sanitize them, the same way as for the
other binary writers and dump().

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/error_handler_t.md

@@ -12,14 +12,13 @@ enum class error_handler_t {
 This enumeration is used to choose how to treat ill-formed UTF-8 in a string value or object key:
 
 - [`dump`](dump.md) uses it while serializing a `basic_json` value to text.
-- [`to_cbor`](to_cbor.md), [`to_ubjson`](to_ubjson.md), [`to_bjdata`](to_bjdata.md), and [`to_bson`](to_bson.md) use it
-  while serializing a `basic_json` value to that binary format; none of CBOR, UBJSON, BJData, or BSON requires a
-  decoder to reject ill-formed UTF-8, so the library can check on write instead. Their default is `keep`, as no
-  binary writer checked before this parameter was added, or `strict` if
-  [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled. `to_msgpack` and
-  `to_bon8` do not take this parameter: MessagePack's specification explicitly allows a string to contain ill-formed
-  UTF-8, so `to_msgpack` always passes it through, while BON8 always validates, since UTF-8 lead bytes are structural
-  to that format.
+- [`to_cbor`](to_cbor.md), [`to_msgpack`](to_msgpack.md), [`to_ubjson`](to_ubjson.md), [`to_bjdata`](to_bjdata.md),
+  and [`to_bson`](to_bson.md) use it while serializing a `basic_json` value to that binary format. Their default is
+  `keep`, as no binary writer checked before this parameter was added. CBOR, UBJSON, BJData, and BSON require valid
+  UTF-8, so for these four the default is `strict` if [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md)
+  is enabled; MessagePack's specification explicitly allows a string to contain ill-formed UTF-8, so `to_msgpack`
+  stays at `keep`. `to_bon8` does not take this parameter: BON8 always validates, since UTF-8 lead bytes are
+  structural to that format.
 - [`from_cbor`](from_cbor.md), [`from_msgpack`](from_msgpack.md), [`from_ubjson`](from_ubjson.md),
   [`from_bjdata`](from_bjdata.md), and [`from_bson`](from_bson.md) use it while parsing that binary format, to decide
   whether to check a string value or object key for well-formed UTF-8 at all; by default (`keep`) they do not, as no

docs/mkdocs/docs/api/basic_json/to_msgpack.md

@@ -2,11 +2,14 @@
 
 ```cpp
 // (1)
-static std::vector<std::uint8_t> to_msgpack(const basic_json& j);
+static std::vector<std::uint8_t> to_msgpack(const basic_json& j,
+                                            const error_handler_t error_handler = error_handler_t::keep);
 
 // (2)
-static void to_msgpack(const basic_json& j, detail::output_adapter<std::uint8_t> o);
-static void to_msgpack(const basic_json& j, detail::output_adapter<char> o);
+static void to_msgpack(const basic_json& j, detail::output_adapter<std::uint8_t> o,
+                       const error_handler_t error_handler = error_handler_t::keep);
+static void to_msgpack(const basic_json& j, detail::output_adapter<char> o,
+                       const error_handler_t error_handler = error_handler_t::keep);
 ```
 
 Serializes a given JSON value `j` to a byte vector using the MessagePack serialization format. MessagePack is a binary
@@ -25,6 +28,13 @@ The exact mapping and its limitations are described on a [dedicated page](../../
 `o` (in)
 :   output adapter to write serialization to
 
+`error_handler` (in)
+:   how to treat a string or object key in `j` that is not valid UTF-8; see [`error_handler_t`](error_handler_t.md).
+    The default, `keep`, writes the ill-formed bytes to the output as is, as every version of `to_msgpack` did before
+    this parameter was added and as the MessagePack specification allows; `strict` throws; `replace`/`ignore` sanitize
+    it the same way [`dump`](dump.md) would. Unlike the other binary writers, the default stays `keep` even if
+    [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled.
+
 ## Return value
 
 1. MessagePack serialization as a byte vector
@@ -42,6 +52,8 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
 - Throws [`out_of_range.415`](../../home/exceptions.md#jsonexceptionout_of_range415) if the subtype of a binary value
   exceeds 255, the maximum of the MessagePack ext type; example:
   `"subtype 70000 is too large for the MessagePack ext type (max 255)"`
+- Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if a string or object key in `j` is
+  not valid UTF-8 and `error_handler` is `strict`
 
 ## Complexity
 
@@ -76,6 +88,8 @@ Linear in the size of the JSON value `j`.
 
 - Added in version 2.0.9.
 - Throws `out_of_range.412` and `out_of_range.415` since version 3.13.0.
+- Added `error_handler` parameter in version 3.13.0. Its default, `keep`, writes the bytes of a string or object key
+  that is not valid UTF-8 unchanged, as before.
 - Fixed in version 3.13.0 to serialize `number_integer_t`/`number_unsigned_t` pairs of different width correctly;
   before, integers could be serialized with the wrong value if `number_integer_t` was narrower than
   `number_unsigned_t`.

docs/mkdocs/docs/api/macros/json_strict_binary_utf8.md

@@ -15,7 +15,7 @@ always takes precedence.
 The macro does not affect:
 
 - [`to_msgpack`](../basic_json/to_msgpack.md): the MessagePack specification allows a `str` value to contain bytes that
-  are not valid UTF-8, so it always writes them unchanged.
+  are not valid UTF-8, so its `error_handler` always defaults to `keep`.
 - [`to_bon8`](../basic_json/to_bon8.md): BON8 always checks, because the UTF-8 lead bytes mark where a string ends.
 - The binary readers ([`from_cbor`](../basic_json/from_cbor.md), [`from_msgpack`](../basic_json/from_msgpack.md),
   [`from_ubjson`](../basic_json/from_ubjson.md), [`from_bjdata`](../basic_json/from_bjdata.md),

docs/mkdocs/docs/features/binary_formats/messagepack.md

@@ -163,8 +163,10 @@ The library maps MessagePack types to JSON value types as follows:
     round-trips through `from_msgpack(to_msgpack(j))` byte for byte. Passing `error_handler_t::strict` makes
     `from_msgpack()` check anyway and throw
     [`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113) for ill-formed UTF-8, and
-    `replace`/`ignore` sanitize the string instead of keeping it. `to_msgpack()` itself has no `error_handler`
-    parameter and always writes `str` bytes as-is, since the specification permits it. However,
+    `replace`/`ignore` sanitize the string instead of keeping it. `to_msgpack()` also writes `str` bytes as-is by
+    default, since the specification permits it; its [`error_handler`](../../api/basic_json/to_msgpack.md) parameter
+    can be set to `strict` to throw [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) instead, or
+    to `replace`/`ignore` to sanitize the string, for instance for a decoder that rejects ill-formed UTF-8. However,
     [`dump()`](../../api/basic_json/dump.md) still requires valid UTF-8 and throws
     [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for a value read this way with the
     default `keep` handler, unless an error handler is passed that replaces or ignores the ill-formed bytes.

docs/mkdocs/docs/home/exceptions.md

@@ -755,7 +755,8 @@ The `dump()` function only works with UTF-8 encoded strings; that is, if you ass
 The binary writers [`to_cbor()`](../api/basic_json/to_cbor.md), [`to_ubjson()`](../api/basic_json/to_ubjson.md),
 [`to_bjdata()`](../api/basic_json/to_bjdata.md), and [`to_bson()`](../api/basic_json/to_bson.md) throw this exception
 as well for a string value or object key that is not valid UTF-8 if their `error_handler` is `strict` (the default if
-[`JSON_STRICT_BINARY_UTF8`](../api/macros/json_strict_binary_utf8.md) is enabled).
+[`JSON_STRICT_BINARY_UTF8`](../api/macros/json_strict_binary_utf8.md) is enabled). So does
+[`to_msgpack()`](../api/basic_json/to_msgpack.md) if `error_handler_t::strict` is passed.
 
 !!! failure "Example message"