Files
json/docs/mkdocs/docs/api/basic_json/to_ubjson.md
T
Niels Lohmann 73aea7ce54 Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget (#6083)
* Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget

The whole-input budget for the elements of optimized UBJSON arrays of
type Z, T and F (1048576 plus 4 per byte read) rejects files written by
3.12.0 that have long arrays of these types, such as 1100 rows of 1000
nulls with a `$Z` annotation, and there was no way to read them. The
per-byte rate is now the macro JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE.

Semantics:
- default 4, so the default build is unchanged (the generated code of
  from_ubjson is identical at -O2 up to source line numbers);
- the value must be a positive integer: 0, negative, non-integral, bool,
  character and wider-than-uintmax_t values fail to compile with a
  static_assert that names the macro;
- a value that does not fit into std::size_t is the largest size_t, so
  one spelling works for 32- and 64-bit targets;
- the limit base + rate * bytes_read saturates at the largest size_t
  instead of wrapping around, for any rate and input size; a rate of
  131072 (2^20 / 8) or more cannot be exceeded by any input (the
  shortest header of an array of more than 32767 elements is 8 bytes),
  which switches the whole-input budget off;
- the limit of one array (1048576) and the base (1048576) stay fixed
  constants, so the writer fallback still follows the base only and a
  reader with any rate reads everything the writer produces;
- the macro is not part of the ABI tag and must have the same value in
  every translation unit; it is documented that way.

The memory the arrays can take in a DOM is about 20 bytes per rate and
byte of input (measured, macOS, containers of 9 * rate elements:
25 bytes per input byte at rate 1, 81 at 4, 312 at 16, 1226 at 64, 5493
at 256) plus about 33 MB for the base. A 3.12.0 row of N values has a
header of b bytes (5 up to 255, 6 up to 32767, 8 above), so a table is
read completely if N <= rate * b, and otherwise up to
(1048576 + h * rate, h = outer header: 6, 7 or 9 bytes) / (N - rate * b) rows; 1100 rows of 1000 nulls need
a rate of 8. The new page of the macro documents the formulas, the
measurements, and two examples that are run by the documentation checks.

The tests of the budget move to tests/src/unit-ubjson_valueless.cpp, which
derives every boundary from the macro and is built with the rates 1,
1000000 and 0xFFFFFFFFFFFFFFFF in addition to the default (the last two
check that nothing is refused and that the limit does not wrap around);
they fail with the unchanged header for rate 1, 1000000 and the maximum.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Refuse char8_t for JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE and clarify its documentation

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-11-11 22:37:35 +01:00

5.7 KiB

nlohmann::basic_json::to_ubjson

// (1)
static std::vector<std::uint8_t> to_ubjson(const basic_json& j,
                                           const bool use_size = false,
                                           const bool use_type = false,
                                           const error_handler_t error_handler = error_handler_t::keep);

// (2)
static void to_ubjson(const basic_json& j, detail::output_adapter<std::uint8_t> o,
                      const bool use_size = false, const bool use_type = false,
                      const error_handler_t error_handler = error_handler_t::keep);
static void to_ubjson(const basic_json& j, detail::output_adapter<char> o,
                      const bool use_size = false, const bool use_type = false,
                      const error_handler_t error_handler = error_handler_t::keep);

Serializes a given JSON value j to a byte vector using the UBJSON (Universal Binary JSON) serialization format. UBJSON aims to be more compact than JSON itself, yet more efficient to parse.

  1. Returns a byte vector containing the UBJSON serialization.
  2. Writes the UBJSON serialization to an output adapter.

The exact mapping and its limitations are described on a dedicated page.

Parameters

j (in)
JSON value to serialize
o (in)
output adapter to write serialization to
use_size (in)
whether to add size annotations to container types; optional, #!cpp false by default.
use_type (in)
whether to add type annotations to container types (must be combined with #!cpp use_size = true); optional, #!cpp false by default.
error_handler (in)
how to treat a string or object key in j that is not valid UTF-8; see error_handler_t. The default, keep, writes the ill-formed bytes to the output as is, as every version of to_ubjson did before this parameter was added; strict throws; replace/ignore sanitize it the same way dump would. If JSON_STRICT_BINARY_UTF8 is enabled, the default is strict instead.

Return value

  1. UBJSON serialization as a byte vector
  2. (none)

Exception safety

Strong guarantee: if an exception is thrown, there are no changes in the JSON value.

Exceptions

  • Throws other_error.502 if use_type is true and use_size is false, and j contains a non-empty array, object, or binary value.
  • Throws type_error.316 if a string or object key in j is not valid UTF-8 and error_handler is strict (the default only if JSON_STRICT_BINARY_UTF8 is enabled)
  • Throws type_error.321 if j or a value nested in it is discarded; example: "cannot serialize discarded value to UBJSON"

Complexity

Linear in the size of the JSON value j.

Examples

??? example "Example: serialize a JSON value to UBJSON"

The example shows the serialization of a JSON value to a byte vector in UBJSON format.
 
```cpp
--8<-- "examples/to_ubjson.cpp"
```

Output:

```json
--8<-- "examples/to_ubjson.output"
```

??? example "Example: other_error.502 exception"

The example shows how requesting type annotations (`use_type`) without size annotations (`use_size`) throws an
exception, because type-optimized containers can only be read back with a preceding size.

```cpp
--8<-- "examples/to_ubjson__exception.cpp"
```

Output:

```json
--8<-- "examples/to_ubjson__exception.output"
```

See also

  • UBJSON - the format, its byte layout and how values are mapped
  • Binary formats - overview and comparison of the supported binary formats
  • Binary values - how binary values and subtypes are handled
  • from_ubjson - create a JSON value from an input in UBJSON format
  • JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE - set how many elements of arrays of null, true, or false a reader accepts per byte of input
  • to_cbor - create a CBOR serialization of a JSON value
  • to_msgpack - create a MessagePack serialization of a JSON value
  • to_bson - create a BSON serialization of a JSON value
  • to_bjdata - create a BJData serialization of a JSON value
  • to_bon8 - create a BON8 serialization of a JSON value

Version history

  • Added in version 3.1.0.
  • Added error_handler parameter in version 3.13.0. Its default, keep, writes the bytes of a string or object key that is not valid UTF-8 unchanged, as before; strict (the default if JSON_STRICT_BINARY_UTF8 is enabled) throws type_error.316.
  • Writes arrays of Z, T, or F without the $ type annotation ([#n Z Z ...) once their elements in one call add up to more than 1,048,576, in version 3.13.0, so that from_ubjson can read the output back. Such output is larger: 270,000 rows of 100 null values take 26.8 MB instead of 1.35 MB. The limit does not depend on JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE: a reader accepts at least 1,048,576 elements whatever it is set to, so the writer follows that base only.
  • Throws type_error.321 for a discarded value since version 3.13.0; previously, a discarded value nested in an array or object was silently skipped, producing invalid UBJSON.