docs: correct and complete the BJData documentation

Verify the BJData chapter against the library and the Draft 2/3/4
specifications and fix what did not match:

- bjdata.md: correct the list of unused markers (N, not Z), the ND-array
  example (3x2, not 2x3), the conditions under which an object is packed
  as an ND-array (exactly three keys, row vectors, float-only elements for
  single/double, char 0..127, byte only in Draft 3), the binary table row
  and notes (which flag/draft combinations write [$B), the "$" type
  restriction (Draft 3, Z is excluded too), the H marker in the
  deserialization table, and the claim of a complete deserialization
  mapping. Document the supported drafts, that Draft 4 (SoA containers,
  extension type E) and column-major ND-arrays are rejected, ND-arrays of
  half floats, duplicate object keys, and link the Draft 3 specifications
  instead of master.
- Add an API page for basic_json::bjdata_version_t with an example, and
  link it from the nav, the API index and to_bjdata.
- Update binary_formats/index.md and binary_values.md, which still claimed
  that BJData has no binary type, and fix a wrong byte comment there.
- Document parse_error.115 and the out_of_range.408 cases for from_bjdata,
  the precise other_error.502 condition for to_bjdata, and reword the note
  on truncated unsigned integers in exceptions.md (only ND-array elements
  were truncated before 3.13.0).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann committed 2026-10-25 20:05:49 +01:00
1 parent 6755ad6ded
commit 4b0cf7d76d
11 files changed
+191 -44

No files matched your search

@@ -0,0 +1,48 @@
# <small>nlohmann::basic_json::</small>bjdata_version_t
```cpp
enum class bjdata_version_t
{
draft2,
draft3
};
```
This enumeration is used in [`to_bjdata`](to_bjdata.md) to choose the version of the BJData specification to write:
draft2
: BJData Draft 2 (the default). Binary values are written as arrays of `U` integers, and ND-arrays with
`"_ArrayType_": "byte"` are written as regular objects, since the `B` marker is not part of Draft 2.
draft3
: BJData Draft 3. Binary values are written as `B` elements: as an optimized binary array (`[$B#`) if both `use_size`
and `use_type` are set, and as an array of `B`-prefixed elements otherwise. ND-arrays with
`"_ArrayType_": "byte"` are written as packed arrays (`[$B#[...]`).
Only the writer is affected by the version. [`from_bjdata`](from_bjdata.md) reads the markers of both drafts.
## Examples
??? example
The example below shows how the different values of the `bjdata_version_t` influence the serialization of a binary
value with [`to_bjdata`](to_bjdata.md).
```cpp
--8<-- "examples/bjdata_version_t.cpp"
```
Output:
```json
--8<-- "examples/bjdata_version_t.output"
```
## See also
- [to_bjdata](to_bjdata.md) serializes a JSON value to BJData
- [BJData](../../features/binary_formats/bjdata.md) - the article on the BJData format
## Version history
- Added in version 3.12.0.
@@ -82,8 +82,11 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
- Throws [parse_error.112](../../home/exceptions.md#jsonexceptionparse_error112) if a parse error occurs
- Throws [parse_error.113](../../home/exceptions.md#jsonexceptionparse_error113) if a string could not be parsed
successfully, or if a string value or object key is not valid UTF-8 and `error_handler` is `strict`
- Throws [parse_error.115](../../home/exceptions.md#jsonexceptionparse_error115) if the text of a high-precision
number (`H`) is not a valid number
- Throws [out_of_range.408](../../home/exceptions.md#jsonexceptionout_of_range408) if the size of an optimized container
or n-dimensional array cannot be represented by `std::size_t`
cannot be represented by `std::size_t` or exceeds the container's `max_size()`, or if the product of the dimensions of
an n-dimensional array overflows
## Complexity
+1
View File
@@ -121,6 +121,7 @@ The class satisfies the following concept requirements:
- [**json_pointer**](../json_pointer/index.md) - JSON Pointer implementation
- [**json_serializer**](json_serializer.md) - type of the serializer to for conversions from/to JSON
- [**error_handler_t**](error_handler_t.md) - type to choose behavior on decoding errors
- [**bjdata_version_t**](bjdata_version_t.md) - type to choose the BJData version to write
- [**cbor_tag_handler_t**](cbor_tag_handler_t.md) - type to choose how to handle CBOR tags
- **initializer_list_t** - type for initializer lists of `basic_json` values
- [**input_format_t**](input_format_t.md) - type to choose the format to parse
+5 -2
View File
@@ -43,7 +43,8 @@ The exact mapping and its limitations are described on a [dedicated page](../../
`#!cpp false` by default.
`version` (in)
: which version of BJData to use (see note on "Binary values" on [BJData](../../features/binary_formats/bjdata.md));
: which version of BJData to use, see [`bjdata_version_t`](bjdata_version_t.md) and the note on "Binary values" on
[BJData](../../features/binary_formats/bjdata.md);
optional, `#!cpp bjdata_version_t::draft2` by default.
`error_handler` (in)
@@ -64,7 +65,9 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
## Exceptions
- Throws [`other_error.502`](../../home/exceptions.md#jsonexceptionother_error502) if `use_type` is true and `use_size`
is false, and `j` contains a non-empty array, object, or binary value.
is false, and `j` contains a non-empty array or object, or a non-empty binary value (with
`version = bjdata_version_t::draft3`, any binary value, including an empty one).
Scalars, empty arrays, and empty objects do not throw.
- Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if a string or object key in `j` is
not valid UTF-8 and `error_handler` is `strict` (the default only if
[`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled)
@@ -0,0 +1,33 @@
#include <iomanip>
#include <iostream>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
void print_hex(const char* label, const std::vector<std::uint8_t>& v)
{
std::cout << label << ':';
for (auto b : v)
{
std::cout << ' ' << std::hex << std::setw(2) << std::setfill('0') << static_cast<int>(b);
}
std::cout << std::dec << std::setfill(' ') << '\n';
}
int main()
{
// a binary value
json j = json::binary({0xCA, 0xFE, 0xBA, 0xBE});
// Draft 2 (the default) writes an array of integers
auto v2 = json::to_bjdata(j, true, true, json::bjdata_version_t::draft2);
print_hex("draft2", v2);
// Draft 3 writes an optimized binary array
auto v3 = json::to_bjdata(j, true, true, json::bjdata_version_t::draft3);
print_hex("draft3", v3);
// reading yields an array for Draft 2 and a binary value for Draft 3
std::cout << json::from_bjdata(v2) << '\n';
std::cout << json::from_bjdata(v3) << '\n';
}
@@ -0,0 +1,4 @@
draft2: 5b 24 55 23 69 04 ca fe ba be
draft3: 5b 24 42 23 69 04 ca fe ba be
[202,254,186,190]
{"bytes":[202,254,186,190],"subtype":null}
@@ -2,9 +2,9 @@
The [BJData format](https://neurojson.org) was derived from and improved upon
[Universal Binary JSON(UBJSON)](https://ubjson.org) specification (Draft 12). Specifically, it introduces an optimized
array container for efficient storage of N-dimensional packed arrays (**ND-arrays**); it also adds 5 new type markers -
`[u] - uint16`, `[m] - uint32`, `[M] - uint64`, `[h] - float16` and `[B] - byte` - to unambiguously map common binary
numeric types; furthermore, it uses little-endian (LE) to store all numerics instead of big-endian (BE) as in UBJSON to
array container for efficient storage of N-dimensional packed arrays (**ND-arrays**); it also adds four new type markers -
`[u] - uint16`, `[m] - uint32`, `[M] - uint64` and `[h] - float16` - to unambiguously map common binary numeric types
(Draft 3 adds `[B] - byte`); furthermore, it uses little-endian (LE) to store all numerics instead of big-endian (BE) as in UBJSON to
avoid unnecessary conversions on commonly available platforms.
Compared to other binary JSON-like formats such as MessagePack and CBOR, both BJData and UBJSON demonstrate a rare
@@ -15,7 +15,26 @@ simple processing.
!!! abstract "References"
- [BJData Specification](https://neurojson.org/bjdata/draft2)
- [BJData Specification (Draft 3)](https://neurojson.org/bjdata/draft3)
- [BJData Specification (Draft 2)](https://neurojson.org/bjdata/draft2)
!!! info "Supported drafts"
The library supports BJData Draft 2 (the default) and Draft 3, except for column-major N-dimensional arrays (see
below). The draft used for writing is selected with the
[`version` parameter](../../api/basic_json/bjdata_version_t.md) of
[`to_bjdata`](../../api/basic_json/to_bjdata.md); [`from_bjdata`](../../api/basic_json/from_bjdata.md) has no such
parameter and reads the markers of both drafts.
!!! warning "Draft 4 and column-major ND-arrays are not supported"
BJData Draft 4 (April 2026) adds structure-of-arrays (SoA) containers (`[$` or `{$` followed by a schema object)
and the extension type `E`. `from_bjdata` does not implement them and throws
[`parse_error.112`](../../home/exceptions.md#jsonexceptionparse_error112) for such input; `to_bjdata` never writes
them.
Likewise, column-major ND-arrays of Draft 3 (the `#` marker followed by an array that wraps the dimension vector)
are rejected with [`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113).
## Byte layout
@@ -82,7 +101,7 @@ The library uses the following mapping from JSON values types to BJData types ac
| string | *with shortest length indicator* | string | `S` |
| array | *see notes on optimized format/ND-array* | array | `[` |
| object | *see notes on optimized format* | map | `{` |
| binary | *see notes on binary values* | array | `[$B` |
| binary | *see notes on binary values* | array | `[$B` (Draft 3 with `use_size` and `use_type`), `[U` or `[B` otherwise |
!!! success "Complete mapping"
@@ -107,10 +126,16 @@ The library uses the following mapping from JSON values types to BJData types ac
!!! info "Unused BJData markers"
The following markers are not used in the conversion:
The following markers are not written:
- `Z`: no-op values are not created.
- `C`: single-byte strings are serialized with `S` markers.
- `N`: no-op values are not created (they are skipped when reading).
- `C`: single-character strings are serialized with `S` markers (`C` is read, and written for the elements of
ND-arrays with `"_ArrayType_": "char"`).
- `h`: floating-point numbers are written as `D`, not as float16 (`h` is read).
- `B`: only written in Draft 3 mode.
- `d` is only written for `single` ND-arrays or if `number_float_t` is `float`; `H` is only written for integers
wider than 64 bits (custom `number_integer_t`/`number_unsigned_t`). An unsigned integer above 9223372036854775807
is written as `M`.
!!! info "NaN/infinity handling"
@@ -142,7 +167,7 @@ The library uses the following mapping from JSON values types to BJData types ac
BJData extends UBJSON's optimized array **size** marker to support ND-arrays of uniform numerical data types
(referred to as *packed arrays*). For example, the 2-D `uint8` integer array `[[1,2],[3,4],[5,6]]`, stored as nested
optimized array in UBJSON `[ [$U#i2 1 2 [$U#i2 3 4 [$U#i2 5 6 ]`, can be further compressed in BJData to
`[$U#[$i#i2 2 3 1 2 3 4 5 6` or `[$U#[i2 i3] 1 2 3 4 5 6`.
`[$U#[$i#i2 3 2 1 2 3 4 5 6` or `[$U#[i3 i2] 1 2 3 4 5 6`.
To maintain type and size information, ND-arrays are converted to JSON objects following the **annotated array
format** (defined in the [JData specification (Draft 3)][JDataAAFmt]), when parsed using
@@ -152,11 +177,14 @@ The library uses the following mapping from JSON values types to BJData types ac
```json
{
"_ArrayType_": "uint8",
"_ArraySize_": [2,3],
"_ArraySize_": [3,2],
"_ArrayData_": [1,2,3,4,5,6]
}
```
ND-arrays with the element type `h` (float16) are returned with `"_ArrayType_": "half"`. `to_bjdata` does not pack
`"half"` objects; they are written as regular objects, which read back as an equal value.
Likewise, when a JSON object in the above form is serialized using
[`to_bjdata`](../../api/basic_json/to_bjdata.md), it is automatically converted into a compact BJData ND-array.
@@ -167,34 +195,39 @@ The library uses the following mapping from JSON values types to BJData types ac
object; otherwise it is serialized as a regular JSON object, so the annotation is never lost in a round trip. This requires
all of the following:
- the object has exactly the three keys `"_ArrayType_"`, `"_ArraySize_"`, and `"_ArrayData_"` (any other key, e.g.,
`"_ArrayOrder_"`, makes it a regular object),
- `"_ArrayType_"` is one of `uint8`, `int8`, `uint16`, `int16`, `uint32`, `int32`, `uint64`, `int64`, `single`,
`double`, `char`, or `byte`,
`double`, `char`, or (only with `version = bjdata_version_t::draft3`, since the `B` marker is not part of Draft 2)
`byte`; the names are case-sensitive,
- `"_ArraySize_"` is an array, since the dimensions are written as the ND-array header's length,
- `"_ArraySize_"` has at least two entries and is not a 1×N row vector (first entry 1), since other shapes are
parsed back as a regular array,
- `"_ArraySize_"` has at least two entries and is not a two-entry row vector (`[1,N]`), since other shapes are
parsed back as a regular array (`[1,2,2]` is packed, `[1,4]` is not),
- every entry of `"_ArraySize_"` is a positive integer, and their product is representable as a `std::size_t`,
- `"_ArrayData_"` is an array holding exactly that many elements, and
- every element of `"_ArrayData_"` is a number of the kind named by `"_ArrayType_"`: for the integer types, a
value that fits the named width; for `double`, any value; for `single`, a value that survives narrowing to
`float` and back without change (for instance, `0.1` does not, since it is not exactly representable as
`float`).
- every element of `"_ArrayData_"` is a number of the kind named by `"_ArrayType_"`: for the integer types, an
integer (a floating-point number such as `1.0` does not qualify) that fits the named width (for `char`, 0..127);
for `double`, a floating-point number (an integer such as `1` does not qualify); for `single`, a floating-point
number that survives narrowing to `float` and back without change (for instance, `0.1` does not, since it is not
exactly representable as `float`).
An annotated object is always read back with its keys in the order shown above, `"_ArrayType_"`, `"_ArraySize_"`,
`"_ArrayData_"`, regardless of the order the ND-array's header stores them in on the wire. This matters for
`ordered_json`, whose comparison takes key order into account.
With `ordered_json`, an annotated object read from BJData has its keys in the order `"_ArrayType_"`,
`"_ArraySize_"`, `"_ArrayData_"`; with `json`, they are sorted like those of any other object.
The current version of this library does not yet support automatic detection of and conversion from a nested JSON
array input to a BJData ND-array.
[JDataAAFmt]: https://github.com/NeuroJSON/jdata/blob/master/JData_specification.md#annotated-storage-of-n-d-arrays
[JDataAAFmt]: https://github.com/NeuroJSON/jdata/blob/Draft-3/JData_specification.md#annotated-storage-of-n-d-arrays
!!! info "Restrictions in optimized data types for arrays and objects"
Due to diminished space saving, hampered readability, and increased security risks, in BJData, the allowed data
types following the `$` marker in an optimized array and object container are restricted to
Due to diminished space saving, hampered readability, and increased security risks, in BJData (since Draft 3), the
allowed data types following the `$` marker in an optimized array and object container are restricted to
**non-zero-fixed-length** data types. Therefore, the valid optimized type markers can only be one of
`UiuImlMLhdDCB`. This also means other variable (`[{SH`) or zero-length types (`TFN`) can not be used in an
optimized array or object in BJData.
`UiuImlMLhdDCB`. This also means other variable (`[{SH`) or zero-length types (`TFZN`) can not be used in an
optimized array or object in BJData. The library enforces this restriction for both drafts. With `use_type`, the
elements must have the same marker (`[1,200,3]` is not optimized, since it mixes `i` and `U`); booleans, null,
strings, arrays, and objects are never optimized with `$`.
!!! info "Binary values"
@@ -202,14 +235,16 @@ The library uses the following mapping from JSON values types to BJData types ac
in optimized arrays to designate binary data. This means that, unlike UBJSON, binary data can be both serialized and
deserialized.
To preserve compatibility with BJData Draft 2, the Draft 3 optimized binary array must be explicitly enabled using
the `version` parameter of [`to_bjdata`](../../api/basic_json/to_bjdata.md).
To preserve compatibility with BJData Draft 2, the Draft 3 optimized binary array must be explicitly enabled: pass
`version = bjdata_version_t::draft3` together with `use_size = true` and `use_type = true` to
[`to_bjdata`](../../api/basic_json/to_bjdata.md). With Draft 3 and only `use_size` (or neither), a binary value is
written as an array of `B` elements.
In Draft2 mode (default), if the JSON data contains the binary type, the value stored as a list of integers, as
In Draft 2 mode (default), if the JSON data contains the binary type, the value is stored as a list of integers, as
suggested by the BJData documentation. In particular, this means that the serialization and the deserialization of
JSON containing binary values into BJData and back will result in a different JSON object.
[BJDataBinArr]: https://github.com/NeuroJSON/bjdata/blob/master/Binary_JData_Specification.md#optimized-binary-array
[BJDataBinArr]: https://github.com/NeuroJSON/bjdata/blob/Draft-3/Binary_JData_Specification.md#optimized-binary-array
??? example "Example: serialize JSON values to BJData, with and without size/type optimization"
@@ -236,6 +271,7 @@ The library maps BJData types to JSON value types as follows:
| float16 | number_float | `h` |
| float32 | number_float | `d` |
| float64 | number_float | `D` |
| high-precision number | number_integer, number_unsigned or number_float | `H` |
| uint8 | number_unsigned | `U` |
| int8 | number_integer | `i` |
| uint16 | number_unsigned | `u` |
@@ -254,7 +290,18 @@ The library maps BJData types to JSON value types as follows:
!!! success "Complete mapping"
The mapping is **complete** in the sense that any BJData value can be converted to a JSON value.
The mapping is **complete** for the BJData types of Draft 2 and Draft 3 with one exception: column-major
N-dimensional arrays are rejected with [`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113).
The Draft 4 constructs (structure-of-arrays containers and the extension type `E`) are not supported either.
The digits of an `H` value are parsed like a JSON number: integers that fit 64 bits become `number_integer` or
`number_unsigned`, anything else a `number_float`; invalid text throws
[`parse_error.115`](../../home/exceptions.md#jsonexceptionparse_error115).
!!! info "Duplicate object keys"
If an object contains the same key more than once, `from_bjdata` accepts it and keeps the last value; with
`ordered_json`, the key keeps the position of its first occurrence.
!!! warning "Ill-formed UTF-8 in string values and object keys"
@@ -18,7 +18,7 @@ to efficiently encode JSON values to byte vectors and to decode such vectors.
| Format | Serialization | Deserialization |
|-------------|-----------------------------------------------|----------------------------------------------|
| BJData | complete | complete |
| BJData | complete | incomplete: no column-major ND-arrays, no Draft 4 SoA/extension types |
| BON8 | incomplete: no unsigned integers above int64 | complete |
| BSON | incomplete: top-level value must be an object | incomplete, but all JSON types are supported |
| CBOR | complete | incomplete, but all JSON types are supported |
@@ -29,7 +29,7 @@ to efficiently encode JSON values to byte vectors and to decode such vectors.
| Format | Binary values | Binary subtypes |
|-------------|---------------|-----------------|
| BJData | not supported | not supported |
| BJData | supported (Draft 3 only) | not supported |
| BON8 | not supported | not supported |
| BSON | supported | supported |
| CBOR | supported | supported |
+7 -4
View File
@@ -132,8 +132,11 @@ is an integer or `null`.
### BJData
[BJData](binary_formats/bjdata.md) neither supports binary values nor subtypes and proposes to serialize binary values
as an array of uint8 values. The library implements this translation.
[BJData](binary_formats/bjdata.md) Draft 3 supports binary values as optimized arrays of `B` (byte) elements, but no
subtypes. With `version = bjdata_version_t::draft3` and both `use_size` and `use_type` set, `to_bjdata` writes a binary
value as `[$B#...`, and `from_bjdata` reads it back as a binary value (without subtype). In all other cases, including
the default Draft 2 mode, the library follows the Draft 2 proposal and writes an array of integers (`U` in Draft 2, `B`
in Draft 3), which is read back as an array.
??? example "Example: serialize a binary value to BJData"
@@ -171,7 +174,7 @@ as an array of uint8 values. The library implements this translation.
```c
0x7B // '{'
0x23 0x69 0x01 // '#' 'i' type of the array elements: unsigned integers
0x23 0x69 0x01 // '#' 'i' 1 number of object members
0x69 0x06 // i 6 (length of the key)
0x62 0x69 0x6E 0x61 0x72 0x79 // "binary"
0x5B // '[' array
@@ -180,7 +183,7 @@ as an array of uint8 values. The library implements this translation.
0xCA 0xFE 0xBA 0xBE // content
```
Note that subtype (42) is **not** serialized and that BJData has **no binary type**, and deserializing `v` would
Note that subtype (42) is **not** serialized and that, in Draft 2 mode, BJData has **no binary type**, and deserializing `v` would
yield the following value:
```json
+9 -5
View File
@@ -418,13 +418,16 @@ The parsing of the corresponding BSON record type is not implemented (yet).
### json.exception.parse_error.115
A UBJSON high-precision number could not be parsed.
A UBJSON or BJData high-precision number could not be parsed.
!!! failure "Example message"
!!! failure "Example messages"
```
[json.exception.parse_error.115] parse error at byte 5: syntax error while parsing UBJSON high-precision number: invalid number text: 1A
```
```
[json.exception.parse_error.115] parse error at byte 5: syntax error while parsing BJData high-precision number: invalid number text: 1A
```
### json.exception.parse_error.116
@@ -1025,7 +1028,7 @@ exception, because they serialize integers beyond 64 bits as high-precision numb
- Since version 3.12.0, integer numbers beyond int64 are serialized as uint64 BSON numbers.
Before version 3.13.0, CBOR, MessagePack, and BSON silently truncated integers wider than 64 bits, and BJData
truncated unsigned integers wider than 64 bits.
truncated unsigned integers wider than 64 bits that were elements of ND-arrays with `"_ArrayType_": "uint64"`.
!!! failure "Example messages"
@@ -1233,8 +1236,9 @@ A JSON Patch operation 'test' failed. The unsuccessful operation is also printed
### json.exception.other_error.502
[`to_ubjson`](../api/basic_json/to_ubjson.md) and [`to_bjdata`](../api/basic_json/to_bjdata.md) were called with
`use_type = true` but `use_size = false` for a value that contains a non-empty array, object, or binary value. UBJSON
requires a size marker (`#`) after a type marker (`$`); scalars and empty containers do not throw.
`use_type = true` but `use_size = false`, and the value contains a non-empty array or object (or, with BJData Draft 3,
any binary value). UBJSON requires a size marker (`#`) after a type marker (`$`); scalars and empty arrays or objects
do not throw.
!!! failure "Example message"
+1
View File
@@ -171,6 +171,7 @@ nav:
- 'at': api/basic_json/at.md
- 'back': api/basic_json/back.md
- 'begin': api/basic_json/begin.md
- 'bjdata_version_t': api/basic_json/bjdata_version_t.md
- 'binary': api/basic_json/binary.md
- 'binary_t': api/basic_json/binary_t.md
- 'boolean_t': api/basic_json/boolean_t.md