amorfo77/json
1
0 Fork 0
mirror of https://github.com/nlohmann/json.git synced 2026-10-28 08:45:43 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: correct and complete the BJData documentation

Browse Source 
Verify the BJData chapter against the library and the Draft 2/3/4
specifications and fix what did not match:

- bjdata.md: correct the list of unused markers (N, not Z), the ND-array
  example (3x2, not 2x3), the conditions under which an object is packed
  as an ND-array (exactly three keys, row vectors, float-only elements for
  single/double, char 0..127, byte only in Draft 3), the binary table row
  and notes (which flag/draft combinations write [$B), the "$" type
  restriction (Draft 3, Z is excluded too), the H marker in the
  deserialization table, and the claim of a complete deserialization
  mapping. Document the supported drafts, that Draft 4 (SoA containers,
  extension type E) and column-major ND-arrays are rejected, ND-arrays of
  half floats, duplicate object keys, and link the Draft 3 specifications
  instead of master.
- Add an API page for basic_json::bjdata_version_t with an example, and
  link it from the nav, the API index and to_bjdata.
- Update binary_formats/index.md and binary_values.md, which still claimed
  that BJData has no binary type, and fix a wrong byte comment there.
- Document parse_error.115 and the out_of_range.408 cases for from_bjdata,
  the precise other_error.502 condition for to_bjdata, and reword the note
  on truncated unsigned integers in exceptions.md (only ND-array elements
  were truncated before 3.13.0).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann committed 2026-10-26 06:53:52 +01:00
1 parent 31538a91ed
commit b3ab23e11c
11 files changed
+191 -44
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/mkdocs/docs/features/binary_formats/bjdata.md
+77 -30
View File
@@ -2,9 +2,9 @@
The [BJData format](https://neurojson.org) was derived from and improved upon
[Universal Binary JSON(UBJSON)](https://ubjson.org) specification (Draft 12). Specifically, it introduces an optimized
array container for efficient storage of N-dimensional packed arrays (**ND-arrays**); it also adds 5 new type markers -
`[u] - uint16`, `[m] - uint32`, `[M] - uint64`, `[h] - float16` and `[B] - byte` - to unambiguously map common binary
numeric types; furthermore, it uses little-endian (LE) to store all numerics instead of big-endian (BE) as in UBJSON to
array container for efficient storage of N-dimensional packed arrays (**ND-arrays**); it also adds four new type markers -
`[u] - uint16`, `[m] - uint32`, `[M] - uint64` and `[h] - float16` - to unambiguously map common binary numeric types
(Draft 3 adds `[B] - byte`); furthermore, it uses little-endian (LE) to store all numerics instead of big-endian (BE) as in UBJSON to
avoid unnecessary conversions on commonly available platforms.
Compared to other binary JSON-like formats such as MessagePack and CBOR, both BJData and UBJSON demonstrate a rare
@@ -15,7 +15,26 @@ simple processing.
!!! abstract "References"
- [BJData Specification](https://neurojson.org/bjdata/draft2)
- [BJData Specification (Draft 3)](https://neurojson.org/bjdata/draft3)
- [BJData Specification (Draft 2)](https://neurojson.org/bjdata/draft2)
!!! info "Supported drafts"
The library supports BJData Draft 2 (the default) and Draft 3, except for column-major N-dimensional arrays (see
below). The draft used for writing is selected with the
[`version` parameter](../../api/basic_json/bjdata_version_t.md) of
[`to_bjdata`](../../api/basic_json/to_bjdata.md); [`from_bjdata`](../../api/basic_json/from_bjdata.md) has no such
parameter and reads the markers of both drafts.
!!! warning "Draft 4 and column-major ND-arrays are not supported"
BJData Draft 4 (April 2026) adds structure-of-arrays (SoA) containers (`[$` or `{$` followed by a schema object)
and the extension type `E`. `from_bjdata` does not implement them and throws
[`parse_error.112`](../../home/exceptions.md#jsonexceptionparse_error112) for such input; `to_bjdata` never writes
them.
Likewise, column-major ND-arrays of Draft 3 (the `#` marker followed by an array that wraps the dimension vector)
are rejected with [`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113).
## Byte layout
@@ -82,7 +101,7 @@ The library uses the following mapping from JSON values types to BJData types ac
| string | *with shortest length indicator* | string | `S` |
| array | *see notes on optimized format/ND-array* | array | `[` |
| object | *see notes on optimized format* | map | `{` |
| binary | *see notes on binary values* | array | `[$B` |
| binary | *see notes on binary values* | array | `[$B` (Draft 3 with `use_size` and `use_type`), `[U` or `[B` otherwise |
!!! success "Complete mapping"
@@ -107,10 +126,16 @@ The library uses the following mapping from JSON values types to BJData types ac
!!! info "Unused BJData markers"
The following markers are not used in the conversion:
The following markers are not written:
- `Z`: no-op values are not created.
- `C`: single-byte strings are serialized with `S` markers.
- `N`: no-op values are not created (they are skipped when reading).
- `C`: single-character strings are serialized with `S` markers (`C` is read, and written for the elements of
ND-arrays with `"_ArrayType_": "char"`).
- `h`: floating-point numbers are written as `D`, not as float16 (`h` is read).
- `B`: only written in Draft 3 mode.
- `d` is only written for `single` ND-arrays or if `number_float_t` is `float`; `H` is only written for integers
wider than 64 bits (custom `number_integer_t`/`number_unsigned_t`). An unsigned integer above 9223372036854775807
is written as `M`.
!!! info "NaN/infinity handling"
@@ -142,7 +167,7 @@ The library uses the following mapping from JSON values types to BJData types ac
BJData extends UBJSON's optimized array **size** marker to support ND-arrays of uniform numerical data types
(referred to as *packed arrays*). For example, the 2-D `uint8` integer array `[[1,2],[3,4],[5,6]]`, stored as nested
optimized array in UBJSON `[ [$U#i2 1 2 [$U#i2 3 4 [$U#i2 5 6 ]`, can be further compressed in BJData to
`[$U#[$i#i2 2 3 1 2 3 4 5 6` or `[$U#[i2 i3] 1 2 3 4 5 6`.
`[$U#[$i#i2 3 2 1 2 3 4 5 6` or `[$U#[i3 i2] 1 2 3 4 5 6`.
To maintain type and size information, ND-arrays are converted to JSON objects following the **annotated array
format** (defined in the [JData specification (Draft 3)][JDataAAFmt]), when parsed using
@@ -152,11 +177,14 @@ The library uses the following mapping from JSON values types to BJData types ac
```json
{
"_ArrayType_": "uint8",
"_ArraySize_": [2,3],
"_ArraySize_": [3,2],
"_ArrayData_": [1,2,3,4,5,6]
}
```
ND-arrays with the element type `h` (float16) are returned with `"_ArrayType_": "half"`. `to_bjdata` does not pack
`"half"` objects; they are written as regular objects, which read back as an equal value.
Likewise, when a JSON object in the above form is serialized using
[`to_bjdata`](../../api/basic_json/to_bjdata.md), it is automatically converted into a compact BJData ND-array.
@@ -167,34 +195,39 @@ The library uses the following mapping from JSON values types to BJData types ac
object; otherwise it is serialized as a regular JSON object, so the annotation is never lost in a round trip. This requires
all of the following:
- the object has exactly the three keys `"_ArrayType_"`, `"_ArraySize_"`, and `"_ArrayData_"` (any other key, e.g.,
`"_ArrayOrder_"`, makes it a regular object),
- `"_ArrayType_"` is one of `uint8`, `int8`, `uint16`, `int16`, `uint32`, `int32`, `uint64`, `int64`, `single`,
`double`, `char`, or `byte`,
`double`, `char`, or (only with `version = bjdata_version_t::draft3`, since the `B` marker is not part of Draft 2)
`byte`; the names are case-sensitive,
- `"_ArraySize_"` is an array, since the dimensions are written as the ND-array header's length,
- `"_ArraySize_"` has at least two entries and is not a 1×N row vector (first entry 1), since other shapes are
parsed back as a regular array,
- `"_ArraySize_"` has at least two entries and is not a two-entry row vector (`[1,N]`), since other shapes are
parsed back as a regular array (`[1,2,2]` is packed, `[1,4]` is not),
- every entry of `"_ArraySize_"` is a positive integer, and their product is representable as a `std::size_t`,
- `"_ArrayData_"` is an array holding exactly that many elements, and
- every element of `"_ArrayData_"` is a number of the kind named by `"_ArrayType_"`: for the integer types, a
value that fits the named width; for `double`, any value; for `single`, a value that survives narrowing to
`float` and back without change (for instance, `0.1` does not, since it is not exactly representable as
`float`).
- every element of `"_ArrayData_"` is a number of the kind named by `"_ArrayType_"`: for the integer types, an
integer (a floating-point number such as `1.0` does not qualify) that fits the named width (for `char`, 0..127);
for `double`, a floating-point number (an integer such as `1` does not qualify); for `single`, a floating-point
number that survives narrowing to `float` and back without change (for instance, `0.1` does not, since it is not
exactly representable as `float`).
An annotated object is always read back with its keys in the order shown above, `"_ArrayType_"`, `"_ArraySize_"`,
`"_ArrayData_"`, regardless of the order the ND-array's header stores them in on the wire. This matters for
`ordered_json`, whose comparison takes key order into account.
With `ordered_json`, an annotated object read from BJData has its keys in the order `"_ArrayType_"`,
`"_ArraySize_"`, `"_ArrayData_"`; with `json`, they are sorted like those of any other object.
The current version of this library does not yet support automatic detection of and conversion from a nested JSON
array input to a BJData ND-array.
[JDataAAFmt]: https://github.com/NeuroJSON/jdata/blob/master/JData_specification.md#annotated-storage-of-n-d-arrays
[JDataAAFmt]: https://github.com/NeuroJSON/jdata/blob/Draft-3/JData_specification.md#annotated-storage-of-n-d-arrays
!!! info "Restrictions in optimized data types for arrays and objects"
Due to diminished space saving, hampered readability, and increased security risks, in BJData, the allowed data
types following the `$` marker in an optimized array and object container are restricted to
Due to diminished space saving, hampered readability, and increased security risks, in BJData (since Draft 3), the
allowed data types following the `$` marker in an optimized array and object container are restricted to
**non-zero-fixed-length** data types. Therefore, the valid optimized type markers can only be one of
`UiuImlMLhdDCB`. This also means other variable (`[{SH`) or zero-length types (`TFN`) can not be used in an
optimized array or object in BJData.
`UiuImlMLhdDCB`. This also means other variable (`[{SH`) or zero-length types (`TFZN`) can not be used in an
optimized array or object in BJData. The library enforces this restriction for both drafts. With `use_type`, the
elements must have the same marker (`[1,200,3]` is not optimized, since it mixes `i` and `U`); booleans, null,
strings, arrays, and objects are never optimized with `$`.
!!! info "Binary values"
@@ -202,14 +235,16 @@ The library uses the following mapping from JSON values types to BJData types ac
in optimized arrays to designate binary data. This means that, unlike UBJSON, binary data can be both serialized and
deserialized.
To preserve compatibility with BJData Draft 2, the Draft 3 optimized binary array must be explicitly enabled using
the `version` parameter of [`to_bjdata`](../../api/basic_json/to_bjdata.md).
To preserve compatibility with BJData Draft 2, the Draft 3 optimized binary array must be explicitly enabled: pass
`version = bjdata_version_t::draft3` together with `use_size = true` and `use_type = true` to
[`to_bjdata`](../../api/basic_json/to_bjdata.md). With Draft 3 and only `use_size` (or neither), a binary value is
written as an array of `B` elements.
In Draft2 mode (default), if the JSON data contains the binary type, the value stored as a list of integers, as
In Draft 2 mode (default), if the JSON data contains the binary type, the value is stored as a list of integers, as
suggested by the BJData documentation. In particular, this means that the serialization and the deserialization of
JSON containing binary values into BJData and back will result in a different JSON object.
[BJDataBinArr]: https://github.com/NeuroJSON/bjdata/blob/master/Binary_JData_Specification.md#optimized-binary-array
[BJDataBinArr]: https://github.com/NeuroJSON/bjdata/blob/Draft-3/Binary_JData_Specification.md#optimized-binary-array
??? example "Example: serialize JSON values to BJData, with and without size/type optimization"
@@ -236,6 +271,7 @@ The library maps BJData types to JSON value types as follows:
| float16 | number_float | `h` |
| float32 | number_float | `d` |
| float64 | number_float | `D` |
| high-precision number | number_integer, number_unsigned or number_float | `H` |
| uint8 | number_unsigned | `U` |
| int8 | number_integer | `i` |
| uint16 | number_unsigned | `u` |
@@ -254,7 +290,18 @@ The library maps BJData types to JSON value types as follows:
!!! success "Complete mapping"
The mapping is **complete** in the sense that any BJData value can be converted to a JSON value.
The mapping is **complete** for the BJData types of Draft 2 and Draft 3 with one exception: column-major
N-dimensional arrays are rejected with [`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113).
The Draft 4 constructs (structure-of-arrays containers and the extension type `E`) are not supported either.
The digits of an `H` value are parsed like a JSON number: integers that fit 64 bits become `number_integer` or
`number_unsigned`, anything else a `number_float`; invalid text throws
[`parse_error.115`](../../home/exceptions.md#jsonexceptionparse_error115).
!!! info "Duplicate object keys"
If an object contains the same key more than once, `from_bjdata` accepts it and keeps the last value; with
`ordered_json`, the key keeps the position of its first occurrence.
!!! warning "Ill-formed UTF-8 in string values and object keys"
docs/mkdocs/docs/features/binary_formats/index.md
+2 -2
View File
@@ -18,7 +18,7 @@ to efficiently encode JSON values to byte vectors and to decode such vectors.
| Format | Serialization | Deserialization |
|-------------|-----------------------------------------------|----------------------------------------------|
| BJData | complete | complete |
| BJData | complete | incomplete: no column-major ND-arrays, no Draft 4 SoA/extension types |
| BON8 | incomplete: no unsigned integers above int64 | complete |
| BSON | incomplete: top-level value must be an object | incomplete, but all JSON types are supported |
| CBOR | complete | incomplete, but all JSON types are supported |
@@ -29,7 +29,7 @@ to efficiently encode JSON values to byte vectors and to decode such vectors.
| Format | Binary values | Binary subtypes |
|-------------|---------------|-----------------|
| BJData | not supported | not supported |
| BJData | supported (Draft 3 only) | not supported |
| BON8 | not supported | not supported |
| BSON | supported | supported |
| CBOR | supported | supported |
docs/mkdocs/docs/features/binary_values.md
+7 -4
View File
@@ -132,8 +132,11 @@ is an integer or `null`.
### BJData
[BJData](binary_formats/bjdata.md) neither supports binary values nor subtypes and proposes to serialize binary values
as an array of uint8 values. The library implements this translation.
[BJData](binary_formats/bjdata.md) Draft 3 supports binary values as optimized arrays of `B` (byte) elements, but no
subtypes. With `version = bjdata_version_t::draft3` and both `use_size` and `use_type` set, `to_bjdata` writes a binary
value as `[$B#...`, and `from_bjdata` reads it back as a binary value (without subtype). In all other cases, including
the default Draft 2 mode, the library follows the Draft 2 proposal and writes an array of integers (`U` in Draft 2, `B`
in Draft 3), which is read back as an array.
??? example "Example: serialize a binary value to BJData"
@@ -171,7 +174,7 @@ as an array of uint8 values. The library implements this translation.
```c
0x7B // '{'
0x23 0x69 0x01 // '#' 'i' type of the array elements: unsigned integers
0x23 0x69 0x01 // '#' 'i' 1 number of object members
0x69 0x06 // i 6 (length of the key)
0x62 0x69 0x6E 0x61 0x72 0x79 // "binary"
0x5B // '[' array
@@ -180,7 +183,7 @@ as an array of uint8 values. The library implements this translation.
0xCA 0xFE 0xBA 0xBE // content
```
Note that subtype (42) is **not** serialized and that BJData has **no binary type**, and deserializing `v` would
Note that subtype (42) is **not** serialized and that, in Draft 2 mode, BJData has **no binary type**, and deserializing `v` would
yield the following value:
```json