mirror of
https://github.com/nlohmann/json.git
synced 2026-07-12 13:35:13 +00:00
Documentation change (#3672)
Co-authored-by: Florian Albrechtskirchinger <falbrechtskirchinger@gmail.com>
This commit is contained in:
Niels Lohmann
GitHub
@@ -1,24 +1,21 @@
|
||||
# BJData
|
||||
|
||||
The [BJData format](https://neurojson.org) was derived from and improved upon
|
||||
[Universal Binary JSON(UBJSON)](https://ubjson.org) specification (Draft 12).
|
||||
Specifically, it introduces an optimized array container for efficient storage
|
||||
of N-dimensional packed arrays (**ND-arrays**); it also adds 4 new type markers -
|
||||
`[u] - uint16`, `[m] - uint32`, `[M] - uint64` and `[h] - float16` - to
|
||||
unambigiously map common binary numeric types; furthermore, it uses little-endian
|
||||
(LE) to store all numerics instead of big-endian (BE) as in UBJSON to avoid
|
||||
[Universal Binary JSON(UBJSON)](https://ubjson.org) specification (Draft 12). Specifically, it introduces an optimized
|
||||
array container for efficient storage of N-dimensional packed arrays (**ND-arrays**); it also adds 4 new type markers -
|
||||
`[u] - uint16`, `[m] - uint32`, `[M] - uint64` and `[h] - float16` - to unambiguously map common binary numeric types;
|
||||
furthermore, it uses little-endian (LE) to store all numerics instead of big-endian (BE) as in UBJSON to avoid
|
||||
unnecessary conversions on commonly available platforms.
|
||||
|
||||
Compared to other binary JSON-like formats such as MessagePack and CBOR, both BJData and
|
||||
UBJSON demonstrate a rare combination of being both binary and **quasi-human-readable**. This
|
||||
is because all semantic elements in BJData and UBJSON, including the data-type markers
|
||||
and name/string types are directly human-readable. Data stored in the BJData/UBJSON format
|
||||
are not only compact in size, fast to read/write, but also can be directly searched
|
||||
or read using simple processing.
|
||||
Compared to other binary JSON-like formats such as MessagePack and CBOR, both BJData and UBJSON demonstrate a rare
|
||||
combination of being both binary and **quasi-human-readable**. This is because all semantic elements in BJData and
|
||||
UBJSON, including the data-type markers and name/string types are directly human-readable. Data stored in the
|
||||
BJData/UBJSON format are not only compact in size, fast to read/write, but also can be directly searched or read using
|
||||
simple processing.
|
||||
|
||||
!!! abstract "References"
|
||||
|
||||
- [BJData Specification](https://neurojson.org/bjdata/draft2)
|
||||
- [BJData Specification](https://neurojson.org/bjdata/draft2)
|
||||
|
||||
## Serialization
|
||||
|
||||
@@ -55,67 +52,59 @@ The library uses the following mapping from JSON values types to BJData types ac
|
||||
|
||||
!!! success "Complete mapping"
|
||||
|
||||
The mapping is **complete** in the sense that any JSON value type can be converted to a BJData value.
|
||||
The mapping is **complete** in the sense that any JSON value type can be converted to a BJData value.
|
||||
|
||||
Any BJData output created by `to_bjdata` can be successfully parsed by `from_bjdata`.
|
||||
Any BJData output created by `to_bjdata` can be successfully parsed by `from_bjdata`.
|
||||
|
||||
!!! warning "Size constraints"
|
||||
|
||||
The following values can **not** be converted to a BJData value:
|
||||
The following values can **not** be converted to a BJData value:
|
||||
|
||||
- strings with more than 18446744073709551615 bytes, i.e., $2^{64}-1$ bytes (theoretical)
|
||||
|
||||
!!! info "Unused BJData markers"
|
||||
|
||||
The following markers are not used in the conversion:
|
||||
The following markers are not used in the conversion:
|
||||
|
||||
- `Z`: no-op values are not created.
|
||||
- `C`: single-byte strings are serialized with `S` markers.
|
||||
|
||||
!!! info "NaN/infinity handling"
|
||||
|
||||
If NaN or Infinity are stored inside a JSON number, they are
|
||||
serialized properly. This behavior differs from the `dump()`
|
||||
function which serializes NaN or Infinity to `null`.
|
||||
|
||||
If NaN or Infinity are stored inside a JSON number, they are serialized properly. This behavior differs from the
|
||||
`dump()` function which serializes NaN or Infinity to `#!json null`.
|
||||
|
||||
!!! info "Endianness"
|
||||
|
||||
A breaking difference between BJData and UBJSON is the endianness
|
||||
of numerical values. In BJData, all numerical data types (integers
|
||||
`UiuImlML` and floating-point values `hdD`) are stored in the little-endian (LE)
|
||||
byte order as opposed to big-endian as used by UBJSON. Adopting LE
|
||||
to store numeric records avoids unnecessary byte swapping on most modern
|
||||
computers where LE is used as the default byte order.
|
||||
A breaking difference between BJData and UBJSON is the endianness of numerical values. In BJData, all numerical data
|
||||
types (integers `UiuImlML` and floating-point values `hdD`) are stored in the little-endian (LE) byte order as
|
||||
opposed to big-endian as used by UBJSON. Adopting LE to store numeric records avoids unnecessary byte swapping on
|
||||
most modern computers where LE is used as the default byte order.
|
||||
|
||||
!!! info "Optimized formats"
|
||||
|
||||
Optimized formats for containers are supported via two parameters of
|
||||
Optimized formats for containers are supported via two parameters of
|
||||
[`to_bjdata`](../../api/basic_json/to_bjdata.md):
|
||||
|
||||
- Parameter `use_size` adds size information to the beginning of a container and
|
||||
removes the closing marker.
|
||||
- Parameter `use_type` further checks whether all elements of a container have the
|
||||
same type and adds the type marker to the beginning of the container.
|
||||
The `use_type` parameter must only be used together with `use_size = true`.
|
||||
- Parameter `use_size` adds size information to the beginning of a container and removes the closing marker.
|
||||
- Parameter `use_type` further checks whether all elements of a container have the same type and adds the type
|
||||
marker to the beginning of the container. The `use_type` parameter must only be used together with
|
||||
`use_size = true`.
|
||||
|
||||
Note that `use_size = true` alone may result in larger representations -
|
||||
the benefit of this parameter is that the receiving side is
|
||||
immediately informed of the number of elements in the container.
|
||||
Note that `use_size = true` alone may result in larger representations - the benefit of this parameter is that the
|
||||
receiving side is immediately informed of the number of elements in the container.
|
||||
|
||||
!!! info "ND-array optimized format"
|
||||
|
||||
BJData extends UBJSON's optimized array **size** marker to support ND-arrays of
|
||||
uniform numerical data types (referred to as *packed arrays*).
|
||||
For example, the 2-D `uint8` integer array `[[1,2],[3,4],[5,6]]`, stored
|
||||
as nested optimized array in UBJSON `[ [$U#i2 1 2 [$U#i2 3 4 [$U#i2 5 6 ]`,
|
||||
can be further compressed in BJData to `[$U#[$i#i2 2 3 1 2 3 4 5 6`
|
||||
or `[$U#[i2 i3] 1 2 3 4 5 6`.
|
||||
BJData extends UBJSON's optimized array **size** marker to support ND-arrays of uniform numerical data types
|
||||
(referred to as *packed arrays*). For example, the 2-D `uint8` integer array `[[1,2],[3,4],[5,6]]`, stored as nested
|
||||
optimized array in UBJSON `[ [$U#i2 1 2 [$U#i2 3 4 [$U#i2 5 6 ]`, can be further compressed in BJData to
|
||||
`[$U#[$i#i2 2 3 1 2 3 4 5 6` or `[$U#[i2 i3] 1 2 3 4 5 6`.
|
||||
|
||||
To maintina type and size information, ND-arrays are converted to JSON objects following the
|
||||
**annotated array format** (defined in the [JData specification (Draft 3)][JDataAAFmt]),
|
||||
when parsed using [`from_bjdata`](../../api/basic_json/from_bjdata.md).
|
||||
For example, the above 2-D `uint8` array can be parsed and accessed as
|
||||
To maintain type and size information, ND-arrays are converted to JSON objects following the **annotated array
|
||||
format** (defined in the [JData specification (Draft 3)][JDataAAFmt]), when parsed using
|
||||
[`from_bjdata`](../../api/basic_json/from_bjdata.md). For example, the above 2-D `uint8` array can be parsed and
|
||||
accessed as
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -126,34 +115,28 @@ The library uses the following mapping from JSON values types to BJData types ac
|
||||
```
|
||||
|
||||
Likewise, when a JSON object in the above form is serialzed using
|
||||
[`to_bjdata`](../../api/basic_json/to_bjdata.md), it is automatically converted
|
||||
into a compact BJData ND-array. The only exception is, that when the 1-dimensional
|
||||
vector stored in `"_ArraySize_"` contains a single integer or two integers with one
|
||||
being 1, a regular 1-D optimized array is generated.
|
||||
[`to_bjdata`](../../api/basic_json/to_bjdata.md), it is automatically converted into a compact BJData ND-array. The
|
||||
only exception is, that when the 1-dimensional vector stored in `"_ArraySize_"` contains a single integer or two
|
||||
integers with one being 1, a regular 1-D optimized array is generated.
|
||||
|
||||
The current version of this library does not yet support automatic detection of and
|
||||
conversion from a nested JSON array input to a BJData ND-array.
|
||||
The current version of this library does not yet support automatic detection of and conversion from a nested JSON
|
||||
array input to a BJData ND-array.
|
||||
|
||||
[JDataAAFmt]: https://github.com/NeuroJSON/jdata/blob/master/JData_specification.md#annotated-storage-of-n-d-arrays)
|
||||
|
||||
!!! info "Restrictions in optimized data types for arrays and objects"
|
||||
|
||||
Due to diminished space saving, hampered readability, and increased
|
||||
security risks, in BJData, the allowed data types following the `$` marker
|
||||
in an optimized array and object container are restricted to
|
||||
**non-zero-fixed-length** data types. Therefore, the valid optimized
|
||||
type markers can only be one of `UiuImlMLhdDC`. This also means other
|
||||
variable (`[{SH`) or zero-length types (`TFN`) can not be used in an
|
||||
optimized array or object in BJData.
|
||||
Due to diminished space saving, hampered readability, and increased security risks, in BJData, the allowed data
|
||||
types following the `$` marker in an optimized array and object container are restricted to
|
||||
**non-zero-fixed-length** data types. Therefore, the valid optimized type markers can only be one of `UiuImlMLhdDC`.
|
||||
This also means other variable (`[{SH`) or zero-length types (`TFN`) can not be used in an optimized array or object
|
||||
in BJData.
|
||||
|
||||
!!! info "Binary values"
|
||||
|
||||
If the JSON data contains the binary type, the value stored is a list
|
||||
of integers, as suggested by the BJData documentation. In particular,
|
||||
this means that the serialization and the deserialization of JSON
|
||||
containing binary values into BJData and back will result in a
|
||||
different JSON object.
|
||||
|
||||
If the JSON data contains the binary type, the value stored is a list of integers, as suggested by the BJData
|
||||
documentation. In particular, this means that the serialization and the deserialization of JSON containing binary
|
||||
values into BJData and back will result in a different JSON object.
|
||||
|
||||
??? example
|
||||
|
||||
@@ -196,8 +179,7 @@ The library maps BJData types to JSON value types as follows:
|
||||
|
||||
!!! success "Complete mapping"
|
||||
|
||||
The mapping is **complete** in the sense that any BJData value can be converted to a JSON value.
|
||||
|
||||
The mapping is **complete** in the sense that any BJData value can be converted to a JSON value.
|
||||
|
||||
??? example
|
||||
|
||||
|
||||
@@ -6,8 +6,8 @@ representation of data types that are not part of the JSON spec. For example, BS
|
||||
|
||||
!!! abstract "References"
|
||||
|
||||
- [BSON Website](http://bsonspec.org) - the main source on BSON
|
||||
- [BSON Specification](http://bsonspec.org/spec.html) - the specification
|
||||
- [BSON Website](http://bsonspec.org) - the main source on BSON
|
||||
- [BSON Specification](http://bsonspec.org/spec.html) - the specification
|
||||
|
||||
|
||||
## Serialization
|
||||
|
||||
@@ -5,13 +5,14 @@ small code size, fairly small message size, and extensibility without the need f
|
||||
|
||||
!!! abstract "References"
|
||||
|
||||
- [CBOR Website](http://cbor.io) - the main source on CBOR
|
||||
- [CBOR Website](http://cbor.io) - the main source on CBOR
|
||||
- [CBOR Playground](http://cbor.me) - an interactive webpage to translate between JSON and CBOR
|
||||
- [RFC 7049](https://tools.ietf.org/html/rfc7049) - the CBOR specification
|
||||
|
||||
## Serialization
|
||||
|
||||
The library uses the following mapping from JSON values types to CBOR types according to the CBOR specification (RFC 7049):
|
||||
The library uses the following mapping from JSON values types to CBOR types according to the CBOR specification
|
||||
([RFC 7049](https://www.rfc-editor.org/rfc/rfc7049.html)):
|
||||
|
||||
| JSON value type | value/range | CBOR type | first byte |
|
||||
|-----------------|--------------------------------------------|-----------------------------------|------------|
|
||||
@@ -61,15 +62,15 @@ see "binary" cells in the table above.
|
||||
|
||||
!!! success "Complete mapping"
|
||||
|
||||
The mapping is **complete** in the sense that any JSON value type can be converted to a CBOR value.
|
||||
The mapping is **complete** in the sense that any JSON value type can be converted to a CBOR value.
|
||||
|
||||
!!! info "NaN/infinity handling"
|
||||
|
||||
If NaN or Infinity are stored inside a JSON number, they are serialized properly. This behavior differs from the normal JSON serialization which serializes NaN or Infinity to `null`.
|
||||
If NaN or Infinity are stored inside a JSON number, they are serialized properly. This behavior differs from the normal JSON serialization which serializes NaN or Infinity to `null`.
|
||||
|
||||
!!! info "Unused CBOR types"
|
||||
|
||||
The following CBOR types are not used in the conversion:
|
||||
The following CBOR types are not used in the conversion:
|
||||
|
||||
- UTF-8 strings terminated by "break" (0x7F)
|
||||
- arrays terminated by "break" (0x9F)
|
||||
@@ -149,7 +150,7 @@ The library maps CBOR types to JSON value types as follows:
|
||||
|
||||
!!! warning "Incomplete mapping"
|
||||
|
||||
The mapping is **incomplete** in the sense that not all CBOR types can be converted to a JSON value. The following CBOR types are not supported and will yield parse errors:
|
||||
The mapping is **incomplete** in the sense that not all CBOR types can be converted to a JSON value. The following CBOR types are not supported and will yield parse errors:
|
||||
|
||||
- date/time (0xC0..0xC1)
|
||||
- bignum (0xC2..0xC3)
|
||||
@@ -161,7 +162,7 @@ The library maps CBOR types to JSON value types as follows:
|
||||
|
||||
!!! warning "Object keys"
|
||||
|
||||
CBOR allows map keys of any type, whereas JSON only allows strings as keys in object values. Therefore, CBOR maps with keys other than UTF-8 strings are rejected.
|
||||
CBOR allows map keys of any type, whereas JSON only allows strings as keys in object values. Therefore, CBOR maps with keys other than UTF-8 strings are rejected.
|
||||
|
||||
!!! warning "Tagged items"
|
||||
|
||||
|
||||
@@ -1,15 +1,18 @@
|
||||
# MessagePack
|
||||
|
||||
MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON. But it's faster and smaller. Small integers are encoded into a single byte, and typical short strings require only one extra byte in addition to the strings themselves.
|
||||
MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON.
|
||||
But it's faster and smaller. Small integers are encoded into a single byte, and typical short strings require only one
|
||||
extra byte in addition to the strings themselves.
|
||||
|
||||
!!! abstract "References"
|
||||
|
||||
- [MessagePack website](https://msgpack.org)
|
||||
- [MessagePack specification](https://github.com/msgpack/msgpack/blob/master/spec.md)
|
||||
- [MessagePack website](https://msgpack.org)
|
||||
- [MessagePack specification](https://github.com/msgpack/msgpack/blob/master/spec.md)
|
||||
|
||||
## Serialization
|
||||
|
||||
The library uses the following mapping from JSON values types to MessagePack types according to the MessagePack specification:
|
||||
The library uses the following mapping from JSON values types to MessagePack types according to the MessagePack
|
||||
specification:
|
||||
|
||||
| JSON value type | value/range | MessagePack type | first byte |
|
||||
|-----------------|------------------------------------------|------------------|------------|
|
||||
@@ -49,22 +52,23 @@ The library uses the following mapping from JSON values types to MessagePack typ
|
||||
|
||||
!!! success "Complete mapping"
|
||||
|
||||
The mapping is **complete** in the sense that any JSON value type can be converted to a MessagePack value.
|
||||
The mapping is **complete** in the sense that any JSON value type can be converted to a MessagePack value.
|
||||
|
||||
Any MessagePack output created by `to_msgpack` can be successfully parsed by `from_msgpack`.
|
||||
Any MessagePack output created by `to_msgpack` can be successfully parsed by `from_msgpack`.
|
||||
|
||||
!!! warning "Size constraints"
|
||||
|
||||
The following values can **not** be converted to a MessagePack value:
|
||||
The following values can **not** be converted to a MessagePack value:
|
||||
|
||||
- strings with more than 4294967295 bytes
|
||||
- byte strings with more than 4294967295 bytes
|
||||
- arrays with more than 4294967295 elements
|
||||
- objects with more than 4294967295 elements
|
||||
- strings with more than 4294967295 bytes
|
||||
- byte strings with more than 4294967295 bytes
|
||||
- arrays with more than 4294967295 elements
|
||||
- objects with more than 4294967295 elements
|
||||
|
||||
!!! info "NaN/infinity handling"
|
||||
|
||||
If NaN or Infinity are stored inside a JSON number, they are serialized properly. function which serializes NaN or Infinity to `null`.
|
||||
If NaN or Infinity are stored inside a JSON number, they are serialized properly in contrast to the
|
||||
[dump](../../api/basic_json/dump.md) function which serializes NaN or Infinity to `null`.
|
||||
|
||||
??? example
|
||||
|
||||
@@ -123,7 +127,7 @@ The library maps MessagePack types to JSON value types as follows:
|
||||
|
||||
!!! info
|
||||
|
||||
Any MessagePack output created by `to_msgpack` can be successfully parsed by `from_msgpack`.
|
||||
Any MessagePack output created by `to_msgpack` can be successfully parsed by `from_msgpack`.
|
||||
|
||||
|
||||
??? example
|
||||
|
||||
@@ -1,10 +1,11 @@
|
||||
# UBJSON
|
||||
|
||||
Universal Binary JSON (UBJSON) is a binary form directly imitating JSON, but requiring fewer bytes of data. It aims to achieve the generality of JSON, combined with being much easier to process than JSON.
|
||||
Universal Binary JSON (UBJSON) is a binary form directly imitating JSON, but requiring fewer bytes of data. It aims to
|
||||
achieve the generality of JSON, combined with being much easier to process than JSON.
|
||||
|
||||
!!! abstract "References"
|
||||
|
||||
- [UBJSON Website](http://ubjson.org)
|
||||
- [UBJSON Website](http://ubjson.org)
|
||||
|
||||
## Serialization
|
||||
|
||||
@@ -36,50 +37,43 @@ The library uses the following mapping from JSON values types to UBJSON types ac
|
||||
|
||||
!!! success "Complete mapping"
|
||||
|
||||
The mapping is **complete** in the sense that any JSON value type can be converted to a UBJSON value.
|
||||
The mapping is **complete** in the sense that any JSON value type can be converted to a UBJSON value.
|
||||
|
||||
Any UBJSON output created by `to_ubjson` can be successfully parsed by `from_ubjson`.
|
||||
Any UBJSON output created by `to_ubjson` can be successfully parsed by `from_ubjson`.
|
||||
|
||||
!!! warning "Size constraints"
|
||||
|
||||
The following values can **not** be converted to a UBJSON value:
|
||||
The following values can **not** be converted to a UBJSON value:
|
||||
|
||||
- strings with more than 9223372036854775807 bytes (theoretical)
|
||||
|
||||
!!! info "Unused UBJSON markers"
|
||||
|
||||
The following markers are not used in the conversion:
|
||||
The following markers are not used in the conversion:
|
||||
|
||||
- `Z`: no-op values are not created.
|
||||
- `C`: single-byte strings are serialized with `S` markers.
|
||||
|
||||
!!! info "NaN/infinity handling"
|
||||
|
||||
If NaN or Infinity are stored inside a JSON number, they are
|
||||
serialized properly. This behavior differs from the `dump()`
|
||||
function which serializes NaN or Infinity to `null`.
|
||||
If NaN or Infinity are stored inside a JSON number, they are serialized properly. This behavior differs from the
|
||||
`dump()` function which serializes NaN or Infinity to `null`.
|
||||
|
||||
!!! info "Optimized formats"
|
||||
|
||||
The optimized formats for containers are supported: Parameter
|
||||
`use_size` adds size information to the beginning of a container and
|
||||
removes the closing marker. Parameter `use_type` further checks
|
||||
whether all elements of a container have the same type and adds the
|
||||
type marker to the beginning of the container. The `use_type`
|
||||
parameter must only be used together with `use_size = true`.
|
||||
The optimized formats for containers are supported: Parameter `use_size` adds size information to the beginning of a
|
||||
container and removes the closing marker. Parameter `use_type` further checks whether all elements of a container
|
||||
have the same type and adds the type marker to the beginning of the container. The `use_type` parameter must only be
|
||||
used together with `use_size = true`.
|
||||
|
||||
Note that `use_size = true` alone may result in larger representations -
|
||||
the benefit of this parameter is that the receiving side is
|
||||
immediately informed on the number of elements of the container.
|
||||
Note that `use_size = true` alone may result in larger representations - the benefit of this parameter is that the
|
||||
receiving side is immediately informed on the number of elements of the container.
|
||||
|
||||
!!! info "Binary values"
|
||||
|
||||
If the JSON data contains the binary type, the value stored is a list
|
||||
of integers, as suggested by the UBJSON documentation. In particular,
|
||||
this means that serialization and the deserialization of a JSON
|
||||
containing binary values into UBJSON and back will result in a
|
||||
different JSON object.
|
||||
|
||||
If the JSON data contains the binary type, the value stored is a list of integers, as suggested by the UBJSON
|
||||
documentation. In particular, this means that serialization and the deserialization of a JSON containing binary
|
||||
values into UBJSON and back will result in a different JSON object.
|
||||
|
||||
??? example
|
||||
|
||||
@@ -117,8 +111,7 @@ The library maps UBJSON types to JSON value types as follows:
|
||||
|
||||
!!! success "Complete mapping"
|
||||
|
||||
The mapping is **complete** in the sense that any UBJSON value can be converted to a JSON value.
|
||||
|
||||
The mapping is **complete** in the sense that any UBJSON value can be converted to a JSON value.
|
||||
|
||||
??? example
|
||||
|
||||
|
||||
Reference in New Issue
Block a user